Vibration-based damage detection methods are popular for structural health monitoring. However, they can only detect fairly large damages. Usually impact pulse, ambient vibrations and sine-wave forces are applied as the excitations. In this paper, we propose the method to use the chaotic excitation to vibrate structures. The attractors built from the output responses are used for the minor damage detection. After the damage is detected, it is further quantified using the Kalman Filter. Simulations are conducted. A 5-story building is subjected to chaotic excitation. The structural responses and related attractors are analyzed. The results show that the attractor distances increase monotonously with the increase of the damage degree. Therefore, damages, including minor damages, can be effectively detected using the proposed approach. With the Kalman Filter, damage which has the stiffness decrease of about 5% or lower can be quantified. The proposed approach will be helpful for detecting and evaluating minor damages at the early stage.