A novel system identification and structural health assessment procedure of steel framed structures with semi-rigid connections is presented in this paper. It is capable of detecting damages at the local element level under normal operating conditions; i.e., serviceability limit state. The procedure is a linear time-domain system identification technique in which the structure responses are required, whereas the dynamic excitation force is not required to identify the structural parameters. The procedure tracks changes in the stiffness properties of all the elements in a structure. It can identify damage-free and damaged structural elements very accurately when excited by different types of dynamic loadings. The method is elaborated with the help of several numerical examples. The results indicate that the proposed algorithm identified the structures correctly and detected the pre-imposed damages in the frames when excited by earthquake, impact, and harmonic loadings. The algorithm can potentially be used for structural health assessment and monitoring of existing structures with minimum disruption of operations. Since the procedure requires only a few time points of response information, it is expected to be economic and efficient.