We present a semi-rigid connection estimation method by using cross modal strain energy method. While rigid or pinned assumptions are adopted for steel frames in traditional modeling via finite element method, the actual behavior of the connections is usually neither. Semi-rigid joints enable connections to be modeled as partially restrained, which improves the quality of the model. To identify the connection stiffness and update the FE model, a newly-developed cross modal strain energy (CMSE) method is extended to incorporate the connection stiffness estimation. Meanwhile, the relations between the correction coefficients for the CMSE method are derived, which enables less modal information to be used in the estimation procedure. To illustrate the capability of the proposed parameter estimation algorithm, a four-story frame structure is demonstrated in the numerical studies. Several cases, including Semi-rigid joint(s) on single connection and on multi-connections, without and with measurement noise, are investigated. Numerical results indicate that an excellent updating is achievable and the connection stiffness can be estimated by CMSE method.