The unscented Kalman filter (UKF) has been developed for nonlinear model parametric identification, and it assumes that the model parameters are symmetrically distributed about their mean values without any constrains. However, the parameters in many applications are confined within certain ranges to make sense physically. In this paper, a constrained unscented Kalman filter (CUKF) algorithm is proposed to improve accuracy of numerical substructure modeling in hybrid testing. During hybrid testing, the numerical models of numerical substructures which are assumed identical to the physical substructures are updated online with the CUKF approach based on the measurement data from physical substructures. The CUKF method adopts sigma points (i.e., sample points) projecting strategy, with which the positions and weights of sigma points violating constraints are modified. The effectiveness of the proposed hybrid testing method is verified by pure numerical simulation and real-time as well as slower hybrid tests with nonlinear specimens. The results show that the new method has better accuracy compared to conventional hybrid testing with fixed numerical model and hybrid testing based on model updating with UKF.