An analytical method is presented for evaluation of the steady state periodic behavior of nonlinear structural systems. This method is based on the substructure synthesis formulation and a harmonic balance procedure, which is applied to the analysis of nonlinear responses. A complex nonlinear system is divided into substructures, of which equations are approximately transformed to modal coordinates including nonlinear term under the reasonable procedure. Then, the equations are synthesized into the overall system and the nonlinear solution for the system is obtained. Based on the harmonic balance method, the proposed procedure reduces the size of large degrees-of-freedom problem in the solving nonlinear equations. Feasibility and advantages of the proposed method are illustrated using the study of the nonlinear rotating machine system as a large mechanical structure system. Results obtained are reported to be an efficient approach with respect to nonlinear response prediction when compared with other conventional methods.