In a transmission or geared rotor system a coupled phenomenon of lateral and torsional vibrations may occur due to the gear meshing effect. Particularly, in high speed or low vibration and low noise applications of geared rotor systems a coupled rotordynamic analysis is required to precisely predict their dynamic characteristics. In this paper a generalized finite element model of a gear pair element is developed, which actively couples the lateral and torsional vibrations due to the gear meshing effect. In the modeling the generalized forces due to the transmission error. geometrical eccentricities. and unbalances in the gear system are also considered. Then. using the developed gear pair element model a coupled unforced rotordynamic analysis is performed with a prototype 800 RT turbo-chiller rotor-bearing system having a hull-pinion speed increasing gear. Results show that the torsional vibration characteristics experience some changes due to the gear meshing and lateral dynamic coupling effect, but that they have no adverse effect and the lateral ones have no practical changes in an operating speed range.