The spindle with a built-in motor can be used to simplify the structure of machine tool system, while the rotor has unbalance mass inevitably. Therefore, it is important to recognize the effect of unbalance mass. This paper presents analysis of dynamic behavior of a high speed spindle with a built-in motor. The spindle is supported by the angular contact ball bearings and the rotor is fixed at the middle of spindle. The spindle used in CNC automatic lathe has been investigated using combined methodologies of finite elements and transfer matrices. The Houbolt method is used for the integration of the system equations and the dynamic behavior of spindle is obtained considering unbalance mass of rotor. Results show that increasing rotational speed of spindle magnifies the whirl responses of spindle seriously. Also the whirl responses of spindle are affected by the other factors such as unbalance mass and bearing stiffness.