Transmission error is an important reason for instability in helical gears. A six-degree-of-freedom dynamic model coupled flexional, torsional and axial motion of a helical gear transmission system, which includes time varying mesh stiffness, bearing supporting stiffness, mesh damping and backlash, is developed, after taking into account the dynamic characteristics and vibration responses of helical gear in three dimensions. Influences of involute contact ratio, bearing supporting stiffness, mesh damping and backlash on the dynamic transmission errors and vibration stability of the helical gear system are investigated using numerical simulation technique. The effects on dynamic transmission errors and stabilities by contact ratio, supporting stiffness and mesh damping as well as gear backlash are analyzed. The intrinsic relationship between above parameters and dynamic transmission errors and stabilities for helical gear system are presented. The stable and unstable regions under different parameters are given. The results in this paper can be helpful to the dynamic and stable design of a helical gear transmission system.