A transient CFD procedure to compute the nonlinear dynamic characteristic of the coupled rotor-seal system was presented in this study. In each time step, the displacement diffusion model was implemented to govern the mesh deformation, and the URANS (Unsteady Reynolds averaged Navier-Stokes) equations were solved to obtain the transient fluid force on the rotor surface for the free vibrations. With the obtained fluid force from the CFD solver, the nonlinear equations of motion for a simplified rotor-seal system were numerically solved on the basis of external user defined routines. During each transient time step, the computed fluid force from the CFD solver and the rotor motion from the user defined routines were transferred to each other timely. The rotor center trajectories, frequency spectrum and projection of Poincar$acute{e}$ section were calculated to investigate the nonlinear dynamic performance of the single disk rotor-seal system. The effects of the rotational speed and pressure ratio on the vibration characteristic of the rotor-seal system were analyzed by the bifurcation theorem. The results show that the coupled rotor-seal system experiences a period-one motion, resonance, periodic-doubling motion, quasi-periodic motion, and finally possible chaotic motion as the rotor speed increases. The pressure ratio has pronounced effect on the frequency response of the first-order critical speed; however, it has little influence on the motion state as well as the frequency response of the rotating speed. Although a constant-clearance annular smooth stator seal was selected as the research object in the current paper, the presented transient CFD method is still available for other complex annular seals, such as labyrinth seal, honeycomb seal, pocket damper seal, etc.