This study focuses on the development of numerical procedure to analyze the nonlinear combustion instabilities in liquid rocket engine. Nonlinear behaviors of acoustic instabilities are characterized by the existence of limit cycle in linearly unstable engines and nonlinear or triggering instability in linearly stable engines. To discretize convective fluxes with high accuracy and robustness, approximated Riemann solver based on characteristics and Euler-characteristic boundary conditions are employed. The present procedure predicts well the transition processes from initial harmonic pressure disturbance to N-like steep-fronted shock wave in a resonant pipe. Longitudinal pressure oscillations within the SSME(Space Shuttle Main Engine) engine have been analyzed using the pressure-sensitive time lag model to account for unsteady combustion response. It is observed that the pressure oscillations reach a limit cycle which is independent of the characteristics of the initial disturbances and depends only on combustion parameters and operating conditions.