Convenient analytical tools for evaluation of the aperiodic and the fluctuating instabilities of the passive residual heat removal system (PRHRS) of an integral reactor are developed and results are discussed from the viewpoint of the system design. First, a static model for the aperiodic instability using the system hydraulic loss relation and the downcomer feedwater heating equations is developed. The calculated hydraulic relation between the pressure drop and the feedwater flow rate shows that several static states can exist with various numbers of water-mode feedwater module pipes. It is shown that the most probable state can exist by basic physical reasoning, that there is no flow rate through the steam-mode feedwater module pipes. Second, a dynamic model for the fluctuating instability due to steam generation retardation in the steam generator and the dynamic interaction of two compressible volumes, that is, the steam volume of the main steam pipe lines and the gas volume of the compensating tank is formulated and the D-decomposition method is applied after linearization of the governing equations. The results show that the PRHRS becomes stabilized with a smaller volume compensating tank, a larger volume steam space and higher hydraulic resistance of the path $a_{ct}$. Increasing the operating steam pressure has a stabilizing effect. The analytical model and the results obtained from this study will be utilized for PRHRS performance improvement.