This paper presents a new algorithm for determining generator operation point for maintaining stability considering generator faults in Jeju-Haenam HVDC system. As the HVDC system consumes reactive power for the transmission of active power substantially, compensation of reactive power is essential. And the HVDC system is operated on frequency control mode. That is to say, the HVDC system almost manages system frequency. Therefore, we recognized that the Jeju system could be unstable if the reactive power consumed by the HVDC is insufficient when out-of-step occurs with large generators. When the solution of power flow analysis does not converge due to the unstable system phenomenon, we have difficulty in establishing countermeasures as the post-fault information is not available. In this paper, for the purpose of overcoming this difficulty in establishing countermeasures, we introduce the CPF(Continuation Power Flow) algorithm. This paper suggests an algorithm for calculating the output limitation of the generator to maintain the stability in case of generator fault in the Jeju system.