In a power system, an out-of-step condition causes a variety of risk such as serious damage to system elements, tripping of loads and generators, mal-operation of relays, etc. Therefore, it is very important to detect the out-of-step condition and take a proper measure. Several out-of-step detection methods have been employed in relays until now. Most common method used for an out-of-step detection is based on the transition time through the blocking impedance area in R-X diagram. Also, the R-R dot out-of-step relay, the out-of-step prediction method and the adaptive out-of-step relay using the equal area criterion (EAC) and Global Positioning Satellite (GPS) technology have been developed. This paper presents the out-of-step detection algorithm using the time variation of the complex power. The complex power is calculated and the mechanical power of the generator is estimated by using the electrical power, and then the out-of-step detection algorithm which is based on the complex power and the estimated mechanical power, is presented. This algorithm may detect the instant when the generator angle passes the Unstable Equilibrium Point (UEP). The proposed out-of-step algorithm is verified and tested by using Alternative Transient Program/Electromagnetic Transient Program (ATP/EMTP) MODELS.