AC generator plays an important role of power system. The large AC generator fault may lead to large impacts or perturbations in power system. And then the protection of a generator has very important role in maintaining stability in a power system. In present, the DFT(discrete Fourier transform) based RDR(ratio differential relay) had been widely applied to a internal fault of a generator stator winding. But DFT has a serious drawback. In the course of transforming a target signal to frequency domain, time information is lost. DWT uses a time-scale region. This paper proposes an advanced fault detection algorithm using DWT(discrete Wavelet transform) to enhance the drawback of conventional DFT based relaying. To evaluate the performance of the proposed relaying, we used the test data which were sampled with 720 [Hz] per cycle and obtained from ATP(alternative transient program) simulation. And we made a comparative study of conventional DFT based RDR and the proposed relaying.