In this study, a method of error compensation for channel gain imbalance, phase imbalance and local oscillator leakage in the modulator of frequency direct conversion is suggested. The compensation of channel imbalance can be carried out by using the received power after transmitting test signal. By applying this method, the phase imbalance conversion with frequency can be easily compensated since this method is rarely affected by the transmission channel. It is confirmed that the algorithm proposed in this study(iteration coefficient=11) converges faster than conventional algorithm(iteration coefficient=43). From the numerical results, the DC-offset, channel gain, phase imbalance compensation coefficient and iteration number converges into($f_1$=0.0199999, $f_2$=-0.050001, $C_{22}$=0.9133, $C_{12}$=-0.0524, N=13) when the local oscillator leakage is not considered. However, it converges into($f_1$=-0.02, $f_2$=-2.2476, $C_{22}$=0.9133, $C_{12}$=-0.0524, N=16) when the local oscillator leakage is considered.