In this paper, as the mobile communication service is widely used and the demand for wireless repeaters is rapidly increasing because of the easiness of extending service areas. But a wireless repeater has a problem the oscillation due to feedback signal. We proposed a new hybrid interference canceller using the adaptive filter with CMA(Constant Modulus Algorithm)-Grouped LMS(Least Mean Square) algorithm in the adaptive interference canceller. The proposed interference canceller has better channel adaptive performance and a lower MSE(Mean Square Error) than conventional structure because it uses the cancellation method of Grouped LMS algorithm. The proposed detector uses the LMS algorithms with two different step size to reduce mean square error and to obtain fast convergence. This structure reduces the number of iterations for the same MSE performance and hardware complexity compared to conventional nonlinear interference canceller.