In this paper, we perform interference cancellation in multiuser MIMO (Multiple Input Multiple Output) relay network with improved Amplify-and-Forward (AF) and Decode-and-Forward (DF) relay protocols. The work of interference cancellation is followed by evolved NodeB (eNB), Relay Node (RN) and User Equipment (UE) to improve the error performance of whole transmission system with the explicit use of relay node. In order to perform interference cancellation, we use Dirty Paper Coding (DPC) and Thomilson Harashima Precoding (THP) allied with detection techniques Zero Forcing (ZF), Minimum Mean Square Error (MMSE), Successive Interference Cancellation (SIC) and Ordered Successive Interference Cancellation (OSIC). These basic techniques are studied and improved in the proposal by using the functions of relay node. The performance is improved by Decode-and-Forward which enhance the cancellation of interference in two layers at the cooperative relay node. The interference cancellation using weighted vectors is performed between eNB and RN. In the final results of the research, we conclude that in contrast with the conventional algorithms, the proposed algorithm shows better performance in lower SNR regime. The simulation results show the considerable improvement in the bit error performance by the proposed scheme in the LTE-Advanced system.