We proposed a method to improve moving object detection capability of Gaussian Mixture Model by suggesting shape adaptive bidirectional block matching algorithm. This method achieves more accurate detection and tracking performance at various motion types such as slow, fast, and bimodal motions than that of Gaussian Mixture Model. Experimental results showed that the proposed method outperformed the conventional methods.