It is possible to improve the accuracy of the motion estimation for a video by applying a variable block size. However, it has limits in the zoom motion estimation. In this paper, we propose a method for estimating the zoom motion with variable block size. The proposed method separates the background within the object picture by depth information obtained from a depth camera, and only the object regions are applied to zoom scale, but the background is not applied. In addition, the object regions select efficiently variable block size mode in consideration of the generated motion vectors and the accuracy of motion estimation at the same time. Simulation results show the accuracy of the motion estimation and the number of motion vectors for the proposed method. It is verified that the proposed method can reduce the number of motion while maintaining the similar accuracy of motion estimation than the conventional motion estimation methods.