In this paper, we present the detection method of moving objects based on two background models. These background models support to understand multi layered environment belonged in images taken by shaking camera and each model is MBM(Multiple Background Model) and TMBM (Temporal Median Background Model). Because two background models are Pixel-based model, it must have noise by camera movement. Therefore correlation coefficient calculates the similarity between consecutive images and measures camera motion vector which indicates camera movement. For the calculation of correlation coefficient, we choose the selected region and searching area in the current and previous image respectively then we have a displacement vector by the correlation process. Every selected region must have its own displacement vector therefore the global maximum of a histogram of displacement vectors is the camera motion vector between consecutive images. The MBM classifies the intensity distribution of each pixel continuously related by camera motion vector to the multi clusters. However, MBM has weak sensitivity for temporal intensity variation thus we use TMBM to support the weakness of system. In the video-based experiment, we verify the presented algorithm needs around 49(ms) to generate two background models and detect moving objects.