Video based surveillance systems based on an active camera require a fast algorithm for real time detection and tracking of local motion in the presence of global motion. This paper presents a new fast and efficient motion detection and tracking algorithm using the displaced frame difference (DFD). In the Proposed algorithm, first, a Previous frame is adaptively selected according to the magnitude of object motion, and the global motion is estimated by using only a few confident matching blocks for a fast and accurate result. Then, a DFD is obtained between the current frame and the selected previous frame displaced by the global motion. Finally, a moving object is extracted from the noisy DFD by utilizing the correlation between the DFD and current frame. We implement this algorithm into an active camera system including a pan-tilt unit and a standard PC equipped with an AMD 800MHz processor. The system can perform the exhaustive search for a search range of 120, and achieve the processing speed of about 50 frames/sec for video sequences of 320$ imes$240. Thereby, it provides satisfactory tracking results.