Object tracking in a real time environment is one of challenging subjects in computer vision area during past couple of years. This paper proposes a method of object detection and tracking using adaptive background estimation in real time environment. To obtain a stable and adaptive background, we combine 3-frame differential method and running average single gaussian background model. Using this background model, we can successfully detect moving objects while minimizing false moving objects caused by noise. In the tracking phase, we propose a matching criteria where the weight of position and inner brightness distribution can be controlled by the size of objects. Also, we adopt a Kalman Filter to overcome the occlusion of tracked objects. By experiments, we can successfully detect and track objects in real time environment.