This ppaer describes the development of the stereo camera system(KASS-1) and the control of the vergence of the stereo camera to fix a gaze on a moving object in real-time using a stereo disparity. The motion energy and the stereo disparity of a moving object from the stereo image are used to control the vergence of stereo camera to keep stereo disparity constant. The disparity from the rotating stereo camera is introduced not only from the moving object but also from the background. In this paper, the background disparity error due to the vergence control of the stereo camera is eliminated by compensation algoithm, and the vergence of steereo camera system can be controlled continuously using the disparity of a moving object only.