This paper proposes a fast face tracking method with only depth information. It is basically a template matching method, but it uses a early termination scheme and a sparse search scheme to reduce the execution time to solve the problem of a template matching method, large execution time. Also a refinement process with the neighboring pixels is incorporated to alleviate the tracking error. The depth change of the face being tracked is compensated by predicting the depth of the face and resizing the template. Also the search area is adjusted on the basis of the resized template. With home-made test sequences, the parameters to be used in face tracking are determined empirically. Then the proposed algorithm and the extracted parameters are applied to the other home-made test sequences and a MPEG multi-view test sequence. The experimental results showed that the average tracking error and the execution time for the home-made sequences by Kinect ($640{ imes}480$) were about 3% and 2.45ms, while the MPEG test sequence ($1024{ imes}768$) showed about 1% of tracking error and 7.46ms of execution time.