In this paper, a robust correlation-based video tracking is proposed to track a moving object in correlated image sequences. A correlation-based video tracking algorithm seeks to align the incoming target image with the reference target block image, but has critical problems, so called a false-peak problem and a drift phenomenon (correlator walk-off. The false-peak problem is generally caused by highly correlated background pixels with similar intensity of a moving target and the drift phenomenon occurs when tracking errors accumulate from frame to frame because of the nature of the correlation process. At first, the false-peaks problem for the ordinary correlation-based video tracking is investigated using a simple mathematical analysis. And, we will suggest a robust selective-attention correlation measure with a gradient preprocessor combined by a drift removal compensator to overcome the walk-off problem. The drift compensator adaptively controls the template block size according to the target size of interest. The robustness of the proposed method for practical application is demonstrated by simulating two real-image sequences.