When size and shape of moving object have been changed, a correlator often accumulates walk-off error. A success of correlation-based tracking largely depends on choosing suitable window size and position and thus transferring the proper reference image to the next frame. For this, we propose the Adaptive Window Algorithm with Four-Direction Sizing Factors (AWA-FSF) for fast adjusting a reference region to enhance reliability of correlation-based image tracking in complex cluttered environments. Since the AWA-FSF is capable of adjusting a reference image size more rapidly and properly, we can minimize the influence of complex background and clutter. In addition, we can finely tune the center point of the reference image repeatedly after main tracking process. Thus we have increased stability and reliability of correlation-based image tracking. We tested performance of the AWA-FSF using 45 real image sequences made of over 3400 images and had the satisfied results for most of them.