This paper proposes an efficient motion compensation algorithm for video sequences with brightness variations. In the proposed algorithm, the brightness variation parameters are estimated and local motions are compensated. To detect the frame with large brightness variations. we employ the frame classification based on the cross entropy between histograms of two successive frames, which can reduce the computational redundancy. Simulation results show that the proposed method yields a higher peak signal to noise ratio (PSNR) than the conventional methods, with a low computational load, when the video scene contains large brightness changes.