In this paper, we proposed a robust object extraction and tracking algorithm in the IR image sequence acquired in the sea environment. In order to extract size-invariant object, we detect horizontal and vertical edges by using DWT and combine it to generate saliency map. To extract object region, binarization technique is applied to saliency map. The correspondences between objects in consecutive frames are defined by the calculating minimum weighted Euclidean distance as a matching measure. Finally, object trajectories are determined by considering false correspondences such as entering object, vanishing objects and false object and so on. The proposed algorithm can find trajectories robustly, which has shown by experimental results.