Existing area-based stereo algorithms rely heavily on rectangular windows for computing correspondence. While the algorithms with the rectangular windows are efficient, they generate relatively large matching errors due to variations of disparity profiles near depth discontinuities and doesnt take into account local deformations of the windows due to projective distortion. In this paper, in order to deal with these problems, a new correlation function with 4 directional line masks, based on robust estimator, is proposed for the selection of potential matching points. These points is selected to consider depth discontinuities and reduce effects on outliers. The proposed matching method finds an arbitrarily-shaped variable window around a pixel in the 3d array which is constructed with the selected matching points. In addition, the method take into account the local deformation of the variable window with a constant disparity, and perform the estimation of sub-pixel disparities. Experiments with various synthetic images show that the proposed technique significantly reduces matching errors both in the vicinity of depth discontinuities and in continuously smooth areas, and also does not be affected drastically due to outlier and noise.