The effect of nonlinear friction in the low velocity is dominant in precise controlled mechanisms and it is difficult to model. This paper is concerned with the compensation for friction using the variable structure system approach as nonmodel based method. The problem of chattering in the sliding mode controller is suppressed by the implementation of the boundary layer concept. And the estimation for friction using sliding mode observer makes the upper bound of matched uncertainty reduced. Accordingly, the effect of chattering can be more suppressed. And the sliding surface is constructed by adding an integral component to the switching function that is made by using error dynamics. This sliding surface guarantees the good tracking performance. Experimental results for a XY table system show that the proposed method has a good performance especially in the low velocity.