Recently, nano-methodology is increasingly important as the ruler for measuring nano-technology, and we applied the linear encoder to nano-methodology. The quadrature output in the linear encoder has an effect on increasing the resolution in some techniques. Already, various interpolation techniques based on the quadrature signal have applied to the precision servo system. In this paper, we propose a new interpolation algorithm for ultra-precision positioning in the low speed with simulation by MATLAB SIMULINK. This method modified previous methods and was properly designed for some given control system. To verify, we first fulfilled the encoder signal test to find main parameters fer the signal transformation, then we proved the proposed interpolation algorithm by experiments, which show that the result of the interpolation algorithm corresponds with the measurement of the laser interferometer in 100 nm unit approximately. In addition, we can get more precise measurement by more accurate and noise-free signal. So we need to compensate imperfections in the encoder signal. After that, we will apply this algorithm to nano positioning system.