In this paper, we study effect of waveness at the optical surface roughness measurement. In generally, waveness components cause errors in calculation of the roughness value of metal surface. We study about surface roughness signals In the frequency domain for separate two signal component of real roughness and waveness by digital signal processing methods. Thereafter, determine low and high Component of frequency spectrum. By this separating frequency value we design liner low and high pass filter which cutoff frequency is 1 Hz. After this process, converted each filtered spectrum by inverse discrete fourier transformation to time domain waveness and real roughness signals We calculate surface roughness value from filtered roughness signals. For evaluate this method, we use five specimens roughness signal which obtained from optical surface roughness measuring system in 3mm/s moving speed with 0.1 mm laser beam spot size As a result, we obtain more linerized roughness value than that of unfiltered roughness signals.