An efficient peak detection algorithm in magnitude spectrum is proposed to distinguish the M-frequency shift keying(FSK) signals from other digitally modulated signal. In addition, recognition of the modulation order estimation of FSK signals is also studied based on the fact that the magnitude spectrum of FSK signals reveals the number of peaks equal to the modulation order. When no a priori information about the signals, we utilize the histogram of the magnitude spectrum to determine the threshold which is important factor in peak detection algorithm. The simulation results show high probability of classification under 500 symbols and signal-to-noise ratio(SNR) higher than 4dB.