In this paper, we investigated possible optical parameters causing deviation of experimentally observed output spectra from theoretically predicted results in a high-order fiber comb filter based on a polarization-diversity loop configuration. They include wavelength dependency of half-wave plates (HWPs) inserted in the filter for wavelength switching and the modal birefringence of single-mode fibers (SMFs) with which optical components comprising the filter are connected. In order to consider the effect of the modal birefringence of the SMF on a filter performance, it is modeled as a low birefringence fiber with an arbitrary orientation angle and birefringence. It is found from the simulation results that the modal birefringence of SMFs strongly affects the spectral characteristics of the filter and decreases the extinction ratio of the filter, compared with the wavelength dependency of HWPs. In particular, it is also confirmed that the spectral deviation and asymmetric distortions of side-lobes in narrow band transmission spectra result mainly from the modal birefringence of SMFs.