S$i_{1-x}$G$e_x$(x=0.2~0.5) amorphous layer(~150nm) was deposited on(001) Si substrate using molecular beam epitaxy and annealed at various temperatures for solid phase epitaxy. The change of microstructure of SiGe film was investigated using X-ray diffractometry and transmission electron microscopy. For the sample annealed at 55$0^{circ}C$ for 30min SiGe film found to be amorphous. Single crystal SiGe film(~30nm) with defects was formed in the sample annealed at 65$0^{circ}C$ for 10min, but amorphous SiGe layer still remained above the SiGe single crystalline layer. According to high resolution lattice image, major defects in the SiGe layer on Si substrate were twins. For the sample annealed at 75$0^{circ}C$ for 10min all the amorphous SiGe was crystallized but SiGe film was found to be polycrystalline. According to our results, amorphous SiGe was crystallized but SiGe film was found to be polycrystalline. According to our results, amorphous SiGe layer on Si was crystallized as single crysal at the first stage of growth, but it changed to polycrystalline at the later stage of growth. The changes of SiGe film from single crystal to polycrystal and the effect of Ge composition on crystallinity of SiGe film were also studied.