Co fine particles were dispersed in Au metal and $AlO_x$ amorphous matrices by vacuum evaporation and rf-sputtering, respectively, thus forming granular composite films having chemical compositions of $Co_{0.59}-Au_{0.41}$ and $Co_{0.52}/(AlO_x$)_{0.48}$</TEX>. The films were annealed at 200~$500^{circ}C$ to increase the size of the Co particles, from 30$AA$ to 180$AA$ in the Au matrix and 40$AA$ to 180$AA$ in the $AlO_x$ matrix, as revealed by X-ray diffraction analysis. The Co metal in as-deposited films have saturation magnetization equivalent to that of bulk Co, which is unchanged by the annealing, showing that the Co metal is not oxidized by the annealing. Magneto-optical Kerr rotation measured at $lambda$=400-900nm for the $Co_{0.59}-Au_{0.41}$ film as deposited is larger than that calculated for the composition. The rotation increases as the film is annealed at $200^{circ}C$ and $300^{circ}C$, approaching to that of bulk Co. The Kerr rotation for the $Co_{0.52}-(AlO_x)_{0.48}$ film as deposited is smaller than that calculated for the composition based on Bruggeman effective medium theory. However, the rotation increases much, exceeding the rotation of the bulk Co as annealed at $300^{circ}C$ and $400^{circ}C$. As a possible origin of the marked magneto-optical enhancement a weak localization of light in granular structure is suggested.