In this study, we analyzed the effect of silicon oxynitride matrix on the optical properties of Au nanoparticles dispersed on composite film and explored the effectiveness of the silicon in fine tuning the refractive index of the composite film for applications in optical waveguide devices. The atomic fraction of nitrogen in $SiO_xN_y$ films was controlled by varying the relative flow ratio of nitrogen gas in reactive sputtering and was evaluated optically using an effective medium theory with Bruggeman geometry consisting of a random mixture between $SiO_2$ and $Si_3N_4$. The Au nanoparticles were embedded in the $SiO_xN_y$ matrix by employing the alternating deposition technique and clearly showed an absorption peak due to the excitation of surface plasmon. With increasing nitrogen atomic fraction in the matrix, the surface plasmon resonance wavelength shifted to a longer wavelength (a red-shift) with an enhanced resonance absorption. These characteristics were interpreted using the Maxwell-Garnett effective medium theory. The formation of a guided mode in a slab waveguide consisting of 3 $mu$m thick Au:$SiO_xN_y$ nanocomposite film was confirmed at the telecommunication wavelength of 1550 nm by prism coupler method and compared with the case of using $SiO_2$ matrix. The use of $SiO_xN_y$ matrix provides an effective way of controlling the mode confinement while maintaining or even enhancing the surface plasmon resonance properties.