New coupling agent or surface modified agent (9-decenoic acid) was used to enhance the compatibility between silica and polystyrene in silica/polystyrene hybrid nanocomposite, synthesized by in situ miniemulsion polymerization. Composites contain well dispersed nanosize silica particles. Related tests and analyses confirmed the success of synthesis. Functionalization of silica by 9-decenoic acid and silica on the polystyrene was confirmed by FTIR. TGA showed presence and amount of silica in final latex. The glass transition temperature of the hybrid nanocomposite was increased with the silica amount. SEM and TEM analysis showed the spherical morphology of PS and composite with an average diameter of 55 nm. The presence of silica within composite was confirmed by EDS attached to the existing TEM.