In this study, we prepare polymer solar cells incorporating organic ligand-modified Ag nanoparticles (O-AgNPs) highly dispersed in the P3HT:PCBM layer. Ag nanoparticles decorated with water-dispersible ligands (WAgNPs) were also utilized as a control sample. The existence of the ligands on the Ag surface was confirmed by FT-IR spectra. Metal nanoparticles with different surface chemistries exhibited different dispersion tendencies. O-AgNPs were highly dispersed even at high concentrations, whereas W-AgNPs exhibited significant aggregation in the polymer layer. Both dispersion and blending concentration of the Ag nanoparticles in P3HT:PCBM matrix had critical effects on the device performance as well as light absorption. The significant changes in short-circuit current density ($J_{SC}$) of the solar cells seemed to be related to the change in the polymer morphology according to the concentration of AgNPs introduced. These findings suggested the importance of uniform dispersion of plasmonic metal nanoparticles and their blending concentration conditions in order to boost the solar cell performance.