We investigate the cosmic evolution of the black hole mass-bulge luminosity relation with a sample of 52 moderate-luminosity AGNs at $z{simeq}0.36$ and $z{simeq}0.57$, corresponding to look-back times of 4 and 6 Gyrs. By employing robust multi-component spectral and structural decomposition methods to the obtained high-quality Keck spectra and high-resolution HST images, black hole masses ($M_{BH}$) are estimated from the Hbeta broad emission line with the 5100A nuclear luminosity, and bulge luminosities ($L_{bul}$) are derived from the surface photometry. Based on these consistent measurements, we constrain the redshift evolution of the $M_{BH}-L_{bul}$ relation by performing the Monte Carlo simulations designed to account for selection effects. We provide implications of our results in terms of the black hole-galaxy co-evolution and discuss possible bulge growth mechanisms.