For the first time, a novel analytical model of the net gain for a Raman-EDFA hybrid optical amplifier (HOA) is proposed and its various parameters optimized using a genetic algorithm. Our method has been shown to be robust in the simultaneous analysis of multiple parameters (Raman length, EDFA length, and pump powers) to obtain large gain. The optimized HOA is further investigated at the system level for the scenario of a 50-channel DWDM system with 0.2-nm channel spacing. With an optimized HOA, a flat gain of >17 dB is obtained over the effective ITU-T wavelength grid with a variation of less than 1.5 dB, without using any gain-flattening technique. The obtained noise figure is also the lowest value ever reported for a Raman-EDFA HOA at reduced channel spacing.