The freak wave, also known as New-Year-Wave in the north Atlantic, is relatively large and spontaneous ocean surface wave that can sink even large ships and destroy maritime structures. To understand oceanic conditions that develop freak waves, we simulated and generated two versions of scale-downed waves (1:64 and 1:42) in a numerical wave tank and compared the results with the experiment in wave flume. Both of the breaking and non-breaking waves were generated in the simulation. The numerical simulation was implemented based on the finite volume method and a genetic optimization algorithm. Random values were assigned as the initial values for the parameter in the control function, which produced signals representing the motion of wave-maker. The same signal obtained from the optimization process was used for both of the simulation and the experiment. By varying the object function and restrictions of the simulation, a best profile of design wave was selected based on the characteristics, height and period of simulated waves. Results showed that the simulation and experiment with the scale of 1:42 agreed better with freak waves in the natural condition. The presented simulation method will contribute to saving the time and cost for conducting subsequent response analyses of motion under freak waves in the course of the model test for ship and maritime structure.