The optimal design of the squeeze film damper (SFD) for rotor system has been studied in previous researches. However, these researches have not been considering jumping or nonlinear phenomena of a rotor system with SFD. This paper represents an optimization technique for linear and nonlinear response of a simple rotor system with SFDs by using a hybrid GA-SA algorithm which combined enhanced genetic algorithm (GA) with simulated annealing algorithm (SA). The damper design parameters are the radius, length and radial clearance of the damper. The objective function is to minimize the transmitted load between SFD and foundation at the operating and critical speeds of the rotor system with SFD which has linear and nonlinear unbalance responses. The numerical results show that the transmitted load of the SFD is greatly reduced in linear and nonlinear responses for the rotor system.