In this paper, optimal selection of the unified power flow controller (UPFC) damping controller parameters in order to improve the power system dynamic response and its stability based on two modified intelligent algorithms have been proposed. These algorithms are based on a modified intelligent particle swarm optimization (PSO) and continuous genetic algorithm (GA). After extraction of UPFC dynamic model, intelligent PSO and genetic algorithms are used to select the effective feedback signal of the damping controller; then, to compare the performance of the proposed UPFC controller in damping the critical modes of a single-machine infinite-bus (SMIB) power system, the simulation results are presented. The comparison shows the good performance of both presented PSO and genetic algorithms in an optimal selection of UPFC damping controller parameters and damping oscillations.