Particle swarm optimization (PSO) and differential evolution (DE) have their similarities and compatibility in the design update process, such that a new design vector is determined by using neighborhood designs under algorithm control parameters. The paper deals with an integrated method of a hybrid PSO (HPSO) algorithm combined with DE in order to refine the optimization performance. PSO and DE also possess common characteristics compared with genetic algorithm (GA). The crossover-and mutation-like operators are suggested in the HPSO. A bounce back method is also implemented to prevent the design from locating out of design spaces during the optimization process. For the purpose of further enhancing the search capabilities, such HPSO is combined with the Q-learning that is one of efficient reinforcement learning methods. Using a number of nonlinear multimodal functions and engineering optimization problems, the proposed algorithms of HPSO and HPSO with Q-learning are compared with PSO, DE and GA.