Proper installation of Flexible AC Transmission Systems (FACTS) devices in existing transmission networks can enable power systems to accommodate more power transfer with less network expansion cost. The problem to maximize transmission system loadability by determining optimal locations and settings for installations of two types of FACTS devices, namely static var compensator (SVC) and thyristor controlled series compensator (TCSC), is formulated as a mixed discrete-continuous nonlinear optimization problem (MDCP). For solving the MDCP, in the paper, the proposed method with fitness sharing technique involved in the updating process of the particle swarm optimization (PSO) algorithm, can diversify the particles over the search regions as much as possible, making it possible to achieve the optimal solution with a big probability. The modified IEEE-14 bus network and a practical power system are used to validate the proposed method.