This paper presents the optimization of an endwall boundary layer fence which improves the aerothermal characteristics of a gas turbine passage, The fence in a gas turbine passage effectively reduces secondary flow loss, which is generated in the cascade of the turbine. The turbine passage was simulated by a $90^{circ}$ turning duct ($Re_D=360,000$). The main purpose of the present investigation was to focus on finding a boundary layer fence with minimum total pressure loss in the passage and heat transfer coefficient on the endwall of the duct. An approximate optimization method was used for the investigation to secure the computational efficiency. Design variables were the fence shape factors (fence thickness, fence height, and fence width) as well as the radial position on the passage endwall. Results indicated that a significant improvement in aerodynamic performance can be achieved through the application of an optimized boundary layer fence. The optimized design reduced the mass-weighted average total pressure loss in the duct by 8.6% relative to the duct without the fence. The area-weighted average maximum Nusselt number on the endwall of the optimized design was decreased by 2.5% in comparison with the duct without the fence.