The planetary gear reducer becomes more and more widely used in machine industries. The planetary gear reducer has a significant role to transmit power to wheel & tire module in the In-wheel system. Thus, the planetary gear reducer should have strong stiffness and durability. In this paper, the contact and bending stresses at the tooth of the planetary gear reducer are analyzed using MASTA, a commercial gear design and analysis software. Stress distribution at the tooth face of the sun, planetary and annulus gears are obtained using the finite element method. The design modification is performed using the response surface method. The usefulness of the design modification and optimization method presented in this paper is verified by comparing the maximum stresses of the original and optimized planetary gear tooth.