When the nonlinearities, such as friction and backlash, are not considered in the controller design, undesirable oscillations can occur in the steady-state response of a control system. This paper deals with a method to reduce oscillations that often appear in the steady-state response of a pendulum system, which is controlled by a state feedback controller based on the linearized system model. With an assumption that the oscillations shown in the steady-state are caused by the Coulomb friction, we improve the performance of stabilization and tracking by estimating and compensating for the Coulomb friction in the pendulum system. Experimental results show that the control performance can be improved sufficiently by the proposed method, when it is applied to an inverted cart pendulum which is a multi-variable unstable system. Furthermore, we could see that the Coulomb friction model used in the estimation of the friction is valid in applying the suggested method.