In recent years, researches on rotary inverted pendulum control systems have been significantly focused due their highly nonlinear dynamics and complicated geometric structures. This paper presents a novel control approach for such systems by means of similarity transformation theory. At first, we represent nonlinear system dynamics to the controllability-formed state space model including a time-varying parameter vector. We establish the state-feedback control configuration based on the transformed model and derive an adaptive control law for adjusting desired characteristic equation. Numerical analysis is achieved to evaluate our control method and demonstrate its superiority by comparing it to the traditional control strategy. Furthermore, real-time control experiment is carried out to test its practical reliability.