Hybrid mass damper systems have recently been introduced as a dynamic vibration absorber to exploit the benefits of both the conventional tuned mass damper system and the active control system. A hybrid system is programmed to function as either a conventional TMD or as an active system according to the wind conditions and the resultant building and damper mass vibration characteristics. This paper deals with the design of the robust controller for the control of the flexible box structure. The control algorithm was devised based on $H_2$(LQG) robust control logic with acceleration feedback and to improve the capability of the controller Kalman Filter was accepted for the system. To test the ability of the robust controller using the linear motor damper system, performance tests and simulations were carried out on the full-scale steel frame structure. Through the performance tests, it was confirmed that acceleration levels are reduced down.