This paper presents a robust tracking controller design method for the track-following system of an optical recording device. A tracking loop gain adjustment algorithm is introduced to accurately estimate the tracking vibration quantity in spite of the uncertainties of the tracking actuator. A minimum tracking open-loop gain is calculated by the estimated tracking vibration quantity and a tolerable limit of tracking error. A robust tracking controller is designed by considering a robust $H_infty$ control problem with the weighting function of a slightly larger gain than the minimum tracking open-loop gain. The proposed controller design method is applied to the track-following system of an optical recording device and is evaluated through the experimental result.