Polishing a die that has free-form surfaces is a time-consuming and tedious job, and requires a considerable amount of high-precision skill. In order to reduce the polishing time and cope with the shortage of skilled workers, a user-friendly automatic polishing system was developed. The polishing system is composed of two subsystems, a three-axis machining center and a two-axis polishing robot. The system has five degrees of freedom and is able to keep the polishing tool in a position normal to the die surface during operation. A sliding mode control algorithm with velocity compensation was proposed to reduce tracking errors. Trajectory tracking experiments showed that the tracking error can be reduced prominently by the proposed sliding mode control compared to a PD (proportional derivative) control. To evaluate the polishing performance of the polishing system and to and the optimal polishing conditions, the polishing experiments were conducted.