High speed and efficiency are highly important factors in handling cargo at ports. The ability to handle cargo at high speed strongly depends on the anti-sway system employed. However, it is not easy to realize a high-performance anti-sway control system due to a number of problems. High-speed operation of container cranes is continuously required for crane operators who work as cargo handlers at ports. Undesirable motion can lower the work efficiency due to the prolonged strain to which operators are subjected. To overcome this problem, we propose an anti-sway system with an image sensor for container cranes. In this system, the image sensor is used for measuring the motion of the spreader, and the measured data are fed back to the controller in real time. The applied image processing technique is a kind of robust template matching method referred to as vector code correlation (VCC), which was devised in order to consider real environmental conditions. The anti-sway system proposed in this paper is a mass damper type system in which a movable mass is installed on the spreader. The actuator acting on the movable mass applies inertial force to the spreader, which results in suppressing undesirable sway motion. The system is simple and can be easily applied to any crane system. In this regard, the useful features and the performance of the proposed anti-sway system are experimentally verified.