Dynamic displacement of structures is an important index for in-service structural condition and behavior assessment, but accurate measurement of structural displacement for large-scale civil structures such as long-span bridges still remains as a challenging task. In this paper, a vision-based dynamic displacement measurement system with the use of digital image processing technology is developed, which is featured by its distinctive characteristics in non-contact, long-distance, and high-precision structural displacement measurement. The hardware of this system is mainly composed of a high-resolution industrial CCD (charge-coupled-device) digital camera and an extended-range zoom lens. Through continuously tracing and identifying a target on the structure, the structural displacement is derived through cross-correlation analysis between the predefined pattern and the captured digital images with the aid of a pattern matching algorithm. To validate the developed system, MTS tests of sinusoidal motions under different vibration frequencies and amplitudes and shaking table tests with different excitations (the El-Centro earthquake wave and a sinusoidal motion) are carried out. Additionally, in-situ verification experiments are performed to measure the mid-span vertical displacement of the suspension Tsing Ma Bridge in the operational condition and the cable-stayed Stonecutters Bridge during loading tests. The obtained results show that the developed system exhibits an excellent capability in real-time measurement of structural displacement and can serve as a good complement to the traditional sensors.