In this paper, we suggest a vision-based lane change control system, which can be applied on the straight road, without additional sensors such as a yaw rate sensor and a lateral accelerometer. In order to reduce the image processing time, we predict a reference line position during lane change using the lateral dynamics and the inverse perspective mapping. The sliding mode control algorithm with a boundary layer is adopted to overcome variations of parameters that significantly affects a vehicle`s lateral dynamics and to reduce chattering phenomenon. However, applying the sliding mode control to the system with a long sampling interval, the stability of a control system may seriously be affected by the sampling interval. Therefore, in this paper, a look ahead offset has been used instead of a lateral offset to reduce the effect of the long sampling interval due to the image processing time. The control algorithm is developed to follow the desired trajectory designed in advance. In the design of the desired trajectory, we take account of the constraints of lateral acceleration and lateral jerk for ride comfort. The performance of the suggested control system is evaluated in simulations as well as field tests.