This paper presents a modified lateral control method for an autonomous vehicle with both look-ahead and look-down sensing systems. To cope with sensor noise and modeling uncertainty in the lateral control of the vehicle, a modified LMI-based H lateral controller was proposed, which uses the look-ahead information of the lateral offset error measured at the front of vehicle and the look-down information of the vehicle yaw angle error between the reference lane and the centerline of the vehicle. To verify the safety and the performance of the lateral control, a scaled-down vehicle was developed, and the positioning of the vehicle was estimated with USAT. The proposed controller, which uses both look-ahead and look-down information, was tested for lane changing and reference lane tracking with both simulation and experiment. The simulation and experimental results show that the proposed controller has better tracking and handling performance compared with a controller that uses only the look-ahead information of the target heading angle error.