The overall driving environment consists of the Traffic environment, vehicle and driver states (TVD). advanced driver assistance Systems (ADAS) must consider not only information on each of the TVD states but also their context. Recent research has focused on making more efficient and effective assistance systems by fusing all the information from the TVD states. Based on this research trend, this paper focuses on decision-level fusion to estimate the level of danger of a warning by using visual information of the traffic environment and the driver state. The driver state consists of the gazing region and the facial feature points that are obtained using the active appearance model (AAM). The traffic environment state consists of time to collision (TTC), time to lane Crossing (TLC), and lane color information, which are obtained from the environment in front of the vehicle, i.e., position of lanes and other vehicles. Warnings against lane-off, collision, and driver inattention are generated by fusing this vision-based information from inside and outside the vehicle. The experimental results prove that our vision-based interactive driver assistance system reduces most useless warnings.