Traditionally, aerial images have been used as main sources for compiling topographic maps. In recent years, lidar data has been exploited as another type of mapping data. Regarding their performances, aerial imagery has the ability to delineate object boundaries but omits much of these boundaries during feature extraction. Lidar provides direct information about heights of object surfaces but have limitations with respect to boundary localization. Considering the characteristics of the sensors, this paper proposes an approach to extracting buildings from lidar and aerial imagery, which is based on the complementary characteristics of optical and range sensors. For detecting building regions, relationships among elevation contours are represented into directional graphs and searched for the contours corresponding to external boundaries of buildings. For generating building models, a wing model is proposed to assemble roof surface patches into a complete building model. Then, building models are projected and checked with features in aerial images. Experimental results show that the proposed approach provides an efficient and accurate way to extract building models.