In this letter, we propose a novel approach to detecting and tracking apartment buildings for the development of a video-based navigation system that provides augmented reality representation of guidance information on live video sequences. For this, we propose a building detector and tracker. The detector is based on the AdaBoost classifier followed by hierarchical clustering. The classifier uses modified Haar-like features as the primitives. The tracker is a motion-adjusted tracker based on pyramid implementation of the Lukas-Kanade tracker, which periodically confirms and consistently adjusts the tracking region. Experiments show that the proposed approach yields robust and reliable results and is far superior to conventional approaches.