With the popularity of digital audio contents, querying and retrieving audio contents efficiently from database has become essential. In this paper, we propose a new index scheme for retrieving audio contents efficiently using audio portions that have been queried frequently. This scheme is based on the observation that users have a tendency to memorize and query a small number of audio portions. Detecting and indexing such portions enables fast retrieval and shows better performance than sequential search-based audio retrieval. Moreover, this scheme is independent of underlying retrieval system, which means this scheme can work together with any other audio retrieval system. We have implemented a prototype system and showed its performance gain through experiments.