The visual system applies previously learned contextual knowledge to facilitate the perception and encoding of scenes. When extrapolation following the employment of contextual information occurs, it sometimes leads to scene memory distortion: people report as if they saw more than they actually had seen. This phenomenon is called the "boundary extension" effect (Intraub & Richardson, 1989). The present study aimed to clarify the effects of contextual information on boundary extension in a more systematic way. Based on the assumption that it is harder to extract contextual information from inverted scenes compared to intact scenes, we presented inverted scenes either during encoding or retrieval to manipulate the level of contextual information and compared the magnitude of boundary extension effect for upright versus inverted scenes. In a series of experiments, we found that scene inversion during encoding, but not during retrieval, significantly reduced boundary extension. Showing reduced memory distortion for inverted scenes, the current study directly demonstrated that access to contextual information is a critical component of scene extrapolation process.