Prebiotic oligosaccharides, with a degree of polymerization (DP) of mostly less than 10, exhibit diverse biological activities that contribute to human health. Currently available prebiotics are mostly derived from disaccharides and simple polysaccharides found in plants. Subtle differences in the structures of oligosaccharides can cause significant differences in their prebiotic properties. Therefore, alternative substances supplying polysaccharides that have more diverse and complex structures are necessary for the development of novel oligosaccharides that have actions not present in existing prebiotics. In this review, we show that structural polysaccharides found in plant cell walls, such as xylans and pectins, are particularly potential resources supplying broadly diverse polysaccharides to produce new prebiotics.