For the autonomy of a mobile robot it is first needed to know its position and orientation. Various methods of estimating the position of a robot have been developed. However, it is still difficult to localize the robot without any initial position or orientation. In this paper we present the method how to make the colored map and how to calculate the position and direction of a robot using the angle data of an omnidirectional image. The wall of the map is rendered with the corresponding color images and the color histograms of images and the coordinates of feature points are stored in the map. Then a mobile robot gets the color omnidirectional image at arbitrary position and orientation, segments it and recognizes objects by multiple color indexing. Using the information of recognized objects robot can have enough feature points and localize itself.