In this paper, we present an algorithm that quickly and effectively estimates orthogonal vanishing points in equirectangular images of urban environment. Our algorithm is based on the RANSAC (RANdom SAmple Consensus) algorithm and on the characteristics of the line segment in the spherical panorama image of the $360^{circ}$ longitude and $180^{circ}$ latitude field of view. These characteristics can be used to reduce the geometric ambiguity in the line segment classification as well as to improve the robustness of vanishing point estimation. The proposed algorithm is validated experimentally on a wide set of images. The results show that our algorithm provides excellent levels of accuracy for the vanishing point estimation as well as line segment classification.