This paper presents a calibration method of an omnidirectional stereo camera which may be essentially performed for distance measurement to a certain point. In the calibration of the omnidirectional stereo camera, the independent calibrations of two cameras or the calibration of a stereo camera having the small baseline is feasible applying many methods studied in the past. However, the baseline should be large enough for long distance measurement by the omnidirectional stereo camera, since it is not easy to calibrate two cameras with a large baseline at the same time. It is because a test pattern for the calibration, which is simultaneously captured by two omnidirectional cameras, appears too small in at least one of the omnidirectional cameras. It causes inaccurate calibration. In this paper, therefore, we propose a calibration method of the omnidirectional stereo camera with a large baseline and empirically verify its feasibility.