Calibration is a prerequisite procedure for employing a camera as a 3D sensor in an automated machines like robots. As accurate sensing is possible only when the vision sensor is calibrated accurately, many different approaches and models have been proposed for increasing calibration accuracy. Particularly an important factor which greatly affects the calibration accuracy is the nonlinearity in the mapping between 3D world and corresponding 2D image. In this paper GMDH algorithm is used to learn the nonlinearity without physical modelling. The technique proposed can be effective in various situations where the levels of noises and characteristics of nonlinear distortion are different. In simulations and an experiment, the proposed technique showed good and reliable results.