The GMDH(Group Method of Data Handling) algorithm can be used to predict the complex nonlinear systems. The traditional GMDH algorithm produces the prdicted output of the system model in the output layer through the input layer and the intermediate layers as the prescribed process. The outputs of each layer are produced only by the outputs of the former layer. However, in the traditional GMDH algorithm, though the optimal structure of each layer is derived, the overall structure may not be derived optimally. To overcome this problem, GMDH prediction model which has the overall optimal structure is constructed by feeding back the error between the predicted output and the real output. This can make the prediction more precise. The capability improvement of the proposed algorithm compared to the traditional algorithm is verified through computer simulation.