Storing and estimating the high order probability distribution of classifiers and class labels is exponentially complex and unmanageable without an assumption or an approximation, so we rely on an approximation scheme using the dependency. In this paper, as an extended study of the second-order dependency-based approximation, the probability distribution is optimally approximated by the third-order dependency. The proposed third-order dependency-based approximation is applied to the combination of multiple classifiers recognizing handwritten numerals from Concordia University and the University of California, Irvine and its usefulness is demonstrated through the experiments.