A rigorous theory for modeling, analysis, optimum nonseparable filter bank in multidimensional M-band quantized subband codec are developed in this paper. Each pdf-optimized quantizer is modeled by a nonlinear gain-plus-additive uncorrelated noise and embedded into the subband structure. We then decompose the analysis/synthesis filter banks into their polyphase components and shift the down-and up-samplers to the right and left of the analysis/synthesis polyphase matrices respectively. Focusing on the slow clock rate signal between the samplers, we derive the exact expression for the output mean square quantization error by using spatial-invariant analysis. We show that this error can be represented by two uncorrelated components : a distortion component due to the quantizer gain, and a random noise component due to fictitious uncorrelated noise at the uantizer. This mean square error is then minimized subject to perfect reconstruction (PR) constraints and the total bit allocation for the entire filter bank. The algorithm gives filter coefficients and subband bit allocations. Numerical design example for the optimum nonseparable orthonormal filter bank is given with a quincunx subsampling lattice.