Polypropylene/layered silicate nanocomposites containing maleic anhydride grafted polypropylene were prepared by melt compounding and their rheological behavior was investigated in shear flow. Transient and steady shear flows were simulated numerically by using the K-BKZ integral constitutive equation along with experimentally determined damping functions under dynamic oscillatory and step strain shear flows. Nonlinear shear responses were predicted with the K-BKZ constitutive equation using two different damping functions such as the Wagner and PSM models. It was observed that PP-g-MAH compatibilized PP/layered silicate nanocomposites have stronger and earlier shear thinning and higher steady shear viscosity than pure PP resin or uncompatibilized nanocomposites at low shear rate regions. Strong damping behavior of the PP/layered silicate nanocomposite was predicted under large step shear strain and considered as a result of the strain-induced orientation of the organoclay in the shear flow. Steady shear viscosity of the pure PP and uncompatibilized nanocomposite predicted by the K-BKZ model was in good agreement with the experimental results at all shear rate regions. However, the model was inadequate to predict the steady shear viscosity of PP-g-MAH compatibilized nanocomposites quantitatively because the K-BKZ model overestimates strain-softening damping behavior for PP/layered silicate nanocomposites.