A modified strain gradient theory is proposed based on the nonhomogeneity of polycrystalline metallic materials. Geometrically necessary dislocations are generated on the slip planes as well as on the grain boundary to accommodate deformation with minimum internal stress. Since amount of the geometrically necessary dislocation depends on the deformation shape, specimen size and grain size, it is an important factor for the modified strain gradient theory and the size effect. This new theory differs from the mechanism based strain gradient plasticity in its consideration of the geometrically necessary dislocations on the grain boundary and free surface effect. This theory provides a possible explanation for conflicting size effect: the smaller can be either harder or softer due to the deformation. Using the proposed theory, analysis of the effect of both specimen and grain size under the plane bulge test of polycrystalline materials is performed.