This paper proposes an interaction field concept based on the field theory of plasticity. Relative deformation between two arbitrary scales, e.g., macro and micro fields, is defined which can be implemented in the crystal plasticity-based constitutive framework. Differential geometrical quantities responsible for describing dislocations and defects in the interaction field are obtained, based on which dislocation density and incompatibility tensors are further derived. It is shown that the explicit interaction exists in the curvature or incompatibility tensor field, whereas no interaction in the torsion or dislocation density tensor field. General expressions of the interaction fields over multiple scales with more than three scale levels are derived and implemented into the present constitutive equation.