Objective Apolipoprotein E (APOE) gene is known to influence cerebral functional connectivity (FC) in Alzheimer’s disease continuum. We investigated association between APOE allotypes and FC, structural connectivity, and cortical thickness in amyloid-PET negative cognitive normal older adults (CN). Methods A total of 188 CN (37 had ε2/ε2 or ε2/ε3 [ε2 group], 113 had ε3/ε3 [ε3 group], and 38 had ε3/ε4 or ε4/ε4 [ε4 group]) were recruited. Voxel-based morphometry and cortical thickness analysis were used to investigate differences in cortical thickness between three APOE allotypes. To investigate integrity of structural connectivity, we analyzed diffusion weighted imaging using fractional anisotropy and mean diffusivity. In terms of FC, differences of FC in default mode network (DMN) among APOE allotypes were measured using functional magnetic resonance imaging. Results There were no significant differences in age, sex, education, cerebral beta-amyloid (Aβ) deposition severity, or neuropsychological profiles. No significant differences were found in cortical thickness and structural connectivity among the APOE allotypes. However, FC within the DMN was significantly lower in ε4 and ε2 carriers compared to ε3 homozygotes. Conclusion This study suggests that both ε4 and ε2 exhibit APOE-associated DMN FC changes before Aβ deposition, structural changes, and neurodegeneration.