The pair-induced quenching (PIQ) effect in a highly doped bismuth oxide-based erbium-doped fiber amplifier (EDFA) was theoretically and experimentally investigated. In the theoretical investigation, the bismuth oxide-based EDFA was modeled as a 6-level amplifier system that incorporated clustering-induced concentration quenching, cooperative up-conversion, pump excited state absorption (ESA), and signal ESA. The relative number of paired ions in a highly doped bismuth oxide EDF was estimated to be ~6.02%, determined by a comparison between the theoretical and the experimentally measured gain values. The impacts of the PIQ on the gain and the noise figure were also investigated.