A novel compact model for a five-port transformer balun is proposed for the efficient circuit design of hybrid balun. Compared to the conventional model, the proposed model provides much faster computation time and more reasonable values for the extracted parameters. The hybrid balun, realized in $0.18;{mu}m$ CMOS, achieves 2.8 dB higher gain and 1.9 dB lower noise figure than its passive counterpart only at a current consumption of 0.67 mA from 1.2 V supply. By employing the hybrid balun, a differential zero-IF receiver is designed in $0.18;{mu}m$ CMOS for IEEE 802.15.4 ZigBee applications. It is composed of a differential cascode LNA, passive mixers, and active RC filters. Comparative investigations on the three receiver designs, each employing the hybrid balun, a simple transformer balun, and an ideal balun, clearly demonstrate the advantages of the hybrid balun in fully differential CMOS RF receivers. The simulated results of the receiver with the hybrid balun show 33 dB of conversion gain, 4.2 dB of noise figure with 20 kHz of 1/f noise corner frequency, and -17.5 dBm of IIP3 at a current consumption of 5 mA from 1.8 V supply.