This paper proposes a novel bit allocation algorithm for H.264 scalable extension(SE) based on a human visual system (HVS) to improve the coding efficiency. The proposed algorithm is consist of two stages: visual weighting model and visual weighting-based bit allocation algorithm. In the first stage, the visual weighting for each macroblock (MB) is analyzed according to the region of interests. Then the adaptation of the visual weighting into the bit allocation routine for each quality layer is performed for improving the visual quality. In the simulation results, it is observed that the proposed scheme can improve the subjective and objective video quality in the same bit rate, compared to the previous scalable video coding in H.264.