Matusoka & Kawara (2010) showed that the number density of the most massive galaxies (log $M/M_{odot}=11.5-12.0$) increases faster than that of the next massive group (log $M/M_{odot}=11.0-11.5$) during 0 < z < 1. This appears to be in contradiction to another important empirical concept of "downsizing". We attempt to understand the two observational findings in the context of the hierarchical merger paradigm using semi-analytic techniques. Our models closely reproduce the result of Matusoka & Kawara (2010). Downsizing can also be understood as larger galaxies have on average smaller assembly ages but larger stellar ages. Our fiducial models further reveal the details on the history of stellar mass growth of massive galaxies. The most massive galaxies (log $M/M_{odot}=11.5-12.0$ at z=0), which are mostly brightest cluster galaxies, obtain roughly 70% of their stellar components via merger accretion. The role of merger accretion monotonically declines with galaxy mass: 45% for log $M/M_{odot}=11.0-11.5$ and 20% for log $M/M_{odot}=10.5-11.0$ at z = 0. The specific accreted stellar mass rates via galaxy mergers decline very slowly during the whole redshift range, while the specific star formation rates sharply decrease with time. In the case of the most massive galaxies, merger accretion becomes the most important channel for the stellar mass growth at z ~ 2. On the other hand, in-situ star formation is always the dominant channel in the $L_*$ galaxies.