A three-dimensional computational fluid dynamics (CFD) model with the renormalization group (RNG) $k-{varepsilon}$ turbulence model is used to examine the effects of difference in building height on flow and pollutant dispersion in asymmetric street canyons. Three numerical experiments with different street canyons formed by two isolated buildings are performed. In the experiment with equal building height, a portal vortex is formed in the street canyon and a typical recirculation zone is formed behind the downwind building. In the experiment with the downwind building being higher than the upwind building, the ambient flow comes into the street canyon at the front of the downwind building and incoming flow diverges strongly in the street canyon. Hence, pollutants released therein are strongly dispersed through the lateral sides of the street canyon. In the experiment with the upwind building being higher than the downwind building, a large recirculation zone is formed behind the upwind building, which is disturbed by the downwind building. Pollutants are weakly dispersed from the street canyon and the residue concentration ratio is largest among the three experiments. This study shows that the difference in upwind and downwind building height significantly influences flow and pollutant dispersion in and around the street canyon.