The adaptive coding schemes in H.264 standardization provide a significant ceding efficiency and some additional features like error resilience and network friendliness. The variable block size motion compensation using multiple reference frames is one of the key H.264 coding elements to provide main performance gain, but also the main culprit that increases the overall computational complexity. For this reason, this paper proposes a selective motion estimation algorithm based on variable block size for fast motion estimation in H.264. After we find the SAD(Sum of Absolute Difference) at initial points using diamond search, we decide whether to perform additional motion search in each block. Simulation results show that the proposed method is five times faster than the conventional full search in case of search range $pm$32.