In transmitting compressed video bit-stream over Internet, packet loss causes error propagation in both spatial and temporal domain, which in turn leads to severe degradation in image quality. In this paper, a new approach for the recovery of lost or erroneous Motion Vector(MV)s by classifying the movements of neighboring blocks by their homogeneity is proposed. MVs of neighboring blocks are classified according to the direction of MVs and a representative value for each class is determined to obtain the candidate MV set. By computing the distortion of the candidates, a MV with the minimum distortion is selected. Experimental results show that the proposed algorithm exhibits better performance in many cases than existing methods.