In this paper, we propose a fast ME (motion estimation) algorithm for MPEG-4 to H.264 Transcoder. Whereas 2 modes ($8{ imes}8$, $16{ imes}16$) are used for ME in MPEG-4 simple profile, ME using 7 modes is supported for further enhanced coding efficiency in H.264. The transcoding speed is affected dominantly by the computational complexity of encoder part in transcoder, where ME module of H.264 encoder has high complexity due to using 7 modes. In order to increase the speed of transcoding between MPEG-4 and H.264, we use 3 PMVs (predicted motion vectors) and the mode information of MBs (macroblocks) provided from the decoder part of transcoder. Since the proposed 3 PMVs are very close to an optimal motion vector, and we consider only some restricted modes according to the MB information transferred from decoder part, the proposed scheme can speed up the transcoding procedure without loss of image quality. We show experimental results which demonstrate the effectiveness of the proposed algorithm, where performance of our scheme is compared with that of the conventional fast algorithm for H.264.