LTE/SAE has presented the handover key management to revoke the compromised keys and to isolate corrupted network devices. In this paper, we identify that the handover key management is vulnerable to de-synchronization attacks, which is jeopardizing the forward secrecy of handover key management. Also, an adversary could prevent the UE from creating the secure link with eNodeB, which is delaying the handover procedure. In this paper, we present a counrermeasure to prevent above attacks, and analyze the performance issues of the proposed protocol.