Damage often occurs on the surface of railway wheel by wheel-rail contact fatigue. It should be removed before reaching wheel failure, because wheel failure can cause derailment with loss of life and property. The increase or decrease of the contact fatigue life by the metal removal of the contact surface were shown by many researchers, but it has not explained precisely why fatigue life increases or decreases. In this study, the effect of metal removal depth on the contact fatigue life for railway wheel has been evaluated by applying finite element analysis. It has been revealed that the residual stress and the plastic flow are the main factors determining the fatigue life. The railway wheel has the initial residual stress formed during the manufacturing process, and the residual stress is changed by thermal stress induced by braking. It has been found that the initial residual stress determines the amount of metal removal depth. Also, the effects of the initial residual stress and metal removal on the contact fatigue lift has been estimated, and an equation is proposed to decide the optimal metal removal depth for maximizing the contact fatigue life.