Objectives: The purpose of this study was to test factor structure and latent mean invariance of general self-esteem between people with and without physical disabilities. Methods: Data were obtained from the 15th wave in Korean Welfare Panel Study. This study assessed the following: a) the goodness of fit of seven alternative factor models for the Rosenberg’s Self-Esteem Scale(RSES); b) the multigroup invariance of global self-esteem and method effects associated with positively and negatively worded items on the RSES between people with and without physical disabilities; c) the latent mean invariances of self-esteem between people with and without physical disabilities, using multigroup structural equation modeling. Results: First, the configural invariance of the RSES and the weak and strong invariances were supported. However, the findings of this study show that the factor structure of the RSES is contaminated by method effects associated with positively and negatively worded items. Second, the mean self-esteem factor of people with physical disabilities was significantly lower than that of people without physical disabilities. Conclusions: The results suggest that there were few differences between people with and without physical disabilities in terms of the factor structure and method effects associated with positively and negatively worded items of the RSES, indicating that people with and without physical disabilities interpret the self-esteem construct of the RSES items similarly.