Abnormal changes in skin color induce significant cosmetic problems and affect quality of life. There are two groups of abnormal change in skin color; hyperpigmentation and hypopigmentation. Hyperpigmentation, darkening skin color by excessive pigmentation, is a major concern for Asian people with yellowe-brown skin. A variety of hypopigmenting agents have been used, but treating the hyperpigmented condition is still challenging and the results are often discouraging. Panax ginseng has been used traditionally in eastern Asia to treat various diseases, due to its immunomodulatory, neuroprotective, antioxidative, and antitumor activities. Recently, several reports have shown that extract, powder, or some constituents of ginseng could inhibit melanogenesis in vivo or in vitro. The underlying mechanisms of antimelanogenic properties in ginseng or its components include the direct inhibition of key enzymes of melanogenesis, inhibition of transcription factors or signaling pathways involved in melanogenesis, decreasing production of inducers of melanogenesis, and enhancing production of antimelanogenic factor. Although there still remain some controversial issues surrounding the antimelanogenic activity of ginseng, especially in its effect on production of proinflammatory cytokines and nitric oxide, these recent findings suggest that ginseng and its constituents might be potential candidates for novel skin whitening agents.