In tangible augmented reality (AR) environments, the user interacts with virtual objects by manipulating their physical counterparts, but he or she often encounters awkward situations in which his or her hands are occluded by the augmented virtual objects, which causes great difficulty in figuring out hand positions, and reduces both immersion and ease of interaction. To solve the problem of such hand occlusion, skin color information has been usefully exploited. In this paper, we propose an approach to simple and effective construction of a skin color map which is suitable for hand segmentation and tangible AR interaction. The basic idea used herein is to obtain hand images used in a target AR environment by simple image subtraction and to represent their color information by a convex polygonal map in the YCbCr color space. We experimentally found that the convex polygonal map is more accurate in representing skin color than a conventional rectangular map. After implementing a solution for resolving hand occlusion using the proposed skin color map construction, we showed its usefulness by applying it to virtual design evaluation of digital handheld products in a tangible AR environment.