This study examined perceived discomforts encountered in residential facilities such as individual houses and apartments, and suggested a procedure for improving the problems in the living residence. The participants consisted of two groups of people: normal and underprivileged people; the underprivileged group was classified into 3 sub-groups: elderly people (over 65 years old), handicapped people and pregnant women. To identify inconvenient places and design factors in the residential housing, a total of 200 subjects (50 for each group) participated in the survey using the 7-point Likert scale. As a result, all the user groups mainly answered a high degree of discomfort due to narrow parking places, doorsills, and height differences between floors. The elderly, the handicapped, and the pregnant women replied that it is uncomfortable to use ramps and utility rooms. In a follow-up study, one-to-one in-depth interviews were conducted with some of the participants to analyze the detailed causes of inconvenient places and design factors. This study also suggested the procedure for solving the problems in various design factors from perspective of universal design, by considering prior studies which focused on a certain user group like the elderly. This procedure was suggested to consider as many user groups as possible at the same time. The results can be used to understand the discomforts of various user groups on residential housing. The procedure can also be used to develop universal design guidelines for the living residence.