This study investigates the difference in the evaluation of foreign-accentedness of English pronunciation across subject groups, evaluated accents, and compared components. This study independently evaluates the prosody and segmentals of the foreign-accented English sentences by pairwise difference rating. Using the prosody swapping technique, segmentals and prosody of the English sentences read by native speakers of American English (one male and one female) were combined with the corresponding segmentals and prosody of the English sentences read by male and female native speakers of Chinese, Japanese or Korean (one male and one female from each native language). These stimuli were evaluated by 4 different subject groups: native speakers of American English, Korean, Chinese, and Japanese. The results showed that the Japanese subject group scored higher in prosody difference than in segmental difference while the other groups scored the other way around. This study is significant in that the attitude toward the difference in segmentals and prosody of the foreign accents of English varies with the native language of the subject group. In other words, for native speakers of some languages, the difference in prosody could have a greater influence on the foreign-accentedness than the difference in segmentals, while for native speakers of other languages the other way around.