The color-based vision systems have been used to recognize our team robots, the opponent team robots and a ball in the robot soccer system. The color-based vision systems have the difficulty in that they are very sensitive to color variations brought by brightness changes. In this paper, a neural network trained with data obtained from various illumination conditions is used to classify colors in the modified YUV color space for the robot soccer vision system. For this, a new method to measure brightness is proposed by use of a color card. After the neural network is constructed, a look-up-table is generated to replace the neural network in order to reduce the computation time. Experimental results show that the proposed color classification method is robust under illumination variations.