This study investigates the effects of visual texture on color emotion and establishes prediction models for color emotion by both physical color properties and visual texture characteristics. A variety of fabrics including silk, cotton, and flax were colored by digital textile printing according to chromatic hue and tone combinations that are evaluated in terms of color emotion. Subjective visual texture ratings are also obtained for gray-colored same fabrics to those used in color emotion tests. As a result, fabric clusters by visual texture factors showed significant differences in color emotion factors that are primarily affected by physical color properties. Finally prediction models for color emotion factors by both physical color properties and visual texture clusters were established, which has a potential to be used to explain color emotion according to the visual texture characteristics of fabrics.