The motion-based video segmentation provides a powerful method of video compression, because it defines a region with similar motion, and it makes video compression system to more efficiently describe motion video. In this paper, we propose the Modified Fuzzy Competitive Learning Algorithm (MFCLA) to improve the traditional K-menas clustering algorithm to implement the motion-based video segmentation efficiently. The segmented region is described with the affine model, which consists of only six parameters. This affine model was calculated with optical flow, describing the movements of pixels by frames. This method could be applied in the low bit rate video transmission, such as video conferencing system.