In this paper, we propose a fast and robust algorithm for fighting behavior detection based on Motion Vectors (MV), in order to solve the problem of low speed and weak robustness in traditional fighting behavior detection. Firstly, we analyze the characteristics of fighting scenes and activities, and then use motion estimation algorithm based on block-matching to calculate MV of motion regions. Secondly, we extract features from magnitudes and directions of MV, and normalize these features by using Joint Gaussian Membership Function, and then fuse these features by using weighted arithmetic average method. Finally, we present the conception of Average Maximum Violence Index (AMVI) to judge the fighting behavior in surveillance scenes. Experiments show that the new algorithm achieves high speed and strong robustness for fighting behavior detection in surveillance scenes.