Traffic anomaly detection is one of important technology that should be considered in network security and administration. In this paper, we propose an abnormal traffic detection mechanism that includes traffic monitoring and traffic analysis. We develop analytical passive monitoring system called WISE-Mon which can inspect traffic behavior. We establish a criterion by analyzing the characteristics of a traffic training set. To detect abnormal traffic, we derive a hyperplane by using Fisher linear discriminant and chi-square distribution as well as the analyzed characteristics of traffic. Our mechanism can support reliable results for traffic anomaly detection and is compatible to real-time detection. In addition, since the trend of traffic can be changed as time passes, the hyperplane has to be updated periodically to reflect the changes. Accordingly, we consider the self-learning algorithm which reflects the trend of the traffic and so enables to increase the pliability of detection probability. Numerical results are presented to validate the accuracy of proposed mechanism. It shows that the proposed mechanism is reliable and relevant for traffic anomaly detection.