The average consensus problem of high-order multi-agent systems with multiple time-varying communication delays is investigated in this paper. By using the idea of state decomposition, the condition for guaranteeing average consensus is converted into verifying the stability of zero equilibrium of disagreement system. Considering multiple time-varying communication delays, Lyapunov-Krasovskii approach in time-domain is employed to analyze the stability of zero equilibrium. With the help of Free-weighting Matrices (FWM) approach, the tolerant upper bounds on communication delays can be obtained through solving feasible linear matrix inequalities (LMIs). Delay-dependent stability criteria for both strongly-connected fixed and switching topologies are provided in the main results. Further, the conclusion is extended to the case of jointly-connected switching topologies. Numerical examples and simulation results are given to demonstrate the effectiveness and the benefit on reducing conservativeness of the proposed method.