Sybil attack has been proved effective in mitigating the P2P botnet, but the impacts of some important parameters were not studied, and no model to estimate the effectiveness was proposed. In this paper, taking Kademlia-based botnets as the example, the model which has the upper and lower bound to estimate the mitigating performance of the Sybil attack is proposed. Through simulation, how three important factors affect the performance of the Sybil attack is analyzed, which is proved consistent with the model. The simulation results not only confirm that for P2P botnets in large scale, the Sybil attack is an effective countermeasure, but also imply that the model can give suggestions for the deployment of Sybil nodes to get the ideal performance in mitigating the P2P botnet.