Behind the popularity of VoIP in these days, it may present significant security challenges in privacy and accounting. Authentication and message encryption are considered to be essential mechanisms in VoIP to be comparable to PSTN. SIP is responsible for setting up a secure call in VoIP. SIP employs TLS, DTLS or IPSec combined with TCP, UDP or SCTP as a security protocol in VoIP. These security mechanisms may introduce additional overheads into the SIP performance. However, this overhead has not been understood in detail by the community. In this paper we present the effect of the security protocol on the performance of SIP by comparing the call setup delays among security protocols. We implement a simulation of the various combinations of three security protocols and three transport layer protocols suggested for SIP. UDP with any combination of security protocols performs a lot better than the combination of TCP. TLS over SCTP may impose higher impact on the performance in average because TLS might have to open secure channels as the same number of streams in SCTP. The reasons for differences in the SIP performances are given.