Reliance on the Internet has introduced Voice over Internet Protocol (VoIP) to various security threats. A reliable security protocol and an authentication scheme are thus required to prevent the aforementioned threats. However, an authentication scheme often demands additional cost and effort. Accordingly, a security framework for known participants in VoIP communication is proposed in this paper. The framework is known as Randomness-Optimized Self-Securing (ROSS), which performs authentication automatically throughout the session by optimizing the uniqueness and randomness of the communication itself. Elliptic Curve Diffie-Hellman (ECDH) key exchange and Salsa20 stream cipher are utilized in the framework correspondingly to secure the key agreement and the communication with low computational cost. Human intelligence supports ROSS authentication process to ensure participant authenticity and communication regularity. The results show that with marginal overhead, the proposed framework is able to secure VoIP communication by performing reliable authentication.