This paper proposes a probability measure for the evaluation of relay performance from two aspects, namely, correct trip and no-unwanted trip. They are developed based on the relationship between relay settings and relay measurements, which follow a Gaussian probability model. The proposed method based on strict mathematical derivation is applied to protection capability evaluation of distance relays under various settings. Considering the specific attributes of each protection zone, the optimal settings are also determined accordingly. The protection capability could demonstrate clearly the relay performance under various settings and the optimal settings could provide good references for engineering applications.