A digital hologram generated by a computer calculation (computer-generated hologram or capture using charge-coupled device [CCD] camera) is one of the most expensive types of content, and its usage is expanding. Thus, it is highly necessary to protect the ownership of digital holograms. This paper presents an efficient visual security scheme for holographic image reconstruction with a low scrambling cost. Most recent studies on optical security concentrate their focus on security authentication using optical characteristics. However, in this paper, we propose an efficient scrambling method to protect a digital hologram. Therefore, we introduce in this paper several scrambling attempts in both the spatial domain and frequency domain on the basis of the results of analyzing the properties of the coefficients in each domain. To effectively hide the image information, 1/4, 1/256, and 1/16,384 of the original digital hologram needs to be scrambled for the spatial-domain scheme, Fresnel-domain scheme, and discrete cosine transform-domain scheme, respectively. The encryption schemes and the analyses in this paper can be expected to be useful in the research on encryption and other works on digital holograms.