Though authentication using biometric techniques has conveniences for people, security problems like the leakage of personal bio-information would be serious. Even if cancelable biometric is a good solution for the problems, only a few biometric authentication scheme with cancelable feature has been published. In this paper, we suggest a face authentication scheme with two security factors: password and face image. Using matching algorithm in the permuted domain, our scheme is designed to be cancelable in the sense that templates that is composed of permutation and weight vector can be changed freely.