The big challenge in current content-based image retrieval systems is to reduce the semantic gap between the low level-features and high-level concepts. In this paper, we have proposed a novel framework for efficient image retrieval to improve the retrieval results significantly as a means to addressing this problem. In our proposed method, we first extracted a strong set of image features by using the dual-tree rotated complex wavelet filters (DT-RCWF) and dual tree-complex wavelet transform (DT-CWT) jointly, which obtains features in 12 different directions. Second, we presented a relevance feedback (RF) framework for efficient image retrieval by employing a support vector machine (SVM), which learns the semantic relationship among images using the knowledge, based on the user interaction. Extensive experiments show that there is a significant improvement in retrieval performance with the proposed method using SVMRF compared with the retrieval performance without RF. The proposed method improves retrieval performance from 78.5% to 92.29% on the texture database in terms of retrieval accuracy and from 57.20% to 94.2% on the Corel image database, in terms of precision in a much lower number of iterations.