Advances of digital device technology such as digital cameras, smart phones and tablets, provide convenience way for people to take pictures during his/her life. Photo data is being spread rapidly throughout the social network, causing the excessive amount of data available on the internet. Photo retrieval is categorized into three types, which are: keyword-based search, example-based search, visualize query-based search. The commonly used multimedia search methods which are implemented on Smart TV are adapting the previous methods that were optimized for PC environment. That causes some features of the method becoming irrelevant to be implemented on Smart TV. This paper proposes a novel Visual Query-based Photo Retrieval Method in Smart TV Environment using a motion sensing input device known as Kinect Sensor. We detected hand gestures using kinect sensor and used the information to mimic the control function of a mouse. The average precision and recall of the proposed system are 81% and 80%, respectively, with threshold value was set to 0.7.