A touchless user interface system is proposed for real-time mobile devices in this work. The proposed system first recognizes the position of pattern to interact various applications based on the scene taken from mobile camera. Then, the proposed system traces the route of the pattern, which estimates the motion of pattern based on the mean-shift method. Based on this information, we control menu and keypad for the various applications. Differ from other user interface systems, the proposed system provides the new experience to even for the user of low-end mobile without additional hardware. Simulation results demonstrate that the proposed algorithm provides better interaction performances than the conventional method, while achieving real-time user interacting for mobile devices.