This paper presents balancing control of inverted pendulum on the ROBOKER arm using visual information. The angle of the inverted pendulum placed on the robot arm is detected by a stereo camera and the detected angle is used as a feedback and tracking error for the controller. Thus, the overall closed loop forms a visual servoing control task. To improve control performance, neural network is introduced to compensate for uncertainties. The learning algorithm of radial basis function(RBF) network is performed by the digital signal controller which is designed to calculate floating format data and embedded on a field programmable gate array(FPGA) chip. Experimental studies are conducted to confirm the performance of the overall system implementation.