This paper presents an algorithm of optimal cutting tool selection for machining of the point-based surface that is defined by a set of surface points rather than parametric polynomial surface equations. As the ball-end and fillet-end mills are generally used for finish machining in a 3-axis computer numerical control machine, the algorithm is applicable for both cutters. The optimum tool would be as large as possible in terms of the cutter radius and/or corner radius which maximise (s) the material removal rate (i.e., minimise (s) the machining time), while still being able to machine the entire point-based surface without gouging any surface point. The gouging are two types: local and global. In this paper, the distance between the cutter bottom and surface points is used to check the local gouging whereas the shortest distance between the surface points and cutter axis is effectively used to check the global gouging. The selection procedure begins with a cutter from the tool library, which has the largest cutter radius and/or corner radius, and then adequacy of the point-density is checked to limit the accuracy of the cutter selection for the point-based surface within tolerance prior to the gouge checking. When the entire surface is gouge-free with a chosen cutting tool then the tool becomes the optimum cutting tool for a list of cutters available in the tool library. The effectiveness of the algorithm is demonstrated considering two examples.