Visual features of a mushroom(Lentinus Edodes, L.) are critical in grading and sorting as most agricultural products are. Because of its complex and various visual features, grading and sorting of mushrooms have been done manually by the human expert. To realize the automatic handling and grading of mushrooms in real time, the computer vision system should be utilized and the efficient and robust processing of the camera captured visual information be provided. Since visual features of a mushroom are distributed over the front and back sides, recognizing sides and states of the stalk including the stalk orientation from the captured image is a prime process in the automatic task processing. In this paper, the efficient and robust recognition process identifying the front and back side and the state of the stalk was developed and its performance was compared with other recognition trials. First, recognition was tried based on the rule set up with some experimental heuristics using the quantitative features such as geometry and texture extracted from the segmented mushroom image. And the neural net based learning recognition was done without extracting quantitative features. For network inputs the segmented binary image obtained from the combined type automatic thresholding was tested first. And then the gray valued raw camera image was directly utilized. The state of the stalk seriously affects the measured size of the mushroom cap. When its effect is serious, the stalk should be excluded in mushroom cap sizing. In this paper, the stalk removal process followed by the boundary regeneration of the cap image was also presented. The neural net based gray valued raw image processing showed the successful results for our recognition task. The developed technology through this research may open the new way of the quality inspection and sorting especially for the agricultural products whose visual features are fuzzy and not uniquely defined.