Electrical discharge machining (EDM) can easily machine the high strength materials. However, EDM is conventionally the up-and-down machining for the mold and hole; grooves and curved surfaces on the side walls of the blind hole are difficult produced in general EDM. This study developed two electrodes that could create the lateral discharge on the workpiece in EDM. One is an elastic electrode, where two thin copper plates are stuck on the two rectangular pieces of spring steel that could produce a lateral deformation in EDM. This electrode uses the driving force to change the lateral shape of the steel plate to perform the side discharge machining; the other one is the lateral feeding electrode, where two inclined electrodes with angle $45^{circ}$ in the up and down position are jostled to create side deformation, and side grooves or side curved surface could be performed in the discharge process. As a result, the elastic electrode performs poor efficiency in the lateral EDM because of the fast wearing of the thin copper plate. However, lateral feeding electrode enhances the discharge effect of the elastic electrode and results in good efficiency of the plow grooves or curved surface produced in the blind hole.