As the electrical power industry is restructured, the electrical power exchange is extended. One of the key information used to determine how much power can be transferred through the network is known as available transfer capability (ATC). To calculate ATC, traditional deterministic approach is based on the severest case, but the approach has the complexity of procedure. Therefore, novel approach for ATC calculation is proposed using cost optimization in this paper Cheju Island interconnected HVDC system with mainland in KEPCO (Korean Electric Power Corporation) systems, and the demand of Cheju Island increases about 10 ($\%$) every year. To supply for increasing demand, the supply of HVDC system must be increased. This paper proposed the optimal transfer capability of HVDC system between Haenam in mainland and Cheju in Chju Island through cost optimization. The cost optimization is considered production cost in Cheju Island, wheeling charge through Haenam-Cheju HVDC system and outage cost with one depth (N-1 contingency)