In residential house, photovoltaic (PV) system among various alternatives in renewable energy system is the most efficient and feasible solution for reducing energy consumption and electricity cost. However, relatively high initial cost make people reluctant to install PV system in their houses. Therefore, in the initial state for PV system installation in the house, it is very important to decide proper capacity of the PV system considering the expected energy usage and solar energy supplying condition with the house. This paper proposes a novel optimization model for deciding appropriate capacity of the PV system for residential house. The objective function of the model is to minimize the annual cost including electricity bill, operation and maintenance cost, and annual fixed cost calculated from the initial installation cost based on capital recovery factor (CRF). The model also shows the optimal inclining angle of PV panels of the system. In this paper, we estimate the PV output using PVWATTS (PV simulator of Office of Energy Efficiency and Renewable Energy) and find optimal solutions by Sequential Quadratic Programming (SQP) method using MATLAB software. The proposed approach is finally applied to a residential model house in Gangneung, Gangwon-Do and verified its feasibility for adopting to PV system design for residential houses.