Software as a service (SaaS) is one of the most-talked about trends in IT. Unlike traditional perpetual licensing model, software applications are sold on subscription bases and services are provided over web by the vendors. It is said that SaaS can make vendors to invest more on R&D than on marketing while offering its customers better quality software applications at lower costs. By empirically comparing vendors providing their software applications either by SaaS or by traditional perpetual licensing model, we examine whether or not SaaS really increases overall customer benefits in terms of cost efficiency, software quality, and customization. We show that SaaS may not provide better quality or cost efficient software applications than perpetual licensing does. Then we provide two practical tools which are useful for customers to evaluate whether SaaS is better than perpetual licensing for the purposes of software applications they want to adopt.