Purpose: Though web-based tests have many advantages, they do not become the main test form in medical education because of their many technical or operational limitations. We have developed a new web-based test system to overcome these limitations and, ultimately, to replace conventional paper and pencil tests. Methods: We developed a web-based test system using a windows 2000 server, mySQL, visual basic 6.0 and ASP script as developing tools. We applied this system to formal examinations. The system was modified and optimized based on users’ demands and problems identified during testing. Results: Since web-based testing was first administered in 2003 to the third-year students during their clinical practice examinations, it has been used in 51 examinations at the College of Medicine in Chungbuk National University. In 2004, it was used in more than a half of the regular examinations at our medical college. At the beginning, lecturers were reluctant to adopt a new test system because of unfamiliarity and fear of additional workload. Shortly, however, they found that they were able to handle the multimedia contents more easily and obtain much information from results analysis without additional work. Examinees also favored web-based tests in presenting multimedia data but had some difficulties in focusing on the test. Overall, examinees’ preference for web‐based tests was comparable to conventional tests. According to user demand, we improved this test system by verifying question formats and making test analysis. Conclusion: Web-based tests can replace paper and pencil tests in medical education. Improvement of this testing system is expected with serious and continuous concerns from examiners.