This paper provides simulation-based results of effect analysis of sample size and sampling periods on accuracy of reliability estimation methods using multiple comparisons with analysis of variance. Sum of squared errors in estimated reliability measures were evaluated through applying seven estimation methods for one-shot systems to simulated quantal-response data. Analysis of variance was implemented to investigate change in these errors according to variations of sample size and sampling periods for each estimation method, and then the effect analysis on accuracy in reliability estimation was performed using multiple comparisons based on sample size and sampling periods. An efficient way to allocate both sample size and sampling periods for reliability estimation tests of one-shot systems is proposed in this paper from the effect analysis results.