To increase the benefit of a pre/post test design, the validity and reliability of the research should be carefully considered, since a pre/post test design require tests to be conducted multiple times. The current study developed a reading fluency test with a pre/post test design that aimed to measure the effect of the intervention on English learners. The developed reading fluency test was piloted twice and subsequently revised according to the pilot test results. According to the results of the pilot studies, the current study a) increased the test validity by measuring not only reading rate but also comprehension and b) strengthened the test reliability by using different reading texts that took into consideration the comparability of the readability of the text used across the pre and post tests.