Score Changes with Repetition of Paper Version(s) of the TOEFL in an Arab Gulf State: A Natural Experiment

Although replaced by the internet-based TOEFL (iBT), the paper-based TOEFL (PBT) was still used in some countries until July 2017. This natural experiment investigates a case where, November 2006–May 2009, a testing center in the Middle East did not follow TOEFL time interval protocol at that time, with 1,575 test takers allowed to attempt paper TOEFL versions as frequently as desired. This study analyzes score increase/decrease with frequency of/time intervals between attempts at taking paper TOEFL version(s). These test takers attempted the TOEFL multiple times, with 48% taking it at least twice, some up to 22 times. Results indicate a negative relationship between number of attempts and score increases/decreases between consecutive tests greater than Standard Error of Measurement (>SEM), and a positive relationship between score change and average time interval. Score decreases showed greater variability, with smaller average time interval than increases. Percentages of individuals’ score decrease averaged in repeater group positively correlated with number of attempts. Consecutive test scores decreased (>SEM) at least once for 900 test takers. Score decreases showed greater variability and significantly smaller average time intervals than increases. There was very strong positive correlation between increase in number of attempts and consecutive score decreases. Repetitions within one week showed greater score variations. These findings raise questions about construct-irrelevant effects of test repetition and time interval on score change. One explanation is that score increases >SEM occurred gradually over more time, in smaller amounts, in keeping with increases in ability through learning, but decreases >SEM occurred in shorter time intervals and larger amounts, possibly reflecting more immediate negative affect-related factors.