Sequential ANOVA: An Efficient Alternative to Fixed Sample Designs

The replication crisis has shown that researchers often collect insufficient sample sizes for their studies or use questionable research practices such as data peeking. Sequential testing procedures are one possible solution to these shortcomings. The Sequential Probability Ratio Tests (SPRTs) offer interesting possibilities within Frequentist Statistics. Here, a likelihood ratio is calculated as a test statistic, which can be computed continuously by adding new data points. After each iterative step of data collection, SPRTs decide whether the evidence is sufficient to accept the null or alternative hypothesis or whether more data points are needed. SPRTs control for alpha and beta error rates and require, as a test specification, a minimum expected effect size. We show in simulation studies that the one-way sequential ANOVA is very efficient compared to a classical fixed ANOVA. In 87% of the simulated cases, the sequential samples are smaller than the fixed samples. On average, 56% of the data can be saved using the sequential design. However, sequential designs show biases in effect size estimation. Thus, we want to discuss the benefits and limitations of sequential testing. The R package sprtt can be used to calculate sequential versions of t-tests and one-way ANOVAs.