Type I error rates for ANOVA, GLM, and GLMM for binary outcomes when simulating from binomial logistic model and Ratcliff diffusion model

Logistic regression models are often recommended for analysing binary response variables, such as accuracy, that commonly arise in psychological research designs. One of the main reasons for this recommendation are simulation studies showing that binomial logistic models outperform ordinary ANOVA models when simulating from a binomial logistic model. However, such a simulation setup is at risk of circularity as the logistic model is both the data generating and winning candidate model. To overcome this limitation, we compared different candidate models when simulating from two data generating models – a binomial logistic model and the Ratcliff diffusion model. For each simulation study, we simulated a two-group between-participants design with 30 participants per condition. We also varied the number of observations per simulated participant, either 1 observation or 100 observations. We then compared the type I error rates (i.e., the proportion of false positive errors) from three popular candidate methods, linear regression (ANOVA), generalised binomial logistic regression (GLM), and generalised binomial logistic mixed models (GLMM). Our results suggested that ANOVA shows the best performance in terms of type I errors across different simulation setups and data generated by both logistic and diffusion models. For the GLM, the type I error rate was around 0.05 only for 1 observation per participant and severely anti-conservative (i.e., too high type I error) for 100 observations. GLMM yielded an acceptable type I error rate with 100 observations per participant but varied amounts of type I errors dependent on the data generation models with 1 observation per participant. When simulating from the logistic model GLMM produced acceptable type I error rates but too high type I error rates when simulating from the diffusion model. Additionally, the type I errors from GLMM with 1 observation per participant increased as the overall performance level approached the boundary of the parameter space. Overall, our results suggest that in terms of type I error rates, ANOVA generally perform better than logistic models in most cases and the performance of logistic models depend exactly on the simulation setup.