Type I error in diffusion models: A drift towards false positives?
Diffusion models are one of the most used tools to analyze reaction times (RT), and their relevance keeps growing overtime. According to these models, evidence is accumulated over time, until a threshold is reached, leading to a response. Contrary to simpler RT analysis approaches, these models are equipped with more parameters to be estimated, such as the drift rate, the threshold or non-decisional factors. This allows a more nuanced understanding of the process underlying the decision and response. Unfortunately, this higher number of parameters can also be problematic. We present a series of three simulations with Ratcliff's diffusion model. Simulation 1 used empirical data, Simulation 2 simulated data based on empirically estimated parameters and Simulation 3 was carried out with simulated data based on common distributions of the parameters. The three simulations show that commonly used statistical analyses in diffusion models can lead to an inflation of the Type I error rate. Different strategies to prevent this problem are discussed, including pre-registration of the analysis, model comparisons and Type I error corrections.
The talk gives the impression that the Type I error rate is above nominal levels when comparing parameters of the diffusion model, but I am not sure this is an accurate representation. I think if at all, there are less Type I errors than expected under the null model. Let's begin with the results of Study 3. The probability of not obtaining a si...