Comparing and exploring modeling solutions to the reliability paradox in conflict tasks

The study of individual differences in cognitive control using conflict tasks such as the Stroop task has proven difficult. Despite robust experimental effects, the reliability of individual differences tends to be low, and correlations between tasks are weak at best. A statistical explanation for this reliability paradox is that individual differences are masked by trial-to-trial variability and are too small to be detected. Modeling recommendations to improve the assessment of individual differences include the use of trial-level hierarchical models that account for trial noise, the use of descriptively more accurate models that account for the skewness of response time data, and the use of models that make cognitively more plausible assumptions, such as race or competitive models. At the same time, we may fall into the trap of overfitting. In this talk, we will compare Bayesian hierarchical models of increasing complexity with respect to their signal-to-noise ratio, i.e., the ratio of the amount of "true" individual differences (i.e., the signal) to the trial-by-trial variability (i.e., the noise). This ratio has been proposed to indicate the degree of attenuation that can be expected in correlational research in the area of cognitive control (typically 1 to 7). By combining the most powerful modeling techniques and using progressively more complex models, can we optimize the signal-to-noise ratio and gain increasing resolution for individual differences?