Statistics

The replication crisis, characterized by issues such as inadequate sample sizes and questionable research practices (QRPs) including data peeking, has prompted the exploration of sequential testing procedures as a potential solution. Notably, Sequential Probability Ratio Tests (SPRTs) are highly efficient compared to traditional fixed designs. In our simulation of 120,000 datasets, the sequential samples were smaller than the fixed samples in 87% of cases. And on average, 56% fewer data points had to be collected with the sequential design. However, QRPs may also emerge in the context of SPRTs. Consequently, we carried out a simulation study in which we applied various strategies aimed at favoring the alternative hypothesis. These ranged from ostensibly well-intentioned or motivated reasoning to outright fraudulent manipulations, including multiple parallel sequential ANOVAs, subgroup and outlier analyses, adjustments in expected effect sizes, reordering of data, and selective data filtering based on their impact on likelihood ratios. Reflecting on our findings, our aim is to discuss the strategies we employed by assessing their severity, and to deepen the understanding of the risks associated with QRPs in the specific context of sequential testing.

This is an in-person presentation on July 20, 2024 (14:00 ~ 14:20 CEST).

Traditional statistical approaches applied to intensive longitudinal data often assume linearity and stationarity. When applied to inherently non-linear processes such as second language acquisition, psychopathology onset and treatment, or emotional valence, these approaches will likely lead to biased conclusions. While various methods for modelling non-linearity like polynomial regression, regression splines, and non-linear dynamic structural equation modelling have gained popularity, many other techniques within the time series literature remain unexplored. Due to the diverse assumptions, inferential capabilities, and types of non-linearity that each of these methods can capture, researchers frequently face challenges in choosing the most suitable approach for their specific context. This talk aims to address this ambiguity by conducting an exhaustive review of the available techniques, ranging from data-driven tools, such as local polynomial regression and Gaussian processes, to fully parametric models within the state space model framework. Further, we conducted simulation studies to compare the efficacy of the different methods in capturing various types of non-linearity at different sample sizes, initially focusing on an n = 1 design. Lastly, we will illustrate what insights and inferences each method can provide by applying them to real intensive longitudinal experience sampling data. This comprehensive analysis intends to empower researchers to choose a statistical method that aligns with their theoretical considerations, research questions, and sample characteristics, when studying non-linear processes. We expect that this will reduce the bias induced by violating the linearity assumption in current experience sampling results and provide novel insights fostering theory development across domains.

This is an in-person presentation on July 20, 2024 (14:20 ~ 14:40 CEST).

Structural equation models (SEMs) are popular tools for investigating structural relationships among latent psychological constructs. In traditional applications, SEMs are estimated on summary scores aggregated across multiple measurements per individual, ignoring the hierarchical structure of the data and assuming that the individual-level data are normally distributed. This approach suffers from two shortcomings. First, failing to account for the nested data structure and the associated measurement uncertainty attenuates estimates of the structural relationships. Second, the assumption of normality is implausible in many applications and fails to provide a substantive psychological account of the processes that give rise to the data. Here we propose a Bayesian hierarchical SEM framework that addresses both limitations. Our approach allows researchers to flexibly model the individual-level data, ranging from the traditional normal distribution to generative cognitive models, such as evidence-accumulation or reinforcement learning models. The joint hierarchical estimation of the individual-level model, and the structural relationships among the latent constructs extracted from it, takes into account measurement uncertainty and hence safeguards against attenuation, and enables the use of Bayesian model selection techniques. We showcase how hierarchical SEMs can be used to test the latent structure of psychological constructs extracted from a psychological model fit to data obtained from multiple conditions, tasks, or data streams.

This is an in-person presentation on July 20, 2024 (14:40 ~ 15:00 CEST).

Cognitive models, such as evidence accumulation models, have become increasingly popular in individual differences research in psychology and neuroscience. By computing correlations among cognitive model parameters, researchers aim to understand how the cognitive processes they represent relate to each other and jointly determine performance. It is generally recognized that cognitive model parameters can be challenging to estimate due to their sloppiness, that is the strong within-participant correlations among the parameters encapsulated in the likelihood of the models. However, it is rarely acknowledged that sloppiness can lead to spurious between-participant correlations and hence result in incorrect substantive conclusions about individual differences. Consider, for instance, the diffusion decision model, a prominent cognitive model of speeded decision making. In the presence of limited between-participant variability in model parameters, the strong negative within-participant correlation between the response caution and non-decision time parameters can masquerade as a between-participant correlation, leading to the spurious conclusion that individuals with more cautious decision making are quicker at encoding the stimulus and executing the corresponding response. In this talk, we explore how single-level parameter estimation, Bayesian and non-Bayesian alike, can result in spurious between-participant correlations reflecting within-participant correlations rather than individual differences. We then show that the appropriately parameterized Bayesian hierarchical model can protect against spurious correlations. We discuss the consequences of this statistical artifact and offer recommendations for identifying and guarding against the resulting inferential biases.

This is an in-person presentation on July 20, 2024 (15:00 ~ 15:20 CEST).

The illusory truth effect refers to the phenomenon that repeated exposure to a statement increases its perceived truthfulness. In truth-effect studies, binary judgments are usually aggregated within subjects, yielding proportions between 0 and 1. These values are then used as the dependent variable in an analysis of variance (ANOVA). However, this procedure has several limitations. First, it assumes that all statements in the study are homogeneous, even though they vary in terms of many properties. Second, proportions are subject to floor and ceiling effects, causing violations of model assumptions such as heteroscedasticity and impossible predictions beyond 0 and 1. Third, the ANOVA approach does not allow to add trial-level predictors. A solution to these issues is generalized linear mixed-effects models (GLMM). The random-effect structure can account for differences both in persons and statements, the use of a link function prevents the model from making impossible predictions, and trial-level predictors can easily be included. GLMM also offers theoretical benefits since the estimated regression coefficients can be interpreted as response bias and discrimination sensitivity in terms of signal detection theory. To compare the results of ANOVA and different GLMM specifications, we re-analyzed 22 openly available datasets from 2018 to 2024. The preliminary results show that GLMMs with random intercepts for subjects only do not solve the problems; conversely, they lead to higher rates of finding significant effects. However, once random intercepts for statements are added, p-values become more conservative.

This is an in-person presentation on July 20, 2024 (15:20 ~ 15:40 CEST).