Statistical Methods

Serial dependence is present in most time series data sets collected in psychological research. This paper investigates the implications of various approaches for handling such serial dependence, when one is interested in the linear effect of a time-varying covariate on the time-varying criterion. Specifically, the serial dependence is either neglected, corrected for by specifying autocorrelated residuals, or modeled by including a lagged version of the criterion as an additional predictor. Using both empirical and simulated data, we showcase that the obtained results depend considerably on which approach is selected. We discuss how these differences can be explained by understanding the restrictions imposed under the various approaches. Based on the insight that all three approaches are restricted versions of an autoregressive distributed lag model, we demonstrate that accessible statistical tools, such as information criteria and likelihood ratio tests can be used to justify a chosen approach empirically.

This is an in-person presentation on July 21, 2023 (15:20 ~ 15:40 UTC).

The replication crisis has shown that researchers often collect insufficient sample sizes for their studies or use questionable research practices such as data peeking. Sequential testing procedures are one possible solution to these shortcomings. The Sequential Probability Ratio Tests (SPRTs) offer interesting possibilities within Frequentist Statistics. Here, a likelihood ratio is calculated as a test statistic, which can be computed continuously by adding new data points. After each iterative step of data collection, SPRTs decide whether the evidence is sufficient to accept the null or alternative hypothesis or whether more data points are needed. SPRTs control for alpha and beta error rates and require, as a test specification, a minimum expected effect size. We show in simulation studies that the one-way sequential ANOVA is very efficient compared to a classical fixed ANOVA. In 87% of the simulated cases, the sequential samples are smaller than the fixed samples. On average, 56% of the data can be saved using the sequential design. However, sequential designs show biases in effect size estimation. Thus, we want to discuss the benefits and limitations of sequential testing. The R package sprtt can be used to calculate sequential versions of t-tests and one-way ANOVAs.

This is an in-person presentation on July 21, 2023 (15:40 ~ 16:00 UTC).

The queueing model of visual search proposed by Li, Schlather, and Erdfelder (in press) is a novel mathematical model that accounts for both accuracy and response time in standard visual search with interpretable parameters. One of the merits of the model is that the probabilities of correct and incorrect responses are specified as analytical functions of the experimental manipulations, namely the set size (i.e., the number of stimuli in the display) and the presence or absence of the target. As a result, the number of model parameters remains constant even if the number of set size levels increases. However, its ability to incorporate quantitative features of the experimental condition comes with the cost that tailor-made goodness of fit tests need to be developed. The application of a standard likelihood ratio test provides too conservative results because the model implies more restrictive patterns than its number of free parameters suggests. In this presentation, I show that the distribution of the commonly used likelihood ratio statistic under the null hypothesis cannot be approximated asymptotically by the chi-square distribution with degrees of freedom that equal the difference in the number of free parameters. I explain the reasons in detail and compare alternative goodness-of-fit measures based on various approaches.

This is an in-person presentation on July 21, 2023 (16:00 ~ 16:20 UTC).

Delta-Plots (DPs) are valuable for analyzing reaction time (RT) experiments. They help to differentiate between various cognitive models and theories and to identify different mechanisms behind observed effects, such as the Simon or Stroop effect. The conventional definition of DPs is based on empirical data (empirical DPs; e.g. Schwarz & Miller, 2012), which contrasts with other statistical definitions relying on population distributions or estimators of population properties. Moreover, the details of the estimation procedure, e.g. the number of bins, can affect the properties of the estimates. While a definition using population distributions exists (distributional DPs; Speckman et al., 2008), it is less common. Nevertheless, we show that the distributional definition in combination with psychological models poses some interesting implications regarding, e.g., the monotonicity of DPs. Unfortunately, it is unclear how these two definitions relate formally, e.g. if empirical DPs can be considered an estimator of distributional DPs. Furthermore, both definitions only concern individual DPs for single participants, but it is open how population DPs should be defined. Consequently, the concept of a DP for a specific task, such as the Simon or Stroop task, is not well-defined. To address some of these issues, we present an algorithm that uses kernel-density estimations of the cumulative density function (CDF) to estimate DPs. Our algorithm leverages the Newton-Raphson method to enable the computation of the empirical DPs at arbitrary RTs. By using a direct estimation of the CDF, our method is closer to the formal definition of DPs based on population distributions and allows for the computation of DPs at any RT. Hence, it offers new ways to generalize individual DPs to population DPs. We also discuss open questions regarding negative DPs (nDPs) for population distributions and propose possible definitions of nDPs that address these questions.

This is an in-person presentation on July 21, 2023 (16:20 ~ 16:40 UTC).

In fitting computational models to data, we represent the data (e.g. reaction time distributions) in terms of a small set of psychologically meaningful latent dimensions (e.g. parameters in an evidence accumulation model). Fitting a model to observed data then involves finding a point in this "cognitive parameter space" that is likely to have generated the data. Typically that is where the mechanistic explanation ends and we do not specify or, indeed, ask how an agent ended up at that point. The (often implicit) assumption is that agents try to maximise some objective function through some optimisation process (e.g. gradient descent). However, we rarely take the agent's perspective and consider the information and cognitive mechanisms available to conduct their search through the parameter space. This search is subject to several constraints. Sampling the objective is necessarily serial, local and time-consuming: objective estimates at a given location are likely to be uncertain and the agent may need several interactions with the environment to reduce this uncertainty. In light of these constraints, we explore a cognitively more plausible (i.e. minimal) search strategy. This strategy is based on local sampling of an objective function and making ordinal comparisons with only the most recently visited location in the parameter space. We report simulation results for the behaviour of this algorithm for optimising and satisficing agents, under a range of boundary conditions (e.g. noise in the objective estimates and the granularity with which objective comparisons can be made). Our overall argument is that identifying the information and mechanisms available to agents for navigating the cognitive parameter space is critical for understanding variation in cognition and behaviour over time, between different environmental conditions and between different populations or individuals.

This is an in-person presentation on July 21, 2023 (16:40 ~ 17:00 UTC).