Simulation Studies for Methodological Research in Psychology

Simulation studies are widely used for evaluating the performance of statistical methods in psychology. However, the quality of simulation studies can vary widely in terms of their design, execution, and reporting. In a review of 100 articles containing a simulation study from three leading methodological journals in psychology, we find that many articles do not provide complete and transparent information about key aspects of the study. To address this problem, we provide a summary of the ADEMP (Aims, Data-generating mechanism, Estimands and other targets, Methods, Performance measures) design and reporting framework that has gained traction in biostatistics in recent years, and adapt it to simulation studies in psychology. Based on this framework, we provide ADEMP-PreReg, a step-by-step template for researchers to use when designing, potentially preregistering, and reporting their simulation studies. We give formulae for estimating common performance measures, their Monte Carlo standard errors, and for calculating the number of simulation repetitions to achieve a desired Monte Carlo standard error. In this presentation, we will further discuss the advantages and disadvantages of simulation study preregistration, and highlight unique aspects and differences to preregistration of other empirical studies that have to be considered for a simulation study preregistration to be useful.