This site uses cookies

By using this site, you consent to our use of cookies. You can view our terms and conditions for more information.

Safe anytime live and leading interim meta-analysis

Alexander Ly
CWI Amsterdam ~ Machine Learning
Dr. Judith ter Schure
Amsterdam University Medical Centre ~ Epidemiology & Data Science
Prof. Peter Grünwald
CWI Amsterdam/Leiden University ~ Machine Learning/Mathematical Institute

In this talk we introduce the Anytime Live and Leading Interim (ALL-IN) meta-analysis approach that allows experimenters to adaptively design multi-lab replication experiments in a safe manner. In this context, safe refers to the fact that the procedure comes with explicit guarantees regarding the tolerable type I error rate *during* data collection. This approach to meta-analysis is based on a so-called meta e-value that combines the currently available evidence across all studies. At any moment in time, experimenters can safely consult this meta e-value to decide whether it is worthwhile to (i) extend the meta-analysis with another (replication) study, (ii) continue recruiting participants in the currently active studies, or (iii) stop the data collection of all studies, because the combined evidence is already compelling. It is important to note, that despite this data-driven approach leading to interdependent studies, the statistical inferences from ALL-IN meta-analyses remain valid. This is unfortunately not the case for conventional meta-analyses based on p-values and confidence intervals, because, when used adaptively, they over-inflate the type I error, thus, have a high chance of mistaking random noise for structural effects. Statistical reliable use of conventional methods, therefore, requires experimenters to confine themselves to rigid designs, in which a meta-analysis is only conducted retrospectively once all --either too few, or too many-- studies are completed. The ALL-IN procedure frees experimenters from the statistical shackles imposed by conventional methods and empowers them with a more flexible, efficient, and safe approach to conducting meta-analyses.



Adaptive design of experiments
evidence-based research
living systematic review

There is nothing here yet. Be the first to create a thread.

Cite this as:

Ly, A., ter Schure, J. A., & Grünwald, P. D. (2023, July). Safe anytime live and leading interim meta-analysis. Abstract published at MathPsych/ICCM/EMPG 2023. Via