Bayesian-frequentist p-values: The best of both worlds

Currently, researchers need to choose between one of two different statistical frameworks, a frequentist or Bayesian approach. Frequentist inference – null hypothesis significance testing – is the de-facto standard. It is computationally relatively cheap and comparatively convenient as it does not require the researcher to specify a prior on the effect to be tested. Bayesian inference is becoming increasingly popular, in large parts due to easy-to-use software such as brms that make it easy to estimate complex models with little programming. In contrast, where Bayesian estimation is convenient even for quite complex models, Bayesian testing via Bayes factors is computationally expensive and rather cumbersome as it requires the specification of a prior that can largely influence results. We evaluate a compromised approach that combines Bayesian estimation with frequentist testing: Bayesian-frequentist p-values, where Bayesian model estimation is combined with frequentist Wald-based p-values. To assess this combination, we examine the type I error rates of Bayesian-frequentist p-values across three different settings: regular analysis of variance (ANOVA), logistic regression, and logistic mixed-model designs. Our results showed that Bayesian models with improper flat priors produced nominal type I error rates mirroring the behaviour of frequentist models across all designs. However, non-zero-centred priors resulted in too high (i.e., anti-conservative) rates of type I errors and zero-centred models produced low (i.e., conservative) rates of type I error, with the degree of conservativity depending on the width of the prior. Overall, our results indicate that frequentist testing can be combined with Bayesian estimation if the prior is relatively non-informative. Bayesian-frequentist p-values offer an attractive alternative to researchers, combining the ease of frequentist testing with the convenience and flexibility of Bayesian estimation.