Assessing the 'paradox' of converging evidence by modeling the joint distribution of individual differences

Davis-Stober and Regenwetter (2019; D&R) showed that even if all predictions of a theory hold in separate experiments, not even a single individual may be described by all predictions jointly. To illustrate this 'paradox' of converging evidence, D&R derived upper and lower bounds on the proportion of individuals for whom all predictions of a theory hold. These bounds reflect extreme positive and negative stochastic dependence of individual differences across predictions. However, psychological theories often make more specific and plausible assumptions, such as that true individual differences are independent or show a certain degree of consistency (e.g., due to a common underlying trait). Based on this psychometric perspective, I extend D&R's conceptual framework by developing a multivariate normal model of individual effects. The model mitigates the 'paradox' of converging evidence even though it does not resolve it. Overall, scholars can improve the scope of their theories by assuming that individual effects are highly correlated across predictions.