Testing sample-based accounts of probability judgments using a ranking task

People's explicit probability judgements often appear to be probabilistically incoherent. The most prominent example of this is the conjunction fallacy (Kahneman & Tversky, 1983). Recently, a growing body of research argues that biases in probability judgements can arise from rational reasoning processes based on mental samples from coherent probability distributions. However, the sample-based normative accounts of probability judgements are mainly investigated in probability estimation tasks. In the current study, a ranking task is used to study people's explicit probability judgements, and more importantly, to test the sample-based normative accounts of probability judgements. In the ranking task, participants are asked to rank four events, A, not-A, B, and not-B, according to their perceived likelihoods of occurrence. Results show a novel probabilistic reasoning bias:  Participants often provide logically impossible rankings, violating the complement rule and the transitive rule. Interestingly, one existing sample-based normative account, namely the Probability Theory plus Noise (PT+N) account (Costello & Watts, 2014), can potentially explain the logical inconsistencies in rankings of events. We formally derive the predictions for rankings from the PT+N account. Our predictions suggest that specific qualitative patterns should appear in people's responses if the logically impossible rankings are solely the products of internal sampling processes instead of inconsistent inherent beliefs.