The wisdom of the crowd when people choose what they rank

The wisdom of the crowd effect is the finding where a group aggregation of the crowd is more accurate on average than any random individual from the crowd. The wisdom of the crowd can be applied to a variety of tasks, such as a ranking one where participants are told to rank a set of items according to some criterion. In past work that applies the wisdom of the crowd to ranking tasks, participants are typically asked to rank all items (Lee et al., 2014) although they have also been asked to rank a random subset of the items (Lee et al., 2022). We consider how aggregated rankings are affected when participants are allowed to choose which items they include in their ranking and which items they exclude. Previous work on binary-choice trivia questions found that wisdom of the crowd aggregates were more accurate when individual participants chose which questions they wanted to answer (Bennett et al., 2018). We develop a Thurstonian model for our novel subset ranking task and evaluate performance by comparing the model-generated ranking to the correct ranking. References: Bennett, S. T., Benjamin, A. S., Mistry, P. K., & Steyvers, M. (2018). Making a wiser crowd: Benefits of individual metacognitive control on crowd performance. Computational Brain & Behavior, 1, 90-99. Lee, M. D., Bradford, N., & Tejeda, H. (2022). Using thurstonian ranking models to find the wisdom of the crowd. Paper presented at meeting of the European Mathematical Psychology Group. Rovereto (TN), Italy. Lee, M. D., Steyvers, M., & Miller, B. (2014). A cognitive model for aggregating people's rankings. PloS one, 9(5), e96431.