How to know what you should know: Implications of the choice of prior distribution on the behavior of adaptive design optimization

Adaptive design optimization (ADO) is a state-of-the-art technique for designing experiments for cognitive modeling (Cavagnaro, Myung, Pitt, and Kujala, 2010). ADO dynamically identifies stimuli that, in expectation, yield the most information about the hypothetical construct of interest (e.g., parameters of a cognitive model). To calculate this expectation, ADO leverages the modeler’s existing knowledge, specified in the form of a prior distribution. “Informative” priors, constructed on the basis of domain knowledge or previous data, have the potential to align the prior with the empirical distribution in the participant population, thereby making ADO maximally efficient. However, if the informative prior is inaccurate, i.e., “misinformative,” then ADO may be led astray, leading to wasted trials and lower efficiency. To play it safe, many researchers turn to “uninformative” priors. Yet, priors chosen on the basis of their predictive agnosticism rather than insight are also unlikely to align with the population distribution, possibly making them equally inefficient. In on-going work, we investigate the consequences of informative, misinformative and uninformative prior distributions on the efficiency of experiments using ADO.