Probabilistic modeling of JAS behavior: an application to mine-countermeasure

Recent studies on human-AI interactions have strived to acquire data about how domain specialists in a task would follow an AI advice. They use the judge-advisor system (JAS), an experimental paradigm formerly interested in regressions, and analyze reliance behaviors on classification tasks. In this stream of research, few studies focused on developing models of influence, as effects found are often mixed, and difficult to generalize. Tejeda et al. (2022) proposed a consistent probabilistic model of this JAS applied on a lay image classification task ; however the reliance matrix they introduce still needs refinement in order to make sense of unexpected results. Iterating on this work, we propose a probabilistic modeling approach to quantify how advice influence decisions. We test on both Tejeda et al.’s dataset (open acces), and on an application of JAS to mine countermeasures. These data comes from an experimental study (soon to be published) performed with a sample of sonar analysts with varied degrees of experience. In this setting, three advisors (2 humans, 1 AI) help the operator classifying sonar images as mines or non-mines. Results show the feasibility of parsimonious models to predict human behavior, aimed at estimating impacts of organizational changes. Our models also allow us to assess whether individuals or data items are the predominant factor in this kind of decisional setting.