Boosting intelligence analysts’ judgment accuracy: what works, what fails?
Article
Mandel, D., Karvetski, C. and Dhami, M. 2018. Boosting intelligence analysts’ judgment accuracy: what works, what fails? Judgment and Decision Making. 13 (6), pp. 607-621.
Type | Article |
---|---|
Title | Boosting intelligence analysts’ judgment accuracy: what works, what fails? |
Authors | Mandel, D., Karvetski, C. and Dhami, M. |
Abstract | A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments. |
Publisher | The Society for Judgment and Decision Making, and The European Association for Decision Making |
Journal | Judgment and Decision Making |
ISSN | 1930-2975 |
Publication dates | |
30 Nov 2018 | |
Publication process dates | |
Deposited | 13 Nov 2018 |
Accepted | 12 Nov 2018 |
Output status | Published |
Publisher's version | |
Copyright Statement | Copyright: © 2018. The authors license this article under the terms of the Creative Commons Attribution 3.0 License. |
Web address (URL) | http://journal.sjdm.org/18/18803/jdm18803.pdf |
Language | English |
https://repository.mdx.ac.uk/item/8805x
Download files
56
total views10
total downloads0
views this month0
downloads this month