Using machine learning to infer reasoning provenance from user interaction log data: based on the data/frame theory of sensemaking

The reconstruction of analysts’ reasoning processes (reasoning provenance) during complex sensemaking tasks can support reflection and decision making. One potential approach to such reconstruction is to automatically infer reasoning from low-level user interaction logs. We explore a novel method for doing this using machine learning. Two user studies were conducted in which participants performed similar intelligence analysis tasks. In one study, participants used a standard web browser and word processor; in the other, they used a system called INVISQUE (Interactive Visual Search and Query Environment). Interaction logs were manually coded for cognitive actions based on captured think-aloud protocol and posttask interviews based on Klein, Phillips, Rall, and Pelusos’s data/frame model of sensemaking as a conceptual framework. This analysis was then used to train an interaction frame mapper, which employed multiple machine learning models to learn relationships between the interaction logs and the codings. Our results show that, for one study at least, classification accuracy was significantly better than chance and compared reasonably to a reported manual provenance reconstruction method. We discuss our results in terms of variations in feature sets from the two studies and what this means for the development of the method for provenance capture and the evaluation of sensemaking systems.

Neesha Kodagoda, Sheila Pontis, Donal Simmie, Simon Attfield, B. L. William Wong, Ann Blandford, Chris Hankin, Using Machine Learning to Infer Reasoning Provenance From User Interaction Log Data: Based on the Data/Frame Theory of Sensemaking, Journal of Cognitive Engineering and Decision Making, Vol 11, Issue 1, pp. 23 - 41. Copyright © 2016 (Human Factors and Ergonomics Society). Reprinted by permission of SAGE Publications.