Measuring the effect of user-generated external representations during sensemaking in electronic environments

Sensemaking has been defined as the process of building understanding when facing complex situations. Creating a representation of a domain of interest is central to sensemaking, whether stored internally as a mental model or externally by using maps or tables, for example. The latter is generally assumed to be helpful, but little is known about the way in which external representations actually affect the sensemaking process. Despite the literature explicitly recognising the significance of external representations during sensemaking, it is not addressed in depth. Yet the prevalence of external representations supported by electronic environments, as developed by interaction designers, certainly warrants such interest. Understanding the role of external representations in the sensemaking processes can aid in the effective design of interactive computer-based tools.
This thesis investigates the way in which the use of external representations alters the process of sensemaking when searching for information in an electronic environment. Studies into sensemaking, which are often qualitative and exploratory in nature, have shed light on information behaviour and the underlying cognition involved in sensemaking, with exploratory characterisations usefully identifying certain key phenomena. By extending into more rigorous quantitative measurement, this study seeks deeper insight into what happens during sensemaking.
Four studies were conducted in the course of this research. Two controlled experiments involved participants searching for documents relevant to a task brief. In one condition, participants created an external representation in the form of a narrative and in another condition; they collected documents, which they judged as relevant, in a folder. The former controlled experiment was an exploratory study conducted to investigate the effects of using external representations in sensemaking, whereas the latter was conducted to replicate and expand on the findings of the former by testing additional hypotheses. The third study aimed at validating a metric of structuredness in the external representations created by users during sensemaking tasks. The fourth aimed to validate a questionnaire developed to measure levels of perceived sensemaking.
The results of this research suggest that the effect of user-generated external representations of sensemaking can be measured quantitatively – by using the developed paradigm, the sensemaking questionnaire and the metric for the measurement of structuredness, and by measuring the percentage of novel search terms. The results of the research also suggest that external user-generated representations involved in individual sensemaking tasks help to increase perceived sensemaking, reduce perceived uncertainty, and increase the generation of novel search terms. Moreover, the results show no relation between structuredness of the created external representations and perceived sensemaking, perceived uncertainty and keyword novelty.
This thesis makes a number of contributions, in that it provides: a paradigm for the study of the process of sensemaking; a reliable instrument to measure perceived sensemaking; a metric for structuredness of external representations created by users during sensemaking; empirical findings clarifying the relation between the effects of user-generated representations and structuredness on keyword novelty, perceived sensemaking and perceived uncertainty; and a theoretical explanation of the effect of user-generated representations on some of the sub-tasks involved in the sensemaking process (keyword novelty).