Evaluation of human interaction with complex systems using a full-scope simulator: lessons learned and methodological issues

Several evaluation methods have been developed in various application domains for studying human interaction with complex systems. Of those, emphasizing the study of human cognitive activities in the context close to real one, researchers have paid much attention to how to use a full-scope simulator in a systematic way. However, studies continue to suffer from methodological problems that could challenge the value of using the simulator. This article describes common methodological issues and principles that have been reported in the literature. Then the article reports three studies using a full-scope simulator of nuclear power plants and explains how the issues were dealt with. Finally, the article describes what has been learned through these studies in methodological perspective, providing recommendations that could be useful for researchers to plan and conduct empirical studies of human interaction with complex systems, particularly using a full-scope simulator.