Assessing a decision support tool for SOC analysts

It is difficult to discern real-world consequences of attacks on an enterprise when investigating network-centric data alone. In recent years, many tools have been developed to help understand attacks using visualization, but few aim to predict real-world consequences. We have developed a visualization tool that aims to improve decision support during attacks in Security Operation Centres (SOCs). Our tool visualizes propagation of risks from sensor alert data to Business Process (BP) tasks. This is an important capability gap present in many SOCs today as most threat detection tools are technology-centric. In this paper we present a user study that assesses our tool's usability and ability to support the analyst. Ten analysts from seven SOCs performed carefully designed tasks related to understanding risks and recovery decision-making. The study was conducted in laboratory conditions with simulated attacks and used a mixed-method approach to collect data from questionnaires, eye tracking and semi-structured interviews. Our findings suggest that relating business tasks to network asset in visualizations can help analysts prioritise response strategies. Finally, our paper also provides an in-depth discussion on user studies conducted with SOC analysts more generally, including lessons learnt, recommendations and a critique of our own study.