Title: Timely Feedback in Unstructured Cybersecurity Exercises,
Abstract: Cyber defence exercises are intensive, hands-on learning events for teams of
professionals who gain or develop their skills to successfully prevent and
respond to cyber attacks. The exercises mimic the real-life, routine operation
of an organization which is being attacked by an unknown offender. Teams of
learners receive very limited immediate feedback from the instructors during
the exercise; they can usually see only a scoreboard showing the aggregated
gain or loss of points for particular tasks. An in-depth analysis of learners'
actions requires considerable human effort, which results in days or weeks of
delay. The intensive experience is thus not followed by proper feedback
facilitating actual learning, and this diminishes the effect of the exercise.
In this initial work, we investigate how to provide valuable feedback to
learners right after the exercise without any unnecessary delay. Based on the
scoring system of a cyber defence exercise, we have developed a new feedback
tool that presents an interactive, personalized timeline of exercise events. We
deployed this tool during an international exercise, where we monitored
participants' interactions and gathered their reflections. The results show
that learners did use the new tool and rated it positively. Since this new
feature is not bound to a particular defence exercise, it can be applied to all
exercises that employ scoring based on the evaluation of individual exercise
objectives. As a result, it enables the learner to immediately reflect on the
experience gained.