abstract: This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to the batch. PER chooses which experiences should be replayed based on how beneficial they will be towards learning. HER learns from failure by substituting the desired goal with the achieved goal and recomputing the reward function. The effectiveness of combinations of these experience replay techniques is tested in a variety of OpenAI gym environments.
author: Hima Tammineedi
Department of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
`firstname.lastname@example.com` Tracy Wan
Department of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
`firstname.lastname@example.com` Neil Xu
Department of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
`email@example.com`
bibliography: citation.bib
title: Advances in Experience Replay

# Introduction

With the rapid introduction of new techniques to reinforcement learning, a smörgåsbord of approaches have emerged, all promising improvements over baseline methods. The goal of our project is to examine some of the most high impact recent advances in reinforcement learning, and see how and whether they can be combined to create a new state of the art standard in performance on OpenAI Gym tasks. Our hope is that we can identify which of these methods, or combination of these methods, has the best performance, and create a new benchmark for these standardized learning environments. We will examine techniques involving two popular and promising ideas in current in reinforcement learning literature: experience replay (ER).

With the advent of the successful utilization of deep neural networks (DNNs) as function approximators in various model-free techniques based upon TD learning, experience replay has become a necessary tool to enhance accurate and generalized learning by DNNs. As a result of the emphasis that has been placed on experience replay, a variety of modifications have emerged in recent years that have individually shown significant increases in convergence speed when applied to DNN-based learning models. The ones that seem to have shown the most dramatic performance improvements are prioritized experience replay (PER) and hindsight experience replay (HER) . Another technique, combined experience replay (CER) has also been shown to improve performance.

To our knowledge, there has been no study on the combination of all the experience replay advances in recent years. Thus, We combine the recent experience replay techniques of HER, PER, and CER in order to show the combined effectiveness in a multitude of environments.

# Background and Related Works

## Deep Q Network

In standard reinforcement set up, an agent interacts with an environment in discrete time steps. At each time t, the agent receives an observation $x_t,$ makes an action $a_t$ and receives a reward $r_t.$ An agent's behavior is defined by a policy $\pi$ which given a state outputs a probability distribution over the possible actions. The process is modeled as a Markov Decision process with state space $\EuScript{S}$ action space $\EuScript{A}$ and an initial state distribution $p(s_0),$ transition dynamics $p(s_{t+1}|s_t,a_t),$ and reward function $r(s_t, a_t).$ The return is defined as sum of discounted reward $\gamma \in [0,1].$ $$R_T = \sigma_{i=t}^{T} \gamma(i-t)r(s_i, a_i)$$ The goal of reinforcement learning is to learn a policy distribution that maximizes the expected reward. The expected reward after taking an action $a_t$ in state $s_t$ following policy $\pi$ is $$Q^{\pi} (s_t, a_t) = E_{r_i \geq t, s_{i>t}~E, a_{i>t}~\pi}[R_t|s_t, a_t]$$ Expanding the expectation gives the Bellman Equation $$Q^{\pi}(s_t, a_t) = E_{r_t, s+1 ~ E} [r(s_t, a_t) + \gamma E_{a_{t+1}~\pi}[Q^{\pi}(s_{t+1}, a_{t+1}]]$$ If the target policy is deterministic, it can be described as a function $\mu: S -> A$ $$Q^{\mu}(s_t, a_t) = E_{r_t, s+1 ~ E} [r(s_t, a_t) + \gamma Q^{\mu}(s_{t+1}, \mu(s_{t+1}))]$$ The expectation depends only on the environment. It is possible to learn $Q^\mu$ off policy, using transitions generated from a different policy.

Q-learning uses the greedy policy $\mu(s) = argmax_a Q(s,a).$ This can be approximated by minimizing the loss of its parametrization using $\theta^{Q}$ $$L(\theta^{Q}) = E_{s_t~p^{\pi}, a_t ~ \pi, r_t~E} [(Q(s_t, a_t | \theta^Q) - y_t)^2]$$ where $$y_t = r(s_t, a_t) + \gamma Q(s_{t+1}, \mu Q(s_{t+1}, \mu(s_{t+1})|\theta^Q)$$ Using replay buffer and a separate target network for calculating t, large neural networks could be used to approximate the Q function. This is known as deep Q learning.

## Deep Deterministic Policy Gradients (DDPG)

The DDPG algorithm uses an actor-critic approach based on the DPG algorithm. The DPG algorithm uses a $\mu(s | \theta^{\mu})$ to specify the deterministic policy by returning an action given a state. The critic $Q(s|a)$ is updated using the Bellman equation. The actor is updated using the following gradient $$\Delta_{\theta \mu} J \approx E_{st~p^{\beta}}[\Delta Q(s,a|\theta^{Q})|_{s=s_t,a=\mu(s_t)}\Delta_{\theta_{\mu}(s|\theta^{\mu})|_{s=s_t}}]$$ Rather than directly copying the weights, the DDPG algorithm creates a copy of the actor and critic network $Q'(s, a|\theta^{Q'})$ and $\mu'(s|\theta^(\theta'))$ and perform target updates. The weights of the networks are updated by having them slowly match the learned network: $\theta' <- \tau\theta + (1 - \tau)\theta'$ with $\tau << 1.$ The target network is constrained to train slowly, making the entire network more stable. In order for the hyper parameters to generalize across environments with different scales of state value, DDPG employs batch normalization that normalizes each dimension across the samples in a unit to have unit mean and variance. It maintain a running average of the mean and variance to use for normalization. Batch normalization is applied on the state input, all layers of the $Q$ and $\mu$ network prior to the action output. It allows the system to learn on different environment with different settings. The exploration policy samples from a noise process in addition to the actor policy. $$\mu'(s_t) = \mu(s_t|\theta_t^{\mu}) + \EuScript{N}$$ The noise process is chosen to suit the environment.

## Hindsight Experience Replay (HER)

In HER, the trained value function takes in not only state $s \in S$ but also a goal $g \in G.$ After experiencing some episode $s_0, s_1, ... s_t,$ each transition is stored in the memory replay buffer with both the original goal and the same transition with the original replaced with an alternative goal. Thus, HER is motivated on the principle that an agent that performs multi-task learning (in this case on all the goals in the goal-space $G$) will learn more quickly, then an agent solely trying to learn the singular, original goal.

In this case, the different set of goals is $m(s_t),$ the goal achieved in the final state of the episode. This is especially useful a sparse reward environment, where an agent training to achieve single goal has trouble receiving any useful reward. By setting the goal to the final state of the episode (or more generally, by setting the goal s.t. the agent achieves a big reward through the episode) the agent will receive much more reward signal in its experiences, and thus learn more quickly.

## Prioritized Experience Replay (PER)

In regular experience replay, all transitions are sampled uniformly. But this does not seem ideal if some transitions do not really help the agent learn, yet we keep sampling them.
In PER , the transitions are sampled according to how helpful they will be for learning. Clearly, we have no way, at present, of knowing exactly how much each transition will help the network in its learning progress, but we can try to get a proxy for it.
Transitions with high expected learning progress, as measured by the magnitude of their temporal-difference (TD) error ($\delta$) are replayed more frequently. The magnitude of the TD error indicates how "surprising" a given transition is since our network did not predict the Q-values well, and so we prioritize these for learning.
However, we don't want to only choose the transitions that have the highest priorities as this can lead to a loss of diversity and thus over-fitting. So we ensure that there is a non-zero sampling probability for all transitions (equation ).

The probability of sampling transition $i$ is $$\label{eq:PER}
    P(i) = \frac{p^{\alpha}_i}{\sum_k p^\alpha_k}$$ where $p^i$ is the priority of transition $i$ (in this case $p_i = |\delta_i|$).
We further use importance sampling weights to correct for the bias introduced by the prioritization changing the original data distribution.

## Combined Experience Replay (CER)

CER is a special case of PER. PER gives the latest transition a higher priority but it is not guaranteed to be replayed immediately. CER deals with this by adding the latest transition into the training batch. As a hyper parameter, the size of the replay buffer is extremely sensitive to the stabilization of the training system. CER attempts to remedy the effect of having a large replay buffer by ensuring that the latest transition is sampled.

## Evaluating experience replay techniques

Previous work has addressed the issue of memory replay size, by either measuring the empirical results of changing the buffer size on Gym and Atari environments , or using analytical techniques to derive a theoretically optimal buffer size dynamically throughout training. These works provide insight into the the hyperparameter of the buffer size of the experience, and provide strategies through which that hyperparameter may be chosen to maximize the convergence speed. Zhang et al. (2017) demonstrates that the choice of buffer size can have a large impact on the sample efficiency of the model being trained, and furthermore, proposes that experience replay can be detrimental if used with improper priority methods since it may delay certain samples that could speed up convergence due to the stochastic nature of the sampling process.

# Methodology

We first implemented the three individual experience replay techniques to establish a baselines for how well they can perform. We then tried the combinations of the various techniques as well. To establish the efficacy of these techniques, we first tested these in conjunction with a deep Q-network (DQN) on the CartPole, MountainCar, and LunarLander environments from OpenAI Gym. For the DQN, we implemented a DQN with target fixing in order to increase stability.

We then extended our methods to use continuous environments. Since DQN cannot generate continuous outputs, we implemented a Deep Deterministic Policy Gradient (DDPG). We tested DDPG on the Pendulum, Continuous Lunar Lander and Continuous Mountain Car environments.

We looked at two metrics when evaluating all of our various model depending on the environment: (1) highest reward after a fixed number of episodes or (2) speed of convergence i.e. how many episodes until convergence. where We define convergence to be the first episode where the frozen policy network can achieve an average reward over 100 episodes that exceeds or is equal to the goal for solving the environment. This is defined to be the following for the environments we utilized:

|                 |                            |
|:----------------|:---------------------------|
| **Environment** | **Reward needed to solve** |
| CartPole-v0     | 200                        |
| MountainCar-v0  | -110                       |
| LunarLander-v2  | 200                        |

For the remaining environments i.e. Acrobot-v1 and Pendulum-v0, we measure the effectiveness of our experience replay strategies by setting an episode limit for each environment and analyzing their performance by taking the best average reward over 100 episodes model from the training rule.

For each environment, we tried every possible combination of combined, prioritized, and hindsight experience replay strategies, which in total resulted in 8 different agents being experimented on at most. We wanted to test, even for simple environments, whether the combination of different experience replay techniques could be counterproductive when utilized simultaneously, or that the techniques could all yield improvements in convergence rate and sample efficiency. However, some tasks are not conducive to the goal based formulation that hindsight experience replay uses. Thus, for CartPole-v0, Acrobot-v1, and Pendulum-v0, we only run variants of combined and experience replay, which totals of 4 different agents being run in those environments

All of our code is available at: <https://github.com/himat/CHAPtER>

# Results

Below are our results for the various environments we tested our experience replay strategies on. Note that we tested every combination of combined (C), prioritized (P), and hindsight (H) experience replays (ER) for each environment, providing an exhaustive search for the interactions between different types of strategies.

| 2-4 | **Environments** |  |  |
|:---|:---|:---|:---|
| **Strategies (Episodes to Convergence)** | *CartPole-v0* | *MountainCar-v0* | *LunarLander-v2* |
| *Baseline* | 4000 | 33000 | 3500 |
| *CER* | 3500 | 15000 | 4500 |
| *PER* | 4000 | N/A | 7000 |
| *HER* | N/A | 15500 | 11000 |
| *CPER* | 5000 | N/A | 19500 |
| *HPER* | N/A | N/A | 10000 |
| *CHER* | N/A | 17500 | 4500 |
| *CHPER* | N/A | N/A | 8000 |

DQN used to solve these discrete action environments (CartPole does not support hindsight goals)

Interestingly, different environments have different strategies that seem to be more optimal than others. Noticeably for LunarLander-v2, the baseline actually performs the best out of all the experience replay strategies. In fact, CPER, HER, and HPER are perform significantly worse than the baseline.

# Discussion

## CartPole-V0

CartPole-v0 is a task of balancing a pole on top of the cart. The cart has access to its position and velocity as state, and can only go left or right for each action. The task is over when the pole falls over, the cart goes out of the boundaries, or 200 time steps are reached, with each step returning 1 reward.

Prioritized and combined replay both ended up being detrimental to CartPole-v0 over the baseline. It may be that CartPole-v0 is an easily solved task, which results in additional techniques only perturbing the training, but show no additional benefit.

## MountainCar-V0

MountainCar-V0 is a one dimensional track between two mountains. The goal is to drive up the mountain to the right. The agent receive a -1 reward for every time step it does not reach the top. The episode terminates when it reaches the top and receives 0 reward. The objective is for the agent to learn to drive back and forth to build momentum that will be enough to push the car up the hill. The observation consists of the car's position and velocity and the action consists of pushing left, pushing right and no push. For hindsight replay, the code passes in the position of the car as the modified goal. The main challenge of mountain car is that the rewards are very sparse since the agent only receive a reward if it reaches the top.

We found that our MountainCar results were not very good, and there is not much of a differentiating factor among the different ER techniques tried, and so we omit the results for brevity.

## LunarLander-v2

LunarLander-v2 is a two dimensional environment featuring a landing pad at (x,y)=(0,0). The goal is to move from the top of the screen to the landing pad without crashing. The agent receives 100-140 points for landing. If it moves away from the landing pad, it loses rewards. If it crashes it will receive -100 and if it lands successfully it receives 100. Each leg ground contact is 10. Firing main engine is -0.3 points each frame. The observation consists of f the x and y coordinates, the x and y velocities, angle, angular velocity, and ground contact information of the lander and the action consists of do nothing, fire left orientation engine, fire main engine, fire right orientation engine.

An important distinguishing feature of the LunarLander-v2 environment and the other environments tested in this paper is that it is the only one that has dense rewards. The reward each step is calculated based on the distance of the lander way from its landing spot. Consequently, this environment may not suffer from the issues that hindsight and prioritized experience replay are best at tackling: sparse rewards and off policy sampling that does not assist the agent. We can observe that the results for techniques involving hindsight and prioritized experience replay do worse than baseline on this task, and this may be due to the non-universality of these techniques. In contrast, combined experience replay performed close to baseline, but still worse than the baseline. This perhaps is on account of the combined strategy being only slightly deviant from the baseline sampling strategy.

## Pendulum-v0

Pendulum-v0 is a two dimensional environment featuring a frictionless pendulum. The goal is to keep the pendulum standing. The precise equation for reward is $-(\theta^2 + 0.1*\theta_{dt}^2 + 0.001*action^2).$ The observation consists of $\cos(\theta), \sin(\theta),$ and $\theta_{dt}.$ The action consists of the joint effort, which ranges between -2.0 and 2.0. For hindsight replay, we pass in the angle of the pendulum as the goal with the original goal being achieving a vertical position.

Unlike all the other environments, Pendulum-v0 has continuous action space. We found that our DDPG algorithm did very poorly on this task. Many times the agent never reached the goal. We were supposed to train pendulum with baseline, P, H, CH, and CHP. Among them, CHER converged the fastest, at 500 episodes. This demonstrates that hindsight replay has great potential in continuous action environment. Further hyperparameter tuning could improve the stability of the results.

# Conclusion

Through this endeavor, we found that the field of reinforcement learning is highly unstable, and testing often requires multiple trials. Unfortunately this means that there was a large amount of variance in our results and so some experience replay methods which worked well at times would work terribly at other times.

Our DDPG algorithm in particular did not perform well, and so we were unable to get substantial results on continuous action environments due to the poor training of the model.

However, in conclusion, we believe that there is a lot of promise in combining the various experience replay techniques proposed in recent years. Hindsight experience replay in particular makes a lot of sense as a standalone technique, since it provides a lot more information to the agent in that the agent can still learn even if it ends in a different final goal state. Combining this with prioritized experience replay should only serve to improve the convergence of the agent by choosing more informative updates. And using the idea of always learning from the most recent experience a la combined experience replay should also add a slight edge. We believe that these techniques should be further explored and exploited in the future.


abstract: Cyber defence exercises are intensive, hands-on learning events for teams of professionals who gain or develop their skills to successfully prevent and respond to cyber attacks. The exercises mimic the real-life, routine operation of an organization which is being attacked by an unknown offender. Teams of learners receive very limited immediate feedback from the instructors during the exercise; they can usually see only a scoreboard showing the aggregated gain or loss of points for particular tasks. An in-depth analysis of learners' actions requires considerable human effort, which results in days or weeks of delay. The intensive experience is thus not followed by proper feedback facilitating actual learning, and this diminishes the effect of the exercise.
  .
  In this initial work, we investigate how to provide valuable feedback to learners right after the exercise without any unnecessary delay. Based on the scoring system of a cyber defence exercise, we have developed a new feedback tool that presents an interactive, personalized timeline of exercise events. We deployed this tool during an international exercise, where we monitored participants' interactions and gathered their reflections. The results show that learners did use the new tool and rated it positively. Since this new feature is not bound to a particular defence exercise, it can be applied to all exercises that employ scoring based on the evaluation of individual exercise objectives. As a result, it enables the learner to immediately reflect on the experience gained.
author: Jan Vykopal; Radek Ošlejšek; Karolína Burská; Kristína Zákopčanová
bibliography: sigcse2018-feedback.bib
title: Timely Feedback in Unstructured Cybersecurity Exercises

# Introduction

Cyber attacks threatening ICT infrastructure have become routine. Their intensity and complexity are growing with the increasing number of interconnected devices exposed to attackers and the influx of new vulnerabilities being revealed each year. Unfortunately, there is a significant global shortage of cybersecurity workers equipped with the skills necessary for preventing or responding to the attacks .

Cyber defence exercises (CDX)  represent a popular type of training that aims to fill this skill gap. They enable participants to experience cyber attacks first-hand with real-life limitations, including a lack of information and resources, the need for communication and making decisions under stress.

CDX are usually intensive, short-term events lasting several days. Tens to hundreds of professional learners participate and are grouped in teams. The target groups are administrators of ICT systems, incident responders, and security managers. The exercises deliver rich immediate experience of ongoing attacks and the opportunity to practice crisis procedures and techniques.

In contrast to structured, step-by-step hands-on training guided by an instructor, teams of learners have to figure out all the issues on their own, in the order they agree on, within the team. These settings simulate real operation but prevent any direct feedback from instructors (exercise organizers). Learners can only presume what they did was correct, what worked and what did not. The only feedback they are given during the exercise is often through an exercise score with no further details about score breakdown. Some exercise organizers provide technical reports after the exercise, which reveal some details highlighting important moments from the perspective of a particular team of learners. The after-action report is sometimes also complemented by a short workshop, which provides an opportunity to discuss the content of the exercise with the instructors in person. Nevertheless, all these methods of feedback are delivered with a significant delay after the actual exercise because they require preparation from the instructors that cannot begin before the end of the exercise.

In this paper, we study whether learners benefit from simple, but individualized feedback provided just after the end of a two-day intensive exercise. In our study, each team was provided with an interactive timeline of its score development during the exercise, with important events emphasized. The timeline was generated automatically from data stored by an existing scoring system. All interactions of exercise participants (mouse clicks and movements) were logged with the scoring timeline. After that, participants were asked to fill out short evaluation questionnaire. The data and answers we obtained show that learners valued the feedback, even though they still lack more details about particular events.

# State of the art

Research on providing feedback in complex and unstructured cybersecurity exercises is very scarce. The following overview is therefore based not only on a review of academic literature but also on technical reports published by organizers and the experience of authors who participated in several CDXs.

One of the world's largest exercise is Locked Shields , which is organized annually by the NATO Cooperative Cyber Defence Centre of Excellence in Tallinn, Estonia. Immediate brief feedback to learners is provided at a so-called "hot wash-up" session right after the exercise since any time lag will diminish the learning impact . Educators provide a summary of the exercise's progression and comment on key moments that drove it. More comprehensive and detailed feedback is available only at a workshop which is held a month later, and furthermore, not all learners come to this event.

Another international exercise organized by NATO is Cyber Coalition. Very brief feedback is provided a day later in-person at a hot wash-up session. More specific information is available only in the form of an after-action report at a workshop, which takes place a month later . A very similar type of feedback, with an even longer delay is provided in Cyber Europe, another international exercise organized by ENISA.

Granåsen and Andersson  focused on measuring team effectiveness in CDXs. They thoroughly analysed system logs, observer reports, and surveys collected during Baltic cyber shield 2010, a multi-national civil-military CDX. They concluded that these multiple data sources provided valuable insight into the exercise's run. However, they did not mention how to use these data for providing feedback. The only feedback provided to learners was during a virtual after-action review the day after the exercise, where only the leaders of learners' and organizers' teams summarized and discussed their experience.

Henshel et al.  also focused on the assessment model and metrics for team proficiency in CDXs. They analysed learners' and observers' input from surveys and intrusion detection logs from the Cyber Shield 2015 exercise. Similarly to the Baltic cyber shield exercise, the feedback provided to learners was not based on an analysis of acquired data since the analysis was done manually and required significant human effort.

Since existing CDXs do not provide any timely and personalized feedback to the learners, we also mention a feedback-related study in learning, which does not concern CDXs. Gibbs and Taylor  focused on the theory that personalized feedback does not hold such importance or a value compared to its time-consumption for the instructor.

# Experiment setup

In this experiment, we studied the behaviour and interactions of participants at a complex cyber defence exercise held on May 23–24, 2017 at Masaryk University, Brno, Czech Republic. The exercise is focused on defending critical information infrastructure (particularly railway infrastructure administration) against skilled and coordinated attackers.

First, the learners got access to the exercise infrastructure to become familiar with virtual hosts in their network. Then, they took part in an intensive exercise where they faced challenges posed by simulated attackers and legitimate users. Right after the end of the exercise, the learners were asked to express their immediate impressions about the exercise in a short post-exercise survey. After a short break, a timeline depicting the score of each team was presented in the exercise portal. Finally, the learners were asked to evaluate the timeline via a very short questionnaire. Time allocated for each phase of the experiment is shown in Table .

| Order | Phase                        | Duration | Day |
|:-----:|:-----------------------------|---------:|:---:|
|   1   | Exercise familirization      |    3 hrs |  1  |
|   2   | Actual exercise              |    6 hrs |  2  |
|   3   | Post-exercise survey         |   5 mins |  2  |
|   4   |  Break                       |  25 mins |  2  |
|   5   | Scoring timeline interaction |  10 mins |  2  |
|   6   | Scoring timeline survey      |   5 mins |  2  |
|   7   |  Quick exercise debriefing   |  15 mins |  2  |

Phases of the exercise with time allocation

## Exercise participants

The experiment involved a Red vs. Blue exercise with 40 participants working in an emulated ICT infrastructure. The structure of the exercise is inspired by the Locked Shield exercise . The participants were divided into four groups according to their role and tasks in the exercise. Their interactions are depicted in Figure .

Twenty professional learners formed five *Blue teams* (T1–T5) which were put into the role of emergency security teams sent into five organizations to recover compromised networks. Each team of 4 learners was responsible for securing the compromised networks, dealing with the attacks, collaborating with other emergency teams, and collaborating with the coordinator of the operation and media representatives. They had to follow the exercise's rules and local cybersecurity law. Each team represented one real cybersecurity response team from one country in Central Europe.

All attacks against infrastructure defended by Blue teams were conducted by *Red team*. This team consisted of cyber security professionals who carefully followed a predefined attack scenario to equally load the Blue teams. This means they should not use any other arbitrary means of attack against the Blue teams. Based on the success of attacks, the Red team assigns penalty points to the Blue teams since the amount of points is based on non-trivial factors that need expert review.

Exercise managers, referees, organizers, and instructors worked in *White team*. During the exercise, this team assigns tasks (called injects) to the Blue teams and thus simulates the requests of many entities, such as legitimate users of the defended organization, the operation coordinator which needs situational reports, media inquiries, and law enforcement agencies. Then the White team assesses the promptness and quality of a Blue teams's reactions to these tasks and assigns penalties and points.

Finally, the *Green team* is a group of operators and system administrators responsible for the exercise infrastructure. They have full access to the exercise network so they can provide assistance to Blue teams in trouble in an exchange for penalty points.

## Exercise phases

Before the actual exercise (Phase 1 in Table ), learners are provided with a background story to introduce them to the situation before they enter the compromised networks. Then they access their part of the emulated network for 3 hours to get familiar with the exercise infrastructure. This is very important since the exercise is not set in a known environment and learners have no previous knowledge about who is who in the fictitious scenario (e. g., users in their organization, a popular news portal, superordinate security team).

The exercise (Phase 2) is driven by a detailed scenario which includes the actions of attackers (Red team) and assignments for the defenders prepared by the organizers (White team). Attackers exploit specific vulnerabilities left in the compromised network in a fixed order. This follows a common attack life cycle in a critical information infrastructure. On top of that, learners should also answer media inquires and requests from users doing their routine job in the defended network. The performance of each Blue team is scored based on successful attacks or their mitigation, the availability of specified critical services and the quality of reporting and communication. The score is either computed automatically from events, processed by the logging infrastructure (e. g., a penalty for inaccessible services) or entered manually (e. g., attacks completed by the Red team). An aggregated score is shown to participants in real-time. Table  shows the structure of the scoreboard and the values of aggregated score.

| Team   | Services | Attacks | Injects |  Users | Access | Total  |
|:-------|---------:|--------:|--------:|-------:|-------:|:------:|
| **T1** |   91,843 |  -8,500 |   9,000 | -1,100 |      0 | 91,243 |
| **T5** |   92,230 |  -5,000 |   3,600 |   -400 |      0 | 90,430 |
| **T2** |   81,280 | -10,750 |   6,425 | -4,000 |      0 | 72,955 |
| **T4** |   74,518 | -11,000 |   6,650 |      0 | -4,000 | 66,168 |
| **T3** |   85,756 | -12,000 |   2,475 | -1,700 | -9,500 | 65,031 |

The scoreboard presented to the learners during the exercise.

Note: Teams are sorted according to their final score.
"Injects" is an abbreviation for communication injects of the White team.

Immediately after the end of the exercise, we asked the learners to evaluate the exercise and their experience by rating several statements using the Likert scale (Phase 3).

## Scoring timeline application

After a break (Phase 4), the score acquired by each team during the exercise was presented to the Blue teams in the form of an interactive application, as shown in Figure  (Phase 5). It provided automatically generated personalized feedback. Members of each team could only see their own scoring timeline with individual exercise events.

The initial score of each team is 100 000 points. In the graph, the main, predominantly descending line represents the development of a team's total score over time. The score is computed from penalties and awarded points that were either recorded automatically for the inaccessibility of required network services or assigned manually by the Red or White team. The colourful dots are interactive and they are related just to the manual rating. The red dots represent Red team penalties, white dots represent the rating of communication injects by the White team, yellow dots indicate the rating of user simulated injects by the White team, and grey dots indicate requests for assistance from the Green team (to grant temporary remote access to a machine or revert to the initial state). Each dot contains textual information that specifies the reason for the rating. This information is shown in each dot's tooltip.

Learners were able to provide us with their reflection on their penalty and awarded points very easily by clicking on the coloured dots and choosing one of predefined options (Phase 5; see right-hand side of Figure 2), e. g., whether they recognized the attack or not, or why they did not respond to the inject of the White team. Moreover, all scoring timeline interactions, including mouse clicks, mouse movements, and selected options were logged. This data, together with answers from a short survey on the scoring timeline (Phase 6) were used to evaluate the usefulness of the timely feedback.

Finally, representatives of the Red and White team provided a short debrief of the exercise ("hot wash-up", Phase 7) from their perspective. They highlighted important breaking points of exercise and pointed out exemplary or interesting decisions and actions took by Blue teams. This part is mentioned here only for completeness, no input from learners was required for the experiment.

# Results

## Post-exercise survey

```latex
\begin{table*}[ht]\caption{Team statistics for the post-exercise and scoring timeline surveys}
\label{tab:questionnaire}
\begin{minipage}{\textwidth}
\begin{center}
\begin{tabular}{lp{7cm}llllll}
\toprule
& {\bf Statement} & {\bf Team 2} & {\bf Team 1} & {\bf Team 5} & {\bf Team 4} & {\bf Team 3} & {\bf Average}\\
\hline \hline
E1: & My knowledge and skills were sufficient.    & 3.75 & 3    & 2.6 & 2.5  & 2 & 3.05\\
E2: & I found exercise difficult for me.          & 3.33 & 3    & 4   & 4.25 & 5 & 3.8\\
E3: & Exercise was well organized and structured. & 2.66 & 3.25 & 3.3 & 4    & 5 & 3.75\\
E4: & Exercise was beneficial and useful to me.   & 2.66 & 3.5  & 4   & 4.5  & 5 & 3.85\\
\hline
F1: & The scoring timeline of my team displayed after the end of the exercise provided useful feedback. & 3.25 & 2.5 & - & 4.25 & 3.66 & 3.53\\
F2: &  Do you have any comments on the scoring timeline? & D & M & - & M & M & \\
\bottomrule
\end{tabular}
\bigskip\\
\footnotesize1 = strongly disagree, 5 = completely agree, D = there was a delay in inserting points by a Red and White team, M = add more details about the depicted events
\end{center}
\end{minipage}
\end{table*}
```

The post-exercise survey was focused on general qualitative aspects of the exercise. Relevant statements to this study are listed in Table  and marked as E1–E4 for further reference. The statistical distribution of individual answers across all teams is depicted in Figure .

We collected answers from all 20 participants. However, four of them did not provide their identification and so their answers were omitted from the team statistics. Team values were computed as the average from a five-point Likert scale answers (1 = strongly disagree, 5 = completely agree). The *Average* column represents the average value across all answers regardless of teams.

The aim of E1 and E2 was to reveal the level of expertise of individual teams and the difficulty of the exercise. Individual answers to E1 significantly varied, which indicates that learners had significantly different expertise. However, the average values calculated for each team reveal that the exercise was well balanced with no extremely weak or strong team. The answers to E2 indicate that the overall difficulty of the exercise was considered as rather high.

Two statements in the questionnaire, E3 and E4, were focused on satisfaction with the organizational aspects and usefulness of the exercise. Both statements brought very similar answers. Teams that were more satisfied with the organization also considered the exercise more beneficial. Even though the opinion differed across the teams, learners considered the exercise rather beneficial and well organized in general.

## Scoring timeline interaction

In order to evaluate the scoring timeline, we were actively recording learners' interactions with a tool. We obtained data from 18 learners, 2 learners were missing due to technical issues. The data consisted of 2,994 individual low-level events (mouse clicks, mouse hovers, etc.). Moreover, we recorded heatmaps of mouse positions on the screen (see an example in Figure ). Putting this data together, we were able to estimate the focus of the learners during their exploration of the scoring timeline. A deeper analysis did not reveal any preference patterns in the sense that learners would be more interested in some kind of objectives, such as Red team attacks, White team injects, or penalties from later critical phases of the exercise. On the contrary, it seems that the learners were interested in all the penalties and awards roughly the same.

Analyses of timestamps revealed that the time spent by individual learners with the scoring timeline ranged between 1m 22s and 8m 27s It is worth noting that there was no significant difference between teams. They spent approximately 3 to 5 minutes with the feedback application on average. We also did not find any relation between the time spent with the timeline and the willingness to provide their reflection on particular penalties or awards. Many learners just spent a long time with only a passive exploration of the timeline.

Seven learners from four teams also gave us an active reflection to penalties in addition to passively exploring the scoring timeline. The values in Table  represent the numbers of collected answers per team and objective type. Blue team 1 is omitted from the table because we got no data from them. These results are discussed in Section .

|                       |           |     |     |     |        |
|:----------------------|:---------:|:---:|:---:|:---:|:------:|
|                       | **Teams** |     |     |     |        |
| **Objectives**        |    T2     | T3  | T4  | T5  | $\sum$ |
| Red team attacks      |     7     | 13  |  5  |  1  |   26   |
| Users injects         |     5     |  7  |  0  |  1  |   13   |
| Communication injects |     0     |  5  |  0  |  2  |   7    |
| Green team assistance |     0     |  4  |  2  |  0  |   6    |

Numbers of responses of each team to objectives.

## Scoring timeline survey

The usefulness of the feedback provided via the scoring timeline was evaluated with a short survey. We got answers from 13 learners (out of 20) because Blue team 5 did not respond at all. The statements and their team statistics are shown in Table  under the labels F1 and F2.

The data shows that the usefulness of the feedback can be considered to be "rather useful" (the average value across all the teams is 3.53). A detailed distribution of answers depicted in Figure  reveals that the most successful team considered the feedback less useful than other teams.

Answers to the open question F2 provided four comments on possible improvements. Three teams requested more details about penalties. They would appreciate knowing "how it happened" in addition to "what happened" (marked as *M* in Table ). One comment objected that "some attacks were happening way sooner in reality than on the timeline" (marked as *D* in Table ). This is true because attack penalties were inserted by the Red team manually with some delay.

# Discussion

The validity of the timely feedback evaluation would be affected by dissatisfaction with the exercise so the learners would not be interested in the feedback at all or they would provide distorted data. Dissatisfaction can be caused by poor organization, unfulfilled expectations, or by a disparity between the difficulty of the exercise and the knowledge of learners. In general, this exercise was considered slightly difficult and was also assessed positively (see Section ). Although the exercise was attended by skilled teams whose members had previous experience with similar exercises, they still considered it rather difficult, challenging and useful. Therefore, we believe that conclusions drawn from the analysis based on this particular exercise and its participants are plausible. In the following part, we put together the results of the post-exercise survey, the scoring timeline survey, and the exploration of the timeline and analyse their mutual relationships.

#### Teams sought out feedback

A deeper analysis of learners' interactions with the scoring timeline shows that all teams were using it intensively, regardless of what reflection they provided in the scoring timeline survey. All teams explored all the penalties depicted in their timeline. They also gave us an evaluation of the majority of displayed objectives. The only exception was Team 4, probably due to technical issues. Team 5, which did not respond to the scoring timeline survey at all, still explored their timeline actively. Team 1 rated the scoring timeline as less useful than other teams. It was the most successful team according to the final score and, therefore, the feedback may not have been so interesting for them because they might have already known about their failures. However, even this team was interested in the timely feedback because they explored the scoring timeline for the longest.

#### A need for more detail

Answers to the open question F2 confirmed our assumption that more precise and more detailed information provided by the timely feedback makes the feedback even more attractive to learners. Nevertheless, the need for more detail can also be indirectly inferred. For example, a further analysis of the White team's injects (users and communication injects) indicates that teams often underestimated time-dependant response to this type of "soft" request. Teams could either consider these requests "annoying" and not so important in comparison with attacks or they could just be too busy with the attacks, for instance. A deeper understanding of their behaviour also requires collecting more detailed and better structured data from the timely feedback.

#### Benefits for instructors

Although the scoring feedback was intended primarily for learners, the previous discussion shows that it is very valuable also for organizers and educators who can learn about the exercise and then fine tune its parameters, e. g., by better scheduling of the White team's injects with respect to the attacks. Since learners are not usually aware of this value, they do not make more effort than necessary into providing quality and valuable information for instructors. However, if providing reflections is intuitive and quick for them, they are motivated to use it. Furthermore, since the feedback is generated automatically, organizers can get valuable data without any additional effort.

#### Limitations of the study

This study is limited in two respects. First, data were obtained from a single exercise with a relatively small group of participants. The reason is that the organization of such a complex exercise is expensive and time-consuming and it is organized rarely for only a narrow group of experts. Nevertheless, we consider the sample to be representative because the learners were highly qualified experts, often with experience in similar exercises and the exercise received a positive rating from them. Second, the timeline evaluation questionnaire was simple and freely structured. However, this was intentional since the survey took place at the end of an exhausting two-day exercise and thus a more sophisticated questionnaire might not have guaranteed more precise results.

# Conclusions and Future work

To best of our knowledge, this paper is the first attempt to study the means of providing feedback to learners participating in cyber defence exercises. The literature review and our own experience showed that feedback provided in state-of-the-art exercises is very limited or delayed. The most often used method is an exercise scoreboard displayed to the learners throughout the exercise. Another method is a short verbal evaluation by exercise observers or organizers right after the end the exercise or its phases. The last commonly used method is after-action reports highlighting key conclusions from a laborious manual analysis of heterogeneous data acquired during the exercise (survey, written communication, scoring and monitoring logs, and checks).

The lack of timely feedback results in the learners having limited opportunity to learn from their experience. They undergo numerous real-life situations during the exercise, but they are not supplied with an explanation why such situations occur. We therefore complemented these means by a novel approach which provides feedback to learners right after the end of the exercise with no additional effort required from educators. Learners can explore a scoring timeline depicting increases and decreases in their team's score and display details about individual events (name and type of the exercise objective, and the number of awarded points or penalty points). Also, they have an opportunity to provide their reflections to educators and indicate their awareness about a particular objective and its solution. The exploration and interaction with the scoring timeline enables learners to reflect on their experience and thus strengthen the learning impact of the exercise.

In order to evaluate this approach, we ran an experiment involving a two-day, complex cyber defence exercise with 24 objectives, and 20 professional learners from five security teams. The results, based on an analysis of user surveys and interactions with the new tool, suggest that learners welcomed the new feature even though the feedback was mined automatically and thus provided a very limited level of detail about particular events.

The experiment also outlined directions for future work. First, learners would appreciate more detail about a particular event in the timeline. This can be easily done by adding a detailed description of the objective related to the event. Another option is to extend the scoring application so that instructors can not only assign points, but also provide a comment on each exercise objective. Second, the timeline could be enriched with a display of the relationship between all exercise objectives. This would highlight that an event that the team may not have been aware of was caused by several previous events they encountered. The context can be then built not only from a time perspective, but also from the topology of the exercise network. Providing information on which host or service was affected by the particular event may help learners to recall the particular situation and understand it better. Finally, the ultimate goal is to provide a "replay function", which would show how attackers proceeded and what could have been done better to prevent or mitigate attacks.