In each iteration the search tree is traversed using the upper confidence bound (UCB) scores of the nodes (eq1). Starting from the root node, the UCB scores of the children are calculated. Here Q is the current sum of previous rewards. N is the number of times the child state has been visited, N-1 is the number of times the parent state has been visited. C is a tunable parameter balancing exploitation and exploration which was set to 1.4 by default. If the selected child is already expanded (i.e. has child nodes), the UCB scores of these are then calculated and the next child selected in an iterative way until an unexpanded leaf node is selected. Actions are stored at the parent level, and the child nodes are first instantiated as node objects by applying the associated action when visited (see below).