We now describe the exact functional form of the policy in RL-CSP. The policy has two parameters per action. One is a constant and the other multiplies the normalised energy, giving a linear action preferences function. This function gives a score for each action; actions with higher scores are chosen with greater probability. Given these two parameters per action, the final step to define the policy is to convert the scores into a probability distribution using the standard soft-max operator, which proceeds by exponentiating the scores and subsequently dividing them by their sum ensuring that the total probability of actions is equal to one (see the Methods section for details).