In RL, the policy maps "states" (in our case these are structures) to "actions" in order to maximise a notion of "reward" (here this is based on the drop in energy from one step to the next). The state space that we use to allow the policy to condition on the current structure is defined by the calculated energy of the structure. The calculated energy is clearly a very important "feature" of the structure, and other features could also be incorporated into the state space. The available actions that we consider are based on the existing CSP codes that we apply our approach to. The reward in the simplest case is the reduction of the calculated energy between the current and the trial structure. In addition to rewarding those actions that produce structures with lower energies, we also introduce penalties to disincentivise certain outcomes such as repeating structures. probability distribution of actions that constitutes a policy. This policy dynamically evolves from one optimisation step Si to another and depends on the energy level Ei of the current structure. The policy is updated after each step according to the reward associated with the outcome of the selected action. The reward function formally captures the desirability of the action choice made and is used by RL-CSP to update the policy. In this illustration, the green and blue actions lead to lower energy structures compared to Ei, which then generate positive rewards. The orange action leads to a structure with an energy higher than Ei and receives a negative reward. This approach leads to an adaptable and dynamic decision making, where action selection depends on structure through its energy (b) and is updated on the fly (c). a shows the policy that defines the probability distribution of actions at step Si when the current structure has the energy Ei. The greyscale potential energy landscape shows the current structure Si as a square and three coloured arrows leading to three local minima that will be attained after applying one of the three actions. b shows how the probability distribution of actions at step Si depends on the energy with Ei identified by the dashed line. c shows how the probability distribution of actions for the structure with energy Ei evolves with time and the dashed line identifies step Si.