In our basic setup, the reward is the negative of the change in energy i.e., the more the action taken reduces the energy, the greater the reward. Since we are trying to find structures with low energy, RL-CSP gets a positive reward when it lowers the energy by changing the structure and a negative reward when it increases the energy. In reinforcement learning, the technique of reward shaping can be employed to encourage the desired behaviour . For this purpose, we developed extensions of the energy drop reward in the form of three penalties (negative rewards). We penalise when the energy calculation fails (e.g., did not converge), because the chosen action was in the end not useful as we cannot proceed from the new structure. We penalise when the new structure has the same energy as the last structure -without a penalty this step would get zero reward but we choose to disincentivise such moves to prevent becoming stuck at a local minimum. Finally, we penalise actions that lead to the structures that have energies that we have seen already, which is to promote general exploration of the potential energy surface. Although the last penalty is more general than the one before, it is practically useful to control these penalties independently, e.g., by removing one completely, or by imposing different penalty levels for zero reward cases where the action does not alter the structure and non-unique energy structures where the action does not promote exploration .