We demonstrate the efficacy of our approach by applying RL to two established CSP codes; FUSE and MC-EMMA , which have both contributed to the discovery of complex materials, including the first known bulk oxide quasicrystal . Given the composition, we measure the success of the RL approach applied to the particular CSP code by the number of steps required to locate the global minimum. Each step involves modifying the current structure with an action followed by the local relaxation of the resulting structure. Indeed, the local relaxation of the trial structure is the most resource intensive operation during a CSP run, so we aim to reduce the number of relaxed structures, i.e., the number of steps needed to achieve the global minimum. For individual experiments, we find that the RL approach is often considerably better than the fixed policies presented here.