The performance of task optimization is shown in Figure . AHC is the most efficient of all RL strategies at all tasks except for maximizing the number of heavy atoms (Figure ). It is particularly better than the other RL strategies during early-stage optimization (e.g., Figure ) and in more difficult objectives (e.g., Figure , f). AHC even outperforms un-regularized RL strategies. This observation is true also for performance by wall time (Figure ), a more practical measure. Intriguingly, AHC seems to achieve maximization towards the end of training in the heavy atom task (seen to a lesser extent with REINVENT 2.0), suggesting it will eventually be able to extrapolate outside the training domain. As AHC uses a considerably smaller batch size than HC and therefore undergoes more frequent network updates, we applied the same batch size to HC to investigate this effect, denoted as HC*.