For further details on REINVENT, see the work by Blaschke et al. REINVENT's Learning Hyperparameters. The same hyperparameters were used for the baseline RL and CL experiments: batch size of 128, learning rate of 0.0001, sigma scalar factor of 128, all scoring function components' weights were set to 1, and using the Adam optimizer. Agent Exploration and Exploitation. Balance between agent chemical space exploration and exploitation was achieved by using a diversity filter (DF) and inception, and learning thresholds (Curriculum Progression Criteria). A DF enforces diverse results by defining buckets with limited size that track the number of compounds sampled possessing the same scaffold. Once a bucket is full, further sampling of compounds with the same scaffold will be penalized. Only compounds that exceed a userdefined total score (based on the score contributions of each component in the scoring function defined) are stored and added to the corresponding bucket. The specific total score threshold used in this work was 0.4. Inception is a form of experience replay to mitigate catastrophic forgetting and can speed up convergence by replaying previously sampled favourable compounds to the agent. TALK ABOUT THRESHOLD HERE For further details on REINVENT, see the work by Blaschke et al. In the baseline RL experiments, an Identical Murcko Scaffold DF (penalizes the agent if the same Bemis-Murcko scaffold is sampled beyond the bucket size) and inception were applied. In contrast, the implementation of CL in REINVENT allows one to initialize separate DFs and inception for the Curriculum Phase and Production Phase. During the Curriculum Phase, the goal is for the agent to acquire intermediate knowledge. Thus, no DF was applied as it can be counterproductive to guiding the agent to favourable areas of chemical space. Moreover, the learning thresholds can control exploration… The effect of this hyperparameter was investigated via the low and high scenarios In the Production Phase, a new inception (previous favourable compounds during the Curriculum Phase cleared) was initialized. Presumably, the agent is in a state of productivity and samples compounds that satisfy the Production Objective. Thus, an Identical Murcko Scaffold DF with a bucket size of 25 was applied to encourage exploration, such that the agent samples from different local minima. ROCS 3D Shape Similarity. ROCS is a 3D shape similarity metric, comprised of two components: 'shape' and 'color'. The components are quantified by the match, if at all, between the volumes occupied and the defined pharmacophoric features between the two ligands, respectively. 24,25 Compounds with similar "shape" and "color" are more likely to exhibit similar properties. The implementation of ROCS in REINVENT is described in detail by Papadopoulos et al. In the CL experiments, the hyperparameters used for ROCS were 1:1 shape:color, giving equal weighting to each component in the final ROCS similarity score.