Finally, the reward is defined as R(a t , s t ) = log π θ Augmented -log π θ Agent . The corresponding gradient expression (Equation ) is now equivalent to the gradient of the loss function (Equation ) up to a constant factor. Further details on the derivation and algorithm is in Appendix K. SMILES Augmentation. SMILES are non-injective and yield different sequence representations given a different atom numbering in the molecular graph, i.e., augmented SMILES. SMILES-based molecular generative models have taken advantage of this to train performant models under low-data regimes, e.g., by artificially increasing the dataset size via data augmentation , and to increase chemical space generalizability by training a Prior model on augmented SMILES. Similar to Bjerrum et al. , we reuse scores obtained from the oracle to update the Agent multiple times by passing different augmented SMILES representations.