Sample Efficiency in Molecular Design. Many existing policy-based RL works for molecular design are based on the REINFORCE 39 algorithm, particularly for models operating on SMILES. Algorithmic alternatives present a unifying theme of using biased gradients to direct the policy towards chemical space with high reward. Neil et al. explored Hill Climbing (HC) and Proximal Policy Optimization (PPO) . Similarly, Atance et al. introduced Best Agent Reminder (BAR) which keeps track of the best agent and reminds the current policy of favorable actions. Thomas et al. introduced Augmented Hill Climbing (AHC) , a hybrid of HC and REINVENT's algorithm, which updates the policy at every epoch using only the top-k generated molecules and shows improved sample efficiency. However, sample efficiency by itself is not sufficient for practical applications of molecular generative models as one should aim to generate diverse molecules that satisfy the objective function. To address this limitation, Bjerrum et al. built directly on REINVENT and introduced Double Loop RL . By performing SMILES augmentation, the policy can be updated numerous times per oracle call. Their results showed improved sample efficiency compared to AHC, while maintaining diverse sampling.