Experience Replay for Molecular Design. Experience replay was first proposed by Lin et al. as a mechanism to replay past experiences to the model so that it can learn from the same experience numerous times. Two paradigms in RL are on-policy and off-policy where the model's actions are dictated by its current policy or a separate policy known as the behavior policy, respectively . Experience replay is usually applied in off-policy methods as past experiences are less likely to be applicable to the current policy. In molecular design, experience replay has been proposed by Blaschke et al. and Korshunova et al. to keep track of the best molecules sampled so far, based on their corresponding reward. Notably, both applications of experience replay are for on-policy learning using the REINFORCE 39 algorithm and only Korshunova et al. empirically show its benefit in sparse reward environments. We note that a similar mechanism was proposed by Putin et al. using an external memory.