In this work, we extend the observations by Korshunova et al. and explicitly show the benefit of experience replay in dense reward environments, i.e., most molecules give at least some reward, for on-policy learning given a static objective function. This static nature means that regardless of the current policy, high-rewarding molecules will always receive the same reward, which supports the efficacy of experience replay in the on-policy setting for molecular generation. Next, we combine elements of HC and SMILES augmentation with experience replay, and propose to update the policy at every fine-tuning epoch using the entire replay buffer. A reward shaping mechanism is introduced by using these extremely biased gradients towards high rewarding chemical space which we show significantly improves sample efficiency. This section describes each component of Augmented Memory (Figure ), which is capable of performing MPO.