In this work, we explicitly show that experience replay is vital for sample efficiency. We propose Augmented Memory which capitalizes on this observation and applies SMILES augmentation on the replay buffer to update the Agent multiple times per oracle call. Compared to existing algorithms, Augmented Memory significantly improves sample efficiency and is able to generate diverse molecules using the newly proposed Selective Memory Purge heuristic. We benchmark Augmented Memory on the PMO benchmark and achieve a new state-of-the-art performance, outperforming the previous state-of-the-art on 19/23 tasks and by a total sum of 0.986 AUC Top-10. Next, we show the practical application of Augmented Memory by mimicking a more realistic drug discovery task. Our method significantly outperforms existing algorithms, as assessed by the property profile of the generated molecules, and can perform MPO. Moreover, we note that in particularly sparse reward landscapes , the enhanced sample efficiency of Augmented Memory may be diminished as it becomes more difficult to populate the replay buffer with high rewarding molecules. Future work will investigate this scenario thoroughly and algorithmic modifications to couple additional local chemical space exploration around high rewarding molecules may better handle sparsity. This work opens up future integration of Augmented Memory with curriculum learning , the use of more expensive oracles given a limited computational budget, and provides further insights into experience replay for molecular generative models.