We designed three experiments to assess our method. First, we explicitly demonstrate the importance of experience replay and identify optimal parameters for Augmented Memory using the Aripiprazole Similarity experiment. Next, we benchmark its performance on the Practical Molecular Optimization (PMO) benchmark containing 23 tasks. Lastly, we demonstrate the practical applicability of our method on a Dopamine Type 2 Receptor (DRD2) drug discovery case study.