As Augmented Memory revolves around exploiting experience replay, we investigate the efficacy of our method when using different buffer sizes (Figure ). We again use the Aripiprazole Similarity task to assess the proposed changes. Interestingly, with the exception of a buffer size of 25, minimal difference is observed between buffer sizes. We posit that a buffer size of 25 is more susceptible to mode collapse as it is increasingly likely that the stored molecules are all identical or similar relative to having a larger buffer size. Conversely, our initial hypothesis was that a larger buffer size would decrease sample efficiency. The rationale is that relatively low rewarding molecules may be stored in the buffer and reinforcing on these low rewarding molecules could be counterproductive. Following experiments (Figure ), this was not the case, at least for the Aripiprazole Similarity task. Given that the differences in the buffer sizes result in minimal difference and that our hypothesis may be true for other objective functions, we decided to use a buffer size of 100 for main result experiments. Next, we were curious if reinforcing the Agent with only the molecules in the replay buffer would be possible. In these experiments, the sampled molecules in a given epoch were only used to reinforce the Agent once and no augmented forms were used to further reinforce the Agent. Interestingly, minimal difference is observed again (Figure ). Since the performance is similar, we hypothesize that using augmented forms of the sampled molecules would act to mitigate against mode collapse. This is in agreement with insights from Arús-Pous et al. that posit SMILES augmentation acts as a regularizer. Therefore, all main result experiments were performed using augmented SMILES from the sampled batch and the buffer. The results in the main text show that experience replay is vital for sample efficiency. In this section, the question we answer is: "can we just perform multiple rounds of Agent update with the entire replay buffer without SMILES augmentation?" If yes, then the benefits of Augmented Memory can be attributed to simply experience replay. The experimental design is as follows: using the Aripiprazole Similarity experiment, perform two rounds of Agent update using the entire buffer (size of 100) without SMILES augmentation. This mirrors the optimal parameters of Augmentation Memory of two augmentation rounds and a buffer size of 100. For statistical rigour, we perform 100 replicates and present the results in Table . Compared to Augmented Memory, the average epochs it takes to reach an average score of 0.5, 0.8, and 0.9 is higher (values with augmentation are shown in parentheses and is from Table ). Importantly, the standard deviation is also much higher, suggesting instability in the runs. This is further supported by some runs not reaching the 0.8 and 0.9 average score thresholds. While monitoring the sampling, we notice that the Agent repeatedly samples the same SMILES, indicating mode collapse. From a probabilistic perspective, the Agent negative log-likelihoods (NLLs) become focused on the replay buffer sequences, suggesting tokenlevel memorization. These insights are supported by previous work from Arús-Pous et al. which explored the effect of SMILES augmentation on the Prior's NLL on the training data. Specifically, they found that training a Prior without SMILES augmentation can cause token-level memorization, such that the NLL for the specific SMILES sequences in the training data are low. This decreases the generalizability of the trained Prior. Bjerrum et al. in their Double Loop RL work also posit that reinforcing the Agent on augmented SMILES prevents sequence-wise mode collapse. Our results are in agreement and we show that SMILES augmentation is necessary to ensure the efficacy of Augmented Memory and is itself a regularizer.