Experience Replay is Vital for Sample Efficiency. We first identified the optimal number of augmentation rounds for Augmented Memory as two for training stability. Increasing the augmentation rounds can further improve sample efficiency but can lead to mode collapse (Appendix A). Next, we compare REINVENT , AHC , BAR , and Double Loop RL 17 with our method. Figure plots the number of oracle calls to explicitly highlight the computational budget. Augmented Memory significantly outperforms all other algorithms and reaches a score of 0.8 with 6,336 oracle calls (average over 100 replicates). Double Loop RL uses experience replay and is the second most sample efficient algorithm and reaches a score of 0.8 after 12,416 ± 1,984 oracle calls (as stated in their paper), which is almost twice the number of oracle calls required compared to our method. Moreover, the key observation we convey is that experience replay improves upon the base algorithm in all cases (Figure ). For example, AHC with the newly implemented experience replay reaches a score of 0.8, but with almost 2.5x the oracle calls (15,552). Our observations around experience replay are supported by previous works . Finally, we show that augmentation is crucial for the enhanced sample efficiency in Appendix D.