Sample efficiency is a limiting factor to enabling more exploration of chemical spaces of interest, such as in drug discovery where the reward is sparse, i.e., finding a needle in the haystack. In this paper, we highlight the importance of experience replay in policy-based RL algorithms for molecular generation. We propose a novel algorithm called Augmented Memory that combines experience replay with SMILES augmentation. We further propose Selective Memory Purge which removes entries in the replay buffer with undesired chemical scaffolds and jointly address sample efficiency and diversity. The main contributions of this paper are: