E Dopamine Type 2 Receptor (DRD2) Case Study: Exploiting AutoDock Vina This section elaborates on the statement that the experimental design of Bjerrum et al. in their Double Loop RL work is insufficient in preventing AutoDock Vina 53 exploitation. Specifically, the drug discovery case study to design potential dopamine type 2 receptor (DRD2) inhibitors was performed using the following objective function: molecular weight (MW) < 500 Da, maximize QED, and minimize docking score. This is in contrast to the objective function proposed by Bjerrum et al. : molecular MW < 500 Da and minimize docking score. We perform a set of experiments comparing the sample efficiency of alternative algorithms including REINVENT , Best Agent Reminder (BAR) , and Augmented Hill Climbing (AHC) using this simplified objective function. Similar to the main result experiments, we ran all experiments enforcing an oracle budget of 9,600 calls and show the distribution of Vina scores from triplicate pooled runs. The following filter was applied: MW < 500 Da and Vina score < -9.4 (the Vina score of the reference drug molecules, risperidone). It is evident that Augmented Memory significantly outperforms other algorithms, generating drastically better docking scores and more molecules passing the filter. Moreover, experience replay improves the performance of all base algorithms. However, we investigate the property profile of the generated molecules in the Augmented Memory experiments and show that the Agent exploits AutoDock Vina in rewarding lipophilic molecules, i.e., all top scoring molecules have extensive aromatic carbon rings (Figure ). These molecules, while possessing excellent Vina scores, are not realistic. As we emphasize the usability of Augmented Memory on more realistic case studies to encourage practical applications, we show the set of experiments which also enforce QED in the main results. QED ensures generated molecules are "drug-like". We end this section by emphasizing that the ability of Augmented Memory to exploit AutoDock Vina is not a weakness and rather, further proves its ability for sample efficient optimization.