Here, we instead investigate accelerating RL for molecular design in an oracle agnostic manner using AL (RL-AL). The RL-AL setting poses unique challenges for surrogate-model based AL relating to the inherent feedback loops in the generative RL setting. We begin with a motivating example that illustrates some of these unique difficulties and general implications of RL-AL compared to traditional AL. We then systematically examine the components of the RL-AL system and design a strategy that can accelerate generative molecular design by a ~23-fold reduction in oracle calls and a ~14-65-fold reduction in CPU time. Next, we introduce a new acquisition strategy that is compatible with the MPO nature of the RL process. Finally, we demonstrate the transferability of our approach across oracle functions and quantify the computational-and wall-time saving of our method.