Although we present our method in this section under the most realistic multi-objective optimization task settings (with experiments in Section 5), our method is easily extendable to other problem settings by modifying the definition of the constraints C P , C N D , and the reward function r accordingly. For example, see Appendix B for the application of our method to the standard constrained penalized log P task and Section 5 for multi-objective molecular optimization under different novelty/diversity metrics.