We trained NGT on a CSL provided by Enamine ("UnREAL", Methods) composed of 74,232 synthons and 21 three-and four-component reactions, resulting in a library with 2,993,132,505,642 products. The initial policy corresponds to a uniform distribution over the Minimum Enumeration Library (MEL) subset of the CSL (Methods). We compared the performance of NGT to a hierarchical exploration (HE) of CSLs . In the case of HE, we substituted the mechanistic docking score with the same deep learning framework used in NGT (Methods). Identical scoring functions in both methods helped us focus on differences attributed to how each method modulates exploration and exploitation of the library.