Chemical structural similarity across training and testing folds can inflate performance in retrospective benchmarking, and creating folds based on chemical clusters or scaffolds can partially, but not fully, alleviate that bias. However, in our study we do not emphasize the cross validation performance metrics and instead use the prospective evaluations to assess hit rates and generalization.