Our experiments show that a more detailed quantification of novelty would be highly desirable. While the FCD detects that these naively generated molecules do not reflect the training distribution, it might be tricked as well by a naive model similar to the AddCarbon model. The currently used metrics do not allow to provide insights whether the molecule generators act in a similar way as our AddCarbon model. What is also worth noting is that many of the methods performing best in the GuacaMol and Moses ) benchmarks are likelihood-based models . For these performance could be evaluated using the likelihood on a hold-out test set, similar to natural language processing. We argue that future studies should report this metric if applicable.