Table compares the performance of the prior models with a baseline method that randomly samples SMILES strings from the training dataset without replacement. By definition, the novelty score of the random baseline is 0.0. The prior models achieved high validity, uniqueness, and novelty scores, exceeding 98% in all cases. In addition, their sphere exclusion diversity (SEDiv) and filters scores were comparable to those of the random baseline, with slight deviations depending on model size. Similarly, the physicochemical property distributions of compounds generated by the prior models closely overlapped with those of the test set, except near the hard cutoff values imposed during the library enumeration (Figure ). The analyzed properties included molecular weight (MW), calculated octanol-water partition coefficient (CLogP), topological polar surface area (TPSA), and quantitative estimate of drug-likeness (QED). No significant differences were observed with respect to model size, suggesting that reproducing these superficial properties was not challenging for the models examined in this study. In contrast, the presence score showed a significant decline with decreasing model size (Table ). This suggests that for chemical language models to generate synthesizable compounds, they must have a relatively high capacity to understand the generative process underlying the training data, i.e., the possible combinations of building blocks and reactions. The 85M parameter model, the largest model examined in this study, achieved a presence score of 0.9072, indicating that more than 90% of the compounds generated by this model are expected to be readily synthesizable. Notably, most of these compounds were not found in the training dataset, as evidenced by a novelty score of 0.9852. To our knowledge, this study is the first to accurately evaluate the ability of chemical language models to generate compounds within the chemical space on which they were trained and to demonstrate the benefits of larger models in this context. Our findings highlight the importance of benchmarking models on well-defined chemical spaces, rather than using existing databases (e.g., ChEMBL 60 ) with unclear generative processes. An interesting direction for future research would be to establish general scaling laws for chemical language models by exploring larger models and more complex chemical spaces than those used in this study.