Default settings Figure shows the results of TPE for ChemGPT models trained on two million molecules from the Molecular Sets (MOSES) dataset. MOSES is significantly smaller than PubChem and is representative of datasets on which chemical generative models are typically trained . Here, we use MOSES to demonstrate how optimal settings for a chemical LLM such as ChemGPT can be quickly discovered using TPE. To enable scaling experiments, we are mainly concerned with settings related to the learning dynamics (e.g., batch size and learning rate), that will significantly impact large-scale training and fluctuate depending on the type of model and the characteristics of the dataset. To demonstrate the effectiveness of TPE, we initialize ChemGPT with the default learning rate and batch size for causal language modeling in Hugging-Face. We then vary the learning rate and batch size and train models with different hyperparameters for 50 epochs. Figure shows the true loss after 50 epochs versus the predicted loss using TPE after only 10 epochs. R 2 = 0.98 for the linear regression (Equation ), and Spearman's rank correlation ρ = 1.0.