Training performance estimation accelerates hyperparameter optimization for new datasets, models, and scales. To conduct extensive scaling experiments, we first need to find reasonable hyperparameters and training settings. Unlike for NLP and CV, there are no default model architectures, datasets, tasks, hyperparameter settings, or training settings for large-scale chemical deep learning. Simply transferring empirical results from other deep learning domains or smaller scale experiments will lead to suboptimal results . Whereas large models and datasets are standard in traditional deep learning application areas, to investigate scaling in deep chemical models, we must lay the groundwork for large-scale experiments. To this end, we first tackle the problem of accelerating HPO in general settings, for new model architectures, heterogeneous datasets, and at scales that have not been previously investigated.