The work presented here invites many directions for future research. Our goal is to enable neural scaling studies and investigate the potential benefits of scaling for chemical deep learning. Future work will investigate the complex relationships between pre-training performance improvements and downstream tasks. Large, pre-trained models can be fine-tuned on any smaller chemical space of interest to investigate the benefits of transfer learning and potential for generalization. Similarly, further study of the representations learned by large pre-trained models is warranted, including the proposal of new benchmarks for evaluating the quality of learned representations beyond downstream property prediction tasks. We were limited by computational and engineering constraints, but much larger chemical models and pre-training datasets clearly warrant investigation; this will require more advanced model parallelism approaches. Finally, the effects of physics-based priors on scaling behavior give a rich description of how the incorporation of physics, known empirical relationships, and other forms of knowledge into machine learning frameworks impact both learning quality and efficiency. Future work in this area is well-poised to yield fundamental advances in scientific machine learning.