In this paper, we develop strategies for scaling deep chemical models and investigate neural scaling behavior in large language models for generative chemical modeling and graph neural networks (GNNs) for machine-learned interatomic potentials. We introduce ChemGPT, a generative pre-trained transformer for autoregressive language modeling of small molecules. We train ChemGPT models with over one billion parameters, using datasets of up to ten million unique molecules. We also examine large, invariant and equivariant GNNs trained on trajectories from molecular dynamics and investigate how physics-based priors affect scaling behavior. To overcome the challenges of hyperparameter tuning at scale in new domains, we extend techniques for accelerating neural architecture search to reduce total time and compute budgets by up to 90% during HPO and neural architecture selection. We identify trends in chemical model scaling with respect to model capacity and dataset size, and show the performance improvements seen with increasing scale. We demonstrate the capability to tune ChemGPT's outputs via "prompt engineering" and sampling strategies. Pre-trained ChemGPT models are also robust, self-supervised representation learners that generalize to previously unseen regions of chemical space and enable embedding-based nearest-neighbor search. The scaling strategies and results enable immediate, significant improvements to model performance, as well as computational and data efficiency for deep chemical models. Our results provide motivation and practical guidance for scaling studies in scientific deep learning, as well as many fruitful new research directions at the intersection of massive scale and physics-informed deep learning.