A key motivation for this work was to begin to investigate scientific deep learning at scale -using large models on massive datasets. Unlike areas such as natural language processing (NLP) and computer vision (CV), where scale has proven to be a key ingredient to recent breakthroughs, scientific domains are built on physics-based priors that impose high levels of structure on data generation processes. For this reason, it is unclear what, if any, benefits massive scale will confer to scientific deep learning. And although there is significant engineering effort required to train and deploy large-scale deep learning models, the study of such models is of inherent scientific interest because these models may display surprising, emergent characteristics that are not predictable by extrapolating from small scales. There is an opportunity in scientific deep learning to anticipate trends towards so-called foundational models, similar to those seen in NLP and CV, to ensure that such important investigations are not limited to a few extremely well-resourced research organizations.