Figure shows the pre-training loss as a function of model and dataset size over many orders of magnitude. Models are trained in a self-supervised, causal language modeling setting and evaluated on next-token prediction for a fixed validation set. Surprisingly, no limitations in performance improvement are seen with increasing scale. The pre-training loss monotonically improves with increasing dataset size up to nearly 10 million molecules. Furthermore, for a fixed data budget, increasing model size provides monotonic improvements to the pre-training loss until the model reaches 1B+ non-embedding parameters. This indicates that even for small datasets, much larger models than were previously considered for deep generative modeling may be useful for pre-training. For the largest dataset considered here, diminishing returns to performance improvements are seen for models above 100 million non-embedding parameters. Interestingly, greater performance improvements are seen with increasing model sizes for smaller datasets than larger ones. For the largest dataset considered, model performance saturates quickly beyond 100 million parameters. However, for the smallest dataset considered, performance plateaus for model sizes between 10 5 -10 7 parameters and then improves considerably. This indicates that for a fixed, small pre-training data budget, significant improvements are possible simply by scaling up the model size. Irrespective of model size, increasing dataset size provides continuous performance improvements with no evidence of diminishing returns for the dataset sizes considered here. Depending on the dataset size, regimes of power-law-like scaling behavior are seen for different ranges of model sizes. Power-law scaling is graphically identifiable as an approximately straight line fit of loss versus model size on a log-log plot. For larger datasets, power law scaling is observed for smaller model sizes. For example, the largest dataset exhibits approximate power law scaling for models between 10 5 and 10 7 non-embedding parameters (Figure A.1). Conversely, for smaller datasets power law scaling is observed for larger models and over a more limited range of model sizes. The smallest dataset exhibits approximate power law scaling for models between 10 7 and 10 8 nonembedding parameters (not shown).