Large Language Models (LLMs). The ChemGPT model architecture is based on the GPT-Neo transformer implementation in Hugging-Face . The model has 24 layers, with variable width, w, where w ∈ and w determines the model size. Model sizes range from 77,600 to 1,208,455,168 non-embedding parameters. The model is trained via stochastic gradient descent (SGD) with the AdamW optimizer, using a learning rate of 2 * 10 -5 , a per-GPU batch size of 8, and a constant learning rate schedule with 100 warmup steps for scaling experiments. Models were trained for 10 epochs in a self-supervised manner, with a cross-entropy loss for causal language modeling.