Update README.md

README.md

@@ -23,7 +23,7 @@ As per the [Roblox website](https://create.roblox.com/docs/assistant/guide), the
 
 # print("Stages of pre-training")
 
-This model was pre-trained in 3 stages.
+This model was continually pre-trained in 3 stages.
 
 - Stage 1: Pre-training on the Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus on 4096 tokens (the maximum olmo 2 can usually reach)
 
@@ -33,6 +33,8 @@ This model was pre-trained in 3 stages.
 !stage 3 and onwards were with added layers. the model started with 16 layers, then we merged another 20 to make the model bigger and deeper!
 - Stage 3: Training on a mix of Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus + wikimedia/wikipedia with rope scaling set to 8, aka **32768** tokens of context. We mixed the wikimedia/wikipedia to hopefully improve the general text and knowledge of the model.
 
+In total, the model was continually pre-trained on up to 1.3B tokens.
+# print("Additional information")
 This repo contains the stage 3 pre-trained/base model.
 
 unsloth was used for training (https://unsloth.ai/)