Update README.md
Browse files
README.md
CHANGED
|
@@ -27,7 +27,7 @@ As per the [Roblox website](https://create.roblox.com/docs/assistant/guide), the
|
|
| 27 |
|
| 28 |
This model was continually pre-trained in 3 stages.
|
| 29 |
|
| 30 |
-
- Stage 1: Pre-training on the Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus on 4096
|
| 31 |
|
| 32 |
- Stage 2: Pre-training on the boatbomber/roblox-info-dump with rope scaling set to 4, so stage 2 was for expanding the context of the model to **16384**.
|
| 33 |
|
|
|
|
| 27 |
|
| 28 |
This model was continually pre-trained in 3 stages.
|
| 29 |
|
| 30 |
+
- Stage 1: Pre-training on the Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus on 4096 context (the maximum olmo 2 can usually reach)
|
| 31 |
|
| 32 |
- Stage 2: Pre-training on the boatbomber/roblox-info-dump with rope scaling set to 4, so stage 2 was for expanding the context of the model to **16384**.
|
| 33 |
|