Update README.md
Browse files
README.md
CHANGED
|
@@ -33,4 +33,6 @@ This model was pre-trained in 3 stages.
|
|
| 33 |
!stage 3 and onwards were with added layers. the model started with 16 layers, then we merged another 20 to make the model bigger and deeper!
|
| 34 |
- Stage 3: Training on a mix of Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus + wikimedia/wikipedia with rope scaling set to 8, aka **32768** tokens of context. We mixed the wikimedia/wikipedia to hopefully improve the general text and knowledge of the model.
|
| 35 |
|
| 36 |
-
This repo contains the stage 3 pre-trained/base model.
|
|
|
|
|
|
|
|
|
| 33 |
!stage 3 and onwards were with added layers. the model started with 16 layers, then we merged another 20 to make the model bigger and deeper!
|
| 34 |
- Stage 3: Training on a mix of Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus + wikimedia/wikipedia with rope scaling set to 8, aka **32768** tokens of context. We mixed the wikimedia/wikipedia to hopefully improve the general text and knowledge of the model.
|
| 35 |
|
| 36 |
+
This repo contains the stage 3 pre-trained/base model.
|
| 37 |
+
|
| 38 |
+
unsloth was used for training (https://unsloth.ai/)
|