Update README.md
Browse files
README.md
CHANGED
|
@@ -34,6 +34,14 @@ This model was continually pre-trained in 3 stages.
|
|
| 34 |
- Stage 3: Training on a mix of Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus + wikimedia/wikipedia with rope scaling set to 8, aka **32768** tokens of context. We mixed the wikimedia/wikipedia to hopefully improve the general text and knowledge of the model.
|
| 35 |
|
| 36 |
In total, the model was continually pre-trained on up to 1.3B tokens.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
# print("Additional information")
|
| 38 |
This repo contains the stage 3 pre-trained/base model.
|
| 39 |
|
|
|
|
| 34 |
- Stage 3: Training on a mix of Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus + wikimedia/wikipedia with rope scaling set to 8, aka **32768** tokens of context. We mixed the wikimedia/wikipedia to hopefully improve the general text and knowledge of the model.
|
| 35 |
|
| 36 |
In total, the model was continually pre-trained on up to 1.3B tokens.
|
| 37 |
+
|
| 38 |
+
# print("Use cases")
|
| 39 |
+
As this is a base model, there isn't much to do with it currently. But, you can fine-tune it on your own datasets to turn it into an instruct - chat type model.
|
| 40 |
+
|
| 41 |
+
# print("Notice")
|
| 42 |
+
|
| 43 |
+
This stage-3 base model did not undergo saftey alignment by us, thus it can generate unethical content. Any outputs generated by the LLM are your responsibility.
|
| 44 |
+
|
| 45 |
# print("Additional information")
|
| 46 |
This repo contains the stage 3 pre-trained/base model.
|
| 47 |
|