Create non-lore-README.md
Browse files- non-lore-README.md +61 -0
non-lore-README.md
ADDED
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Bigger Body 12b
|
2 |
+

|
3 |
+
A roleplay-focused pseudo full-finetune of Mistral Nemo Instruct.
|
4 |
+
The successor to the Ink series.
|
5 |
+
|
6 |
+
## Testimonials
|
7 |
+
> First impressions (temp 1, min-p .05-.1)
|
8 |
+
> - It passes my silly logic tests (read: me trolling random characters)
|
9 |
+
> - Haven't seen any slop yet
|
10 |
+
> - Writes short and snappy replies
|
11 |
+
> - ...yet not *too* short, like Mahou, and can write longer responses if the context warrants it
|
12 |
+
> - Follows card formatting instructions
|
13 |
+
>
|
14 |
+
> If this holds up to 16K it will be constantly in the hopper alongside Mag-Mell for me. I'm biased towards shorter responses with smarts. :)
|
15 |
+
|
16 |
+
\- Tofumagate
|
17 |
+
|
18 |
+
## Dataset
|
19 |
+
The Bigger Body (referred to as Ink v2.1, because that's still the internal name) mix is absolutely disgusting. It's even more cursed than the original Ink mix.
|
20 |
+
|
21 |
+
<details>
|
22 |
+
<summary>(Public) Original Datasets</summary>
|
23 |
+
|
24 |
+
<!-- Start Generation Here -->
|
25 |
+
<ul>
|
26 |
+
<li><a href="https://huggingface.co/datasets/Fizzarolli/limarp-processed">Fizzarolli/limarp-processed</a></li>
|
27 |
+
<li><a href="https://huggingface.co/datasets/Norquinal/OpenCAI">Norquinal/OpenCAI</a> - <code>two_users</code> split</li>
|
28 |
+
<li><a href="https://huggingface.co/datasets/allura-org/Celeste1.x-data-mixture">allura-org/Celeste1.x-data-mixture</a></li>
|
29 |
+
<li><a href="https://huggingface.co/datasets/mapsila/PIPPA-ShareGPT-formatted-named">mapsila/PIPPA-ShareGPT-formatted-named</a></li>
|
30 |
+
<li><a href="https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following">allenai/tulu-3-sft-personas-instruction-following</a></li>
|
31 |
+
<li><a href="https://huggingface.co/datasets/readmehay/medical-01-reasoning-SFT-json">readmehay/medical-01-reasoning-SFT-json</a></li>
|
32 |
+
<li><a href="https://huggingface.co/datasets/LooksJuicy/ruozhiba">LooksJuicy/ruozhiba</a></li>
|
33 |
+
<li><a href="https://huggingface.co/datasets/shibing624/roleplay-zh-sharegpt-gpt4-data">shibing624/roleplay-zh-sharegpt-gpt4-data</a></li>
|
34 |
+
<li><a href="https://huggingface.co/datasets/CausalLM/Retrieval-SFT-Chat">CausalLM/Retrieval-SFT-Chat</a></li>
|
35 |
+
<li><a href="https://huggingface.co/datasets/ToastyPigeon/fujin-filtered-instruct">ToastyPigeon/fujin-filtered-instruct</a></li>
|
36 |
+
</ul>
|
37 |
+
</details>
|
38 |
+
|
39 |
+
## Quants
|
40 |
+
TODO!
|
41 |
+
|
42 |
+
## Recommended Settings
|
43 |
+
Chat template: Mistral *v7-tekken* (NOT v3-tekken !!!! the main difference is that v7 has specific `[SYSTEM_PROMPT]` and `[/SYSTEM_PROMPT]` tags)
|
44 |
+
Recommended samplers (not the be-all-end-all, try some on your own!):
|
45 |
+
- Temp 1.25 / MinP 0.1
|
46 |
+
|
47 |
+
## Hyperparams
|
48 |
+
### General
|
49 |
+
- Epochs = 2
|
50 |
+
- LR = 1e-5
|
51 |
+
- LR Scheduler = Cosine
|
52 |
+
- Optimizer = [Apollo-mini](https://github.com/zhuhanqing/APOLLO)
|
53 |
+
- Optimizer target modules = `all_linear`
|
54 |
+
- Effective batch size = 16
|
55 |
+
- Weight Decay = 0.01
|
56 |
+
- Warmup steps = 50
|
57 |
+
- Total steps = 920
|
58 |
+
|
59 |
+
## Credits
|
60 |
+
Humongous thanks to the people who created the data. I would credit you all, but that would be cheating ;)
|
61 |
+
Big thanks to all Allura members for testing and emotional support ilya /platonic
|