Fizzarolli commited on
Commit
65d94c0
·
verified ·
1 Parent(s): 44b97e6

Create non-lore-README.md

Browse files
Files changed (1) hide show
  1. non-lore-README.md +61 -0
non-lore-README.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Bigger Body 12b
2
+ ![image/png](Z7EP8PNEYT29NBYH0FS0PKKMX0.jpeg)
3
+ A roleplay-focused pseudo full-finetune of Mistral Nemo Instruct.
4
+ The successor to the Ink series.
5
+
6
+ ## Testimonials
7
+ > First impressions (temp 1, min-p .05-.1)
8
+ > - It passes my silly logic tests (read: me trolling random characters)
9
+ > - Haven't seen any slop yet
10
+ > - Writes short and snappy replies
11
+ > - ...yet not *too* short, like Mahou, and can write longer responses if the context warrants it
12
+ > - Follows card formatting instructions
13
+ >
14
+ > If this holds up to 16K it will be constantly in the hopper alongside Mag-Mell for me. I'm biased towards shorter responses with smarts. :)
15
+
16
+ \- Tofumagate
17
+
18
+ ## Dataset
19
+ The Bigger Body (referred to as Ink v2.1, because that's still the internal name) mix is absolutely disgusting. It's even more cursed than the original Ink mix.
20
+
21
+ <details>
22
+ <summary>(Public) Original Datasets</summary>
23
+
24
+ <!-- Start Generation Here -->
25
+ <ul>
26
+ <li><a href="https://huggingface.co/datasets/Fizzarolli/limarp-processed">Fizzarolli/limarp-processed</a></li>
27
+ <li><a href="https://huggingface.co/datasets/Norquinal/OpenCAI">Norquinal/OpenCAI</a> - <code>two_users</code> split</li>
28
+ <li><a href="https://huggingface.co/datasets/allura-org/Celeste1.x-data-mixture">allura-org/Celeste1.x-data-mixture</a></li>
29
+ <li><a href="https://huggingface.co/datasets/mapsila/PIPPA-ShareGPT-formatted-named">mapsila/PIPPA-ShareGPT-formatted-named</a></li>
30
+ <li><a href="https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following">allenai/tulu-3-sft-personas-instruction-following</a></li>
31
+ <li><a href="https://huggingface.co/datasets/readmehay/medical-01-reasoning-SFT-json">readmehay/medical-01-reasoning-SFT-json</a></li>
32
+ <li><a href="https://huggingface.co/datasets/LooksJuicy/ruozhiba">LooksJuicy/ruozhiba</a></li>
33
+ <li><a href="https://huggingface.co/datasets/shibing624/roleplay-zh-sharegpt-gpt4-data">shibing624/roleplay-zh-sharegpt-gpt4-data</a></li>
34
+ <li><a href="https://huggingface.co/datasets/CausalLM/Retrieval-SFT-Chat">CausalLM/Retrieval-SFT-Chat</a></li>
35
+ <li><a href="https://huggingface.co/datasets/ToastyPigeon/fujin-filtered-instruct">ToastyPigeon/fujin-filtered-instruct</a></li>
36
+ </ul>
37
+ </details>
38
+
39
+ ## Quants
40
+ TODO!
41
+
42
+ ## Recommended Settings
43
+ Chat template: Mistral *v7-tekken* (NOT v3-tekken !!!! the main difference is that v7 has specific `[SYSTEM_PROMPT]` and `[/SYSTEM_PROMPT]` tags)
44
+ Recommended samplers (not the be-all-end-all, try some on your own!):
45
+ - Temp 1.25 / MinP 0.1
46
+
47
+ ## Hyperparams
48
+ ### General
49
+ - Epochs = 2
50
+ - LR = 1e-5
51
+ - LR Scheduler = Cosine
52
+ - Optimizer = [Apollo-mini](https://github.com/zhuhanqing/APOLLO)
53
+ - Optimizer target modules = `all_linear`
54
+ - Effective batch size = 16
55
+ - Weight Decay = 0.01
56
+ - Warmup steps = 50
57
+ - Total steps = 920
58
+
59
+ ## Credits
60
+ Humongous thanks to the people who created the data. I would credit you all, but that would be cheating ;)
61
+ Big thanks to all Allura members for testing and emotional support ilya /platonic