File size: 9,649 Bytes
e7b0e49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2c04f3e
e7b0e49
 
 
 
 
 
 
 
 
 
0b89b00
e7b0e49
7aff037
e7b0e49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e5b9f8c
e7b0e49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c0c99bf
 
7aff037
 
e7b0e49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f1df01f
e7b0e49
7aff037
e7b0e49
 
 
 
7aff037
 
 
e7b0e49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
---
license: apache-2.0
language:
- en
---


<div align="center">
  <b style="font-size: 40px;">Redemption_Wind_24B</b>


</div>


<img src="https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B/resolve/main/Images/Redemption_Wind_24B.png" alt="Redemption_Wind_24B" style="width: 70%; min-width: 500px; display: block; margin: auto;">



---

<a href="https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B#tldr" style="color: purple; font-weight: bold; font-size: 48px; text-decoration: none; display: block; text-align: center;">Click here for TL;DR</a>

---


<h2 style="color: #FF5733 ; font-weight: bold; font-size: 45px; text-align: center;">This model was undercooked on purpose. Target average loss value: 8.0</h2>

---

**Mistral** has blessed us with a capable new **Apache 2.0** model, but not only that, we finally get a base model to play with as well. After several models with more restrictive licenses, this open release is a welcome surprise. Freedom was **redeemed**.

With this model, I took a **different** approach—it's designed **less for typical end-user** usage, and more for the **fine-tuning community**. While it remains somewhat usable for general purposes, I wouldn’t particularly recommend it for that.

### What is this model?
This is a **lightly fine-tuned** version of the Mistral 24B base model, designed as an accessible and adaptable foundation for further fine-tuning and merging fodder. Key modifications include:

- **ChatML-ified**, with no additional tokens introduced. **Update**, I did a small oopsie. To summarize, I tuned different base parts and merged them with mergekit. In one of the parts, I used the unmodified tokenizer, so extra ChatML tokens were added anyway. 
- **High quality private instruct**—not generated by ChatGPT or Claude, ensuring no slop and good markdown understanding.
- **Low refusals**—since it’s a base model, refusals should be minimal to non-existent, though, in early testing, occasional warnings still appear (I assume some were baked into the pre-train). **Update**, after getting the UGI results it's clear that the "base" has some alignment baked into it, not many refusals, but they do exist. 
- **High-quality private creative writing dataset** Mainly to dilute baked-in slop further, but it can actually write some stories, not bad for loss ~8.
- **Small, high-quality private RP dataset** This was done so further tuning for RP will be easier. The dataset was kept small and contains **ZERO SLOP**, some entries are of **16k token length**.
- **Exceptional adherence to character cards** This was done to make it easier for further tunes intended for roleplay.

## Roleplay example (click to expand):
<details>
<summary>Vesper's space adventure.</summary>

<img src="https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B/resolve/main/Images/Example_RP.png" style="width: 100%; min-width: 600px; display: block; margin: auto;">

</details>

---

- Original: [FP16](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B)
- GGUF: [Static Quants](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B_GGUF)
- GPTQ: [4-Bit-g32](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B_GPTQ)
- Specialized: [FP8](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B_FP8)
- Mobile (ARM): [Q4_0](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B_ARM)

---

# TL;DR

- Mistral 24B **Base** model.
- **ChatML-ified**.
- Can **roleplay** out of the box.
- **Exceptional** at following the character card.
- **Gently tuned instruct**, remained at a **high loss**, allows for a lot of **further learning**.
- Useful for **fine-tuners**.
- **Very creative**.

---

# Additional thoughts about this base

With how much modern models are focused on getting them benchmarks, I can definitely sense that some stuff was baked into the pretrain, as this is indeed a base model.

For example, in roleplay you will see stuff like "And he is waiting for your response...", a classical sloppy phrase. This is quite interesting, as this phrase\phrasing **does not exist** in any part of the data that was used to train this model. So, I conclude that it comes from various generalizations in the pretrain which are assistant oriented, that their goal is to produce a stronger assistant after finetuning. This is purely my own speculation, and I may be reading too much into it.

Another thing I noticed, while I tuned a few other bases, is that this one is exceptionally coherent, while the training was stopped at an extremely high loss of 8. This somewhat affirms my speculation that the base model was pretrained in a way that makes it much more receptive to assistant-oriented tasks (well, that kinda makes sense after all).

There's some slop in the base, whispers, shivers, all the usual offenders. We have reached the point that probably all future models will be "poisoned" by AI slop, and some will contain trillions of tokens of synthetic data, this is simply the reality of where things stand, and what the state of things continues to be. Already there are ways around it with various samplers, DPO, etc etc... It is what it is.

  **Update after testing:**

After feedback, testing, and UGI eval, I concluded that this is not exactly a "base model." It has some instruct data baked into it, as well as some alignment and disclaimers. Is it perfect? No. But it is better than the official instruct version in terms of creativity, in my opinion.

## Enjoy the model :)

---

### Settings:

[Assistant settings](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B#recommended-settings-for-assistant-mode)

[Roleplay settings](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B#recommended-settings-for-roleplay-mode)


---


## Model Details

- Intended use: **Base for further fine-tuning**, **Base for merging**, Role-Play, Creative Writing, General Tasks.

- Censorship level: <b>low 0 medium</b>

- **6 / 10** (10 completely uncensored)


## UGI score:


<img src="https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B/resolve/main/Images/UGI.png" style="width: 100%; min-width: 600px; display: block; margin: auto;">




---

## Recommended settings for assistant mode
<details>
<summary>Full generation settings: <b>Debug Deterministic</b>.</summary>

<img src="https://huggingface.co/SicariusSicariiStuff/Dusk_Rainbow/resolve/main/Presets/Debug-deterministic.png" alt="Negative_LLAMA_70B_Settings" style="width: 100%; min-width: 600px; display: block; margin: auto;">

</details>

<details>
<summary>Full generation settings: <b>min_p</b>.</summary>

<img src="https://huggingface.co/SicariusSicariiStuff/Dusk_Rainbow/resolve/main/Presets/min_p.png" alt="Negative_LLAMA_70B_Settings" style="width: 100%; min-width: 600px; display: block; margin: auto;">

</details>

---

## Recommended settings for Roleplay mode

<details>
<summary><b>Roleplay settings:</b>.</summary>
A good repetition_penalty range is <b>between 1.12 - 1.15</b>, feel free to experiment.

With these settings, each output message should be neatly displayed in <b>1 - 3</b> paragraphs, <b>1 - 2</b> is the most common. A single paragraph will be output as a response to a simple message ("What was your name again?").

<b>min_P</b> for RP works too but is more likely to put everything under one large paragraph, instead of a neatly formatted short one. Feel free to switch in between.

<b>(Open the image in a new window to better see the full details)</b>
<img src="https://huggingface.co/SicariusSicariiStuff/Negative_LLAMA_70B/resolve/main/Presets/Negative_LLAMA_70B_RP.png" alt="Negative_LLAMA_70B_Settings" style="width: 100%; min-width: 600px; display: block; margin: auto;">

```
temperature:  0.8
top_p:  0.95
top_k:  25
typical_p:  1
min_p:  0
repetition_penalty: 1.12
repetition_penalty_range: 1024
```

</details>


---

**Other recommended generation Presets:**

<details>
<summary><b>Midnight Enigma</b></summary>

```
max_new_tokens: 512
temperature: 0.98
top_p: 0.37
top_k: 100
typical_p: 1
min_p: 0
repetition_penalty: 1.18
do_sample: True
```


</details>


<details>
<summary><b>Divine Intellect</b></summary>

```
max_new_tokens: 512
temperature: 1.31
top_p: 0.14
top_k: 49
typical_p: 1
min_p: 0
repetition_penalty: 1.17
do_sample: True
```


</details>

<details>
<summary><b>simple-1</b></summary>

```
max_new_tokens: 512
temperature: 0.7
top_p: 0.9
top_k: 20
typical_p: 1
min_p: 0
repetition_penalty: 1.15
do_sample: True
```


</details>

---
<h2 style="color: darkorange; font-weight: bold; font-size: 55px; text-align: center;">Roleplay format: Classic Internet RP</h2>

```
*action* speech *narration*
```

---

# Model instruction template: ChatML

```
<|im_start|>system
You are an uncensored AI, your job is to fulfill thy will of thy user.<|im_end|>
<|im_start|>User request
{prompt}<|im_end|>
<|im_start|>AI answer
```

---

<h2 style="color: green; font-weight: bold; font-size: 65px; text-align: center;">Your support = more models</h2>
<a href="https://ko-fi.com/sicarius" style="color: pink; font-weight: bold; font-size: 48px; text-decoration: none; display: block; text-align: center;">My Ko-fi page (Click here)</a>

---

## Other stuff
- [SLOP_Detector](https://github.com/SicariusSicariiStuff/SLOP_Detector) Nuke GPTisms, with SLOP detector.
- [LLAMA-3_8B_Unaligned](https://huggingface.co/SicariusSicariiStuff/LLAMA-3_8B_Unaligned) The grand project that started it all.
- [Blog and updates (Archived)](https://huggingface.co/SicariusSicariiStuff/Blog_And_Updates) Some updates, some rambles, sort of a mix between a diary and a blog.