Update README.md

README.md

@@ -36,12 +36,11 @@ inference:
 
 ## Summary
 
-Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch.
+Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch on a large corpus of text in English.
 
-The following versions of this model are available:
+It's available in the following formats: [Safetensors](https://huggingface.co/Felladrin/Minueza-32M-Base), [GGUF](https://huggingface.co/Felladrin/gguf-Minueza-32M-Base), and [ONNX](https://huggingface.co/Felladrin/onnx-Minueza-32M-Base).
 
-- [Minueza-32M-Base](https://huggingface.co/Felladrin/Minueza-32M-Base): The base model, trained from scratch on a large corpus of text in English.
-- [Minueza-32M-Chat](https://huggingface.co/Felladrin/Minueza-32M-Chat): A version of the base model fine-tuned on conversational datasets.
+And it's being released alongside a fine-tuned version trained in conversational datasets: [Minueza-32M-Chat](https://huggingface.co/Felladrin/Minueza-32M-Chat)
 
 ## Intended Uses
 
@@ -68,37 +67,9 @@ The model was trained on a subset of each of the following non-synthetic dataset
 
 The subsets were interleaved to form the final training corpus of approximately 650 million tokens.
 
-## Usage
-
-This is a pre-trained foundation model. For your task, you will likely want to perform application-specific fine-tuning.
-
-Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.
-
-Having that said, here's how you can run it:
-
-```python
-from transformers import pipeline
-
-generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")
-
-prompt = "The best way to improve your health is"
-
-output = generate(
-    prompt,
-    max_new_tokens=256,
-    do_sample=True,
-    temperature=0.72,
-    top_p=0.73,
-    top_k=50,
-    repetition_penalty=1.176,
-)
-
-print(output[0]["generated_text"])
-```
-
 ## Model Architecture
 
-Trained on a context window of 2048 tokens, this is a transformer model with the Mistral architecture, which includes Grouped-Query Attention, Sliding-Window Attention, and Byte-fallback BPE tokenizer.
+This is a transformer model with the Mistral architecture, trained on a context window of 2048 tokens.
 
 | Configuration           | Value |
 | :---------------------- | :---- |
@@ -110,7 +81,7 @@ Trained on a context window of 2048 tokens, this is a transformer model with the
 | num_key_value_heads     | 4     |
 | vocab_size              | 32002 |
 
-## Training
+The pretraining was made with these hyperparameters and frameworks:
 
 | Hyperparameter              | Value                                         |
 | :-------------------------- | :-------------------------------------------- |
@@ -122,7 +93,6 @@ Trained on a context window of 2048 tokens, this is a transformer model with the
 | total_train_batch_size      | 8                                             |
 | optimizer                   | Adam with betas=(0.9,0.999) and epsilon=1e-08 |
 | lr_scheduler_type           | linear                                        |
-| num_epochs                  | 1.0                                           |
 
 | Framework    | Version     |
 | :----------- | :---------- |
@@ -131,6 +101,60 @@ Trained on a context window of 2048 tokens, this is a transformer model with the
 | Datasets     | 2.16.1      |
 | Tokenizers   | 0.15.1      |
 
+## Fine-tuning
+
+The recommended settings for fine-tuning this model are the following.
+
+For Supervised Fine-Tuning:
+
+| Hyperparameter              | Value                                         |
+| :-------------------------- | :-------------------------------------------- |
+| learning_rate               | 2e-5                                          |
+| total_train_batch_size      | 24                                            |
+| max_seq_length              | 2048                                          |
+| weight_decay                | 0                                             |
+| warmup_ratio                | 0.02                                          |
+
+For Direct Preference Optimization:
+
+| Hyperparameter              | Value                                         |
+| :-------------------------- | :-------------------------------------------- |
+| learning_rate               | 7.5e-7                                        |
+| total_train_batch_size      | 6                                             |
+| max_length                  | 2048                                          |
+| max_prompt_length           | 1536                                          |
+| max_steps                   | 200                                           |
+| weight_decay                | 0                                             |
+| warmup_ratio                | 0.02                                          |
+
+## Usage
+
+This is just a base model. For your task, you will likely want to perform application-specific fine-tuning as recommended above.
+
+Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.
+
+Having that said, here's how you can run it:
+
+```python
+from transformers import pipeline
+
+generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")
+
+prompt = "The best way to improve your health is"
+
+output = generate(
+    prompt,
+    max_new_tokens=256,
+    do_sample=True,
+    temperature=0.72,
+    top_p=0.73,
+    top_k=50,
+    repetition_penalty=1.176,
+)
+
+print(output[0]["generated_text"])
+```
+
 ## License
 
 This model is licensed under the [Apache License 2.0](https://huggingface.co/Felladrin/Minueza-32M-Base/resolve/main/license.txt).