Model save

README.md

@@ -1,17 +1,17 @@
 ---
-base_model: mistralai/Ministral-8B-Instruct-2410
+base_model: RyanYr/reflect_single_mini8B_SftT12
 library_name: transformers
-model_name: reflect_single_mini8B_SftT12
+model_name: reflect_single_mini8B_Om2SftT012_Om2G8kOm2Ag40kIpsdpT012_b0.5
 tags:
 - generated_from_trainer
 - trl
-- sft
+- dpo
 licence: license
 ---
 
-# Model Card for reflect_single_mini8B_SftT12
+# Model Card for reflect_single_mini8B_Om2SftT012_Om2G8kOm2Ag40kIpsdpT012_b0.5
 
-This model is a fine-tuned version of [mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410).
+This model is a fine-tuned version of [RyanYr/reflect_single_mini8B_SftT12](https://huggingface.co/RyanYr/reflect_single_mini8B_SftT12).
 It has been trained using [TRL](https://github.com/huggingface/trl).
 
 ## Quick start
@@ -20,16 +20,16 @@ It has been trained using [TRL](https://github.com/huggingface/trl).
 from transformers import pipeline
 
 question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="RyanYr/reflect_single_mini8B_SftT12", device="cuda")
+generator = pipeline("text-generation", model="RyanYr/reflect_single_mini8B_Om2SftT012_Om2G8kOm2Ag40kIpsdpT012_b0.5", device="cuda")
 output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
 print(output["generated_text"])
 ```
 
 ## Training procedure
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yyr/huggingface/runs/k6r5v7mf)
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yyr/huggingface/runs/mvbd6d22)
 
-This model was trained with SFT.
+This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
 
 ### Framework versions
 
@@ -41,7 +41,18 @@ This model was trained with SFT.
 
 ## Citations
 
+Cite DPO as:
 
+```bibtex
+@inproceedings{rafailov2023direct,
+    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
+    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
+    year         = 2023,
+    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
+    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
+    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
+}
+```
 
 Cite TRL as:
     

last_checkpoint/config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "mistralai/Ministral-8B-Instruct-2410",
+  "_name_or_path": "RyanYr/reflect_single_mini8B_SftT12",
   "architectures": [
     "MistralForCausalLM"
   ],

last_checkpoint/model-00001-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c7a4faad9100f644e1b74119d3f13805ce35092f45d5ca157971921cfc6f4e5e
+oid sha256:9e9ec8b955ab27329222edd0e71f5f053b9438e704d8a5076655fa8361ea9a71
 size 4983016096

last_checkpoint/model-00002-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1e5c814c08f612620c8eff8158154bfe2af199acb528555d1674994fd0a3d746
+oid sha256:4a36dc1c87b9c96995c0313d6c131294d1e0ebae51790b045da6f6264e7cb600
 size 4999836776

last_checkpoint/model-00003-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8b1a3b7cd1f492d247e6ab24cd994a880b351e34f9c5b9809e73eaa369de4941
+oid sha256:8b16fcd33e712553eceac4f9d9d52fe5f72bdbde05da15a19cd64eb556adccdc
 size 4983067960

last_checkpoint/model-00004-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f07a0b4b5309063a0aa422dde492465d94d500a7f70b55a372486f9b61ab222f
+oid sha256:a7f8024f898d2ab88528faeb87506ce355a77de6e8135708001aaf0d12adec2a
 size 1073750144

last_checkpoint/tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:14fc76e7d62f937dce5abc546fc76d251f007317817d2360e5c5dbdb90bbd87b
-size 17078417
+oid sha256:3537f74c84bc8e8cbb95a43947b1bdc89bb143d548bdd2a28ac2c3bbf51971ed
+size 17078318

last_checkpoint/tokenizer_config.json

@@ -8017,9 +8017,13 @@
   "clean_up_tokenization_spaces": false,
   "eos_token": "</s>",
   "legacy": true,
+  "max_length": 16384,
   "model_max_length": 16384,
   "pad_token": "[PAD]",
+  "stride": 0,
   "tokenizer_class": "LlamaTokenizer",
+  "truncation_side": "left",
+  "truncation_strategy": "longest_first",
   "unk_token": "<unk>",
   "use_default_system_prompt": false
 }

last_checkpoint/training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71effb09366401ed9fd0b1aacef6ad75b7449d279f821127a6d78447eeac56b2
-size 6968
+oid sha256:942a478820cabe004a36a544a3ccfa81413200dd7bfbb7599f3a7be00633b46d
+size 8056