Cyrile commited on
Commit
700a117
·
verified ·
1 Parent(s): 2ea9fa9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -2
README.md CHANGED
@@ -13,8 +13,28 @@ tags:
13
  Converted version of [Mixtral Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) to 4-bit using bitsandbytes. For more information about the model, refer to the model's page.
14
 
15
  ### Impact on performance
16
- Impact of quantization on a set of models. The impact of quantization is negligible.
17
- ![constellation](https://i.postimg.cc/T20mBPxD/constellation.png)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
 
19
  ### Prompt Pattern
20
  Here is a reminder of the command pattern to interact with the model if the add_special_tokens option is disabled (otherwise, do not include the BOS symbol and space at the beginning of the sequence):
 
13
  Converted version of [Mixtral Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) to 4-bit using bitsandbytes. For more information about the model, refer to the model's page.
14
 
15
  ### Impact on performance
16
+ Impact of quantization on a set of models.
17
+
18
+ Evaluation of the model was conducted using the PoLL (Pool of LLM) technique, assessing performance on **100 French questions** with scores aggregated from six evaluations
19
+ (two per evaluator). The evaluators included GPT-4o, Gemini-1.5-pro, and Claude3.5-sonnet.
20
+
21
+ **Performance Scores (on a scale of 5):**
22
+ | Model | Score | # params |
23
+ |---------------------------------------------:|:-------:|:--------:|
24
+ | gpt-4o | 4.13 | N/A |
25
+ | mistralai/Mixtral-8x7B-Instruct-v0.1 | 3.71 | 46.7b |
26
+ | cmarkea/Mixtral-8x7B-Instruct-v0.1-4bit | 3.68 | 46.7b |
27
+ | gpt-3.5-turbo | 3.66 | 175b |
28
+ | TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ | 3.56 | 46.7b |
29
+ | mistralai/Mistral-7B-Instruct-v0.2 | 1.98 | 7.25b |
30
+ | cmarkea/bloomz-7b1-mt-sft-chat | 1.69 | 7.1b |
31
+ | cmarkea/bloomz-3b-dpo-chat | 1.68 | 3b |
32
+ | cmarkea/bloomz-3b-sft-chat | 1.51 | 3b |
33
+ | croissantllm/CroissantLLMChat-v0.1 | 1.19 | 1.3b |
34
+ | cmarkea/bloomz-560m-sft-chat | 1.04 | 0.56b |
35
+ | OpenLLM-France/Claire-Mistral-7B-0.1 | 0.38 | 7.25b |
36
+
37
+ The impact of quantization is negligible.
38
 
39
  ### Prompt Pattern
40
  Here is a reminder of the command pattern to interact with the model if the add_special_tokens option is disabled (otherwise, do not include the BOS symbol and space at the beginning of the sequence):