---
license: llama3.1
language:
- en
library_name: transformers
tags:
- mergekit
- merge
base_model:
- meta-llama/Meta-Llama-3.1-70B-Instruct
- NousResearch/Hermes-3-Llama-3.1-70B
- abacusai/Dracarys-Llama-3.1-70B-Instruct
- VAGOsolutions/Llama-3.1-SauerkrautLM-70b-Instruct
---


![image/png](https://cdn-uploads.huggingface.co/production/uploads/649dc85249ae3a68334adcc6/yDDOz1fsWfSviCGtCh3f3.png)
**Brinebreath-Llama-3.1-70B**
=====================================

I made this since I started having some problems with Cathallama. This seems to behave well during some days testing.

**Notable Performance**

* 7% overall success rate increase on MMLU-PRO over LLaMA 3.1 70b at Q4_0
* Strong performance in MMLU-PRO categories overall
* Great performance during manual testing

**Creation workflow**
=====================
**Models merged**
* meta-llama/Meta-Llama-3.1-70B-Instruct
* NousResearch/Hermes-3-Llama-3.1-70B
* abacusai/Dracarys-Llama-3.1-70B-Instruct
* VAGOsolutions/Llama-3.1-SauerkrautLM-70b-Instruct

```
flowchart TD
    A[Hermes 3] -->|Merge with| B[Meta-Llama-3.1]
    C[Dracarys] -->|Merge with| D[Meta-Llama-3.1]
    B -->| | E[Merge]
    D -->| | E[Merge]
    G[SauerkrautLM] -->|Merge with| E[Merge]
    E[Merge] -->| | F[Brinebreath]
```

![image/png](https://cdn-uploads.huggingface.co/production/uploads/649dc85249ae3a68334adcc6/3cjOUfghMD2GvxL7a3SOh.png)

**Testing**
=====================

**Hyperparameters**
---------------

* **Temperature**: 0.0 for automated, 0.9 for manual
* **Penalize repeat sequence**: 1.05
* **Consider N tokens for penalize**: 256
* **Penalize repetition of newlines**
* **Top-K sampling**: 40
* **Top-P sampling**: 0.95
* **Min-P sampling**: 0.05

**LLaMAcpp Version**
------------------

* b3600-1-g2339a0be
* -fa -ngl -1 -ctk f16 --no-mmap

**Tested Files**
------------------

* Brinebreath-Llama-3.1-70B.Q4_0.gguf
* Meta-Llama-3.1-70B-Instruct.Q4_0.gguf


**Manual testing**

| Category | Test Case | Brinebreath-Llama-3.1-70B.Q4_0.gguf | Meta-Llama-3.1-70B-Instruct.Q4_0.gguf |
| --- | --- | --- | --- |
| **Common Sense** | Ball on cup | OK | OK | 
|  | Big duck small horse | OK | OK |
|  | Killers | OK | OK |
|  | Strawberry r's | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
|  | 9.11 or 9.9 bigger | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
|  | Dragon or lens | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
|  | Shirts | OK | <span style="color: red;">KO</span> |
|  | Sisters | OK | <span style="color: red;">KO</span> |
|  | Jane faster | OK | OK |
| **Programming** | JSON | OK | OK |
|  | Python snake game | OK | <span style="color: red;">KO</span> |
| **Math** | Door window combination | OK | <span style="color: red;">KO</span> |
| **Smoke** | Poem |  OK | OK |
|  | Story |  OK | OK |

*Note: See [sample_generations.txt](https://huggingface.co/gbueno86/Brinebreath-Llama-3.1-70B/blob/main/sample_generations.txt) on the main folder of the repo for the raw generations.*

**MMLU-PRO**

| Model | Success % |
| --- | --- |
| Brinebreath-3.1-70B.Q4_0.gguf | **49.0%** |
| Meta-Llama-3.1-70B-Instruct.Q4_0.gguf | 42.0% |


| MMLU-PRO category| Brinebreath-3.1-70B.Q4_0.gguf | Meta-Llama-3.1-70B-Instruct.Q4_0.gguf |
| --- | --- | --- |
| Business | **45.0%** | 40.0% |
| Law | **40.0%** | 35.0% |
| Psychology | **85.0%** | 80.0% |
| Biology | **80.0%** | 75.0% |
| Chemistry | **50.0%** | 45.0% |
| History | **65.0%** | 60.0% |
| Other | **55.0%** | 50.0% |
| Health | **70.0%** | 65.0% |
| Economics | **80.0%** | 75.0% |
| Math | **35.0%** | 30.0% |
| Physics | **45.0%** | 40.0% |
| Computer Science | **60.0%** | 55.0% |
| Philosophy | **50.0%** | 45.0% |
| Engineering | **45.0%** | 40.0% |

Note: MMLU-PRO Overall tested with 100 questions. Categories testes with 20 questions from each category.

**PubmedQA**

 Model Name | Success% |
| --- | --- |
| Brinebreath-3.1-70B.Q4_0.gguf| **71.00%** |
| Meta-Llama-3.1-70B-Instruct.Q4_0.gguf | 68.00% |


Note: PubmedQA tested with 100 questions.


**Request**
--------------
If you are hiring in the EU or can sponsor a visa, PM me :D

PS. Thank you mradermacher for the GGUFs!