README.md

metadata

tags:
  - merge
  - mergekit
  - NousResearch/Meta-Llama-3-8B-Instruct
base_model:
  - NousResearch/Meta-Llama-3-8B-Instruct

Aura-llama

Now that the cute anime girl has your attention.

UPDATE: Model has been fixed

Aura-llama is using the methodology presented by SOLAR for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications with continued pretraining. Using the solar paper as a base, I integrated Llama-3 weights into the upscaled layers, and In the future plan to continue training the model.

Aura-llama is a merge of the following models to create a base model to work from:

• meta-llama/Meta-Llama-3-8B-Instruct
• meta-llama/Meta-Llama-3-8B-Instruct

Merged Evals (Has Not Been Finetuned):

Aura-llama

• Avg: ?
• ARC: ?
• HellaSwag: ?
• MMLU: ?
• T-QA: ?
• Winogrande: ?
• GSM8K: ?

🧩 Configuration

dtype: float16
merge_method: passthrough
slices:
- sources:
  - layer_range: [0, 12]
    model: NousResearch/Meta-Llama-3-8B-Instruct
- sources:
  - layer_range: [8, 20]
    model: NousResearch/Meta-Llama-3-8B-Instruct
- sources:
  - layer_range: [16, 28]
    model: NousResearch/Meta-Llama-3-8B-Instruct
- sources:
  - layer_range: [24, 32]
    model: NousResearch/Meta-Llama-3-8B-Instruct