README.md

---
language: en
library_name: transformers
license: llama3.1
base_model: meta-llama/Llama-3.1-8B-Instruct
pipeline_tag: text-generation
tags:
- llama-3.1
- instruction-tuned
datasets:
- OpenAssistant/oasst1
- databricks/databricks-dolly-15k
- Open-Orca/OpenOrca
- mlabonne/open-perfectblend
- tatsu-lab/alpaca
model-index:
  - name: utkmst/chimera-beta-test2-lora-merged
    results:
      - task:
          type: text-generation
        dataset:
          type: leaderboard
          name: Overall Leaderboard
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4440
            verified: true 
          - name: acc
            type: acc
            value: 0.2992
            verified: true 
          - name: exact_match
            type: exact_match
            value: 0.0951
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: bbh
          name: BBH (Big Bench Hard)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4773
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: gpqa
          name: GPQA (Google-Patched Question Answering)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.3036
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: math
          name: Math
        metrics:
          - name: exact_match
            type: exact_match
            value: 0.0951
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: mmlu_pro
          name: MMLU-Pro
        metrics:
          - name: acc
            type: acc
            value: 0.2992
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: musr
          name: MUSR (Multi-Step Reasoning)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4113
            verified: true 
---

# utkmst/chimera-beta-test2-lora-merged

## Model Description
This model is a fine-tuned version of Meta's Llama-3.1-8B-Instruct model, created through LoRA fine-tuning on multiple instruction datasets, followed by merging the adapter weights with the base model.

## Architecture
- **Base Model**: meta-llama/Llama-3.1-8B-Instruct
- **Size**: 8.03B parameters
- **Type**: Decoder-only transformer
- **Format**: SafeTensors (full precision)

## Training Details
- **Training Method**: LoRA fine-tuning followed by adapter merging
- **LoRA Configuration**:
  - Rank: 8 
  - Alpha: 16 
  - Trainable modules: Attention layers and feed-forward networks
- **Training Hyperparameters**:
  - Learning rate: 2e-4 
  - Batch size: 2 
  - Training epochs: 1 
  - Optimizer: AdamW with constant scheduler

## Intended Use
This model is designed for:
- General purpose assistant capabilities
- Question answering and knowledge retrieval
- Creative content generation
- Instructional guidance

## Limitations
- Base model limitations including potential hallucinations and factual inaccuracies
- Limited context window compared to larger models
- Knowledge cutoff from the base Llama-3.1 model
- May exhibit biases present in training data
- Performance on specialized tasks may vary

## Usage with Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
tokenizer = AutoTokenizer.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
```
## License

This model inherits the license from Meta's Llama 3.1.