File size: 3,694 Bytes
217e4bc
3218d26
 
 
 
 
217e4bc
 
 
 
 
 
 
5ba6652
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
217e4bc
 
 
 
 
3218d26
217e4bc
5ba6652
3218d26
5ba6652
 
 
 
 
217e4bc
5ba6652
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
217e4bc
5ba6652
 
 
 
 
 
217e4bc
 
 
 
 
 
 
 
 
 
 
3218d26
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
---
language: en
library_name: transformers
license: llama3.1
base_model: meta-llama/Llama-3.1-8B-Instruct
pipeline_tag: text-generation
tags:
- llama-3.1
- instruction-tuned
datasets:
- OpenAssistant/oasst1
- databricks/databricks-dolly-15k
- Open-Orca/OpenOrca
- mlabonne/open-perfectblend
- tatsu-lab/alpaca
model-index:
  - name: utkmst/chimera-beta-test2-lora-merged
    results:
      - task:
          type: text-generation
        dataset:
          type: leaderboard
          name: Overall Leaderboard
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4440
            verified: true 
          - name: acc
            type: acc
            value: 0.2992
            verified: true 
          - name: exact_match
            type: exact_match
            value: 0.0951
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: bbh
          name: BBH (Big Bench Hard)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4773
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: gpqa
          name: GPQA (Google-Patched Question Answering)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.3036
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: math
          name: Math
        metrics:
          - name: exact_match
            type: exact_match
            value: 0.0951
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: mmlu_pro
          name: MMLU-Pro
        metrics:
          - name: acc
            type: acc
            value: 0.2992
            verified: true 
      - task:
          type: text-generation
        dataset:
          type: musr
          name: MUSR (Multi-Step Reasoning)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4113
            verified: true 
---

# utkmst/chimera-beta-test2-lora-merged

## Model Description
This model is a fine-tuned version of Meta's Llama-3.1-8B-Instruct model, created through LoRA fine-tuning on multiple instruction datasets, followed by merging the adapter weights with the base model.

## Architecture
- **Base Model**: meta-llama/Llama-3.1-8B-Instruct
- **Size**: 8.03B parameters
- **Type**: Decoder-only transformer
- **Format**: SafeTensors (full precision)

## Training Details
- **Training Method**: LoRA fine-tuning followed by adapter merging
- **LoRA Configuration**:
  - Rank: 8 
  - Alpha: 16 
  - Trainable modules: Attention layers and feed-forward networks
- **Training Hyperparameters**:
  - Learning rate: 2e-4 
  - Batch size: 2 
  - Training epochs: 1 
  - Optimizer: AdamW with constant scheduler

## Intended Use
This model is designed for:
- General purpose assistant capabilities
- Question answering and knowledge retrieval
- Creative content generation
- Instructional guidance

## Limitations
- Base model limitations including potential hallucinations and factual inaccuracies
- Limited context window compared to larger models
- Knowledge cutoff from the base Llama-3.1 model
- May exhibit biases present in training data
- Performance on specialized tasks may vary

## Usage with Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
tokenizer = AutoTokenizer.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
```
## License

This model inherits the license from Meta's Llama 3.1.