File size: 2,670 Bytes
2a00395 ff19ed3 2a00395 ff19ed3 2a00395 ff19ed3 2a00395 cc5927f 8416351 cc5927f 8416351 cc5927f 2a00395 ff19ed3 cc5927f 2a00395 ff19ed3 2a00395 ff19ed3 2a00395 ff19ed3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
---
library_name: transformers
tags:
- paper-summarization
- lora
- peft
- llama
license: mit
datasets:
- armanc/scientific_papers
language:
- en
metrics:
- rouge
base_model:
- meta-llama/Llama-3.2-1B-Instruct
pipeline_tag: summarization
---
# **Llama-PaperSummarization-LoRA**
## **Model Details**
This is a **LoRA fine-tuned adapter** built on [**meta-llama/Llama-3.2-1B-Instruct**](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct). It is designed for scientific paper summarization tasks and leverages **Low-Rank Adaptation (LoRA)** to enhance model performance efficiently while maintaining a low computational overhead.
### **Performance comparison**
| Model | ROUGE-1 | ROUGE-2 | ROUGE-3 | ROUGE-L |
|---------------------------|----------|----------|----------|----------|
| **Llama-3.2-1B-Instruct** | 36.69 | 7.47 | 1.95 | 19.36 |
| **Llama-PaperSummarization-LoRA** | **41.56** | **11.31** | **2.67** | **21.86** |
The model was evaluated on a **6K-sample test set** using **ROUGE scores** with beam search (beam size = 4).
### **How to load**
```python
from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
peft_model_id = "gabe-zhang/Llama-PaperSummarization-LoRA"
model = PeftModel.from_pretrained(base_model, peft_model_id)
model.merge_and_unload()
```
## **Dataset**
The model was fine-tuned on the [**armanc/scientific_papers**](https://huggingface.co/datasets/armanc/scientific_papers) dataset. Below are the details of the dataset splits:
- **Training Set**: 20K samples
- **Validation Set**: 6K samples
- **Test Set**: 6K samples
## **LoRA Configuration**
- **Trainable Parameters**: 850K (~7% of base model parameters)
- **Context Length**: 10K tokens
- **Rank**: 8
- **Target Modules**: Query and Value matrices
- **Optimization Settings**:
- Gradient Accumulation: 4 steps
- Training Steps: 5K
### **Training Setup**
- **Hardware**: NVIDIA RTX A6000 GPU
- **Evaluation Frequency**: Every 20 steps
- **Training Duration**: 28 hours
- **Training Scripts**: [gabe-zhang/paper2summary](https://github.com/gabe-zhang/paper2summary)
## **License**
This repository contains a **LoRA fine-tuned adapter** derived from the Llama 3.2 model.
- **Llama 3.2 Materials**: Governed by the [Llama 3.2 Community License](./LICENSE_Llama3.2).
- **All other content**: Licensed under the [MIT License](./LICENSE).
### **Attribution**
- The model prominently incorporates Llama 3.2 as its base.
- "Built with Llama" is displayed as required by the Llama 3.2 Community License.
|