|
--- |
|
library_name: transformers |
|
tags: |
|
- paper-summarization |
|
- lora |
|
- peft |
|
- llama |
|
license: mit |
|
datasets: |
|
- armanc/scientific_papers |
|
language: |
|
- en |
|
metrics: |
|
- rouge |
|
base_model: |
|
- meta-llama/Llama-3.2-1B-Instruct |
|
pipeline_tag: summarization |
|
--- |
|
|
|
# **Llama-PaperSummarization-LoRA** |
|
|
|
## **Model Details** |
|
This is a **LoRA fine-tuned adapter** built on [**meta-llama/Llama-3.2-1B-Instruct**](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct). It is designed for scientific paper summarization tasks and leverages **Low-Rank Adaptation (LoRA)** to enhance model performance efficiently while maintaining a low computational overhead. |
|
|
|
|
|
### **Performance comparison** |
|
| Model | ROUGE-1 | ROUGE-2 | ROUGE-3 | ROUGE-L | |
|
|---------------------------|----------|----------|----------|----------| |
|
| **Llama-3.2-1B-Instruct** | 36.69 | 7.47 | 1.95 | 19.36 | |
|
| **Llama-PaperSummarization-LoRA** | **41.56** | **11.31** | **2.67** | **21.86** | |
|
|
|
The model was evaluated on a **6K-sample test set** using **ROUGE scores** with beam search (beam size = 4). |
|
|
|
|
|
### **How to load** |
|
```python |
|
from transformers import LlamaForCausalLM, AutoTokenizer |
|
from peft import PeftModel |
|
|
|
base_model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct") |
|
peft_model_id = "gabe-zhang/Llama-PaperSummarization-LoRA" |
|
model = PeftModel.from_pretrained(base_model, peft_model_id) |
|
model.merge_and_unload() |
|
``` |
|
|
|
|
|
## **Dataset** |
|
The model was fine-tuned on the [**armanc/scientific_papers**](https://huggingface.co/datasets/armanc/scientific_papers) dataset. Below are the details of the dataset splits: |
|
- **Training Set**: 20K samples |
|
- **Validation Set**: 6K samples |
|
- **Test Set**: 6K samples |
|
|
|
|
|
## **LoRA Configuration** |
|
- **Trainable Parameters**: 850K (~7% of base model parameters) |
|
- **Context Length**: 10K tokens |
|
- **Rank**: 8 |
|
- **Target Modules**: Query and Value matrices |
|
- **Optimization Settings**: |
|
- Gradient Accumulation: 4 steps |
|
- Training Steps: 5K |
|
|
|
### **Training Setup** |
|
- **Hardware**: NVIDIA RTX A6000 GPU |
|
- **Evaluation Frequency**: Every 20 steps |
|
- **Training Duration**: 28 hours |
|
- **Training Scripts**: [gabe-zhang/paper2summary](https://github.com/gabe-zhang/paper2summary) |
|
|
|
|
|
## **License** |
|
This repository contains a **LoRA fine-tuned adapter** derived from the Llama 3.2 model. |
|
- **Llama 3.2 Materials**: Governed by the [Llama 3.2 Community License](./LICENSE_Llama3.2). |
|
- **All other content**: Licensed under the [MIT License](./LICENSE). |
|
|
|
### **Attribution** |
|
- The model prominently incorporates Llama 3.2 as its base. |
|
- "Built with Llama" is displayed as required by the Llama 3.2 Community License. |
|
|
