---
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- code
datasets:
- shukdevdatta123/twitter_sentiment_preprocessed
language:
- en
base_model: distilbert/distilbert-base-uncased
library_name: transformers
license: cc-by-nd-4.0
---
# DistilBERT-base-uncased LoRA Text Classification Model

## Model Description
This model is a fine-tuned version of `distilbert-base-uncased` on an unspecified dataset. It achieves the following results on the evaluation set:

- **Loss:** 0.4649
- **Accuracy:** 84.16%

## Intended Uses & Limitations
This is a text-classification based model.

## Training and Evaluation Data
Look below for more details about the performances.

## Steps to follow
- Installing the Libraries
- Loading the Dataset from HuggingFace
- Train_test Split the Dataset
- Model
- Preprocess Data
- Evaluation
- Apply untrained base model("distilbert-base-uncased") to text
- Train Model using LoRA
- Generate Prediction
- Save the Model and the Tokenizer
- Load the Model and the Tokenizer to test
- Push Model to HuggingFaceHub




### Training Hyperparameters
The following hyperparameters were used during training:

- **Learning Rate:** 0.001
- **Train Batch Size:** 4
- **Eval Batch Size:** 4
- **Seed:** 42
- **Optimizer:** Adam with betas=(0.9,0.999) and epsilon=1e-08
- **LR Scheduler Type:** Linear
- **Number of Epochs:** 10

### Training Results
| Epoch | Training Loss | Validation Loss | Validation Accuracy |
|-------|---------------|-----------------|---------------------|
| 1.0   | 0.5924        | 0.5523          | 78.45%              |
| 2.0   | 0.5983        | 0.5236          | 80.29%              |
| 3.0   | 0.5703        | 0.4498          | 79.56%              |
| 4.0   | 0.5526        | 0.4976          | 80.66%              |
| 5.0   | 0.5326        | 0.4317          | 80.85%              |
| 6.0   | 0.5851        | 0.4562          | 82.87%              |
| 7.0   | 0.5466        | 0.4713          | 81.95%              |
| 8.0   | 0.5494        | 0.5072          | 82.50%              |
| 9.0   | 0.5748        | 0.4802          | 82.87%              |
| 10.0  | 0.5001        | 0.4649          | 84.16%              |

## Framework Versions
- **PEFT:** 0.12.0
- **Transformers:** 4.42.4
- **PyTorch:** 2.4.0+cu121
- **Datasets:** 2.21.0
- **Tokenizers:** 0.19.1

# Dataset Viewer

You can view the dataset using the following link:

[View Twitter Sentiment Preprocessed Dataset](https://huggingface.co/datasets/shukdevdatta123/twitter_sentiment_preprocessed/)

Simply click the link to open the dataset viewer in your browser.

# Model Viewer

You can view the model using the following link:

[View Model in HuggingFace](https://huggingface.co/shukdevdatta123/distilbert-base-uncased-lora-text-classification/)

Simply click the link to open the model file in your browser.

Check out the "Fine-tune LLM.pptx" file for the theory behind this code.

# Github Repository

You can view the github using the following link:

[View GitHub Repository](https://github.com/shukdevtroy/Fine-Tune-LLM-using-LoRA-on-custom-dataset/)

Simply click the link to open the github repo in your browser.

Check out the "Fine-tune LLM.pptx" file in the GitHub repo for the theory behind this code.