<jupyter_start><jupyter_text>Peft model evaluation using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness)In this notebook, we are going to learn how to evaluate the finetuned lora model on the hellaswag task using lm-eval-harness toolkit.<jupyter_code># Install LM-Eval
!pip install -q datasets evaluate lm_eval<jupyter_output>[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip<jupyter_text>First we will check the accuracy score on the hellaswag task for the base bert without finetuning<jupyter_code>import lm_eval


output = lm_eval.simple_evaluate(model = 'hf',
                        model_args = {
                            'pretrained' : 'bert-base-cased',
                            'dtype' : 'bfloat16'},
                        tasks = 'hellaswag',
                        device = 'cuda:0',
                        batch_size = 128,
                        log_samples = False)
output["results"]<jupyter_output>2024-11-01:20:45:03,210 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-11-01:20:45:03,211 INFO     [evaluator.py:188] Initializing hf model, with arguments: {'pretrained': 'bert-base-cased', 'dtype': 'bfloat16'}
2024-11-01:20:45:03,213 INFO     [huggingface.py:129] Using device 'cuda:0'
2024-11-01:20:45:03,450 INFO     [huggingface.py:481] Using model type 'default'
2024-11-01:20:45:03,741 INFO     [huggingface.py:365] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
If you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`
2024-11-01:20:45:15,862 INFO     [task.py:415] Building contexts for hellaswag on rank 0...
100%|██████████| 10042/10042 [00:02<00:00, 4477.77it/s]
2024-11-01:20:45:18,875 INFO     [evaluator.py:489] Running loglikelihood requests
Running loglikelihood requests: 100%|██████████| 40168/40[...]<jupyter_text>Now lets try to finetune the bert on the imdb dataset (this is for demonstration and finetuning on imdb may not increase the scores on hellaswag task)<jupyter_code># Import necessary libraries
import evaluate
import numpy as np
from datasets import load_dataset
from transformers import AutoTokenizer, BertForSequenceClassification, Trainer, TrainingArguments

from peft import LoraConfig, TaskType, get_peft_model
# Configure LoRA for Sequence Classification
lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,        # Set task type to sequence classification
    target_modules=["query", "key"]    # Specify target modules for LoRA tuning
)

# Initialize the BERT model for sequence classification
model = BertForSequenceClassification.from_pretrained(
    'bert-base-cased',
    num_labels = 2
)

# Wrap the model with LoRA configuration
model = get_peft_model(model, lora_config)

model.print_trainable_parameters()

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
# load the dataset
dataset = load_dataset("imdb")

def tokenize_function(row):
    return tokenizer(row["text"], padding="max_length", truncation = True)

tokenized_datasets = dataset.map(tokenize_function, batched = True)

train_dataset = tokenized_datasets["train"]
eval_dataset = tokenized_datasets["test"]
# Define a function to compute evaluation metrics

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    metric = evaluate.load("accuracy")
    return metric.compute(predictions = predictions, references = labels)
# Configure training arguments
training_args = TrainingArguments("bert-lora-imdb",
    eval_strategy="epoch",
    per_device_train_batch_size=32, # decrease this for OOM error
    per_device_eval_batch_size=64,
    save_strategy="epoch",
    learning_rate=2e-3,
    num_train_epochs=5,
    weight_decay=0.01,
    load_best_model_at_end=True,
    do_eval=True,
    do_predict=True,
    metric_for_best_model="accuracy",
    report_to="none")

# Initialize the Trainer for the model training loop
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
)

#start training
trainer.train()<jupyter_output>13%|█▎        | 500/3910 [08:16<56:48,  1.00it/s]<jupyter_text>Now take the finetuned lora checkpoint and check the accuracy score on hellaswag task.<jupyter_code># use the path of your checkpoint here
output = lm_eval.simple_evaluate(model = 'hf',
                        model_args = {
                          'pretrained' : 'bert-base-cased',
                          'peft' : './bert-lora-imdb/checkpoint-3910',
                          'dtype' : 'bfloat16'},
                        tasks = 'hellaswag',
                        device = 'cuda:0',
                        batch_size = 128,
                        log_samples = False)

output["results"]<jupyter_output>2024-11-01:23:37:57,640 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-11-01:23:37:57,641 INFO     [evaluator.py:188] Initializing hf model, with arguments: {'pretrained': 'bert-base-cased', 'peft': './bert-lora-imdb/checkpoint-3910', 'dtype': 'bfloat16'}
2024-11-01:23:37:57,643 INFO     [huggingface.py:129] Using device 'cuda:0'
2024-11-01:23:37:57,891 INFO     [huggingface.py:481] Using model type 'default'
2024-11-01:23:37:58,161 INFO     [huggingface.py:365] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
If you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`
2024-11-01:23:38:10,295 INFO     [task.py:415] Building contexts for hellaswag on rank 0...
100%|██████████| 10042/10042 [00:02<00:00, 4453.89it/s]
2024-11-01:23:38:13,313 INFO     [evaluator.py:489] Running loglikelihood requests
Running logli[...]