How to measure performance of a pretrained HuggingFace language model?
I am pretraining a GPT2LMHeadModel using Trainer as follows:
training_args = TrainingArguments(
    output_dir=str(project_root / 'models/bn-gpt2/'),
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=4,
    fp16=True,
    optim=&quot;adafactor&quot;,
    eval_steps=400,
    save_steps=800,
    warmup_steps=500,
    evaluation_strategy=&quot;steps&quot;,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['test'],
)

trainer.train()

I want to measure the performance of my pre-trained model using perplexity or accuracy metrics during and after training. I have found some ways to measure these for individual sentences, but I cannot find a way to do this for the complete model. My goal is to create a next word prediction model for my native language using GPT2 training from scratch.

If I understand it correctly then this tutorial shows how to calculate perplexity for the entire test set. If I see it correctly they use the entire test corpus as one string connected by linebreaks, which might have to do with the fact that perplexity uses a sliding window which uses the text that came previous in the corpus. I personally did not calculate perplexity for a model yet and am not an expert at this. In any case you could average the sentence score into a corpus score, although there might be issues with the logic of how that metric works as well as the weighting since sentences can have a different number of words, see this explaination.
Also I'm not sure if you are already aware of this but there is also a pretrained GPT-2 model available for Bengali on huggingface.
