Why accuracy of finetune transformer model is less when evaluated after loading from disk, than during training?
Are you sure you are saving a checkpoint at each time you evaluate during training? At the end of training, when loading the best model, you will load the best saved checkpoint. If there is no checkpoint for the best model version, you will end up loading some other version, which might explain the drop in accuracy.
I am finetuning a transformer model and during the training cycle, evaluating it at each epoch. The best model is selected based on the highest evaluation accuracy among all epochs. Once the training cycle is completed and the best model is dumped to the disk, I try to regenerate that validation accuracy. I am unable to regenerate the exact validation accuracy reported by the training phase. I am getting a 3% to 4% drop in accuracy on the same evaluation data.
(For regeneration, I am calling the same evaluation function and passing it model and dataset. Nothing else changed for evaluation accuracy regeneration)