title: Save only best weights with huggingface transformers
body: <p>Currently, I'm building a new transformer-based model with huggingface-transformers, where attention layer is different from the original one. I used <code>run_glue.py</code> to check performance of my model on GLUE benchmark. However, I found that Trainer class of huggingface-transformers saves all the checkpoints that I set, where I can set the maximum number of checkpoints to save. However, I want to save only the weight (or other stuff like optimizers) with <strong>best</strong> performance on validation dataset, and current Trainer class doesn't seem to provide such thing. (If we set the maximum number of checkpoints, then it removes older checkpoints, not ones with worse performances). <a href="https://github.com/huggingface/transformers/issues/2675" rel="noreferrer">Someone already asked about same question on Github</a>, but I can't figure out how to modify the script and do what I want. Currently, I'm thinking about making a custom Trainer class that inherits original one and change the <code>train()</code> method, and it would be great if there's an easy and simple way to do this. Thanks in advance.</p>