PyTorch Lightning - How to automatically reload last checkpoint when loss unexpectedly spikes?
I'm facing a problem where during training, my loss will unexpectedly spike, like so:

When this happens, I want to automatically reload the last checkpoint, reset the optimizer and resume training. How do I do this?
Edit: I tried training with fp64 precision and the unstable learning problem still occurred albeit later in training.

you could write a callback where you can check the spike and load the last checkpoint. Please, let me know if this helps!
