How to do gradient clipping in pytorch?
Well, I met with same err. I tried to use the clip norm but it doesn't work.
I don't want to change the network or add regularizers. So I change the optimizer to Adam, and it works.
Then I use the pretrained model from Adam to initate the training and use SGD + momentum for fine tuning. It is now working.
What is the correct way to perform gradient clipping in pytorch?
I have an exploding gradients problem.