What can be the cause of a sudden explosion in the loss when training a CNN (Deeplab) I am training the following deeplab CNN:
https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss:


The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:


I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
Only after 50k steps the loss starts exploding, before that it is remarkably stable.
So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?
One possible cause is a high learning rate. High values of this hyperparameter usually cause updates that are too drastic, and therefore divergence from the optimum.

Please keep in mind this is only a suggestion, your problem might be due to completely different reasons. Try different learning rates and schedules, in order to understand if that's the case. You can check this very good post on how to set the learning rate.

Also, consider using early stopping to save some training time.