Can accuracy become worse on the training set with more epochs? I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule that more epochs always decrease the accuracy on the training set. In other words, is the accuracy function over epochs a non-ascending function? If not, then why?
Yes, accuracy measured on the training set over epochs is not monotonically decreasing.

Possible reasons:


Software bug
Nan due to division by zero / log (0) / overflow
Too big learning rate