Why not continue next epoch with best weights I've seen the following happen in our training: (numbers are paraphrased)


epoch 10 - val loss 0.0500 - loss 0.0450
epoch 11 - val loss 0.0400 - loss 0.0400
epoch 12 - val loss 0.0420 - loss 0.0410


Now clearly epoch 11 is better than epoch 12. I assumed epoch 13 would start with the weights of epoch 11, but have been told they do not.

In my opinion worst of all, our training pipeline is set up with ReduceLROnPlateau lowering the learning weight after 4 epochs of no improvement. Now say epoch 15 has a validation loss of 0.0415 and a loss of 0.0405, we could've better trained on with the weights of epoch 11.

Why are new epochs not started with the weights of the previous best epoch?
A new epoch starts with the same weights as the previous epoch but with different mini-batch of data. Model performance will have small oscillations across mini-batches for that reason.

Always starting each epoch with the best overall weights might result in staying in a local optimum. Gradient descent is already prone to staying in a local optimum, at the expense of finding a better optimum. Encouraging continued exploration of weights is one of the goals of training.

There is always a chance that both training and validation loss will do down. It is okay that training performance gets worse, there is no need for monotonic improvement in training. 

At the end of the training, you should select the weights that have the lowest validation loss.