When training a deep learning network (say with TensorFlow or similar), it is common to train for a fixed set of samples and hope the results get better with longer training. But this assumes monotonically increasing accuracy, which as shown below, is clearly not true. As shown below, stopping at "2.0" would have had ten percentage points better accuracy. Is there any common procedure for picking off better models and saving them. 
 In other words a Peak-Detection routine. Perhaps, keeping track of the test-accuracy throughout the training session and saving off the model (checkpoint?) whenever the accuracy was higher than a previous value.  

Questions:

1. What is best practice for picking off the best model?
2. Does TF have an methods for this?
3. Is there merit to continuing to train even longer (maybe a lot longer), on the theory that the optimizer will eventually find a better solution.
4. Is a checkpoint the best save method?

Thanks.
[![Original graph](https://i.stack.imgur.com/z0XtU.png)](https://i.stack.imgur.com/z0XtU.png)

**EDIT:** Thanks to the recommendations from @Wontonimo, the improved accuracy results are shown below. The following changes were made:

* reduce the learning rate in the Adam optimizer from 0.003 to 0.001
* add two additional dropout layers (prob=0.5)
* shuffle-select training frames from the full data set (instead of sequence-selecting)
* increase training iterations by 50%.

With these changes it seems that it would be advantageous to continue training even further. And possibly add more regularization.

[![New training graph](https://i.stack.imgur.com/3uBl3.png)](https://i.stack.imgur.com/3uBl3.png)