The dynamics of the validation perplexity and its difference from the training perplexity (Fig. ) suggests that training should be stopped after 10 to 12 epochs. To guide the eye, we also plotted the difference between the perplexities on the validation and training sets for the current epoch (Fig. , "Perplexity Validation -Train"), and the difference of the perplexity on the validation set at the current and previous epochs of training (Fig. , "Delta Perplexity Validation"). Evidently, the validation perplexity stabilizes after 10 to 12 epochs. Note, however, that the early stopping criterion is not applicable here, because the validation perplexity continues decreasing very slowly even around epoch 32, and one would not stop training with this criterion even at such a late stage.