What to do after selecting a model with cross-validation?
When 'overtraining' is not a problem (as in it will not diverge if you use more time), just use all your data and the empirically found optimal hyper parameters. In case of neural networks this is not the case (although in my experience, a lot of architectures converge to a specific test error, and take a long time to end up diverging again). I see a few options that you could try:


Most obvious one is keeping a (small) validation set around to use as indicator for early stopping (don't think of this as throwing away data, you still use it to train your network better)
Use same weight initalization as one of your folds and run for the same amount of epochs, same initialization should make convergence rate more similar than new random initialization
Keep all the cross validation models and use them in an ensemble instead of retraining the full model