Cross Validation how to determine when to Early Stop? When using "K-Fold Cross Validtion" for Neural Net, do we:


Pick and save initial weights of the network randomly (let's call it $W_0$)
Split data into $N$ equal chunks
Train model on $N-1$ chunks, validating against the left-out chunk (the $K$'th chunk)
Get validation error and revert the weights back to $W_0$
shift $K$ by 1 and repeat from 3.
Average-out the validtion errors, to get a much better understanding of how network will generalize using this data.
Revert back to $W_0$ one last time, and train the network using the ENTIRE dataset 




Question 1:

I realize 7 is possible, because we have a very good understanding of how network will generalize with the help of step 6. - Is this assumption correct?

Question 2:

Is reverting back to the initial $W_0$ a necessity, else we would overfit? (revert like we do in step 4. and 7.) 

Question 3, most important:

Assume we've made it to step 7, and will train the model using ENTIRE data. By now, we don't intend to validate it after we will finish. In that case how do we know when to stop training the model during step 7? 

Sure, we can train with same number of epochs as we did during Cross validation. But then how can we be sure that Cross Validation was trained with an appropriate number of epochs in the first place? 

Please notice - during steps 3, 4, 5 we only had $K$'th chunk to evaluate Training vs Validation loss. $K$'th chunk is very small, so during the actual Cross-Validation it was unclear when to Early-Stop... To make things worse, it will be even more difficult in case of Leave-One-Out (also know as All-But-One), where K is simply made from a single training example
First, I would like to emphasize, that cross-validation on itself does not give you any insights about overfitting. This would require comparing training and validation errors over the epochs. Typically you make such comparison with your eye and you can start with one train/validation split.

Question 1: By getting validation error N times, you develop a reasonable (whether it is very or not very good is a question) understanding of how your network will perform (= what error it will give) on the new unseen data. 

Often you do cross validation as a part of grid search of hyper-parameters. Then averaging errors at step 6 is mainly for choosing the best hyper-parameters: you believe that the hyper-parameters are best if the corresponding network produces the smallest average validation error. Simultaneously this error is your estimation on what error the final model will give you on the new data.

If you want, you can proceed with your exploration and compare validation errors (for one and the same hyper-parameter set) with each other, calculate standard deviation in order to get further insights. 

The concept "model generalizes well" is more related to the absence of overfitting. To make this conclusion, you need to compare train and validation errors. If they are close to each other, then the model generalizes well. This is not directly related to cross validation.

Question 2:  The only purpose to take the whole dataset is to train the model on more data. And more data is always good. On the other side, if you are happy with one of the models produced during cross-validation, you can take it.

Question 3: You can take the number of epochs as one of the parameters in grid-search of hyper-parameters. This search usually goes with cross-validation inside. When at step 7 you take the whole data set, you take more data. Thus, overfitting, if at all, at this stage can only be reduced. If it bothers you, that each chunk is small, replace K-fold cross validation with, for example, K times 50/50 train/test splits. And I would never worry about leave-one-out. It was developed for small (very small) datasets. For Neural Net to be good, you typically need large or very large dataset.