title: How is cross validation implemented?
body: <p>I'm currently trying to train a neural network using cross validation, but I'm not sure if I'm getting how cross validation works. I understand the concept, but I can't totally see yet how the concept translates to code implementation. The following is a description of what I've got implemented, which is more-or-less guesswork.</p>
  
  <p>I split the entire data set into K-folds, where 1 fold is the validation set, 1 fold is the testing set, and the data in the remaining folds are dumped into the training set.</p>
  
  <p>Then, I loop K times, each time reassigning the validation and testing sets to other folds. Within each loop, I continuously train the network (update the weights) using only the training set until the error produced by the network meets some threshold. <strong>However, the error that is used to decide when to stop training is produced using the validation set, not the training set.</strong> After training is done, the error is once again produced, but this time using the testing set. This error from the testing set is recorded. <strong>Lastly, all the weights are re-initialized (using the same random number generator used to initialize them originally) or reset in some fashion to undo the learning that was done before moving on to the next set of validation, training, and testing sets.</strong></p>
  
  <p>Once all K loops finish, the errors recorded in each iteration of the K-loop are averaged.</p>
  
  <p>I have bolded the parts where I'm most confused about. Please let me know if I made any mistakes!</p>