Reporting test result for cross-validation with Neural Network
Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result?


Technically yes this is incorrect process for reporting an unbiased test metric. This could be a bad over-estimation of performance if cv results are noisy and vary randomly epoch-to-epoch. You should in theory treat the early stopping epoch number same as your other hyper-parameters, discover a good value from the lower-level cross-validation and stick with it. 

There is a problem with my suggestion though - the early stopping epoch number is sensitive to training data set size, and you just increased the size when you changed from lower level cross-validation to the higher level one. So you might get an unbiased measure at the expense of significantly worse results.

First, take a look at your learning curves. Just how sensitive are the cv results to epoch number? If they are not sensitive - no obvious over-fit over a reasonable range of epoch numbers - then just pick a mid-range fixed number of epochs that applies to all folds. That way you will have your unbiased estimate, and probably not compromised on model quality.

Alternatively, you may just have to be happy knowing that your test estimate is biased, but you have reduced the variance significantly by using 10-fold CV, and have only searched one hyper-parameter at the top level (not all three). It may only be slightly biased, and still a good estimate. The smoother and less jittery the learning curves are, the more you can get away with this - but sadly you won't get a measure of the bias, so you'll never be 100% sure.