Training error is oscillating, cross validation error is continuously decreasing I created a two layered fully connected neural network as a part of a recommendation engine (after I use embedding layers for products and users). I have been trying to tune the hyper-parameters for the past couple of weeks now. Additionally I am using 5 fold cross validation and 'mse' as the error metric. Below is the curve for training and cross-validation error that I obtain post training on a GPU for 10 epochs. 



I am not sure what really to make of this, fundamentally. The model doesn't overfit or underfit the data. What is even more perplexing is that the validation error is across five folds. Is this an inherent bias in the data and alludes to the fact that the data is structured such that a weak model does well on unseen data ?

Any pointers or comments will be greatly helpful. Thanks !
By definition, the training error should always decrease with a proper learning rate until it reaches his plateau, then starts to oscillate because it's crossing forward and backward the minimum of the loss function.

Instead, what you expect from the validation error is that it decrease with the training error, and at some point it starts to increase, sign of over fitting.

And it is not unusual that at the begin of the training you found the validation error lower than the training error.

From your plot I would say:


Train and validation errors are simply swapped: the red line is actually the training error, and the blue one is the validation error;
Stop training just after the crossing point. This because the validation set is just a set where you tune your parameters, and not the set of the "Revealed Truth", so it is also prone to over fitting.


If you make a third dataset ("test" dataset), and you try to check your metric (accuracy?) on it, I bet that the model trained after two epochs performs better than the model trained with ten epochs.