Neural Network unseen data performance I started dabbling in neural networks quite recently and encountered a situation which is quite strange (at least with my limited knowledge).

The problem I'm using a NN is a regression problem which tried to predict the sales of a product for a particular kind of promotion in an FMCG.

Although the data is not strictly time series, it still has some time related attributes like a prediction may depend on a similar kind of promotion last year (which I've modelled using feature engineering).

Now to the problem I'm having:

I took the data from 2015-2017,augumented it by adding small amount of noise, shuffled it, and ran it through a neural network(I think the architecture is not important, but let me know if it is, I'll try and post it).


Optimizer: Adam with a decay
Loss Function : Volume weighted mape
Validation set : Randomly selected 20% of the data


The network trained well and gave me errors as small as 8%. And both training and validation set errors decreased together and did not indicate an overfit.

But the kicker is that when I applied the algorithm to new unseen data(2018, first half), the errors rose to 55%. .

I tried researching the issue on Google but didn't find anything useful. What is happening here? Am I doing something wrong?
Did you split your „original“ data into Train/Dev/Test?
if not, try splitting your data into three parts. Train the model on your Train set, tune your hyperparameters on your Dev set and test your network on the Test set. The validation error doesn’t give an unbiased error, because the hyperparameters are tuned to it. That’s why you should always test your model on a given Test set. If the error is also low for the Test set I would guess you have another problem:
Without knowing too much about your approach and assuming you have no bugs, my guess is that the high error is because your Validation and Test set come from different distributions. I assume that maybe sales in 2015-2017 are quite different to those in 2018.
You should also take a look at Andrew Ng's explanation of this particular problem.