Difference in Model Performance when using Validation set/ Testing set
I have implemented a PyTorch NN code for classification and regression.

Classification:
a) Use stratifiedKfolds for cross-validation (K=10- means 10 fold-cross validation)

I divided the data: as follows:
Suppose I have 100 data: 10 for testing, 18 for validation, 72 for training.

b) Loss function = CrossEntropy

c) Optimization = SGD

d) Early Stopping where waittime = 100 epochs.

Problem is:
Baseline Accuracy = 51%
Accuracy on Training set = 100%
Accuracy on validation set = 90%
Accuracy on testing set = 72%


I don’t understand what are the reasons behind the huge performance difference in Testing data/ Validation data?

How can I solve this problem?

Regression:

a) use the same network structure

b) loss function = MSELoss

c) Optimization = SGD

d) Early Stopping where wait-time = 100 epochs.

e) Use K-fold for cross-validation.

I divided the data: as follows:

Suppose I have 100 data: 10 for testing, 18 for validation, 72 for training.

Problem is:
Baseline MSE= 14.0
Accuracy on Training set = 0.0012
Accuracy on validation set = 6.45
Accuracy on testing set = 17.12


I don’t understand what are the reasons behind the huge performance difference in Testing data/ Validation data?

How can I solve these problems? or Is this an obvious thing for NN/ depend on particular dataset?

You have a large gap between training and validation performance, and between validation and test performance. There are two issues to explore:


Differences in the distribution. We assume that train / val / test sets are all drawn from the same distribution, and so have similar characteristics. A well trained model should perform equally well on the val and test datasets. If you are dataset really is only 10 samples for test and 18 for val, there is a high chance that the samples selected will skew one/both of these datasets, so that they no longer have similar characteristics. Therefore the difference between your val and test performance could just be chance: Your test set just happens to be much harder. You could test this by manual inspection.
Overfitting to val: However, I think it is more likely that you have experimented with different architectures, training regimes, etc, and have tweaked parameters to get the best performance on you validation set. This means that you have overfit your model to your val set. The test set is a truer reflection of your model's accuracy.


Your training accuracy is very high for both problems, and there is a large gap between training and validation performance. You are therefore overfitting to the training data, so need to train less, or introduce more stringent regularisation.
