How many features do I select when doing feature selection for regression algorithms? Is R2 and RMSE good measures of success for overfitting?
You have overfitting when your model corresponds too closely to the training data and may therefore fail to fit additional data or predict future observations reliably. Basically, when the performance on training (or validation) set are much more better than on test set.
You have the opposite case: performance on test set much better than validation set.
This can happen if the two sets do not come from the same distribution (how did you divide train/validation/test set?). In this case, the data of the test set could be much easier to predict.
Another possibility is that the size of the test set is too small.
My suggestion is: shuffle your dataset; divide in 70% training and 30% test set. Do cross validation on training set. Compute the R2 on both sets.