Comparing results of neural net on two subsets of features
Since you have mentioned that using the different feature extraction methods, you are only getting slightly different feature sets, so the results are also similar. Also since your LSTM model is then also getting almost similar RMSE values, the models are able to generalize well and learn similarly and extract important information from all the datasets.
The best model depends on your future data, the computation time and load of different methods and how well they will last in production. Setting a seed is not really a good idea in neural nets. The basic idea is that your model should be able to reach the optimal weights no matter how they start. If your models are always getting similar results, in most cases, it is a good thing.
I am running a LSTM model on a multivariate time series data set with 24 features. I have ran feature extraction using a few different methods (variance testing, random forest extraction, and Extra Tree Classifier). Different methods have resulted in a slightly different subset of features. I now want to test my LSTM model on all subsets to see which gives the best results.
My problem is that the test/train RMSE scores for my 3 models are all very similar, and every time I run my model I get slightly different answers. This question is coming from a person who is naive and still learning the intricacies of neural nets, so please help me understand: in a case like this, how do you go about determining which model is best? Can you do seeding for neural nets? Or some type of averaging over a certain amount of trials?