How to determine feature importance in a neural network? I have a neural network to solve a time series forecasting problem.  It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features.  The performance of the model is average and I would like to investigate whether adding or removing features will improve the performance.  I have constructed the neural network using keras.

The features I have included are:


The historical data
quarterly lagged series of the historical data (4 series)
A series of the change in value each week
Four time invariant features tiled to extend the length of the series. (another 4 series)


I am aware I could run the model many times changing the combination of features included each time. However, along with tuning the hyperparameters (for it might be that 8 features works really well with one set of hyperparameters but not with another set) this is really a lot of possible combinations.

Is there any separate way that I can use to guage if a feature is likely to add value to the model or not?

I am particuarly concerned that I have four time-invariant features being fed into the model which is designed to work with time varying data and I would like a way to measure their impact and if they add anything or not?
Don't remove a feature to find out its importance, but instead randomize or shuffle it.

Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way.

Here's the theory behind my suggestion: feature importance