How to determine feature importance in a neural network?
Don't remove a feature to find out its importance, but instead randomize or shuffle it.

Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way.

Here's the theory behind my suggestion: feature importance