What to optimize for when asked to find the most important features
Ideally you should select features in the same way you select the best model hyperparameters, with a validation set. 

You are interested in how the features work on unseen data, not your training data.

Depending on your feature selection algorithm, feature search will be intractable with larger data.  Naive feature selection is O(2^n) AND some model hyperparameters could be dependent on your choice of features.