How do you, analytically, show you are not using too many features? One of the managers at my company asked if there is a I could include a metric demonstrates that the my model is not using too many Features/Variables. Is there a metric or best practice that does this? Do you have a good way of communicating this to the business?
Specifically, she asked if I could include AIC about my models. After I asked some questions though, this is not quite what management is looking for because, unless I'm mistaken, AIC is only useful for comparing models with the same features.
In any case, I was able to use train-test validation for the model I was presenting, so AIC would have been overkill. To deal with this problem in practice I create a baseline model which uses all the variables, and then pair down as many variables as I can without sacrificing too much performance (on the train and test sets).
My approach would be quite similar to yours with a baseline, maybe just a bit more general:
Assuming it's practical (i.e. the training process is not too costly), you could train/test multiple models with various number of features, e.g. 10%, 20%...,100% of the features. For every subset you use an appropriate feature selection method, preferably something like genetic algorithm but it might be too costly. If it's not possible a simple individual feature selection method, e.g. using information gain, but it's likely not to be optimal. Once all the models have been trained, tested and evaluated, a plot of the performance as a function of the number of features should (hopefully) show a curve which increases less and less but never reaches the plateau. If it does reach the plateau before your chosen number of features, it means that you could actually cut down the number features.