Title: Evaluating Feature Importance Estimates,
Abstract: Estimating the influence of a given feature to a model prediction is
challenging. We introduce ROAR, RemOve And Retrain, a benchmark to evaluate the
accuracy of interpretability methods that estimate input feature importance in
deep neural networks. We remove a fraction of input features deemed to be most
important according to each estimator and measure the change to the model
accuracy upon retraining. The most accurate estimator will identify inputs as
important whose removal causes the most damage to model performance relative to
all other estimators. This evaluation produces thought-provoking results -- we
find that several estimators are less accurate than a random assignment of
feature importance. However, averaging a set of squared noisy estimators (a
variant of a technique proposed by Smilkov et al. (2017)), leads to significant
gains in accuracy for each method considered and far outperforms such a random
guess.