Feature selection problems have been extensively studied for linear estimation, for instance, Lasso, but less emphasis has been placed on feature selection for non-linear functions. In this study, we propose a method for feature selection in high-dimensional non-linear function estimation problems. The new procedure is based on minimizing the @math norm of the vector of indicator variables that represent if a feature is selected or not. Our approach relies on the continuous relaxation of Bernoulli distributions, which allows our model to learn the parameters of the approximate Bernoulli distributions via gradient descent. This general framework simultaneously minimizes a loss function while selecting relevant features. Furthermore, we provide an information-theoretic justification of incorporating Bernoulli distribution into our approach and demonstrate the potential of the approach on synthetic and real-life applications.
@cite_24: We propose a practical method for @math norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of @math regularization. However, since the @math norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected @math regularized objective is differentiable with respect to the distribution parameters. We further propose the distribution for the gates, which is obtained by stretching'' a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.
@cite_12: We introduce instancewise feature selection as a methodology for model interpretation. Our method is based on learning a function to extract a subset of features that are most informative for each given example. This feature selector is trained to maximize the mutual information between selected features and the response variable, where the conditional distribution of the response variable given the input is the model to be explained. We develop an efficient variational approximation to the mutual information, and show the effectiveness of our method on a variety of synthetic and real data sets using both quantitative metrics and human evaluation.
The two most related works to this study are @cite_24 and @cite_12 . @cite_24 introduces the hard concrete distribution. They show that incorporating it into a deep neural network sparsifies the learned weights. The authors demonstrate how this sparsification leads to fast convergence and improved generalization. In this study, we focus on a different goal. By applying the stochastic gates only to the input layer, we demonstrate how the hard concrete distribution is useful for parametric feature selection. This allows us to increase the feature size up to a number of thousands of features (as shown in the Section ); this high dimensional regime is common in a field such as bioinformatics. In @cite_12 , the Gumbel-softmax trick is used to develop a framework for interpreting a pre-trained model @cite_12 . The main difference between our work and @cite_12 is that their method is focused on finding a subset of features given a particular instance, whereas our method aims to construct a subset of features based on all the training examples. This approach is more appropriate when we want to apply feature selection methods to data that are known to have consistent important features such as gene data.