title: Feature Selection from High-Dimensional Data with Very Low Sample Size: A Cautionary Tale
authors: Ludmila I. Kuncheva Clare E. Matthews Álvar Arnaiz-González Juan J. Rodríguez
text: 
  
  
  Nonparallel Voice Conversion with 
  Augmented Classifier 
  Star Generative Adversarial Networks
  
      Hirokazu Kameoka, 
  
  Takuhiro Kaneko,
  Kou Tanaka,
  and Nobukatsu Hojo
  H. Kameoka, T. Kaneko, K. Tanaka, 
  and N. Hojo are with NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Atsugi, Kanagawa, 243-0198 Japan (e-mail: hirokazu.kameoka.uh@hco.ntt.co.jp).
  
  
  This work was supported by 
  JSPS KAKENHI 17H01763 and JST CREST Grant Number JPMJCR19A3, Japan. 
  
      December 30, 2023
  =====================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
  
  	In classification problems, the purpose of feature selection is to identify a small, highly discriminative subset of the original feature set. 
  	In many applications, the dataset may have thousands of features and only a few dozens of samples (sometimes termed `wide'). This study is a cautionary tale demonstrating why feature selection in such cases may lead to undesirable results. In view to highlight the sample size issue, we derive the required sample size for declaring two features different. Using an example, we illustrate the heavy dependency between feature set and classifier, which poses a question to classifier-agnostic feature selection methods. However, the choice of a good selector-classifier pair is hampered by the low correlation between estimated and true error rate, as illustrated by another example. While previous studies raising similar issues validate their message with mostly synthetic data, here we carried out an experiment with 20 real datasets. We created an exaggerated scenario whereby we cut a very small portion of the data (10 instances per class) for feature selection and used the rest of the data for testing. The results reinforce the caution and suggest that it may be better to refrain from feature selection from very wide datasets rather than return misleading output to the user.
  
  
  
  
  
  
  § INTRODUCTION
  
  
  
  Feature selection is a long-standing theme in pattern recognition and machine learning, which inspires researchers to this day <cit.>. 
  It is a prolific field where new methods and techniques appear continually and are applied to emerging tasks such as multi-label learning <cit.> and big data <cit.>.
  Recent feature selection approaches draw predominantly from   information-theory <cit.>, ensembles <cit.>, evolutionary learning <cit.>, and deep-learning <cit.>. A bibliographic search on Web-of-Science[<https://wok.mimas.ac.uk/>] reveals that feature selection is a popular research area with nearly 25 000 publications containing “feature selection” in title or abstract in the past 10 years (as of 7th August 2019). Web-of-Science gives a conservative estimate; hence the real number of publications is likely much larger. Alongside the general-purpose feature selection algorithms <cit.>, domain-specific methods are being developed for fields such as bioinformatics <cit.>, text categorization <cit.>, and multimedia <cit.>. Recent surveys and reviews provide much needed guidance of supervised <cit.>, semi-supervised <cit.>, and unsupervised <cit.> feature selection.
  
  Present day data is typically characterised by a large number of features, prime examples of which are bioinformatics <cit.> and multimedia <cit.>. In all this wealth of literature, cautionary tales about the inadequacy of feature selection for very small-sized data are often overlooked. 
  
  Selecting a subset of features from data with a very small number of samples and a large number of features poses a serious challenge <cit.>. While not universally accepted, it is convenient to term such data sets `wide' <cit.>.
  To gauge the scale of growth of the problem, consider a study by Murray in 1977, where a set of 157 features was described as `embarrassingly large' <cit.>. 
  In today's terms, a wide dataset would have a few dozens of instances and possibly thousands of features. 
  
  The main problem with wide datasets is the possibility of dramatically overfitting the data. Added to this problem is another one, which we have no control over – the representativeness of the available sample. Overfitting a non-representative sample may return a useless feature set. In the spirit of Occam's razor, the feature selection method will have to be kept to the simplest possible choice. This eliminates a vast and very successful category of feature selection methods, whose main focus is the traversing of the possible feature subsets. This category includes evolutionary algorithms as well other nature-inspired feature selection methods such as swarm optimisation, particle optimisation, ant colony, bees, grey wolf, cuttlefish, bat, and many more <cit.>. Such algorithms can be very successful for high-dimensional data but only when the sample size is adequate. In many studies, both recent and past, sophisticated algorithms for selecting features out of a set of a few thousands are evaluated on relatively small data sets. While we are considering an extreme case, where each class contains only 10 examples, the concerns we raise here still apply, albeit to a lesser extent.
  
  The success of a feature selection attempt from a wide dataset depends on several factors: the underlying probability distributions (some problems could be easy to solve), the sample size (number of instances), the dimensionality (number of features), the chosen method for feature selection (how well it discovers good feature subsets, how robust it is to overfitting, how accurately the criterion of interest is evaluated), and the classifier subsequently recommended to the user. 
  The literature on feature selection from wide datasets identifies the following caveats:
  
  
  	
    * Low-quality subset. More often than not, feature selection algorithms produce feature subsets whose classification error is not close to that of the optimal feature set <cit.>, and sometimes is far in excess of it <cit.>. 
  	
  	Alternatively, due to the very small sample size, spurious feature subsets could be returned to the user, reporting, at the same time, a deceptively low error rate. 
  	
  	
    * Low correlation between estimated and true error. The true error does not correlate well with its estimate even for the best error estimators <cit.>. It has been argued that the quality of the estimate of the error criterion is  more important than the feature selection method itself <cit.>.  
  	
  	
  	
    * Inadequate feature selection protocol. Many studies including recent ones are oblivious to the `peeking' practice whereby feature selection is carried out first using the available data, and then the chosen subset is estimated through some cross-validation protocol on the same data. The inadequacy of this protocol has been flagged many times over the years <cit.>, warning the practitioners about the high optimistic bias of the classification error. The `proper' protocol would include the feature selection followed by the classification in the cross-validation loop, thereby evaluating the error of the whole process.  (The `proper/wrong' protocols <cit.> are called respectively `IN/OUT' by Rafaeilzadeh et al. <cit.> and `PART/ALL' by Aldehim and Wang <cit.>.) Probably due to its computational burden, the `proper' protocol has not been widely adopted yet.   
  
  
  The alarming conclusions from these observations are that: 
  
  	
    * From 1) and 2): The subset of features returned to the user may not be adequate.
  	
    * From 3): The classification accuracy/error predicted for the selected subset may be deceptively low.
  
  
  This study is intended as a cautionary tale. Our contributions are summarised as follows:
  
  
  	
    * We show that the necessary sample size for declaring two features significantly different is very large, which is a warning against selecting features from very small-size data.
  	
    * We exemplify issues 1) and 2) and examine the source of the discrepancy between between training and testing error.
  	
    * Using a synthetic example, we demonstrate a flaw of the classifier-agnostic feature selection methods.
  	
    * To demonstrate the grounds for our concern, we carry out an experimental study with 20 real, benchmark datasets. This is in contrast with  previous studies on feature selection from wide data which are mostly carried out on synthetic, well-behaved data <cit.>.
  
  
  The rest of the paper is organised as follows. Most of the related work is summarised in this introduction, and where suitable in the following sections. Section <ref> explains the problem and the various types of error in feature selection. The effect of sample size is discussed in Section <ref>. The experiment is presented in Section <ref>, and the conclusion, in Section <ref>.
  
  
  
  
  
  
  § SAMPLE SIZE FOR DECLARING TWO FEATURES DIFFERENT
  
  
  
  How big a sample do we need for feature selection? An interesting study on the required number of samples needed for returning a reliable set of features was published by Ein-Dor et al. <cit.>. These studies came from the area of clinical cancer research where the task is to analyse gene expression data and predict the outcome for a given patient, that is, the potential for relapse and for metastasis. The authors were  inspired by the observation that several research groups had published vastly different sets of predictive genes obtained for the same clinical types of 	patients. The reported lists of genes  differed widely and had only very few genes in common. The authors build a statistical model of the degree of intersection of two sets of features selected from different samples. The implied feature selection method is a filter which ranks the features according to the absolute value of their correlation with the (binary) class label variable. 
  
  The authors assume that the correlation ρ between feature x_i and the (binary) class label variable y is a measure of quality of the feature. They proceed to transform this correlation into a new variable z_i = tanh^-1 (ρ(x_i,y)), which has a normal distribution. In gene expression data, it may be reasonable to assume that only positive correlation is informative. We note, however, that negative and positive correlations are equally valuable in general feature selection. (Should we, however, transform |ρ| instead of ρ, we lose the normal distribution property of z_i, and the assumptions for the derivations do not hold.) The staggering conclusion of Ein-Dor et al.'s  work <cit.> is that thousands of instances are needed even for moderate agreement between the lists (disputed recently on the ground of Gaussianity and sparsity of the data <cit.>). 
  
  Here we present a similarly discouraging result for just two features.
  
  
  
   §.§ Counting estimator
  
  An error estimator is a natural alternative to correlation as a measure of a feature quality, although it may suffer from the inconvenience of the discretisation for small values of the sample size N. Let p_i be the probability that a classifier built on feature x_i will label correctly a randomly drawn object from the distribution of the problem at hand. Assuming only two outputs (correct/incorrect), we can form a Bernoulli variable w_i, whose probability of success is p_i. Consider features x_1 and x_2. The task is to pick the better feature.  Without loss of generality, let p_1 > p_2. In this case, we should prefer x_1 to x_2. 
  
  Let Z be a random sample of N objects described by features x_1 and x_2. Table <ref> gives the probabilities for the four possible combinations of values of the two Bernoulli variables.
  
  
  
  
  
  
  
   
  
  To decide on the sample size N needed to ascertain a difference between w_1 and w_2, we can apply a McNemar test. The statistic for this test is calculated as 
  
      χ^2 = (B-C)^2/B+C.
  
  Under the null hypothesis of marginal homogeneity (p_1=p_2, also expressed as b+d=c+d), χ^2 follows a chi-squared distribution with one degree of freedom. Approximating the counts in Table <ref> as N×probability, we obtain
  
      χ^2 = (Nb-Nc)^2/Nb+Nc
      =N(p_1-p_2)^2/p_1+p_2-2d
  
  where d is the probability of both features being correct for a randomly drawn object. Thus d≤min(p_1,p_2)=p_2 and also d≥ p_1+p_2-1. This probability can be thought of as expressing the correlation between the two features. When x_1 and x_2 are independent, d=p_1p_2.
  
  Given p_1, p_2 and d, and choosing the level of significance α, we can approximate the necessary sample size N by the inverse of the chi-squared cumulative distribution function F_χ(.).
  
  We can subsequently rearrange equation (<ref>) to obtain
  
      N = F_χ^-1(1-α)  p_1+p_2-2d/(p_1-p_2)^2 .
  
  
  Clearly, the size will depend on p_1 and p_2. A large difference p_1-p_2 will need fewer objects. The value will also depend on the values of p_1 and p_2 as well as on the agreement pattern measured by d.  
  
  As an example, suppose that we want to estimate the sample size N so that with probability 0.95 we can discover the difference and pick the correct feature x_1. Let p_1=0.85 and p_2=0.80. Assuming independence, we have d=p_1p_2=0.68. Then
  
      N = F_χ^-1(1-0.05)0.85+0.80-2× 0.68/(0.85-0.80)^2
      =445.6 .
  
  
  
  
  Figure <ref> shows the required size N for picking feature x_1 from the pair (x_1,x_2). The sizes are plotted as functions of p_1 for two levels of significance, α=0.05 and α = 0.01. For this example we chose p_2 = p_1-0.05 and assumed independence between w_1 and w_2, hence d=p_1p_2.
  
  
  
  
  Equation (<ref>) shows an interesting effect of the relationship between the Bernoulli variables w_1 and w_2 on the required sample size. For fixed α, p_1, and p_2, when the agreement between the feature evaluators (d) increases, N decreases. In other words, for larger agreement, fewer observations are needed to pick feature x_1 with the same certainty. Indeed, if we take the maximum possible agreement d=p_2=0.80 in the above example, this leaves c=0 and b=0.05. In this case, Eqn.(<ref>) returns N = 77. While some dependency between w_1 and w_2 is expected, the sample size required to pick confidently between just two features is very large. Exactly the same argument is valid when we compare two sets of features. However, generalising this result to sets of more than two elements (thousands of features!) will amount to quite large values of N. Even mitigated by high dependency between the variables w_i, these sample counts expose the inadequacy of feature selection from very small samples, which reinforces the doubts and warnings published elsewhere <cit.>.
  
  
  
   §.§ Smoothed estimator
  
  
  Assume now that instead of the binary variable w_i for feature x_i, we have a continuous-valued variable v_i estimating the probability of correct classification for the value of x_i of the given object, where the classifier is built on feature X_i∈ X alone. Denote this probability by P(C_i|x_i). Unfortunately, we cannot assume that such probabilities would adhere to a normal distribution. Assume again that x_1 is better than x_2 (p_1=E[v_1]>p_2=E[v_2]). In this case, a Wilcoxon signed-rank test can be used. This non-parametric test will have less statistical power than a parametric test such as the t-test. Thus, the Wilcoxon test may not be able to detect an existing difference for normally distributed variables, which would be detectable by the parametric test. This implies that larger samples would be needed in the same set-up to detect the difference through the non-parametric test. Then, we can use the normal distribution assumption to illustrate the magnitude of N, noting that this is a lower estimate.
  
  
  
  
  
  
  
  
  
  Assuming v_1 and v_2 both have a normal distribution,
  
  
  a paired t-test can be applied to ascertain that p_1>p_2. For large N (which we expect here), the t-distribution approaches the normal distribution. Denote by v the difference v=v_1-v_2.  Assuming that v_1 and v_2 are jointly normally distributed, variable v has normal distribution too with mean μ_v=p_1-p_2 and variance σ_v=σ_1^2+σ_2^2-2σ_12, where σ_i^2 is the variance of v_i, i=1,2, and σ_12 is the covariance between v_1 and v_2. If we measure v̅ as an average from sample Z of size N, we can use the cumulative distribution function of the standard normal distribution Φ to estimate critical values 
  
  
  
  
  
  for a chosen level of significance α, and estimate the required values of N_c for the continuous-valued case[For the normal distribution we can use only one tail, which implies that we can replace α in Eqn.(<ref>) with α/2, leading to an even higher requirement for N. ]
  
      N_c = Φ^-1(1-α)  σ_1^2+
      	σ_2^2-2σ_12/(p_1-p_2)^2 .
  
  
  This equation is largely similar to Eqn.(<ref>), and leads to the same conclusions about the considerable required sample sizes and the mitigating effect of positive covariance between the variables estimating the features' merit.
  
  
  
  
  
  
  
  
  
  
  
  
  If ascertaining difference between two features requires sample sizes of this magnitude, selecting more features would require many thousands of instances. This resonates with the previous studies, and gives an early argument for our cautionary tale.
  
  
  
  
  § LOW CORRELATION BETWEEN ESTIMATED AND TRUE ERROR
  
  
  
  
  To define the problem formally, consider a labelled dataset Z={(_1,y_1), …, (_N,y_N)} where _j∈R^n are objects represented as points in some n-dimensional space, and y_j∈Ω={ω_1,…,ω_c} are class labels, j=1,…,N. The dataset is drawn randomly from a joint distribution p(,y). The task of feature selection is to reduce the feature space R^n to a space of a much lower dimensionality d ≪ n by dropping dimensions of the original space.   
  
  
  
   §.§ Types of error in feature selection from a finite sample
  
  Let X={X_1,X_2,...,X_n} be the feature set. The first issue we consider is what we measure and return as the result of the feature selection. Assume that a classifier model C has been chosen for this problem. Denote by y the true label of instance . Let C(|Z) be the output of the classifier trained on dataset Z for instance . To compact the notation, let Z_a^b be a dataset Z of cardinality a and containing only feature set b⊆ X. Denote by E[ζ] an operator which calculates the expectation of ζ over the respective feature space and probability distribution.
  
  Assume also that we have access to the whole population of interest, denoted Z_∞^X. For a selected S⊆ X, consider the probability that C is correct, denoted Pr(C(|Z_∞^S)≠ y). Classifier C is fixed, which means that it will output only one class label for . The uncertainty, measured by this probability, comes from the fact that x may originate from different classes with probability P(y = ω_i|), i=1,…,c. Then the error that we would like to return to the user together with feature subset set S, given the chosen classifier C, is
  
  	
      e(S,C) = E[Pr(C(|Z_∞^S)≠ y
      	)]
      	= ∫_'∈R^|S| Pr(C(|Z_∞^S)≠ y)   p(')   d'   ,
  
  
  y is the true label of , ' is the restriction of  using only the features in set S, and p(') is its probability density function marginalised across X∖ S. Ideally, we would like to return e to the user together with S but in reality we have only some estimate ê (Table <ref>). 
  
  Feature selection algorithms look for the optimal feature subset S^*⊆ X for the problem at hand and for the chosen C
  
      S^*= min_S {e(S,C)} ,
  
  and hence e^*= e(S^*,C) is the true error of the best subset, given classifier C (Table<ref>). We can remark here that finding this error is predicated upon having access to the whole population of interest, as well as being able to identify S^*. The latter assumes that we have a perfect feature selection algorithm at hand. 
  
  Suppose that we can only afford to sample Z of cardinality N, i.e., Z_N^X. Theoretically, a classifier trained on a finite sample will be less accurate than the one trained on Z_∞^X. Furthermore, the optimal subset of features S^* may not be identifiable from the sampled Z_N^X. Instead, we may have another optimal set S_N^*⊆ X. It is possible that S^*≠ S_N^* because the classifier trained on the smaller sample may not be identical to the ideal classifier, and will be either equivalent or more likely inferior to it. In the absence of an infinite dataset, the purpose of feature selection is to return to the user S_N^* and the respective error e_N^* (Table<ref>).
  
  Furthermore, our feature selection algorithm may not be able to identify the best subset for the sampled dataset. The reason is that the estimates of the errors from such a small dataset typically have a very large variance. Therefore, we cannot measure the true error of each feature subset, and will likely return  a non-optimal subset S and its error ê. The inferior subset may come as a result of either a suboptimal feature selection procedure (e.g., a sequential forward selection) of a poor estimate of the accuracy of the subset, so that even exhaustive search will return a suboptimal S. Most likely, the inferior result is due to the combination of both.
  
  
  
  By the argument of optimality, we have:
  
      e^*≤ e_N^* ≤ e.
  
  
  However, ê could be anywhere with respect to the above errors. In a perfect scenario, S^*=S and e_N^*= e = ê. Our main argument in this study is that, for wide datasets, ê is a dangerously poor estimate of e, leading to spurious choices of S with misleading estimates ê.
  
  
  
  
   §.§ Error estimators (finite testing sample)
  
  
  To select S and calculate the estimate of e we re-test the classifier on Z using some acceptable training-testing protocol. Thus, instead of the true e, the returned quantity will be an estimate, sometimes quite far off.
  
  The resubstitution estimate ê^R evaluates the error on the training data and is known to be optimistically biased <cit.>. The leave-one-out error ê^LOO applied on Z after selecting Ŝ will likely be also optimistically biased, more so for smaller datasets <cit.>. The `right' protocol requires a subset of features is selected and evaluated from each cross-validation fold. The error is estimated as the average of the errors in the individual folds. A 10-fold LOO will result in 10 different subsets S. Therefore, after ê is estimated, a single S is selected using the whole of Z. This protocol gives an estimate ê^r-LOO which was found to be only slightly pessimistically biased <cit.>. 
  On average across different samples Z, we expect
  
      ê^R≤ê^LOO≤ê^r-LOO.
  
  
  It has been argued that the quality of the estimate of the error criterion is  more important in feature selection than the feature selection method itself <cit.>.  Various estimators have been proposed over the years to complement the conventional resubstitution and cross-validation estimators. Among the alternative estimators are the bootstrap estimator <cit.>, the bolstered estimator <cit.>, and smoothed estimators <cit.>, the latter being specifically useful for small datasets <cit.>. Comparisons between the estimators <cit.> reveal that the time-consuming ones such as bootstrap 632+ and the bolstered estimator have an edge over the competitors but the general relationship between the true error and the estimated error is poor <cit.>. While most of these findings are based on simulated data, there is no reason to expect much different results with real data. 
  
  In this study we chose the smoothed error estimator because it is computationally more efficient than bootstrap and bolster estimators, and dramatically reduces the number of ties in comparison with the counting estimator. The smoothed estimator relies on the ability of the classifier model to output estimates of posterior probabilities for the classes. In other words, instead of recording 0 (correct classification) and 1 (error), for each  we store the estimate of the probability of predicting an incorrect class. These probabilities are subsequently averaged over the testing instances to arrive at a single estimate of the error. Arguably, the quality of the estimates of the probabilities will suffer from the low sample size as well. Nonetheless, the smoothed estimator gives us better flexibility in choosing the feature subset. 
  
  
  
  
  
   §.§ An example
  
  Here we illustrate the difficulty of feature selection from wide datasets using an example from the work by Reunanen, 2003 <cit.>. A small sample was taken from the `sonar' dataset from the UCI repository <cit.> as Z with 10 instances from each of the two classes. The rest of the dataset (188 instances) was used as the unseen testing data, from which we estimate a version of the `true' error. The data has two balanced classes (47%/54%) and 60 numerical features.
  
  First we rank the features using the Symmetric Uncertainty ranker <cit.>. Next we evaluate all 1 023 combinations of the top 10 features (2^10, excluding the empty set) with respect to the linear discriminant classifier (LDC). Four estimates were calculated: (i) ê^R Resubstitution error (LDC trained and tested on Z); (ii) ê^LOO Leave-One-Out (LOO) error on Z; (iii) ê^s-LOO Smoothed LOO error on Z; and (iv) ê' `True' error (LDC trained on Z and evaluated on the testing set). 
  
  Figure <ref> shows the first set of results. In each plot, an estimator (i)-(iii) is plotted against the true error (iv). Each feature subset is a point on the plot, hence there are 1 023 points in each plot. If the estimator from Z was ideal, all points would lie on the diagonal shown in each plot. 
  
  
  
  The array next to each cloud of points contains the indices of the features in the best set according to the respective error estimator. The features are numbered from 1 to 10 according to their rank in the list. If there was a tie at the minimum error (nearly inevitable for ê^R and ê^LOO), the sets of minimum cardinality were identified among the tied sets, and a random set was chosen among these. The best set according to ê' is shown in each plot for comparison.
  
  This illustration eliminates the effect of a possible sub-optimal search because we have carried out an exhaustive search on the top 10 features. The dramatic discrepancy between the feature sets found by the estimators and the true best set are solely owed to the inadequate error estimates. As seen in the plot (reinforcing finding in past literature <cit.>), there is no visible relationship between the true error and its estimates. Thus, not only is the returned feature subset a poor match of the best set, but the error estimate predicted by the estimator is highly optimistically biased for this Z. Table <ref> shows the error rates for the three estimators.
  
  
  
  We should note that the discrepancy between the returned feature sets identified by different estimators may not be entirely the estimators' fault. It is possible that low error rate is achievable by more than one feature subset. The value of the estimate, however, should be a good approximation of the true error, which was not observed in the experiment. 
  
  This example suggests a slight advantage of the smoothed LOO estimator over the standard LOO. First, the chances of a tie are greatly reduced, and, second, only this estimator led to a smaller true error than that using all 10 top features. Still, the main message here is the evident lack of relationship between predicted and true error for very low-sample-size datasets.
  
  
  
   §.§ Search method (suboptimal search)
  
  Suppose that we have an oracle error estimator which correctly predicts the generalisation error rate of a classifier trained on the given dataset Z. It is well known that the exhaustive search through all subsets of features is the only method guaranteeing that the set with the minimum error will be found <cit.>. Due to its computational intensity (2^n-1 repeats of training a classifier and evaluating its error) this approach is feasible only for small cardinality of the feature space, for example, up to 15 features. A simpler alternative approach is a “heavy” random search whereby random feature subsets are sampled and evaluated. More intricate procedures such as floating forward selection (SFFS) <cit.> and evolutionary algorithms <cit.> are also possible if the initial feature set is reduced to a smaller set as a preliminary step. 
  
  
  In summary, imperfect error estimation in combination with a non-optimal search procedure may result in selecting a spurious feature set with an inadequate error estimate. This problem is exacerbated by the small sample size. In this paper we argue that, for wide datasets, we should stop at ranking the features in X and returning the subset of the top d features. Search procedures to reduce this set further (a wrapper approach) most often lead to over-using the data and reaching arbitrary conclusions. 
  
  
  
   §.§ Classifier dependence
  
  Note that in the preceding discussion, the classifier model C is chosen and fixed. This brings us to the classifier issue: feature selection is intrinsically classifier-dependent. A classifier-independent approach <cit.> may work when the classes are clearly separable in some feature subspace so that any classifier model will find the separation “easy”. However, this approach may overlook highly useful feature subspaces which favour a specific classifier model. 
  
  An example is shown in Figure <ref>.  The two features, assessed individually, are nearly useless. However, the features in plot (a) give together zero LOO error for the linear discriminant classifier (LDC), and LOO error 1 (all objects are classified incorrectly) for the nearest neighbour classifier (1NN). The opposite case is shown in plot (b).[The MATLAB code for generating the example datasets and calculating the LOO errors is available at <https://github.com/LucyKuncheva/Feature-Selection>. ] The choice of a classifier model will ultimately determine the importance of the pair of features.
  
  In this study we will use seven classifiers that broadly cover the spectrum of most used classifiers in wide datasets.
  Even though different classifiers have different sensitivity to changing the feature space, we will assume that, in general, our conclusions will generalise across different classifier models.
  
  
  
  
  
  
  
  § EXPERIMENTS
  
  
  In this experiment, we artificially simulate very low-sample data sets. To this end, we sample only 10 objects per class as our data set Z_10^X. This is an extreme scenario which allows us to demonstrate the issues of concern listed in the introduction. A perfect collection of datasets would contain only large data sets, that is, data of high dimensionality and a large sample size. Since our sample size is very small, we would have large remaining data to use in place of Z_∞^S. Unfortunately, we could not identify a sufficiently large collection of large datasets, which led us to using some wide data sets too. Nonetheless, since our N=10c is extremely small, a reasonably-sized data for testing is left even in the relatively wide datasets in our experiment.
  
  
  
  
   §.§ Datasets
  
  Previous studies were mostly confined to simulated data with known (and well-behaved) distributions. Here we use 19 real and 1 artificial[Madelon is an artificial dataset, nonetheless it is a complex problem because is multivariate and highly non-linear.] datasets sourced from <http://featureselection.asu.edu/datasets.php> <cit.>. All datasets were chosen so that we have a sufficient number of samples to allow for calculating a reasonable estimate of the true error e after cutting out a sample Z of size N. Where the dataset contained more than two classes, we selected the two most frequent classes. Table <ref> shows the properties of the datasets. 
  
  
  
  
  
  
   §.§ Methods
  
  For each dataset we repeated the following procedure 10 times (“sampling runs”): 
  
  	
    * Choose a classifier C and a ranker method R.
  	
    * Take a random sample Z from the dataset containing 10 instances from each class.
  	
    * Rank the features in Z by the ranker method R, and store the top 20 features in set S.
  	
    * Apply each of the Selection Schemes from the list below. 
  
  
  Using the chosen classifier C and the smoothed leave-one-out estimator as the evaluation criterion (e^s-LOO), we applied the Selection Schemes listed below. Shown in parentheses is the number of required evaluations of the LOO error. Note that each such evaluation requires N training and testing iterations.
  
  List of selection schemes:
  
  
  	
    * ALL. All features in the original dataset. (1) 
  	
    * Top3. Top three features from the ranked list. (1)
  	
    * Top10. Top 10 features. (1)
  	
    * Top20. Top 20 features. (1)
  	
    * Best3. Best three features selected through enumerating all combinations of three features out of the top 20. ((20 3) = 1140)
  	
    * EX10. Best subset found through exhaustive search among all subsets of the top 10 features. (2^10 = 1024) Note: For the empty set we used the prior probabilities estimated from the sample.
  	
    * RND20. Best subset identified through random search from the 20 features. 1024 evaluations  were carried out to tally with EX10. At each evaluation, a feature subset was created switching each feature from the top 20 `on' or `of' with probability 0.5. (1024) 
  
  
  The classifier models C which we used here were:
  
  
  	
    * Nearest neighbour (1NN).
  	
    * Decision tree (DT).
  	
    * Linear discriminant classifier (LDC).
  	
    * Naïve Bayes (NB).
  	
    * Random Forest (RF).
  	
    * Support Vector Machine with a Gaussian kernel (SVMG).
  	
    * Support Vector Machine with a linear kernel (SVML).
  
  
  We ran the experiments in the WEKA environment <cit.>.
  
  To calculate the smoothed estimator we used the posterior probabilities as provided by WEKA. The exception is 1NN where we applied the softmax transformation, it gives smoother probabilities.
  In WEKA, for DT the probabilities are obtained using the class distribution of the training examples in the corresponding leaf. For LDC, the probabilities are obtained using  Bayes' rule on the estimated Gaussians. 
  For NB, the probabilities for the attributes are obtained from the normal distribution. For RF, the probabilities are the averages of the trees probabilities. For SVMG and SVML with two classes, the probabilities are discrete, in {0, 1}.
  
  
  
  
  
  
  
  
  
  
  The WEKA implementation of five ranking methods were used for determining the top 20 features: 
  
  
  	
    * Symmetrical Uncertainty <cit.>.
  	
    * Random Forest <cit.>.
  	
    * ReliefF <cit.>.
  	
    * SVM <cit.>.
  	
    * SVMRFE <cit.>.
  
  
  Thus, for each dataset we ran [7 (classifiers) × 5 (rankers) × 6 (selectors) × 10 (sampling runs)] + [7 (classifiers) × 1 (selector = ALL) × 10 (sampling runs)] = 2 170  experiments. A full set of results is presented in the supplementary material.
  
  
  
   §.§ Results
  
  
  The main result from our experiment is presented in Table <ref>. For a given combination of classifier and ranker, <C,R>, we collated the errors  e^s-LOO for the 10 sampling runs for each dataset produced by the 7 selectors. Thus, we constructed a matrix of size 20 (datasets) × 10 (sampling runs) = 200 rows, and 7 (selectors) columns. We are interested in comparing the selectors across datasets and sampling runs. Since the errors may not be commensurable across different datasets, we calculated the ranks of the selectors for each row of the table. The best selector method received rank 1, and the worst, rank 7. The averaged ranks across datasets and runs are shown in the table. We also colour-coded the cells of the table. Bright red corresponds to the best selectors and dark blue, to the worst. The best result in each row is highlighted in boldface.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  For each <C,R> combination we ran the Friedman test (a non-parametric version of balanced two-way ANOVA) to determine whether the differences between the selectors were significant. As we have 10 runs for the same dataset, the block version of the test was applied[We used the MATLAB Statistics Toolbox implementation of the Friedman test.].  
  
  
  
  The Friedman test was applied six times for each row of the table. We sorted the selectors from best to worst based on their average rank. Then we checked for difference between selectors at places 1 and 2, then between selectors at places 1, 2, and 3, then between 1, 2, 3, and 4, and so on. 
  Take for example, the second row of the table where the classifier is 1NN and the Ranker is ReliefF. The test did not find a difference between the best selector (Top20) and the second best (RND20) significant at level 0.05. This is why they are both shown in boxes in Table <ref>. When we add the third-ranked selector however (Top10), the p-value is below the significance level. Obviously, once discovered, the difference is propagated down the list.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  The results in Table <ref> give support to our cautionary tale. We observe that the selector's merit depends strongly on the classifier. The importance of the pair (feature-selector, classifier) has been often overlooked. Many studies propose new criteria for filter selection methods but it is not clear how those criteria will translate into classification accuracy using all returned features together. Striving for a consensus of classifiers, we may miss an extremely valuable subset of features which gives perfect results with just one classifier model. In reality, pairing a classifier with the feature set leads to dramatic differences. The two alternatives are: return S which has moderate performance across many classifiers versus return S which has excellent performance with one specific classifier.  Our example in Figure <ref> demonstrates this effect on a contrived example but Table <ref> gives the real-data perspective. With LDC and SVML, using ALL features is the winning strategy. In other words, any feature selection makes the result worse. For 1NN, NB and SVMG, the Top20 selector is the winner followed by RND20.  Interestingly, the best selector for 1NN, NB and SVMG is the worst selector for LDC, followed closely by RND20, once again highlighting the importance of classifier choice. 
  
  
  
  
  There is little consistency among the ranking of the selectors across different classifiers. On average, ALL seems to dominate over the other selectors. On the other hand, with small exceptions, the behaviour of the selectors seems consistent over the ranking methods. This can be seen by the similar colours of the columns within the block of each classifier. For example, Top3 is worse than Top20 for 1NN, NB, RF, SVMG and SVML for any ranking method. This reinforces our message that developing new feature ranking algorithms may not be the best way forward to solving the wide data problem. Choosing Classifier, Ranker and Selector together holds a lot more promise.
  
  We chose glyph plots to visualise the rest of the results. The spokes on each plot correspond to the 20 datasets ordered as in Table <ref>. Figure <ref> shows the ranks of the seven classifiers for the 20 datasets. The ranks are averaged across runs, rankers and selectors. The better classifier in this experiment was NB, and the worst was LDC.
  
  
  
  Figure <ref> presents a similar view but this time the average is across runs, classifiers and selectors. Thus, the glyph plot shows the relationship between the different ranker methods. The legend is arranged from the worst method (the one with the largest area) to the best. Although we nominated ALL to be a selector, we can treat it as a ranker as well. Thus we included it in the comparison. According to the plot, the best strategy is ALL, followed by ReliefF and SVM-RFE. 
  
  Figure <ref> shows the glyph plot for the selectors. Again, ALL wins, followed by Top20 and Top10.
  
  
  Finally, for each data set, we arranged the 217 possible combinations of Classifier, Ranker and Selector as a matrix with 217 rows and 10 columns. Each column was the result of a run. We applied the Friedman test again to determine whether there is a statistically significant difference between the combinations. The top 35 combinations are shown in Table <ref>. Even with this small number of runs, the test clearly separated two of the combinations: both with ALL features, and both representing a linear classification rule (SVML and LDC)!
  
  
  The results from this experiment lead to the following overall recommendation. For very wide data sets, even for moderately-wide ones, it may be advisable to refrain from feature selection altogether. If we are prepared to accept a possible sacrifice of classification accuracy and go ahead with the feature selection, it is important to choose carefully the classifier model in conjunction with the selector and the ranking method. The combination of classifier and selector seems to be the more important factor than the ranking method. This is a curious result because a large proportion of the literature is devoted to perfecting the ranking (and selection) algorithms.
  
  
  
  
  
  
  
  § CONCLUSION
  
  
  
  This paper is a warning against feature selection from datasets with very low sample size and a large number of original features, termed `wide' datasets. We describe the problem of estimating classification error as a criterion for feature selection and the accompanying caveats. A theoretical argument on the task of choosing one of two features amplifies the concern about the necessary sample size for a correct decision. It turns out that even for this simple problem, we may need hundreds of instances for a standard level of certainty in the right choice. An experiment with 20 real datasets sets apart our study from previous studies of similar nature, where the claims are usually illustrated by synthetic data with known (and well-behaved) distributions. 
  
  
  
  The misconception that practitioners often rely on is that cross-validation will prevent overfitting in all circumstances. Valid as this may be for standard datasets, for wide datasets, cross-validation estimates offer only a marginal remedy. We have used the leave-one-out (LOO) estimate in all our experiments and examples, but the true error (estimated from the left-aside large portion of the data) did not correlate well with the estimates. This is evidenced by the fact that the smallest LOO error (guaranteed by the exhaustive search EX10) did not lead to the best testing error. This point was also illustrated by the example in Section <ref> (Figure <ref>).
  
  Our experimental results revealed that considering the classifier that should be subsequently used with the selected features is paramount. The choice of classifier may determine whether we should attempt feature selection at all. In our experiment, the two linear classifiers performed best with ALL features, discouraging any of the feature selection alternatives. 
  
  In all cases, the computationally intense selectors were found significantly inferior to the best selectors. This comes as no surprise, and is due to `overusing' the data, which is the major deficiency of wide datasets. 
  
  But all is not lost! ALL (no selection) is not the best method for all datasets as seen in Figures <ref> and <ref>. In trying to answer the general question, we have bypassed another crucial one: What is best for my wide dataset? This is a difficult question in its own right, and deserves a separate study. 
  
  Admittedly, our experiment exaggerated the problem by creating an unrealistic scenario with only 10 samples per class in the sample. The problems will naturally be mitigated with increase of the sample size. An interesting future research line would be to evaluate the severity of the issues as a function of the sample size N. This brings the question of whether stability of the selected subsets may be an indicator of whether feature selection should be attempted and which selectors and classifiers should be preferred.
  
  
  
  
  § ACKNOWLEDGEMENTS
  
  This work was done under project RPG-2015-188 funded by The Leverhulme Trust, UK, project TIN2015-67534-P (MINECO/FEDER, UE) of the Ministerio de Economía y Competitividad of the Spanish Government, and project BU085P17 (JCyL/FEDER, UE) of the Junta de Castilla y León (both projects cofinanced through European Union FEDER funds).
  The third author was supported by the Mobility Grant CAS19/00100 from the Ministerio de Ciencia, Innovación y Universidades of the Spanish Government.
  
  
  
  
  
  
  10
  	
  	Abe06
  	Naoto Abe and Mineichi Kudo.
  	Non-parametric classifier-independent feature selection.
  	Pattern Recognition, 39:737–746, 2006.
  	
  	Aldehim17
  	Ghadah Aldehim and Wenjia Wang.
  	Determining appropriate approaches for using data in feature
  	selection.
  	International Journal of Machine Learning and Cybernetics,
  	8:915–918, 2017.
  	
  	Bennasar15
  	Mohamed Bennasar, Yulia Hicks, and Rossitza Setchi.
  	Feature selection using joint mutual information maximisation.
  	Expert Systems with Applications, 42(22):8520 – 8532, 2015.
  	
  	BolonCanedo14
  	V. Bolón-Canedo, N. Sánchez-Maroño, A. Alonso-Betanzos, J.M.
  	Benítez, and F. Herrera.
  	A review of microarray datasets and applied feature selection
  	methods.
  	Information Sciences, 282:111 – 135, 2014.
  	
  	BolonCanedo19
  	Verónica Bolón-Canedo and Amparo Alonso-Betanzos.
  	Ensembles for feature selection: A review and future trends.
  	Information Fusion, 52:1 – 12, 2019.
  	
  	Bolon13
  	Verónica Bolón-Canedo, Noelia Sánchez-Maroño, and Amparo
  	Alonso-Betanzos.
  	A review of feature selection methods on synthetic data.
  	Knowledge and Information Systems, 34(3):483–519, Mar 2013.
  	
  	Bolon15
  	Veronica Bolón-Canedo, Noelia Sánchez-Maroño, and Amparo
  	Alonso-Betanzos.
  	Recent advances and emerging challenges of feature selection in the
  	context of big data.
  	Knowledge-Based Systems, 86:33–45, 2015.
  	
  	Bommert20
  	Andrea Bommert, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and
  	Michel Lang.
  	Benchmark for filter methods for feature selection in
  		high-dimensional classification data.
  	Computational Statistics and Data Analysis, 143:106839, 2020.
  	
  	BragaNeto04
  	Ulisses Braga-Neto and Edward Dougherty.
  	Bolstered error estimation.
  	Pattern Recognition, 37(6):1267 – 1281, 2004.
  	
  	BragaNeto04c
  	Ulisses M. Braga-Neto and Edward R. Dougherty.
  	Is cross-validation valid for small-sample microarray
  		classification?
  	Bioinformatics, 20(3):374–380, 02 2004.
  	
  	BragaNeto10
  	Ulisses M. Braga-Neto and Edward R. Dougherty.
  	Exact correlation between actual and estimated errors in discrete
  	classification.
  	Pattern Recognition Letters, 31:407–412, 2010.
  	
  	Breiman01
  	L. Breiman.
  	Random forests.
  	Machine Learning, 45:5–32, 2001.
  	
  	Brown12
  	Gavin Brown, Adam Pocock, Mingjie Zhao, and Mikel Lujan.
  	Conditional likelihood maximisation: A unifying framework for
  	information theoretic feature selection.
  	Journal of Machine Learning Research, 13:27–66, 2012.
  	
  	Cai18
  	Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang.
  	Feature selection in machine learning: A new perspective.
  	Neurocomputing, 300:70 – 79, 2018.
  	
  	Chandrashekar14
  	Girish Chandrashekar and Ferat Sahin.
  	A survey on feature selection methods.
  	Computers and Electrical Engineering, 40(1):16–28, 2014.
  	
  	Cover77
  	Thomas M. Cover.
  	On the possible orderings in the measurement selection problem.
  	IEEE Transactions on Systems, Man and Cybernetics,
  	7(9):657–661, 1977.
  	
  	Deng19
  	Xuelian Deng, Yuqing Li, Jian Weng, and Jilian Zhang.
  	Feature selection for text classification: A review.
  	Multimedia Tools and Applications, 78(3):3797–3816, Feb 2019.
  	
  	Dernoncourt14
  	David Dernoncourt, Blaise Hanczar, and Jean-Daniel Zucker.
  	Analysis of feature selection stability on high dimension and small
  	sample data.
  	Computational Statistics and Data Analysis, 71:681–693, 2014.
  	
  	Devroye96
  	Luc Devroye, László Györfi, and Gábor Lugosi.
  	A Probabilistic Theory of Pattern Recognition, volume 31 of
  	Stochastic Modelling and Applied Probability.
  	Springer, 1997.
  	
  	Diciotti13
  	S. Diciotti, S. Ciulli, M. Mascalchi, M. Giannelli, and N. Toschi.
  	The “peeking” effect in
  	supervised feature selection on diffusion tensor imaging data.
  	American Journal of Neuroradiology, 34(9):E107–E107, 2013.
  	
  	Ditzler12
  	Gregory Ditzler, Gail Rosen, and Robi Polikar.
  	Information theoretic feature selection for high dimensional
  		metagenomic data.
  	In Proceedings - IEEE International Workshop on Genomic Signal
  		Processing and Statistics, pages 143–146. IEEE, 2012.
  	
  	Dougherty10
  	Edward R. Dougherty, Chao Sima, Jianping Hua, Blaise Hanczar, and Ulisses M.
  	Braga-Neto.
  	Performance of error estimators for classification.
  	Current Bioinformatics, 5:53–67, 2010.
  	
  	UCI
  	Dheeru Dua and Casey Graff.
  	UCI machine learning repository.
  	
  	Efron97
  	Bradley Efron and Robert Tibshirani.
  	Improvements on cross-validation: The 632+ bootstrap method.
  	Journal of the American Statistical Association,
  	92(438):548–560, 1997.
  	
  	EinDor06
  	Liat Ein-Dor, Or Zuk, and Eytan Domany.
  	Thousands of samples are needed to generate a robust gene lists for
  	predicting outcome in cancer.
  	Proceedings of the National Academy of Sciences of the United
  		States of America, 103(15):5923–5928, 2006.
  	
  	EinDor06a
  	Liat Ein-Dor, Or Zuk, and Eytan Domany.
  	Thousands of samples are needed to generate a robust gene lists for
  	predicting outcome in cancer: Supporting text.
  	Proceedings of the National Academy of Sciences of the United
  		States of America, 103(15), 2006.
  	
  	Fan09
  	Jianqing Fan, Richard Samworth, and Yichao Wu.
  	Ultrahigh Dimensional Feature Selection: Beyond the Linear Model.
  	Journal of Machine Learning Research, 10:2013–2038, 2009.
  	
  	Glick78
  	Ned Glick.
  	Additive estimators for probabilities of correct classification.
  	Pattern Recognition, 10(3):211–222, 1978.
  	
  	Guyon02
  	I Guyon, J Weston, S Barnhill, and V Vapnik.
  	Gene selection for cancer classification using support vector
  	machines.
  	Machine Learning, 46:389––422, 2002.
  	
  	Hall03
  	M. A. Hall and G. Holmes.
  	Benchmarking attribute selection techniques for discrete class data
  	mining.
  	IEEE Transactions on Knowledge and Data Engineering,
  	15(6):1437–1447, Nov 2003.
  	
  	Hall09
  	Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann,
  	and Ian H. Witten.
  	The WEKA data mining software: an update.
  	SIGKDD Explor. Newsl., 11(1):10–18, November 2009.
  	
  	Hancer18
  	Emrah Hancer, Bing Xue, Mengjie Zhang, Dervis Karaboga, and Bahriye Akay.
  	Pareto front feature selection based on artificial bee colony
  	optimization.
  	Information Sciences, 422:462–479, 2018.
  	
  	Izetta17
  	Javier Izetta, Pablo F Verdes, and Pablo M Granitto.
  	Improved multiclass feature selection via list combination.
  	Expert Systems With Applications, 88:205–216, 2017.
  	
  	Jacobovic18
  	Royi Jacobovic.
  	Are thousands of samples really needed to generate robust gene-list
  	for prediction of cancer outcome?
  	arXiv e-prints, page arXiv:1701.03159, Dec 2016.
  	
  	Kim09
  	Ji-Hyun Kim.
  	Estimating classification error rate: Repeated cross-validation,
  	repeated hold-out and bootstrap.
  	Computational Statistics and Data Analysis, 53:3735–3745,
  	2009.
  	
  	Kira92
  	Kenji Kira and Larry A Rendell.
  	A practical approach to feature selection.
  	In Proceedings of the ninth international workshop on Machine
  		learning, pages 249–256, 1992.
  	
  	Knocke86
  	James D Knocke.
  	The robust estimation of classification error rates.
  	Computers and Mathematics with Applications, 12(2A):253–260,
  	1986.
  	
  	Krawczuk16
  	Jerzy Krawczuk and Tomasz Łukaszuk.
  	The feature selection bias problem in relation to high-dimensional
  	gene data.
  	Artificial Intelligence in Medicine, 66:63–71, 2016.
  	
  	KunchevaPR18
  	Ludmila I. Kuncheva and Juan J. Rodríguez.
  	On feature selection protocols for very low-sample-size data.
  	Pattern Recognition, 81:660–673, 2018.
  	
  	Lee17
  	Pui Yi Lee, Wei Ping Loh, and Jeng Feng Chin.
  	Feature selection in multimedia: The state-of-the-art review.
  	Image and Vision Computing, 67:29–42, 2017.
  	
  	Li16
  	Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino,
  	Jiliang Tang, and Huan Liu.
  	Feature selection: A data perspective.
  	ACM Comput. Surv., 50(6):94:1–94:45, December 2017.
  	
  	Li17a
  	Jundong Li and Huan Liu.
  	Challenges of feature selection for big data analytics.
  	IEEE Intelligent Systems, 32(2):9–15, 2017.
  	
  	Li17
  	Yun Li, Tao Li, and Huan Liu.
  	Recent advances in feature selection and its applications.
  	Knowledge and Information Systems, 53(3):551–577, 2017.
  	
  	Liu10
  	Huan Liu, Hiroshi Motoda, Rudy Setiono, and Zheng Zhao.
  	Feature selection: An ever evolving frontier in data mining.
  	In Huan Liu, Hiroshi Motoda, Rudy Setiono, and Zheng Zhao, editors,
  	Proceedings of the Fourth International Workshop on Feature Selection in
  		Data Mining, volume 10 of Proceedings of Machine Learning Research,
  	pages 4–13, Hyderabad, India, Jun 2010. PMLR.
  	
  	Miao16
  	Jianyu Miao and Lingfeng Niu.
  	A Survey on Feature Selection.
  	Procedia Computer Science, 91:919–926, 2016.
  	
  	Murray77
  	Gordon D. Murray.
  	A cautionary note on selection of variables in discriminant analysis.
  	Journal of the Royal Statistical Society. Series C,
  	26(3):246–250, 1977.
  	
  	Pereira18
  	Rafael B. Pereira, Alexandre Plastino, Bianca Zadrozny, and Luiz H. C.
  	Merschmann.
  	Categorizing feature selection methods for multi-label
  	classification.
  	Artificial Intelligence Review, 49(1):57–78, Jan 2018.
  	
  	Pudil94
  	Pavel Pudil, Jana Novovičová, and Josef Kittler.
  	Floating search methods in feature selection.
  	Pattern Recognition Letters, 15(11):1119–1125, 1994.
  	
  	Ramirez18
  	S. Ramírez-Gallego, H. Mouriño-Talín, D. Martínez-Rego,
  	V. Bolón-Canedo, J. M. Benítez, A. Alonso-Betanzos, and
  	F. Herrera.
  	An information theory-based feature selection framework for big data
  	under Apache Spark.
  	IEEE Transactions on Systems, Man, and Cybernetics: Systems,
  	48(9):1441–1453, Sep. 2018.
  	
  	Rafaeilzadeh07
  	Payam Refaeilzadeh, Lei Tang, and Huan Liu.
  	On comparison of feature selection algorithms.
  	In AAAI Workshop - Technical Report, volume WS-07-05, pages
  	34–39, 2007.
  	
  	Reunanen03
  	Juha Reunanen.
  	Overfitting in making comparisons between variable selection methods.
  	Journal of Machine Learning Research, 3:1371–1382, 2003.
  	
  	Robnik03
  	Marko Robnik-Šikonja and Igor Kononenko.
  	Theoretical and empirical analysis of ReliefF and RReliefF.
  	Machine Learning, 53(1):23–69, Oct 2003.
  	
  	Saeys07
  	Yvan Saeys, Iñaki Inza, and Pedro Larrañaga.
  	A review of feature selection techniques in bioinformatics.
  	Bioinformatics, 23(19):2507–2517, 08 2007.
  	
  	Schulerud04
  	Helene Schulerud and Fritz Albregtsen.
  	Many are called, but few are chosen. feature selection and error
  	estimation in high dimensional spaces.
  	Computer Methods and Programs in Biomedicine, 73:91–99, 2004.
  	
  	Sheikhpour17
  	Razieh Sheikhpour, Mehdi Agha Sarram, Sajjad Gharaghani, and Mohammad Ali Zare
  	Chahooki.
  	A survey on semi-supervised feature selection methods.
  	Pattern Recognition, 64:141 – 158, 2017.
  	
  	Sima05
  	Chao Sima, Sanju Attoor, Ulisses Brag-Neto, James Lowey, Edward Suh, and
  	Edward R. Dougherty.
  	Impact of error estimation on feature selection.
  	Pattern Recognition, 38:2472–2482, 2005.
  	
  	Sima05b
  	Chao Sima, Sanju Attoor, Ulisses Braga-Neto, James Lowey, Edward Suh, and
  	Edward R. Dougherty.
  	Impact of error estimation on feature selection.
  	Pattern Recognition, 38:2472–2482, 2005.
  	
  	Sima05a
  	Chao Sima, Ulisses Braga-Neto, and Edward R. Dougherty.
  	Superior feature-set ranking for small samples using bolstered error
  	estimation.
  	Bioinformatics, 21(7), 2005.
  	
  	Sima06
  	Chao Sima and Edward R. Dougherty.
  	What should be expected from feature selection in small-sample
  	settings.
  	Bioinformatics, 22(19):2430–2436, 2006.
  	
  	Singhi06
  	Surendra K. Singhi and Huan Liu.
  	Feature subset selection bias for classification learning.
  	In Proceedings of the 23rd International Conference on Machine
  		Learning, ICML '06, pages 849–856, New York, NY, USA, 2006. ACM.
  	
  	Smialwski10
  	Pawel Smialowski, Dmitrij Frishman, and Stefan Kramer.
  	Pitfalls of supervised feature selection.
  	Bioinformatics, 26(3):440–443, 2010.
  	
  	Solorio19
  	Saúl Solorio-Fernández, J. Ariel Carrasco-Ochoa, and José Fco.
  	Martínez-Trinidad.
  	A review of unsupervised feature selection methods.
  	Artificial Intelligence Review, Jan 2019.
  	
  	Spolaor16
  	Newton Spolaôr, Maria Carolina Monard, Grigorios Tsoumakas, and
  	Huei Diana Lee.
  	A systematic review of multi-label feature selection and a new method
  	based on label construction.
  	Neurocomputing, 180:3 – 15, 2016.
  	Progress in Intelligent Systems Design.
  	
  	Storcheus15
  	Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar.
  	A survey of modern questions and challenges in feature extraction.
  	In JMLR: Workshop and Conference Proceedings, pages 1–18,
  	2015.
  	
  	Tuv09
  	Eugene Tuv, Alexander Borisov, George Runger, and Kari Torkkola.
  	Feature selection with ensembles, artificial variables, and
  		redundancy elimination.
  	Journal of Machine Learning Research, 10:1341–1366, 2009.
  	
  	Varoquax18
  	Gaël Varoquaux.
  	Cross-validation failure: Small sample sizes lead to large error
  	bars.
  	NeuroImage, 180(A):68–77, 2018.
  	
  	Varoquaux18
  	Gaël Varoquaux.
  	Cross-validation failure: Small sample sizes lead to large error
  	bars.
  	NeuroImage, 180:68 – 77, 2018.
  	New advances in encoding and decoding of brain signals.
  	
  	Vergara14
  	Jorge R. Vergara and Pablo A. Estévez.
  	A review of feature selection methods based on mutual information.
  	Neural Computing and Applications, 24(1):175–186, Jan 2014.
  	
  	Xue16
  	Bing Xue, Mengjie Zhang, Will N. Browne, and Xin Yao.
  	A survey on evolutionary computation approaches to feature selection.
  	IEEE Transactions on Evolutionary Computation,
  	20(4):606–626, 2016.
  	
  	Zawbaa18
  	Hossam M. Zawbaa, E. Emary, Crina Grosan, and Vaclav Snasel.
  	Large-dimensionality small-instance set feature selection: A hybrid
  		bio-inspired heuristic approach.
  	Swarm and Evolutionary Computation, 42(September 2016):29–42,
  	2018.
  	
  	Zou15
  	Q. Zou, L. Ni, T. Zhang, and Q. Wang.
  	Deep learning based feature selection for remote sensing scene
  	classification.
  	IEEE Geoscience and Remote Sensing Letters, 12(11):2321–2325,
  	Nov 2015.
  	
  
  
  