there are multiple objectives in a feature selection task. Many common criteria incorporate a measure of accuracy, penalised by the number of features selected. Examples include Akaike information criterion (AIC) and Mallows's Cp, which have a penalty of 2 for each added feature. AIC is based on information theory, and is effectively derived via the maximum entropy principle. Other criteria are Bayesian information criterion (BIC), which uses a penalty of for each added feature, minimum description length (MDL) which asymptotically uses , Bonferroni / RIC which use , maximum dependency feature selection, and a variety of new criteria that are motivated by false discovery rate (FDR), which use something close to . A maximum entropy rate criterion may also be used to select the most relevant subset of features. Structure learning Filter feature selection is a specific case of a more general paradigm called structure learning. Feature selection finds the relevant feature set for a specific target variable whereas structure learning finds the relationships between all the variables, usually by expressing these relationships as a graph. The most common structure learning algorithms assume the data is generated by a Bayesian Network, and so the structure is a directed graphical model. The optimal solution to the filter feature selection problem is the Markov blanket of the target node, and in a Bayesian Network, there is a unique Markov Blanket for each node. Information Theory Based Feature Selection Mechanisms There are different Feature Selection mechanisms around that utilize mutual information for scoring the different features. They usually use all the same algorithm: Calculate the mutual information as score for between all features () and the target class () Select the feature with the largest score (e.g. ) and add it to the set of selected features () Calculate the score which might be derived from the mutual information Select the feature with the largest score and add it to the set of select features (e.g. ) Repeat 3. and 4. until a certain number of features is selected (e.g. ) The simplest approach uses the mutual information as the "derived" score. However, there are different approaches, that try to reduce the redundancy between features. Minimum-redundancy-maximum-relevance (mRMR) feature selection Peng et al. proposed a feature selection method that can use either mutual information, correlation, or distance/similarity scores to select features. The aim is to penalise a feature's relevancy by its redundancy in the