term on the diagonal of . Conditional mutual information Another score derived for the mutual information is based on the conditional relevancy: where and . An advantage of is that it can be solved simply via finding the dominant eigenvector of , thus is very scalable. also handles second-order feature interaction. Joint mutual information In a study of different scores Brown et al. recommended the joint mutual information as a good score for feature selection. The score tries to find the feature, that adds the most new information to the already selected features, in order to avoid redundancy. The score is formulated as follows: The score uses the conditional mutual information and the mutual information to estimate the redundancy between the already selected features () and the feature under investigation (). Hilbert-Schmidt Independence Criterion Lasso based feature selection For high-dimensional and small sample data (e.g., dimensionality > and the number of samples < ), the Hilbert-Schmidt Independence Criterion Lasso (HSIC Lasso) is useful. HSIC Lasso optimization problem is given as where is a kernel-based independence measure called the (empirical) Hilbert-Schmidt independence criterion (HSIC), denotes the trace, is the regularization parameter, and are input and output centered Gram matrices, and are Gram matrices, and are kernel functions, is the centering matrix, is the -dimensional identity matrix (: the number of samples), is the -dimensional vector with all ones, and is the -norm. HSIC always takes a non-negative value, and is zero if and only if two random variables are statistically independent when a universal reproducing kernel such as the Gaussian kernel is used. The HSIC Lasso can be written as where is the Frobenius norm. The optimization problem is a Lasso problem, and thus it can be efficiently solved with a state-of-the-art Lasso solver such as the dual augmented Lagrangian method. Correlation feature selection The correlation feature selection (CFS) measure evaluates subsets of features on the basis of the following hypothesis: "Good feature subsets contain features highly correlated with the classification, yet uncorrelated to each other". The following equation gives the merit of a feature subset S consisting of k features: Here, is the average value of all feature-classification correlations, and is the average value of all feature-feature correlations. The CFS criterion is defined as follows: The and variables are referred to as correlations, but are not necessarily Pearson's correlation coefficient or Spearman's ρ. Hall's dissertation uses neither of these, but