In statistics, kernel Fisher discriminant analysis (KFD), also known as generalized discriminant analysis and kernel discriminant analysis, is a kernelized version of linear discriminant analysis (LDA). It is named after Ronald Fisher. Linear discriminant analysis Intuitively, the idea of LDA is to find a projection where class separation is maximized. Given two sets of labeled data, and , we can calculate the mean value of each class, and , as where is the number of examples of class . The goal of linear discriminant analysis is to give a large separation of the class means while also keeping the in-class variance small. This is formulated as maximizing, with respect to , the following ratio: where is the between-class covariance matrix and is the total within-class covariance matrix: The maximum of the above ratio is attained at as can be shown by the Lagrange multiplier method (sketch of proof): Maximizing is equivalent to maximizing subject to This, in turn, is equivalent to maximizing , where is the Lagrange multiplier. At the maximum, the derivatives of with respect to and must be zero. Taking yields which is trivially satisfied by and Extending LDA To extend LDA to non-linear mappings, the data, given as the points can be mapped to a new feature space, via some function In this new feature space, the function that needs to be maximized is where and Further, note that . Explicitly computing the mappings and then performing LDA can be computationally expensive, and in many cases intractable. For example, may be infinitely dimensional. Thus, rather than explicitly mapping the data to , the data can be implicitly embedded by rewriting the algorithm in terms of dot products and using kernel functions in which the dot product in the new feature space is replaced by a kernel function,. LDA can be reformulated in terms of dot products by first noting that will have an expansion of the form Then note that where The numerator of can then be written as: Similarly, the denominator can be written as with the component of defined as is the identity matrix, and the matrix with all entries equal to . This identity can be derived by starting out with the expression for and using the expansion of and the definitions of and With these equations for the numerator and denominator of , the equation for can be rewritten as Then, differentiating and