complete vector space feature maps rkhs regularization identify with each element the linear function gb xb and define the inner product on as hgb gg :bg in this wayg behaves in exactly the same way as (is isomorphic tothe space equipped with the euclidean inner product (dot productthe latter is hilbert spacebecause it is complete with respect to the euclidean norm see exercise for further discussion let us now turn to our "runningpolynomial regression example where the feature vector [ uu - ]=ph(uis itself vector-valued function of another feature thenthe space of functions hb ph( ) is hilbert spacethrough the identification hb in factthis is true for any feature mapping ph [ph ( )ph ( )]this can be further generalized by considering feature maps ku where each ku is real-valued function ku (von the feature space as we shall soon see (in secp tion )functions of the form = bi kvi (ulive in hilbert space of functions called reproducing kernel hilbert space (rkhsin section we introduce the notion of rkhs formallygive specific examplesincluding the linear and gaussian kernelsand derive various useful propertiesthe most important of which is the representer theorem applications of such spaces include the smoothing splines (section )gaussian process regression (section )kernel pca (section )and support vector machines for classification (section the rkhs formalism also makes it easier to treat the important topic of regularization the aim of regularization is to improve the predictive performance of the best learner in some class of functions by adding penalty term to the training loss that penalizes learners that tend to overfit the data in the next section we introduce the main ideas behind regularizationwhich then segues into discussion of kernel methods in the subsequent sections regularization let be the hilbert space of functions over which we search for the minimizergt of the training loss ` (goftenthe hilbert space is rich enough so that we can find learner gt within such that the training loss is zero or close to zero consequentlyif the space of functions is sufficiently richwe run the risk of overfitting one way to avoid overfitting is to restrict attention to subset of the space by introducing non-negative functional rwhich penalizes complex models (functionsin particularwe want to find functions such that ( thus we can formulate the quintessential supervised learning problem asmin {` (gg (gc( the solution (argminof which is our learner when this optimization problem is convexit can be solved by first obtaining the lagrangian dual function ( :min {` (gl( (gc)gg ridge regression and then maximizing (lwith respect to see section in order to introduce the overall ideas of kernel methods and regularizationwe will proceed by exploring ( in the special case of ridge regressionwith the following running example