example (ridge regressionridge regression is simply linear regression with squared-norm penalty functional (also called regularization functionor regularizersuppose we have training set {(xi yi ) }with each xi and we use squared-norm penalty with regularization parameter thenthe problem is to solve (yi (xi )) kgk min gg = regularizer regularization parameter ( where is the hilbert space of linear functions on as explained in section we can identify each with vector andconsequentlykgk hbbi kbk the above functional optimization problem is thus equivalent to the parametric optimization problem  yi > kbk ( minp br = whichin the notation of further simplifies to xb kbk br minp ( in other wordsthe solution to ( is of the form xbwhere bsolves ( (or equivalently ( )observe that as the regularization term becomes dominant and consequently the optimal becomes identically zero the optimization problem in ( is convexand by multiplying by the constant / and setting the gradient equal to zerowe obtain (xb yn ( if these are simply the normal equationsalbeit written in slightly different form if the matrix xx gi is invertible (which is the case for any see exercise )then the solution to these modified normal equations is (xx gi )- xy when using regularization with respect to some hilbert space git is sometimes useful to decompose into two orthogonal subspacesh and saysuch that every can be uniquely written as cwith hc cand hhci such is said to be the direct sum of and hand we write decompositions of this form become useful when functions in are penalized but functions in are not we illustrate this decomposition with the ridge regression example where one of the features is constant termwhich we do not wish to penalize example (ridge regression (cont )suppose one of the features in example is the constant which we do not wish to penalize the reason for this is to ensure that when the optimal becomes the "constantmodelg(xb rather than the "zeromodelg( let us alter the notation slightly by considering the feature vectors to be of the form [ ]where [ ]we thus have featuresrather direct sum