to obtain good estimate of (ubased on the training set {(ui yi ) }we minimize the outcome of the training loss ( ) (yi (ui )) ` (hn = ( over suitable set of candidate functions let us take the set of polynomial functions in of order ( : - ( for and parameter vector [ ]this function class contains the best possible (ue[ ufor note that optimization over is parametric optimization problemin that we need to find the best optimization of ( over is not straightforwardunless we notice that ( is linear function in in particularif we map each feature to feature vector [ uu - ]then the right-hand side of ( can be written as the function (xxbwhich is linear in (as well as bthe optimal (uin for then corresponds to the function (xxbin the set of linear functions from to rwhere [ - - ]thusinstead of working with the set of polynomial functions we may prefer to work with the set of linear functions this brings us to very important idea in statistical learningexpand the feature space to obtain linear prediction function let us now reformulate the learning problem in terms of the new explanatory (featurevariables xi [ ui uip- ] it will be convenient to arrange these feature vectors into matrix with rows > > - - ( un unp- collecting the responses {yi into column vector ythe training loss ( can now be written compactly as ky xbk ( to find the optimal learner ( in the class we need to find the minimizer of ( ) argmin ky xbk ( which is called the ordinary least-squares solution as is illustrated in figure to find bwe choose xb to be equal to the orthogonal projection of onto the linear space spanned by the columns of the matrix xthat isxb pywhere is the projection matrix ordinary least-squares projection matrix