where ni= xi we assume from now on that the data comes from general -dimensional distribution with mean vector and some covariance matrix the covariance matrix is by definition equal to the expectation of the random matrix xxand can be estimated from the data xn via the sample average xi xi sn = as is covariance matrixwe may conduct the same analysis for as we did for in the previous section specificallysuppose ud is an svd of and let uk be the matrix whose columns are the principal componentsthat isthe columns of corresponding to the largest diagonal elements in note that we have used instead of to be compatible with the previous section the transformation zi uk > xi maps each vector xi rd (thuswith featuresto vector zi rd lying in the subspace spanned by the columns of uk with respect to this basisthe point zi has representation zi > (uk > xi > xi rk (thus with featuresthe corresponding covariance matrix of the zi is diagonal the diagonal elements { `of can be interpreted as standard deviations of the data (that isthe trace of in the directions of the principal components the quantity `= ` is thus measure for the amount of variance in the data the proportion `/ indicates how much of the variance in the data is explained by the `-th principal component another way to look at pca is by considering the questionhow can we best project the data onto -dimensional subspace in such way that the total squared distance between the projected points and the original points is minimalfrom section we know that any orthogonal projection to -dimensional subspace vk can be represented by matrix uk > where uk [ uk and the { kare orthogonal vectors of length that span vk the above question can thus be formulated as the minimization programmin ,uk = kxi uk > xi ( now observe that kxi uk > xi ( > uk > )(xi uk > xi = = xx kxi xi uk > xi tr( > uu>xi = = = `= { = xx > uuxi xi uc `= = `= where we have used the cyclic property of trace (theorem and the fact that uk > can be written as `= uu>it follows that the minimization problem( is equivalent to the maximization problem max > ( ,uk `=