# Writing PCA objective function in scalar and matrix forms

These posts explains the PCA objective in matrix form in very nice way:

But I am still confused about the scalar notation in the Introduction to Statistical Learning, page 376:

$\quad\quad\quad\quad\quad\quad\quad\quad$ I think $\frac 1 n \sum_{i=1}^n\sum_{j=1}^p x_{ij}^2$ makes up the covariance matrix, but how are the $\phi_{j1}$ related to the "weights"?

Could anyone help me to build the connection between these two notations?

## 1 Answer

Let us recapitulate what is written in the posts that you linked to.

Let $\mathbf X$ be your $n\times p$ data matrix and let it be centered, i.e. the sum of each column equals zero. The $p\times p$ covariance matrix is then equal to $\mathbf C = \frac{1}{n}\mathbf X^\top \mathbf X$.

Let $\mathbf w$ be a unit $p$-dimensional vector, i.e. $\mathbf w \in \mathbb R^p$ such that $\mathbf w^\top \mathbf w = 1$. Then the projection of the data onto the axis spanned by this vector is given by $\mathbf z = \mathbf {Xw}$. Its mean is zero, and hence its variance is simply the average squared value, i.e. $$\operatorname{Var}(\mathbf z) = \frac{1}{n}\mathbf w^\top \mathbf X^\top\mathbf{Xw}=\mathbf w^\top \mathbf {Cw}.$$ This is the PCA objective function.

Now we can spell out the matrix multiplications. The projection is given by $$z_i = \sum_j X_{ij}w_j,$$ and its average squared value is $$\operatorname{Var}(z_i)=\frac{1}{n}\sum_i z_i^2 = \frac{1}{n}\sum_i\Big[\sum_j X_{ij}w_j\Big]^2,$$ which is the formula given in your book.

Note: $\frac 1 n \sum_{i=1}^n\sum_{j=1}^p X_{ij}^2$ is not a covariance matrix, contrary to what you wrote; this expression is a single number (squared Frobenius norm of $\mathbf X$), not a matrix element. For the element of covariance matrix you should write $$C_{ij}=\frac 1 n \sum_{i=1}^n\sum_{j=1}^p X_{ki}X_{kj}.$$
{"ft_lang_label":"__label__en","ft_lang_prob":0.7602742,"math_prob":0.9999825,"size":1352,"snap":"2019-35-2019-39","text_gpt3_token_len":453,"char_repetition_ratio":0.15133531,"word_repetition_ratio":0.0,"special_character_ratio":0.31730768,"punctuation_ratio":0.093103446,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":1.0000098,"pos_list":[0,1,2],"im_url_duplicate_count":[null,3,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-08-22T05:21:01Z\",\"WARC-Record-ID\":\"<urn:uuid:7f33123c-76f5-4434-b753-7138f08ee5fd>\",\"Content-Length\":\"136053\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:20a751dc-8751-4c61-b9d9-b34e33255844>\",\"WARC-Concurrent-To\":\"<urn:uuid:c802e29c-3903-4a1d-bec7-0cd7f55ae417>\",\"WARC-IP-Address\":\"151.101.129.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/234037/writing-pca-objective-function-in-scalar-and-matrix-forms/234073\",\"WARC-Payload-Digest\":\"sha1:AMXZBGUEKDNAPIRLR4F6AOMYZ7HJ5PXX\",\"WARC-Block-Digest\":\"sha1:DU7FCHGMP3XO3CXU7OSKAS5L2GOQJ5UQ\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-35/CC-MAIN-2019-35_segments_1566027316783.70_warc_CC-MAIN-20190822042502-20190822064502-00465.warc.gz\"}"}