PCA algorithm problems - Python
First question: computing $\Sigma$

Actually, you perform the same computation but using a matrix operation instead of a scalar operation. You might be mislead by your notation of the feature matrix $X$ that you write $x$ because in fact
$$X = (x_j^{(i)})_{i,j} =\Big(x^{(1)} ... x^{(i)} ... x^{(n)}\Big)^T$$
that is to say, $i$-th row of the matrix $X$ contains $x^{(i)}$ the $i$-th sample.

So, when you want to compute the empirical covariance matrix $\Sigma$ (which is indeed a $n \times n$ matrix), you have:
$$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T = \frac{1}{n} X^T X$$
you can check that this is exactly the same computation.

In your code, you are actually computing directly $\Sigma$ using the matrix operation (i.e. $\Sigma = \frac{1}{n} X^T X$) and this does give you a $n \times n$ matrix. So, both your implementation and the formula is correct.

Second question: reconstructing $X$

Your new feature matrix $Z$ is a $n \times 3$ matrix according to your code. In fact, as your code does not show the original size of the feature space, we will see the two cases here:


$X$ is also a $n \times 3$ matrix, then you do not perform any kind of dimension reduction and you should have $X_{new} = X$ (at least in theory, in practice you might have a very small numerical approximation error but it will basically be the same)
$X$ is a $n \times d$ matrix with $d &gt; 3$, then you do perform a dimension reduction and in this case you get rid of some information contained in the original data and in this case $X_{new} \neq X$


In your case, I guess that you do have $d &gt; 3$ and so you're in the second situation.

Third question: when to apply PCA

Well that depends in fact...

First of all PCA performs an SVD which can become very expensive if you have a lot of features. In fact, there are several way to compute PCA, your way is the closer to the mathematical theory but in practice computing an SVD directly on $X$ avoids computing $X^T X$ which is expensive and allows to retrieve the same space. But in any case, in some situation, it can be expensive and time consuming and thus not practical.

As PCA sorts the new feature space according to the value of the eigenvalues, i.e. of the variance of the new directions, removing the last direction might in some cases remove some noise which can help. However, at the same time, it can throw away valuable discrimative information, this is why other methods such as LDA can be interesting.

So to summarize, the answer to the question is no, it is not a good choice to use it everytime, it depends of your problem.