Understanding how distributed PCA works
The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is

For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.
When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.
How then is the covariance matrix calculated in such case when each node have only partial data?

I shall address that, which hopefully will clear the matter to a degree.
Let us look at a common form of covariance calculation, $\frac{1}{n}\sum(x-\bar{x})(y-\bar{y})$
This requires you to calculate the following:

$\bar{x}$
$\bar{y}$
$x-\bar{x}$ and $y-\bar{y}$
Multiply the $(x-\bar{x})$ and $(y-\bar{y})$

in a distributed manner. The rest is simple, let us say I have 100 datat points (x,y), which is distributed to 10 Apache Spark workers, each getting 10 data points.
Calculating the $\bar{x}$ and $\bar{y}$: Each worker will add $x/y$ values of 10 data points and divide this by 10 to arrive at partial mean of $x/y$ (this is the map function). Then the Spark master will run the aggregation step (in Spark DAG of the job) where the partial means from all 10 workers are taken and again added, then divided by 10 to arrive at the final $\bar{x}$ or $\bar{y}$ (the aggregate/reduce operation)
Calculating the $(x-\bar{x}) \cdot (y-\bar{y})$: Same way, distribute the data points, broadcast the $\bar{x}$ and $\bar{y}$ values to all the workers and the calculate the partial $(x-\bar{x}) \cdot (y-\bar{y})$, again run aggregation to get $\sum (x-\bar{x})(y-\bar{y})$
The above method is used for distributed calculation, you shall get the covariance, for multi-dimensional data, you can get the covariance matrix.
The point is to distribute the calculation for stages that can be distributed and then centralize the calculation stages that cannot be distributed. That is in effect one of the important aspect of Spark architecture.
Hope this helps.