Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x?
Incremental PCA typically only reduces the memory footprint of the algorithm. So it's used when your full dataset doesn't fit into memory. The time complexity remains the same.

If you want to speed up the computation you can look into randomized matrix decompositions. Sklearn, for example, implements it with the keyword svd_solver='randomized' where you only approximate the singular values you will ultimately keep.