Issue with Spark SVD I have the following dataset with the dimensions: 

Rows: 41174

Columns: 439316

The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.

However, I get an error saying that 


  Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition


But I ran the same computeSVD on the following dataset and it ran perfectly fine.

Rows: 3502

Columns: 103301

In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?
After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer.

In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968&gt;2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888&gt;2147483647$.

I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big! 

References

https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html