title: What is the fastest way to calculate partial SVD of a huge matrix using Python libraries?
body: <p>I have a large (500k x 300k) dense matrix for which I need to find the top approx. 50 singular values (high accuracy is not important) and corresponding singular vectors. I have access to Amazon SageMaker instances which enables me to use up to 256GB RAM, however the matrix takes up about 600GB so it cannot be kept in memory. </p>
  
  <p>To circumvent this issue, calculating matrix-vector products with this matrix is fairly fast and parallelizable using Tensorflow. </p>
  
  <p>I have looked into some related questions such as <a href="https://stats.stackexchange.com/questions/41259/how-to-compute-svd-of-a-huge-sparse-matrix">https://stats.stackexchange.com/questions/41259/how-to-compute-svd-of-a-huge-sparse-matrix</a> 
  , however the whole workflow is implemented in Python/Tensorflow and therefore it would be a significant overhead to use R. </p>
  
  <p>First, I tried using <code>numpy.linalg.svd</code> which can utilize multi-core execution, however as I mentioned I cannot fit the whole matrix into memory and I don't need a full SVD (which is the only option since it uses LAPACK).</p>
  
  <p>Second, I tried <code>scipy.sparse.linalg.svds</code>, which uses ARPACK and takes a <code>LinearOperator</code> instead of a matrix. This solved the memory issues, however this ARPACK implementation seems to use a single core for SVD (even though the matrix-vector multiplication runs on multiple cores) and therefore extremely slow. The fact that the matrix is dense makes the problem much worse. </p>
  
  <p>I would really appreciate any input how problems like this are usually tackled. I might have overlooked some built-in functionality of <code>numpy</code> or <code>scipy</code>, but I am also open to other practical algorithms, not available in Python libraries, for partial SVD that might help with the memory issue and would do the job in a reasonable amount of time (couple of days maximum).<br>
  Thank you for the answer in advance!</p>