title: Any efficient way to update very large matrix?
body: <p>I have a very large matrix, which is a 2D array with few hundred rows but around 2 million columns. My application needs to update this matrix row by rows with one painful constraint. To update a specific row, it needs to wait all columns of previous rows updated. And the process is very slow.</p>
  
  <p>For example:</p>
  
  <pre><code>matrix = [[0 for x in xrange(2000000)] for y in xrange(300)]
  for i in xrange(1, 300):
      for j in xrange(2000000):
          k = a random column in row i-1
          matrix[i][j] = matrix[i-1][k] * simple_function(i) # some calculation
  </code></pre>
  
  <ul>
  <li>To update <code>matrix[i][j]</code> , I need to have the value of row <code>i-1</code></li>
  </ul>
  
  <p>My initial thought is to use multi-process approach to parallelize <code>j</code> loop in each <code>i</code> round. However, the calculation in <code>j</code> loop is too light. The process creation cost is much higher than calculation (I also tried process pool).</p>
  
  <p>Second thought is to use thread, it works well with almost no performance gain due to GIL limitation. </p>
  
  <p>I would like to know if there's any other approach that can accelerate the my code. Thank you.</p>
  
  <p>BTW, I know Cython can work without GIL. But the calculation function needs to access a Python object and it will takes lots of works to modify the original code.</p>