title: Multiprocessing with large array in state
body: <p>I've got a Class which stores a large <code>numpy</code> array in the state. This is causing <code>multiprocessing.Pool</code> to become extremely slow. Here's a MRE:</p>
  
  <pre><code>from multiprocessing import Pool
  import numpy
  import time
  from tqdm import tqdm
  
  class MP(object):
      def __init__(self, mat):
          self.mat = mat
  
      def foo(self, x):
          time.sleep(1)
          return x*x + self.mat.shape[0]
  
      def bar(self, arr):
          results = []
          with Pool() as p:
              for x in tqdm(p.imap(self.foo, arr)):
                  results.append(x)
          return results
  
  if __name__ == '__main__':
      x = numpy.arange(8)
      mat = numpy.random.random((1,1))
      h = MP(mat)
      res = h.bar(x)
      print(res)
  </code></pre>
  
  <p>I've got 4 cores on CPU, which means that this code should (and does) run in approximately 2 seconds. (The <code>tqdm</code> shows the 2 seconds as a progress bar, it's not really necessary to this example). However, in the main program, if I do <code>mat = numpy.random.random((10000,10000))</code>, it takes forever to run. I suspect this is because <code>Pool</code> is making copies of <code>mat</code> for each worker, but I'm not sure how this works because <code>mat</code> is in the state of the Class, and not directly involved in the <code>imap</code> call. So, my questions are:</p>
  
  <ol>
  <li>Why is this behavior happening? (i.e., how does <code>Pool</code> work within a Class? What exactly does it pickle? What copies are made, and what is passed by reference?)</li>
  <li>What is a viable workaround to this problem?</li>
  </ol>
  
  <p><strong>Edit</strong>: Modified <code>foo</code> to make use of <code>mat</code>, which is more representative of my real problem.</p>