title: multiprocessing with large data
body: <p>I am using <code>multiprocessing.Pool()</code> to parallelize some heavy computations.</p>
  
  <p>The target function returns a lot of data (a huge list). I'm running out of RAM.</p>
  
  <p>Without <code>multiprocessing</code>, I'd just change the target function into a generator, by <code>yield</code>ing the resulting elements one after another, as they are computed.</p>
  
  <p>I understand multiprocessing does not support generators -- it waits for the entire output and returns it at once, right? No yielding. Is there a way to make the <code>Pool</code> workers yield data as soon as they become available, without constructing the entire result array in RAM?</p>
  
  <p>Simple example:</p>
  
  <pre><code>def target_fnc(arg):
     result = []
     for i in xrange(1000000):
         result.append('dvsdbdfbngd') # &lt;== would like to just use yield!
     return result
  
  def process_args(some_args):
      pool = Pool(16)
      for result in pool.imap_unordered(target_fnc, some_args):
          for element in result:
              yield element
  </code></pre>
  
  <p>This is Python 2.7.</p>