title: How to get partial data from the function in multiprocessing?
body: <p>I have code like below:</p>
  <pre><code>while True:
      print('Getting Links')
      links = get_links(LIMIT)
      if len(links) == 0:
          break
      if len(links) &gt; 0:
          with Pool(POOL_COUNT) as p:
              result = p.map(parse, links)
  </code></pre>
  <p>The multiple processes are being spawned that are fetching thousands of records. For sake of performance I do not want to put <code>INSERT</code> query within the <code>parse</code> function and want to store data once it is available in <code>result</code>. Now the issue is that the script could die anytime due to network error, Keyboard interrupt or anything that will ruin all efforts made to scrape thousands of URLs.</p>
  <p>I want to know is it possible that if some exception occur or anything, I could get the records scraped so far across the process and store them? Right now if exception occur or I kill script, the <code>result</code> is always null.</p>