title: python multiprocessing running out of memory work around?
body: <p>I have a script that runs for about 3 hours. If I use multiprocessing, it takes about 20 minutes. The problem is the memory. </p>
  
  <p>Say there is 10 tasks in the script. If I run it on a single core, it will go all the way through. But if I use multiprocessing, I have to stop it half way through to free up the memory and start the second half manually.</p>
  
  <p>I've tried to use garbage <code>collect</code>, and using something like <code>popen</code> but it doesn't help. </p>
  
  <p>I'm running the script in pycharm. Not sure if that matters on how the memory is handled.</p>
  
  <p>I have 32GB of RAM. When the script starts, I'm already using 6GB. I'm running 64-bit Python 3.6. I have a 12-core processor.</p>
  
  <p>I'm running windows and just using <code>Pool</code> from <code>multiprocessing</code>.</p>
  
  <p>Running out of memory would make the script bomb. Thus I need to stop it half way and start it again from the 2nd half.</p>
  
  <p>The question is how to run the script all the way through using multiprocessing without using fewer cores?</p>