title: ipyparallel's LoadBalancedView bloats memory, how can I avoid that?
body: <p>This issue may be related to <a href="https://github.com/ipython/ipyparallel/issues/207" rel="nofollow noreferrer">https://github.com/ipython/ipyparallel/issues/207</a> which is also not marked as solved, yet. <br>
  I also opened this issue here <a href="https://github.com/ipython/ipyparallel/issues/286" rel="nofollow noreferrer">https://github.com/ipython/ipyparallel/issues/286</a></p>
  
  <p>I want to execute multiple tasks in parallel using python and ipyparallel in a jupyter notebook and using 4 local engines by executing <code>ipcluster start</code> in a local console. 
  Besides that one can also use <code>DirectView</code>, I use <code>LoadBalancedView</code> to map a set of tasks. Each task takes around 0.2 seconds (can vary though) and each task does a MySQL query where it loads some data and then processes it. <br>
  Working with ~45000 tasks works fine, however, my memory grows really high. This is actually bad because I want to run another experiment with over 660000 tasks which I can't run anymore because it bloats up my memory limit of 16 GB and then the memory swapping on my local drive starts. However, when using the <code>DirectView</code> my memory grows relatively small and is never full. But I actually need <code>LoadBalancedView</code>. <br>
  Even when running a minimal working example without database query this happens (see below).</p>
  
  <p>I am not perfectly familiar with the ipyparallel library but I've read something about logs and caches that the ipcontroler does which may cause this. I am still not sure if it is a bug or if I can change some settings to avoid my problem.</p>
  
  <h3>Running a MWE</h3>
  
  <p>For my Python 3.5.3 environment running on Windows 10 I use the following (recent) packages:</p>
  
  <ul>
  <li>ipython 6.1.0 </li>
  <li>ipython_genutils 6.1.0 </li>
  <li>ipyparallel 6.0.2</li>
  <li>jupyter 1.0.0</li>
  <li>jupyter_client 4.4.0</li>
  <li>jupyter_console 5.0.0</li>
  <li>jupyter_core 4.2.0</li>
  </ul>
  
  <p>I would like the following example to work for <code>LoadBalancedView</code> <strong>without the immense memory growth</strong> (if possible at all):</p>
  
  <ul>
  <li>Start <code>ipcluster start</code> on a console</li>
  <li><p>Run a jupyter notebook with the following three cells:</p>
  
  <pre><code>&lt;1st cell&gt;
  import ipyparallel as ipp
  rc = ipp.Client()
  lview = rc.load_balanced_view()
  
  &lt;2nd cell&gt;
  %%px --local
  import time
  
  &lt;3rd cell&gt;
  def sleep_here(i):
      time.sleep(0.2)
      return 42
  
  amr = lview.map_async(sleep_here, range(660000))
  amr.wait_interactive()
  </code></pre></li>
  </ul>