title: Using ThreadPoolExecutor with reduced memory footprint
body: <p>I am using <code>ThreadPoolExecutor</code> in order to download a huge (~400k) amount of keyframe images. Keyframes names are stored in text file (let's say <em>keyframes_list.txt</em>).</p>
  
  <p>I have modified the example provided in the <a href="https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example" rel="nofollow noreferrer">documentation</a> and it seems to work flawlessly with one exception: as it is clear the example passes every link to a <code>future</code> object which are all passed to an iterable (<code>dict()</code> to be precise). This iterable is passed as argument to <code>as_completed()</code> function to check when a <code>future</code> is completed. This of course requires a huge amount of text loaded at once in memory. My python process for this task takes up 1GB of RAM. </p>
  
  <p>The full code is shown below:</p>
  
  <pre><code>import concurrent.futures
  import requests
  
  def download_keyframe(keyframe_name):
      url = 'http://server/to//Keyframes/{}.jpg'.format(keyframe_name)
      r = requests.get(url, allow_redirects=True)
      open('path/to/be/saved/keyframes/{}.jpg'.format(keyframe_name), 'wb').write(r.content)
      return True
  
  keyframes_list_path = '/path/to/keyframes_list.txt'
  future_to_url = {}
  with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
      with open(keyframes_list_path, 'r') as f:
          for i, line in enumerate(f):
              fields = line.split('\t')
              keyframe_name = fields[0]
              future_to_url[executor.submit(download_keyframe, keyframe_name)] = keyframe_name
      for future in concurrent.futures.as_completed(future_to_url):
          keyframe_name = future_to_url[future]
          try:
              future.result()
          except Exception as exc:
              print('%r generated an exception: %s' % (keyframe_name, exc))
          else:
              print('Keyframe: {} was downloaded.'.format(keyframe_name))
  </code></pre>
  
  <p>So, my question is how could I provide both an iterable and also keep memory footprint low. I have considered using <code>queue</code> but I am not sure it's cooperating with <code>ThreadPoolExecutor</code> smoothly. Is there an easy way to control the amount of <code>future</code>s submitted to <code>ThreadPoolExecutor</code>?</p>