title: Concurrent download and processing of large files in python
body: <p>I have a list of URLs for large files to <strong>download</strong> (e.g. compressed archives), which I want to <strong>process</strong> (e.g. decompress the archives). </p>
  
  <p>Both download and processing take a long time and processing is heavy on disk IO, so I want to have <strong>just one of each to run at a time</strong>. Since the two tasks take about the same time and do not compete for the same resources, I want to download the next file(s) while the last is being processed.</p>
  
  <p>This is a variation of the <strong><a href="https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem" rel="nofollow noreferrer">producer-consumer problem</a></strong>.</p>
  
  <p>The situation is similar to <a href="https://stackoverflow.com/q/12474182/512111">reading and processing images</a> or <a href="https://stackoverflow.com/questions/37825218/fastest-way-to-read-and-process-100-000-urls-in-python">downloading loads of files</a>, but my downloader calls are not (yet) picklable, so I have not been able to use multiprocessing, and both tasks take about the same time.</p>
  
  <p>Here is a dummy example, where both download and processing are blocking:</p>
  
  <pre><code>import time
  import posixpath
  
  def download(urls):
      for url in urls:
          time.sleep(3)  # this is the download (more like 1000s) 
          yield posixpath.basename(url)
  
  def process(fname):
      time.sleep(2)  # this is the processing part (more like 600s)
  
  urls = ['a', 'b', 'c']
  for fname in download(urls):
      process(fname)
      print(fname)
  </code></pre>
  
  <p>How could I make the two tasks concurrent? Can I use <code>yield</code> or <code>yield from</code> <a href="https://stackoverflow.com/questions/9708902/in-practice-what-are-the-main-uses-for-the-new-yield-from-syntax-in-python-3">in a smart way</a>, perhaps in combination with <a href="https://docs.python.org/2/library/collections.html#collections.deque" rel="nofollow noreferrer"><code>deque</code></a>? Or must it be <a href="https://docs.python.org/3.4/library/asyncio.html" rel="nofollow noreferrer"><code>asyncio</code></a> with <code>Future</code>?</p>