title: How to parallelize file downloads?
body: <p>I can download a file at a time with:</p>
  
  <pre><code>import urllib.request
  
  urls = ['foo.com/bar.gz', 'foobar.com/barfoo.gz', 'bar.com/foo.gz']
  
  for u in urls:
    urllib.request.urlretrieve(u)
  </code></pre>
  
  <p>I could try to <code>subprocess</code> it as such:</p>
  
  <pre><code>import subprocess
  import os
  
  def parallelized_commandline(command, files, max_processes=2):
      processes = set()
      for name in files:
          processes.add(subprocess.Popen([command, name]))
          if len(processes) &gt;= max_processes:
              os.wait()
              processes.difference_update(
                  [p for p in processes if p.poll() is not None])
  
      #Check if all the child processes were closed
      for p in processes:
          if p.poll() is None:
              p.wait()
  
  urls = ['http://www.statmt.org/wmt15/training-monolingual-nc-v10/news-commentary-v10.en.gz',
  'http://www.statmt.org/wmt15/training-monolingual-nc-v10/news-commentary-v10.cs.gz', 
  'http://www.statmt.org/wmt15/training-monolingual-nc-v10/news-commentary-v10.de.gz']
  
  parallelized_commandline('wget', urls)
  </code></pre>
  
  <p>Is there any way to parallelize <code>urlretrieve</code> without using <code>os.system</code> or <code>subprocess</code> to cheat?</p>
  
  <p>Given that I must resort to the "cheat" for now, is <code>subprocess.Popen</code> the right way to download the data? </p>
  
  <p>When using the <code>parallelized_commandline()</code> above, it's using multi-thread but not multi-core for the <code>wget</code>, is that normal? Is there a way to make it multi-core instead of multi-thread?</p>