multithreaded file download in python and updating in shell with download progress
Question: in an attempt to learn multithreaded file download I wrote this piece of cake:

    
    
    import urllib2
    import os
    import sys
    import time
    import threading
    
    urls = ["http://broadcast.lds.org/churchmusic/MP3/1/2/nowords/271.mp3",
    "http://s1.fans.ge/mp3/201109/08/John_Legend_So_High_Remix(fans_ge).mp3",
    "http://megaboon.com/common/preview/track/786203.mp3"]
    
    url = urls[1]
    
    def downloadFile(url, saveTo=None):
        file_name = url.split('/')[-1]
        if not saveTo:
            saveTo = '/Users/userName/Desktop'
        try:
            u = urllib2.urlopen(url)
        except urllib2.URLError , er:
            print("%s" % er.reason)
        else:
    
            f = open(os.path.join(saveTo, file_name), 'wb')
            meta = u.info()
            file_size = int(meta.getheaders("Content-Length")[0])
            print "Downloading: %s Bytes: %s" % (file_name, file_size)
            file_size_dl = 0
            block_sz = 8192
            while True:
                buffer = u.read(block_sz)
                if not buffer:
                    break
    
                file_size_dl += len(buffer)
                f.write(buffer)
                status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
                status = status + chr(8)*(len(status)+1)
                sys.stdout.write('%s\r' % status)
                time.sleep(.2)
                sys.stdout.flush()
                if file_size_dl == file_size:
                    print r"Download Completed %s%% for file %s, saved to %s" % (file_size_dl * 100. / file_size, file_name, saveTo,)
            f.close()
            return
    
    
    def synchronusDownload():
        urls_saveTo = {urls[0]: None, urls[1]: None, urls[2]: None}
        for url, saveTo in urls_saveTo.iteritems():
            th = threading.Thread(target=downloadFile, args=(url, saveTo), name="%s_Download_Thread" % os.path.basename(url))
            th.start()
    
    synchronusDownload()
    

but it seems like for the initiation of the second download it waits for the
first thread and then goes to download the next file, as printed in shell too.

my plan was to begin all downloads simultaneously and print the updated
progress of the files getting downloaded.

Any help will be greatly appreciated. thanks.


Answer: This is a common problem and here are the steps typically taken:

1.) use Queue.Queue to create a queue of all the urls you would like to visit.

2.) Create a class that inherits from threading.Thread. It should have a run
method that grabs a url from the queue and gets the data.

3.) Create a pool of threads based on your class to be "workers"

4.) Don't exit the program until queue.join() has been completed

