Python: Downloading files from HTTP server
Question: I have written some python scripts to download images off an HTTP website, but
because I'm using urllib2, it closes the existing connection and then opens
another before opening another. I don't really understand networking all that
much, but this probably slows things down considerably, and grabbing 100
images at a time would take a considerable amount of time.

I started looking at other alternatives like pycurl or httplib, but found them
complicated to figure out compared to urllib2 and haven't found a lot of code
snippets that I could just take and use.

Simply, how would I establish a persistent connection to a website and
download a number of files and then close the connection only when I am done?
(probably an explicit call to close it)


Answer: since you asked for an httplib snippet:

    
    
    import httplib
    
    images = ['img1.png', 'img2.png', 'img3.png']
    
    conn = httplib.HTTPConnection('www.example.com')
    
    for image in images:
        conn.request('GET', '/images/%s' % image)
        resp = conn.getresponse()
        data = resp.read()
        with open(image, 'wb') as f:
            f.write(data)
    
    conn.close()
    

this would issue multiple (sequential) GET requests for the images in the
list, then close the connection.

