Speed of urllib.urlretrieve vs urllib.urlopen
Question: I am trying to download SEC filings directly from the SEC ftp server. When I
use `urllib.urlretrieve(url,dst)`, it takes significantly longer than when
doing something like `page = urllib.urlopen(url).read()` followed by
`writeFile.write(page)`. As an example:

    
    
    from time import time
    import urllib
    url = 'ftp://ftp.sec.gov/edgar/data/886475/0001019056-13-000804.txt'
    
    t0 = time()
    urllib.urlretrieve(url,'D:/temp.txt')
    t1 = time()
    t = t1-t0
    print "urllib.urlretrieve time = %s" % t
    
    
    t0 = time()
    writefile = open('D:/temp2.txt','w')
    page = urllib.urlopen(url).read()
    writefile.write(page)
    writefile.close()
    t1 = time()
    t = t1-t0
    print "urllib.urlopen time = %s" % t
    

When I run this, I get 33 seconds for `urllib.urlretrieve` and 2.6 seconds for
the `urllib.urlopen` block. If I watch the D drive, the full ~5.6MB is
downloaded very quickly, but then it hangs for ~30 seconds. What is going on
here? I can proceed with my project using the `urllib.urlopen` method, but
would like to know for future projects. I am running Windows 7 professional
64-bit and this is Python 2.7. Thanks in advance for your help.


Answer: Timing is a funny thing, especially considering the stateless environment of
the web.

While I don't have a smoking gun, I would recommend you take a look at the
[source for
urllib](http://svn.python.org/view/python/tags/r27/Lib/urllib.py?view=markup)
(as of 2.7).

You can see at line 69: `def urlopen`, and line 87: `def urlretrieve`. Both
create a `FancyURLopener()`, but call separate functions within the class.

My best guess is the delay revolves around either:

  1. Windows file handlers, opening, closing, etc.
  2. DNS resolution (Less likely, since the file resolves and downloads with 5.6 seconds as you claim.

You could always hack your `urllib.py` source to print out timings of each
sub-function call, even if only temporarily to trace down the hangup. To
locate where your installation is storing urllib.py, use the following:

    
    
    import urllib
    print urllib.__file__
    

