Download file from web in Python 3
Question: I am creating a program that will download a .jar (java) file from a web
server, by reading the URL that is specified in the .jad file of the same
game/application. I'm using Python 3.2.1

I've managed to extract the URL of the JAR file from the JAD file (every JAD
file contains the URL to the JAR file), but as you may imagine, the extracted
value is type() string.

Here's the relevant function:

    
    
    def downloadFile(URL=None):
        import httplib2
        h = httplib2.Http(".cache")
        resp, content = h.request(URL, "GET")
        return content
    
    downloadFile(URL_from_file)
    

However I always get an error saying that the type in the function above has
to be bytes, and not string. I've tried using the URL.encode('utf-8'), and
also bytes(URL,encoding='utf-8'), but I'd always get the same or similar
error.

So basically my question is how to download a file from a server when the URL
is stored in a string type?


Answer: If you want to obtain the contents of a web page into a variable, just `read`
the response of
[**`urllib.request.urlopen`**](http://docs.python.org/dev/library/urllib.request.html#urllib.request.urlopen):

    
    
    import urllib.request
    ...
    url = 'http://example.com/'
    response = urllib.request.urlopen(url)
    data = response.read()      # a `bytes` object
    text = data.decode('utf-8') # a `str`; this step can't be used if data is binary
    

* * *

The easiest way to download and save a file is to use the
[**`urllib.request.urlretrieve`**](http://docs.python.org/dev/library/urllib.request.html#urllib.request.urlretrieve)
function:

    
    
    import urllib.request
    ...
    # Download the file from `url` and save it locally under `file_name`:
    urllib.request.urlretrieve(url, file_name)
    
    
    
    import urllib.request
    ...
    # Download the file from `url`, save it in a temporary directory and get the
    # path to it (e.g. '/tmp/tmpb48zma.txt') in the `file_name` variable:
    file_name, headers = urllib.request.urlretrieve(url)
    

But keep in mind that `urlretrieve` is considered
[legacy](http://docs.python.org/dev/library/urllib.request.html#legacy-
interface) and might become deprecated (not sure why, though).

So the most _correct_ way to do this would be to use the
[**`urllib.request.urlopen`**](http://docs.python.org/dev/library/urllib.request.html#urllib.request.urlopen)
function to return a file-like object that represents an HTTP response and
copy it to a real file using
[**`shutil.copyfileobj`**](http://docs.python.org/dev/library/shutil.html#shutil.copyfileobj).

    
    
    import urllib.request
    import shutil
    ...
    # Download the file from `url` and save it locally under `file_name`:
    with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
        shutil.copyfileobj(response, out_file)
    

If this seems too complicated, you may want to go simpler and store the whole
download in a `bytes` object and then write it to a file. But this works well
only for small files.

    
    
    import urllib.request
    ...
    # Download the file from `url` and save it locally under `file_name`:
    with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
        data = response.read() # a `bytes` object
        out_file.write(data)
    

* * *

It is possible to extract `.gz` (and maybe other formats) compressed data on
the fly, but such an operation probably requires the HTTP server to support
random access to the file.

    
    
    import urllib.request
    import gzip
    ...
    # Read the first 64 bytes of the file inside the .gz archive located at `url`
    url = 'http://example.com/something.gz'
    with urllib.request.urlopen(url) as response:
        with gzip.GzipFile(fileobj=response) as uncompressed:
            file_header = uncompressed.read(64) # a `bytes` object
            # Or do anything shown above using `uncompressed` instead of `response`.
    

