<p>I have a small utility that I use to download an MP3 file from a website on a schedule and then builds/updates a podcast XML file which I've added to iTunes.</p>

<p>The text processing that creates/updates the XML file is written in Python. However, I use wget inside a Windows <code>.bat</code> file to download the actual MP3 file. I would prefer to have the entire utility written in Python.</p>

<p>I struggled to find a way to actually download the file in Python, thus why I resorted to using <code>wget</code>.</p>

<p>So, how do I download the file using Python?</p>

[{'answer_id': 22682, 'author': 'Corey', 'author_id': 1595, 'author_profile': 'https://Stackoverflow.com/users/1595', 'pm_score': 10, 'selected': True, 'text': '<p>Use <a href="https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen" rel="noreferrer"><code>urllib.request.urlopen()</code></a>:</p>\n<pre><code>import urllib.request\nwith urllib.request.urlopen(\'http://www.example.com/\') as f:\n    html = f.read().decode(\'utf-8\')\n</code></pre>\n<p>This is the most basic way to use the library, minus any error handling. You can also do more complex stuff such as changing headers.</p>\n<p>On Python 2, the method is in <a href="http://docs.python.org/2/library/urllib2.html" rel="noreferrer"><code>urllib2</code></a>:</p>\n<pre><code>import urllib2\nresponse = urllib2.urlopen(\'http://www.example.com/\')\nhtml = response.read()\n</code></pre>\n'}, {'answer_id': 22721, 'author': 'Grant', 'author_id': 30, 'author_profile': 'https://Stackoverflow.com/users/30', 'pm_score': 7, 'selected': False, 'text': '<pre class="lang-py prettyprint-override"><code>import urllib2\nmp3file = urllib2.urlopen("http://www.example.com/songs/mp3.mp3")\nwith open(\'test.mp3\',\'wb\') as output:\n  output.write(mp3file.read())\n</code></pre>\n\n<p>The <code>wb</code> in <code>open(\'test.mp3\',\'wb\')</code> opens a file (and erases any existing file) in binary mode so you can save data with it instead of just text.</p>\n'}, {'answer_id': 22723, 'author': 'akdom', 'author_id': 145, 'author_profile': 'https://Stackoverflow.com/users/145', 'pm_score': 4, 'selected': False, 'text': '<p>I agree with Corey, urllib2 is more complete than <a href="http://docs.python.org/lib/module-urllib.html" rel="noreferrer">urllib</a> and should likely be the module used if you want to do more complex things, but to make the answers more complete, urllib is a simpler module if you want just the basics:</p>\n\n<pre><code>import urllib\nresponse = urllib.urlopen(\'http://www.example.com/sound.mp3\')\nmp3 = response.read()\n</code></pre>\n\n<p>Will work fine. Or, if you don\'t want to deal with the "response" object you can call <strong>read()</strong> directly:</p>\n\n<pre><code>import urllib\nmp3 = urllib.urlopen(\'http://www.example.com/sound.mp3\').read()\n</code></pre>\n'}, {'answer_id': 22776, 'author': 'PabloG', 'author_id': 394, 'author_profile': 'https://Stackoverflow.com/users/394', 'pm_score': 10, 'selected': False, 'text': '<p>One more, using <a href="http://docs.python.org/2/library/urllib.html#urllib.urlretrieve" rel="noreferrer"><code>urlretrieve</code></a>:</p>\n<pre><code>import urllib.request\nurllib.request.urlretrieve(&quot;http://www.example.com/songs/mp3.mp3&quot;, &quot;mp3.mp3&quot;)\n</code></pre>\n<p>(for Python 2 use <code>import urllib</code> and <code>urllib.urlretrieve</code>)</p>\n'}, {'answer_id': 10744565, 'author': 'hughdbrown', 'author_id': 10293, 'author_profile': 'https://Stackoverflow.com/users/10293', 'pm_score': 9, 'selected': False, 'text': '<p>In 2012, use the <a href="https://requests.readthedocs.io/" rel="noreferrer">python requests library</a></p>\n\n<pre><code>&gt;&gt;&gt; import requests\n&gt;&gt;&gt; \n&gt;&gt;&gt; url = "http://download.thinkbroadband.com/10MB.zip"\n&gt;&gt;&gt; r = requests.get(url)\n&gt;&gt;&gt; print len(r.content)\n10485760\n</code></pre>\n\n<p>You can run <code>pip install requests</code> to get it.</p>\n\n<p>Requests has many advantages over the alternatives because the API is much simpler. This is especially true if you have to do authentication. urllib and urllib2 are pretty unintuitive and painful in this case.</p>\n\n<hr>\n\n<p>2015-12-30</p>\n\n<p>People have expressed admiration for the progress bar. It\'s cool, sure. There are several off-the-shelf solutions now, including <code>tqdm</code>:</p>\n\n<pre><code>from tqdm import tqdm\nimport requests\n\nurl = "http://download.thinkbroadband.com/10MB.zip"\nresponse = requests.get(url, stream=True)\n\nwith open("10MB", "wb") as handle:\n    for data in tqdm(response.iter_content()):\n        handle.write(data)\n</code></pre>\n\n<p>This is essentially the implementation @kvance described 30 months ago.</p>\n'}, {'answer_id': 16518224, 'author': 'Stan', 'author_id': 2357007, 'author_profile': 'https://Stackoverflow.com/users/2357007', 'pm_score': 5, 'selected': False, 'text': '<p>An improved version of the PabloG code for Python 2/3:</p>\n\n<pre><code>#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nfrom __future__ import ( division, absolute_import, print_function, unicode_literals )\n\nimport sys, os, tempfile, logging\n\nif sys.version_info &gt;= (3,):\n    import urllib.request as urllib2\n    import urllib.parse as urlparse\nelse:\n    import urllib2\n    import urlparse\n\ndef download_file(url, dest=None):\n    """ \n    Download and save a file specified by url to dest directory,\n    """\n    u = urllib2.urlopen(url)\n\n    scheme, netloc, path, query, fragment = urlparse.urlsplit(url)\n    filename = os.path.basename(path)\n    if not filename:\n        filename = \'downloaded.file\'\n    if dest:\n        filename = os.path.join(dest, filename)\n\n    with open(filename, \'wb\') as f:\n        meta = u.info()\n        meta_func = meta.getheaders if hasattr(meta, \'getheaders\') else meta.get_all\n        meta_length = meta_func("Content-Length")\n        file_size = None\n        if meta_length:\n            file_size = int(meta_length[0])\n        print("Downloading: {0} Bytes: {1}".format(url, file_size))\n\n        file_size_dl = 0\n        block_sz = 8192\n        while True:\n            buffer = u.read(block_sz)\n            if not buffer:\n                break\n\n            file_size_dl += len(buffer)\n            f.write(buffer)\n\n            status = "{0:16}".format(file_size_dl)\n            if file_size:\n                status += "   [{0:6.2f}%]".format(file_size_dl * 100 / file_size)\n            status += chr(13)\n            print(status, end="")\n        print()\n\n    return filename\n\nif __name__ == "__main__":  # Only run if this file is called directly\n    print("Testing with 10MB download")\n    url = "http://download.thinkbroadband.com/10MB.zip"\n    filename = download_file(url)\n    print(filename)\n</code></pre>\n'}, {'answer_id': 19011916, 'author': 'anatoly techtonik', 'author_id': 239247, 'author_profile': 'https://Stackoverflow.com/users/239247', 'pm_score': 4, 'selected': False, 'text': '<p>Wrote <a href="https://pypi.python.org/pypi/wget" rel="noreferrer">wget</a> library in pure Python just for this purpose. It is pumped up <code>urlretrieve</code> with <a href="https://bitbucket.org/techtonik/python-wget/src/6859e7b4aba37cef57616111be890fb59631bc4c/wget.py?at=default#cl-330" rel="noreferrer">these features</a> as of version 2.0.</p>\n'}, {'answer_id': 19352848, 'author': 'JD3', 'author_id': 2019895, 'author_profile': 'https://Stackoverflow.com/users/2019895', 'pm_score': 2, 'selected': False, 'text': '<p>This may be a little late, But I saw pabloG\'s code and couldn\'t help adding a os.system(\'cls\') to make it look AWESOME! Check it out : </p>\n\n<pre><code>    import urllib2,os\n\n    url = "http://download.thinkbroadband.com/10MB.zip"\n\n    file_name = url.split(\'/\')[-1]\n    u = urllib2.urlopen(url)\n    f = open(file_name, \'wb\')\n    meta = u.info()\n    file_size = int(meta.getheaders("Content-Length")[0])\n    print "Downloading: %s Bytes: %s" % (file_name, file_size)\n    os.system(\'cls\')\n    file_size_dl = 0\n    block_sz = 8192\n    while True:\n        buffer = u.read(block_sz)\n        if not buffer:\n            break\n\n        file_size_dl += len(buffer)\n        f.write(buffer)\n        status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)\n        status = status + chr(8)*(len(status)+1)\n        print status,\n\n    f.close()\n</code></pre>\n\n<p>If running in an environment other than Windows, you will have to use something other then \'cls\'. In MAC OS X and Linux it should be \'clear\'.</p>\n'}, {'answer_id': 20218209, 'author': 'Zuko', 'author_id': 2114557, 'author_profile': 'https://Stackoverflow.com/users/2114557', 'pm_score': 2, 'selected': False, 'text': '<p>Source code can be:</p>\n\n<pre><code>import urllib\nsock = urllib.urlopen("http://diveintopython.org/")\nhtmlSource = sock.read()                            \nsock.close()                                        \nprint htmlSource  \n</code></pre>\n'}, {'answer_id': 21363808, 'author': 'Marcin Cuprjak', 'author_id': 2454922, 'author_profile': 'https://Stackoverflow.com/users/2454922', 'pm_score': 3, 'selected': False, 'text': '<p>You can get the progress feedback with urlretrieve as well:</p>\n\n<pre><code>def report(blocknr, blocksize, size):\n    current = blocknr*blocksize\n    sys.stdout.write("\\r{0:.2f}%".format(100.0*current/size))\n\ndef downloadFile(url):\n    print "\\n",url\n    fname = url.split(\'/\')[-1]\n    print fname\n    urllib.urlretrieve(url, fname, report)\n</code></pre>\n'}, {'answer_id': 29256384, 'author': 'Sara Santana', 'author_id': 4579638, 'author_profile': 'https://Stackoverflow.com/users/4579638', 'pm_score': 6, 'selected': False, 'text': '<p>use <a href="https://pypi.org/project/wget/" rel="noreferrer">wget module</a>:</p>\n<pre><code>import wget\nwget.download(\'url\')\n</code></pre>\n'}, {'answer_id': 31857152, 'author': 'bmaupin', 'author_id': 399105, 'author_profile': 'https://Stackoverflow.com/users/399105', 'pm_score': 7, 'selected': False, 'text': '<h3>Python 3</h3>\n\n<ul>\n<li><p><a href="https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen" rel="noreferrer"><code>urllib.request.urlopen</code></a></p>\n\n<pre><code>import urllib.request\nresponse = urllib.request.urlopen(\'http://www.example.com/\')\nhtml = response.read()\n</code></pre></li>\n<li><p><a href="https://docs.python.org/3/library/urllib.request.html#urllib.request.urlretrieve" rel="noreferrer"><code>urllib.request.urlretrieve</code></a></p>\n\n<pre><code>import urllib.request\nurllib.request.urlretrieve(\'http://www.example.com/songs/mp3.mp3\', \'mp3.mp3\')\n</code></pre>\n\n<p><strong>Note:</strong> According to the documentation, <code>urllib.request.urlretrieve</code> is a "legacy interface" and "might become deprecated in the future" (thanks <a href="https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python/31857152?noredirect=1#comment107731047_31857152">gerrit</a>)</p></li>\n</ul>\n\n<h3>Python 2</h3>\n\n<ul>\n<li><p><a href="https://docs.python.org/2/library/urllib2.html#urllib2.urlopen" rel="noreferrer"><code>urllib2.urlopen</code></a> (thanks <a href="https://stackoverflow.com/a/22682/399105">Corey</a>)</p>\n\n<pre><code>import urllib2\nresponse = urllib2.urlopen(\'http://www.example.com/\')\nhtml = response.read()\n</code></pre></li>\n<li><p><a href="https://docs.python.org/2/library/urllib.html#urllib.urlretrieve" rel="noreferrer"><code>urllib.urlretrieve</code></a> (thanks <a href="https://stackoverflow.com/a/22776/399105">PabloG</a>)</p>\n\n<pre><code>import urllib\nurllib.urlretrieve(\'http://www.example.com/songs/mp3.mp3\', \'mp3.mp3\')\n</code></pre></li>\n</ul>\n'}, {'answer_id': 33816517, 'author': 'max', 'author_id': 1896222, 'author_profile': 'https://Stackoverflow.com/users/1896222', 'pm_score': 3, 'selected': False, 'text': '<p>If you have wget installed, you can use parallel_sync.</p>\n\n<p>pip install parallel_sync</p>\n\n<pre><code>from parallel_sync import wget\nurls = [\'http://something.png\', \'http://somthing.tar.gz\', \'http://somthing.zip\']\nwget.download(\'/tmp\', urls)\n# or a single file:\nwget.download(\'/tmp\', urls[0], filenames=\'x.zip\', extract=True)\n</code></pre>\n\n<p>Doc:\n<a href="https://pythonhosted.org/parallel_sync/pages/examples.html" rel="noreferrer">https://pythonhosted.org/parallel_sync/pages/examples.html</a></p>\n\n<p>This is pretty powerful. It can download files in parallel, retry upon failure , and it can even download files on a remote machine.</p>\n'}, {'answer_id': 39573536, 'author': 'Jaydev', 'author_id': 4269615, 'author_profile': 'https://Stackoverflow.com/users/4269615', 'pm_score': 4, 'selected': False, 'text': "<p>Following are the most commonly used calls for downloading files in python:</p>\n\n<ol>\n<li><p><code>urllib.urlretrieve ('url_to_file', file_name)</code></p></li>\n<li><p><code>urllib2.urlopen('url_to_file')</code></p></li>\n<li><p><code>requests.get(url)</code></p></li>\n<li><p><code>wget.download('url', file_name)</code></p></li>\n</ol>\n\n<p>Note: <code>urlopen</code> and <code>urlretrieve</code> are found to perform relatively bad with downloading large files (size > 500 MB). <code>requests.get</code> stores the file in-memory until download is complete.  </p>\n"}, {'answer_id': 42764549, 'author': 'Sphynx-HenryAY', 'author_id': 5421147, 'author_profile': 'https://Stackoverflow.com/users/5421147', 'pm_score': 2, 'selected': False, 'text': "<p>urlretrieve and requests.get are simple, however the reality not.\nI have fetched data for couple sites, including text and images, the above two probably solve most of the tasks. but for a more universal solution I suggest the use of urlopen. As it is included in Python 3 standard library, your code could run on any machine that run Python 3 without pre-installing site-package</p>\n\n<pre><code>import urllib.request\nurl_request = urllib.request.Request(url, headers=headers)\nurl_connect = urllib.request.urlopen(url_request)\n\n#remember to open file in bytes mode\nwith open(filename, 'wb') as f:\n    while True:\n        buffer = url_connect.read(buffer_size)\n        if not buffer: break\n\n        #an integer value of size of written data\n        data_wrote = f.write(buffer)\n\n#you could probably use with-open-as manner\nurl_connect.close()\n</code></pre>\n\n<p>This answer provides a solution to HTTP 403 Forbidden when downloading file over http using Python. I have tried only requests and urllib modules, the other module may provide something better, but this is the one I used to solve most of the problems.</p>\n"}, {'answer_id': 43958201, 'author': 'imallett', 'author_id': 688624, 'author_profile': 'https://Stackoverflow.com/users/688624', 'pm_score': 2, 'selected': False, 'text': '<p>I wrote the following, which works in vanilla Python 2 or Python 3.</p>\n\n<hr>\n\n<pre><code>import sys\ntry:\n    import urllib.request\n    python3 = True\nexcept ImportError:\n    import urllib2\n    python3 = False\n\n\ndef progress_callback_simple(downloaded,total):\n    sys.stdout.write(\n        "\\r" +\n        (len(str(total))-len(str(downloaded)))*" " + str(downloaded) + "/%d"%total +\n        " [%3.2f%%]"%(100.0*float(downloaded)/float(total))\n    )\n    sys.stdout.flush()\n\ndef download(srcurl, dstfilepath, progress_callback=None, block_size=8192):\n    def _download_helper(response, out_file, file_size):\n        if progress_callback!=None: progress_callback(0,file_size)\n        if block_size == None:\n            buffer = response.read()\n            out_file.write(buffer)\n\n            if progress_callback!=None: progress_callback(file_size,file_size)\n        else:\n            file_size_dl = 0\n            while True:\n                buffer = response.read(block_size)\n                if not buffer: break\n\n                file_size_dl += len(buffer)\n                out_file.write(buffer)\n\n                if progress_callback!=None: progress_callback(file_size_dl,file_size)\n    with open(dstfilepath,"wb") as out_file:\n        if python3:\n            with urllib.request.urlopen(srcurl) as response:\n                file_size = int(response.getheader("Content-Length"))\n                _download_helper(response,out_file,file_size)\n        else:\n            response = urllib2.urlopen(srcurl)\n            meta = response.info()\n            file_size = int(meta.getheaders("Content-Length")[0])\n            _download_helper(response,out_file,file_size)\n\nimport traceback\ntry:\n    download(\n        "https://geometrian.com/data/programming/projects/glLib/glLib%20Reloaded%200.5.9/0.5.9.zip",\n        "output.zip",\n        progress_callback_simple\n    )\nexcept:\n    traceback.print_exc()\n    input()\n</code></pre>\n\n<hr>\n\n<p>Notes:</p>\n\n<ul>\n<li>Supports a "progress bar" callback.</li>\n<li>Download is a 4 MB test .zip from my website.</li>\n</ul>\n'}, {'answer_id': 44693533, 'author': 'Akif', 'author_id': 950762, 'author_profile': 'https://Stackoverflow.com/users/950762', 'pm_score': 5, 'selected': False, 'text': '<p>Simple yet <code>Python 2 &amp; Python 3</code> compatible way comes with <code>six</code> library:</p>\n\n<pre><code>from six.moves import urllib\nurllib.request.urlretrieve("http://www.example.com/songs/mp3.mp3", "mp3.mp3")\n</code></pre>\n'}, {'answer_id': 47098069, 'author': 'Omer Dagan', 'author_id': 1773706, 'author_profile': 'https://Stackoverflow.com/users/1773706', 'pm_score': 3, 'selected': False, 'text': '<p>If speed matters to you, I made a small performance test for the modules <code>urllib</code> and <code>wget</code>, and regarding <code>wget</code> I tried once with status bar and once without. I took three different 500MB files to test with (different files- to eliminate the chance that there is some caching going on under the hood). Tested on debian machine, with python2.</p>\n\n<p>First, these are the results (they are similar in different runs):</p>\n\n<pre><code>$ python wget_test.py \nurlretrive_test : starting\nurlretrive_test : 6.56\n==============\nwget_no_bar_test : starting\nwget_no_bar_test : 7.20\n==============\nwget_with_bar_test : starting\n100% [......................................................................] 541335552 / 541335552\nwget_with_bar_test : 50.49\n==============\n</code></pre>\n\n<p>The way I performed the test is using "profile" decorator. This is the full code:</p>\n\n<pre><code>import wget\nimport urllib\nimport time\nfrom functools import wraps\n\ndef profile(func):\n    @wraps(func)\n    def inner(*args):\n        print func.__name__, ": starting"\n        start = time.time()\n        ret = func(*args)\n        end = time.time()\n        print func.__name__, ": {:.2f}".format(end - start)\n        return ret\n    return inner\n\nurl1 = \'http://host.com/500a.iso\'\nurl2 = \'http://host.com/500b.iso\'\nurl3 = \'http://host.com/500c.iso\'\n\ndef do_nothing(*args):\n    pass\n\n@profile\ndef urlretrive_test(url):\n    return urllib.urlretrieve(url)\n\n@profile\ndef wget_no_bar_test(url):\n    return wget.download(url, out=\'/tmp/\', bar=do_nothing)\n\n@profile\ndef wget_with_bar_test(url):\n    return wget.download(url, out=\'/tmp/\')\n\nurlretrive_test(url1)\nprint \'==============\'\ntime.sleep(1)\n\nwget_no_bar_test(url2)\nprint \'==============\'\ntime.sleep(1)\n\nwget_with_bar_test(url3)\nprint \'==============\'\ntime.sleep(1)\n</code></pre>\n\n<p><code>urllib</code> seems to be the fastest</p>\n'}, {'answer_id': 48691447, 'author': 'Apurv Agarwal', 'author_id': 6712942, 'author_profile': 'https://Stackoverflow.com/users/6712942', 'pm_score': 4, 'selected': False, 'text': '<p>In python3 you can use urllib3 and shutil libraires.\nDownload them by using pip or pip3 (Depending whether python3 is default or not)</p>\n\n<pre><code>pip3 install urllib3 shutil\n</code></pre>\n\n<p>Then run this code</p>\n\n<pre><code>import urllib.request\nimport shutil\n\nurl = "http://www.somewebsite.com/something.pdf"\noutput_file = "save_this_name.pdf"\nwith urllib.request.urlopen(url) as response, open(output_file, \'wb\') as out_file:\n    shutil.copyfileobj(response, out_file)\n</code></pre>\n\n<p>Note that you download <code>urllib3</code> but use <code>urllib</code> in code</p>\n'}, {'answer_id': 51738373, 'author': 'gzerone', 'author_id': 1070813, 'author_profile': 'https://Stackoverflow.com/users/1070813', 'pm_score': 2, 'selected': False, 'text': '<p>You can use <a href="http://pycurl.io/" rel="nofollow noreferrer">PycURL</a> on Python 2 and 3.</p>\n\n<pre><code>import pycurl\n\nFILE_DEST = \'pycurl.html\'\nFILE_SRC = \'http://pycurl.io/\'\n\nwith open(FILE_DEST, \'wb\') as f:\n    c = pycurl.Curl()\n    c.setopt(c.URL, FILE_SRC)\n    c.setopt(c.WRITEDATA, f)\n    c.perform()\n    c.close()\n</code></pre>\n'}, {'answer_id': 52077420, 'author': 'Robin Dinse', 'author_id': 5096199, 'author_profile': 'https://Stackoverflow.com/users/5096199', 'pm_score': 3, 'selected': False, 'text': '<p>Just for the sake of completeness, it is also possible to call any program for retrieving files using the <code>subprocess</code> package. Programs dedicated to retrieving files are more powerful than Python functions like <code>urlretrieve</code>. For example, <a href="https://www.gnu.org/software/wget/" rel="nofollow noreferrer"><code>wget</code></a> can download directories recursively (<code>-R</code>), can deal with FTP, redirects, HTTP proxies, can avoid re-downloading existing files (<code>-nc</code>), and <a href="https://aria2.github.io/" rel="nofollow noreferrer"><code>aria2</code></a> can do multi-connection downloads which can potentially speed up your downloads.</p>\n\n<pre><code>import subprocess\nsubprocess.check_output([\'wget\', \'-O\', \'example_output_file.html\', \'https://example.com\'])\n</code></pre>\n\n<p>In Jupyter Notebook, one can also call programs directly with the <code>!</code> syntax:</p>\n\n<pre><code>!wget -O example_output_file.html https://example.com\n</code></pre>\n'}, {'answer_id': 53153505, 'author': 'H S Umer farooq', 'author_id': 6454850, 'author_profile': 'https://Stackoverflow.com/users/6454850', 'pm_score': 5, 'selected': False, 'text': '<pre><code>import os,requests\ndef download(url):\n    get_response = requests.get(url,stream=True)\n    file_name  = url.split("/")[-1]\n    with open(file_name, \'wb\') as f:\n        for chunk in get_response.iter_content(chunk_size=1024):\n            if chunk: # filter out keep-alive new chunks\n                f.write(chunk)\n\n\ndownload("https://example.com/example.jpg")\n</code></pre>\n'}, {'answer_id': 60371032, 'author': 'Pedro Lobito', 'author_id': 797495, 'author_profile': 'https://Stackoverflow.com/users/797495', 'pm_score': 3, 'selected': False, 'text': '<p>Late answer, but for <code>python&gt;=3.6</code> you can use:</p>\n\n<pre><code>import dload\ndload.save(url)\n</code></pre>\n\n<hr>\n\n<p>Install <code>dload</code> with:</p>\n\n<pre><code>pip3 install dload\n</code></pre>\n'}, {'answer_id': 60555820, 'author': 'gibbone', 'author_id': 6332373, 'author_profile': 'https://Stackoverflow.com/users/6332373', 'pm_score': 0, 'selected': False, 'text': '<p>I wanted do download all the files from a webpage. I tried <code>wget</code> but it was failing so I decided for the Python route and I found this thread. </p>\n\n<p>After reading it, I have made a little command line application, <a href="https://gist.github.com/gibbbone/acde8ec66ca76f59d70af6ebd8511252" rel="nofollow noreferrer"><code>soupget</code></a>, expanding on the excellent answers of <a href="https://stackoverflow.com/a/22776/6332373">PabloG</a> and <a href="https://stackoverflow.com/a/16518224/6332373">Stan</a> and adding some useful options. </p>\n\n<p>It uses <a href="https://www.crummy.com/software/BeautifulSoup/" rel="nofollow noreferrer">BeatifulSoup</a> to collect all the URLs of the page and then download the ones with the desired extension(s). Finally it can download multiple files in parallel.</p>\n\n<p>Here it is:</p>\n\n<pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\nfrom __future__ import (division, absolute_import, print_function, unicode_literals)\nimport sys, os, argparse\nfrom bs4 import BeautifulSoup\n\n# --- insert Stan\'s script here ---\n# if sys.version_info &gt;= (3,): \n#...\n#...\n# def download_file(url, dest=None): \n#...\n#...\n\n# --- new stuff ---\ndef collect_all_url(page_url, extensions):\n    """\n    Recovers all links in page_url checking for all the desired extensions\n    """\n    conn = urllib2.urlopen(page_url)\n    html = conn.read()\n    soup = BeautifulSoup(html, \'lxml\')\n    links = soup.find_all(\'a\')\n\n    results = []    \n    for tag in links:\n        link = tag.get(\'href\', None)\n        if link is not None: \n            for e in extensions:\n                if e in link:\n                    # Fallback for badly defined links\n                    # checks for missing scheme or netloc\n                    if bool(urlparse.urlparse(link).scheme) and bool(urlparse.urlparse(link).netloc):\n                        results.append(link)\n                    else:\n                        new_url=urlparse.urljoin(page_url,link)                        \n                        results.append(new_url)\n    return results\n\nif __name__ == "__main__":  # Only run if this file is called directly\n    # Command line arguments\n    parser = argparse.ArgumentParser(\n        description=\'Download all files from a webpage.\')\n    parser.add_argument(\n        \'-u\', \'--url\', \n        help=\'Page url to request\')\n    parser.add_argument(\n        \'-e\', \'--ext\', \n        nargs=\'+\',\n        help=\'Extension(s) to find\')    \n    parser.add_argument(\n        \'-d\', \'--dest\', \n        default=None,\n        help=\'Destination where to save the files\')\n    parser.add_argument(\n        \'-p\', \'--par\', \n        action=\'store_true\', default=False, \n        help="Turns on parallel download")\n    args = parser.parse_args()\n\n    # Recover files to download\n    all_links = collect_all_url(args.url, args.ext)\n\n    # Download\n    if not args.par:\n        for l in all_links:\n            try:\n                filename = download_file(l, args.dest)\n                print(l)\n            except Exception as e:\n                print("Error while downloading: {}".format(e))\n    else:\n        from multiprocessing.pool import ThreadPool\n        results = ThreadPool(10).imap_unordered(\n            lambda x: download_file(x, args.dest), all_links)\n        for p in results:\n            print(p)\n</code></pre>\n\n<p>An example of its usage is:</p>\n\n<pre class="lang-sh prettyprint-override"><code>python3 soupget.py -p -e &lt;list of extensions&gt; -d &lt;destination_folder&gt; -u &lt;target_webpage&gt;\n</code></pre>\n\n<p>And an actual example if you want to see it in action:</p>\n\n<pre class="lang-sh prettyprint-override"><code>python3 soupget.py -p -e .xlsx .pdf .csv -u https://healthdata.gov/dataset/chemicals-cosmetics\n</code></pre>\n'}, {'answer_id': 63876738, 'author': 'firebfm', 'author_id': 3290793, 'author_profile': 'https://Stackoverflow.com/users/3290793', 'pm_score': -1, 'selected': False, 'text': "<p>Another way is to call an external process such as curl.exe. Curl by default displays a progress bar, average download speed, time left, and more all formatted neatly in a table.\nPut curl.exe in the same directory as your script</p>\n<pre><code>from subprocess import call\nurl = &quot;&quot;\ncall([&quot;curl&quot;, {url}, '--output', &quot;song.mp3&quot;])\n</code></pre>\n<p>Note: You cannot specify an output path with curl, so do an os.rename afterwards</p>\n"}, {'answer_id': 68224842, 'author': 'Ninja Master', 'author_id': 2693349, 'author_profile': 'https://Stackoverflow.com/users/2693349', 'pm_score': 2, 'selected': False, 'text': '<p>New Api urllib3 based implementation</p>\n<pre><code>&gt;&gt;&gt; import urllib3\n&gt;&gt;&gt; http = urllib3.PoolManager()\n&gt;&gt;&gt; r = http.request(\'GET\', \'your_url_goes_here\')\n&gt;&gt;&gt; r.status\n   200\n&gt;&gt;&gt; r.data\n   *****Response Data****\n</code></pre>\n<p>More info: <a href="https://pypi.org/project/urllib3/" rel="nofollow noreferrer">https://pypi.org/project/urllib3/</a></p>\n'}, {'answer_id': 72255053, 'author': 'uTesla', 'author_id': 13962104, 'author_profile': 'https://Stackoverflow.com/users/13962104', 'pm_score': 2, 'selected': False, 'text': "<p>Use Python Requests in 5 lines</p>\n<pre><code>import requests as req\n\nremote_url = 'http://www.example.com/sound.mp3'\nlocal_file_name = 'sound.mp3'\n\ndata = req.get(remote_url)\n\n# Save file data to local copy\nwith open(local_file_name, 'wb')as file:\n    file.write(data.content)\n</code></pre>\n<p>Now do something with the local copy of the remote file</p>\n"}, {'answer_id': 74097971, 'author': 'thebadgateway', 'author_id': 14927325, 'author_profile': 'https://Stackoverflow.com/users/14927325', 'pm_score': 0, 'selected': False, 'text': '<p>Another possibility is with built-in <a href="https://docs.python.org/3/library/http.client.html" rel="nofollow noreferrer"><code>http.client</code></a>:</p>\n<pre class="lang-py prettyprint-override"><code>from http import HTTPStatus, client\nfrom shutil import copyfileobj\n\n# using https\nconnection = client.HTTPSConnection(&quot;www.example.com&quot;)\nwith connection.request(&quot;GET&quot;, &quot;/noise.mp3&quot;) as response:\n    if response.status == HTTPStatus.OK:\n        copyfileobj(response, open(&quot;noise.mp3&quot;)\n    else:\n        raise Exception(&quot;request needs work&quot;)\n</code></pre>\n<p>The <code>HTTPConnection</code> object is considered “low-level” in that it performs the desired request once and assumes the developer will subclass it or script in a way to handle the nuances of HTTP. Libraries such as <a href="https://requests.readthedocs.io/en/latest/" rel="nofollow noreferrer">requests</a> tend to handle more special cases such as automatically following redirects and so on.</p>\n'}, {'answer_id': 74445704, 'author': 'Mushfirat Mohaimin', 'author_id': 15760624, 'author_profile': 'https://Stackoverflow.com/users/15760624', 'pm_score': 0, 'selected': False, 'text': '<p>You can use <code>keras.utils.get_file</code> to do it:</p>\n<pre><code>from tensorflow import keras\n\npath_to_downloaded_file = keras.utils.get_file(\n    fname=&quot;file name&quot;,\n    origin=&quot;https://www.linktofile.com/link/to/file&quot;,\n    extract=True,\n    archive_format=&quot;zip&quot;,  # downloaded file format\n    cache_dir=&quot;/&quot;,  # cache and extract in current directory\n)\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/22676', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/2109/']