<p>I'm downloading an entire directory from a web server. It works OK, but I can't figure how to get the file size before download to compare if it was updated on the server or not. Can this be done as if I was downloading the file from a FTP server?</p>

<pre><code>import urllib
import re

url = "http://www.someurl.com"

# Download the page locally
f = urllib.urlopen(url)
html = f.read()
f.close()

f = open ("temp.htm", "w")
f.write (html)
f.close()

# List only the .TXT / .ZIP files
fnames = re.findall('^.*&lt;a href="(\w+(?:\.txt|.zip)?)".*$', html, re.MULTILINE)

for fname in fnames:
    print fname, "..."

    f = urllib.urlopen(url + "/" + fname)

    #### Here I want to check the filesize to download or not #### 
    file = f.read()
    f.close()

    f = open (fname, "w")
    f.write (file)
    f.close()
</code></pre>

<hr>

<p>@Jon: thank for your quick answer. It works, but the filesize on the web server is slightly less than the filesize of the downloaded file. </p>

<p>Examples:</p>

<pre><code>Local Size  Server Size
 2.223.533  2.115.516
   664.603    662.121
</code></pre>

<p>It has anything to do with the CR/LF conversion?</p>

[{'answer_id': 5927, 'author': 'Jonathan Works', 'author_id': 309844, 'author_profile': 'https://Stackoverflow.com/users/309844', 'pm_score': 3, 'selected': False, 'text': '<p>The size of the file is sent as the Content-Length header. Here is how to get it with urllib:</p>\n\n<pre><code>&gt;&gt;&gt; site = urllib.urlopen("http://python.org")\n&gt;&gt;&gt; meta = site.info()\n&gt;&gt;&gt; print meta.getheaders("Content-Length")\n[\'16535\']\n&gt;&gt;&gt;\n</code></pre>\n'}, {'answer_id': 5935, 'author': 'dbr', 'author_id': 745, 'author_profile': 'https://Stackoverflow.com/users/745', 'pm_score': 5, 'selected': False, 'text': "<p>Using the returned-urllib-object method <code>info()</code>, you can get various information on the retrieved document. Example of grabbing the current Google logo:</p>\n<pre><code>&gt;&gt;&gt; import urllib\n&gt;&gt;&gt; d = urllib.urlopen(&quot;http://www.google.co.uk/logos/olympics08_opening.gif&quot;)\n&gt;&gt;&gt; print d.info()\n\nContent-Type: image/gif\nLast-Modified: Thu, 07 Aug 2008 16:20:19 GMT  \nExpires: Sun, 17 Jan 2038 19:14:07 GMT \nCache-Control: public \nDate: Fri, 08 Aug 2008 13:40:41 GMT \nServer: gws \nContent-Length: 20172 \nConnection: Close\n</code></pre>\n<p>It's a dict, so to get the size of the file, you do <code>urllibobject.info()['Content-Length']</code></p>\n<pre><code>print f.info()['Content-Length']\n</code></pre>\n<p>And to get the size of the local file (for comparison), you can use the os.stat() command:</p>\n<pre><code>os.stat(&quot;/the/local/file.zip&quot;).st_size\n</code></pre>\n"}, {'answer_id': 5938, 'author': 'Jonathan Works', 'author_id': 309844, 'author_profile': 'https://Stackoverflow.com/users/309844', 'pm_score': 3, 'selected': False, 'text': '<p>Also if the server you are connecting to supports it, look at <a href="http://en.wikipedia.org/wiki/HTTP_ETag" rel="noreferrer">Etags</a> and the <a href="http://en.wikipedia.org/wiki/List_of_HTTP_headers#Requests" rel="noreferrer">If-Modified-Since</a> and <a href="http://en.wikipedia.org/wiki/List_of_HTTP_headers#Requests" rel="noreferrer">If-None-Match</a> headers.</p>\n\n<p>Using these will take advantage of the webserver\'s caching rules and will return a <a href="http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection" rel="noreferrer">304 Not Modified</a> status code if the content hasn\'t changed.</p>\n'}, {'answer_id': 5985, 'author': 'Jonathan Works', 'author_id': 309844, 'author_profile': 'https://Stackoverflow.com/users/309844', 'pm_score': 6, 'selected': True, 'text': '<p>I have reproduced what you are seeing:</p>\n\n<pre><code>import urllib, os\nlink = "http://python.org"\nprint "opening url:", link\nsite = urllib.urlopen(link)\nmeta = site.info()\nprint "Content-Length:", meta.getheaders("Content-Length")[0]\n\nf = open("out.txt", "r")\nprint "File on disk:",len(f.read())\nf.close()\n\n\nf = open("out.txt", "w")\nf.write(site.read())\nsite.close()\nf.close()\n\nf = open("out.txt", "r")\nprint "File on disk after download:",len(f.read())\nf.close()\n\nprint "os.stat().st_size returns:", os.stat("out.txt").st_size\n</code></pre>\n\n<p>Outputs this:</p>\n\n<pre><code>opening url: http://python.org\nContent-Length: 16535\nFile on disk: 16535\nFile on disk after download: 16535\nos.stat().st_size returns: 16861\n</code></pre>\n\n<p>What am I doing wrong here? Is os.stat().st_size not returning the correct size?</p>\n\n<hr>\n\n<p>Edit:\nOK, I figured out what the problem was:</p>\n\n<pre><code>import urllib, os\nlink = "http://python.org"\nprint "opening url:", link\nsite = urllib.urlopen(link)\nmeta = site.info()\nprint "Content-Length:", meta.getheaders("Content-Length")[0]\n\nf = open("out.txt", "rb")\nprint "File on disk:",len(f.read())\nf.close()\n\n\nf = open("out.txt", "wb")\nf.write(site.read())\nsite.close()\nf.close()\n\nf = open("out.txt", "rb")\nprint "File on disk after download:",len(f.read())\nf.close()\n\nprint "os.stat().st_size returns:", os.stat("out.txt").st_size\n</code></pre>\n\n<p>this outputs:</p>\n\n<pre><code>$ python test.py\nopening url: http://python.org\nContent-Length: 16535\nFile on disk: 16535\nFile on disk after download: 16535\nos.stat().st_size returns: 16535\n</code></pre>\n\n<p>Make sure you are opening both files for binary read/write.</p>\n\n<pre><code>// open for binary write\nopen(filename, "wb")\n// open for binary read\nopen(filename, "rb")\n</code></pre>\n'}, {'answer_id': 25502448, 'author': 'Madhusudhan', 'author_id': 3963432, 'author_profile': 'https://Stackoverflow.com/users/3963432', 'pm_score': 3, 'selected': False, 'text': '<p>In Python3:</p>\n\n<pre><code>&gt;&gt;&gt; import urllib.request\n&gt;&gt;&gt; site = urllib.request.urlopen("http://python.org")\n&gt;&gt;&gt; print("FileSize: ", site.length)\n</code></pre>\n'}, {'answer_id': 40957594, 'author': 'ccpizza', 'author_id': 191246, 'author_profile': 'https://Stackoverflow.com/users/191246', 'pm_score': 4, 'selected': False, 'text': '<p>A <a href="http://python-requests.org" rel="noreferrer">requests</a>-based solution using HEAD instead of GET (also prints HTTP headers):</p>\n\n<pre class="lang-python prettyprint-override"><code>#!/usr/bin/python\n# display size of a remote file without downloading\n\nfrom __future__ import print_function\nimport sys\nimport requests\n\n# number of bytes in a megabyte\nMBFACTOR = float(1 &lt;&lt; 20)\n\nresponse = requests.head(sys.argv[1], allow_redirects=True)\n\nprint("\\n".join([(\'{:&lt;40}: {}\'.format(k, v)) for k, v in response.headers.items()]))\nsize = response.headers.get(\'content-length\', 0)\nprint(\'{:&lt;40}: {:.2f} MB\'.format(\'FILE SIZE\', int(size) / MBFACTOR))\n</code></pre>\n\n<h3>Usage</h3>\n\n<blockquote>\n<pre><code>$ python filesize-remote-url.py https://httpbin.org/image/jpeg\n...\nContent-Length                          : 35588\nFILE SIZE (MB)                          : 0.03 MB\n</code></pre>\n</blockquote>\n'}, {'answer_id': 46440161, 'author': 'yukashima huksay', 'author_id': 5120089, 'author_profile': 'https://Stackoverflow.com/users/5120089', 'pm_score': 2, 'selected': False, 'text': "<p>For a python3 (tested on 3.5) approach I'd recommend:</p>\n\n<pre><code>with urlopen(file_url) as in_file, open(local_file_address, 'wb') as out_file:\n    print(in_file.getheader('Content-Length'))\n    out_file.write(response.read())\n</code></pre>\n"}, {'answer_id': 59956757, 'author': 'Hrisimir Dakov', 'author_id': 12198227, 'author_profile': 'https://Stackoverflow.com/users/12198227', 'pm_score': 0, 'selected': False, 'text': "<p>@PabloG Regarding the local/server filesize difference</p>\n<p>Following is high-level illustrative explanation of why it may occur:</p>\n<p>The size on disk sometimes is different from the actual size of the data.\nIt depends on the underlying file-system and how it operates on data.\nAs you may have seen in Windows when formatting a flash drive you are asked to provide 'block/cluster size' and it varies  [512b - 8kb].\nWhen a file is written on the disk, it is stored in a 'sort-of linked list' of disk blocks.\nWhen a certain block is used to store part of a file, no other file contents will be stored in the same blok, so even if the chunk is no occupuing the entire block space, the block is rendered unusable by other files.</p>\n<p>Example:\nWhen the filesystem is divided on 512b blocks, and we need to store 600b file, two blocks will be occupied. The first block will be fully utilized, while the second block will have only 88b utilized and the remaining (512-88)b will be unusable resulting in 'file-size-on-disk' being 1024b.\nThis is why Windows has different notations for 'file size' and 'size on disk'.</p>\n<p>NOTE:\nThere are different pros &amp; cons that come with smaller/bigger FS block, so do a better research before playing with your filesystem.</p>\n"}, {'answer_id': 70382014, 'author': 'Kalob Taulien', 'author_id': 2074077, 'author_profile': 'https://Stackoverflow.com/users/2074077', 'pm_score': 2, 'selected': False, 'text': '<p>For anyone using Python 3 and looking for a quick solution using the <code>requests</code> package:</p>\n<pre class="lang-py prettyprint-override"><code>import requests \nresponse = requests.head( \n    &quot;https://website.com/yourfile.mp4&quot;,  # Example file \n    allow_redirects=True\n)\nprint(response.headers[\'Content-Length\']) \n</code></pre>\n<p>Note: Not all responses will have a <code>Content-Length</code> so your application will want to check to see if it exists.</p>\n<pre class="lang-py prettyprint-override"><code>if \'Content-Length\' in response.headers:\n    ... # Do your stuff here \n</code></pre>\n'}, {'answer_id': 73009100, 'author': 'Harshana', 'author_id': 6952359, 'author_profile': 'https://Stackoverflow.com/users/6952359', 'pm_score': 1, 'selected': False, 'text': '<p>Here is a much more safer way for Python 3:</p>\n<pre class="lang-py prettyprint-override"><code>import urllib.request\nsite = urllib.request.urlopen(&quot;http://python.org&quot;)\nmeta = site.info()\nmeta.get(\'Content-Length\') \n</code></pre>\n<p>Returns:</p>\n<pre><code>\'49829\'\n</code></pre>\n<p><code>meta.get(\'Content-Length\')</code> will return the &quot;Content-Length&quot; header if exists. Otherwise it will be blank</p>\n'}, {'answer_id': 74214042, 'author': 'Eugene Chabanov', 'author_id': 11524604, 'author_profile': 'https://Stackoverflow.com/users/11524604', 'pm_score': 0, 'selected': False, 'text': "<p>Quick and reliable one-liner for Python3 using urllib:</p>\n<pre><code>import urllib\n\nurl = 'https://&lt;your url here&gt;'\n\nsize = urllib.request.urlopen(url).info().get('Content-Length', 0)\n</code></pre>\n<p><code>.get(&lt;dict key&gt;, 0)</code> gets the key from dict and if the key is absent returns 0 (or whatever the 2nd argument is)</p>\n"}, {'answer_id': 74595076, 'author': 'kareem said', 'author_id': 14272080, 'author_profile': 'https://Stackoverflow.com/users/14272080', 'pm_score': 0, 'selected': False, 'text': '<p>you can use requests to pull this data</p>\n<pre><code>\nFile_Name=requests.head(LINK).headers[&quot;X-File-Name&quot;]\n\n#And other useful info** like the size of the file from this dict (headers)\n#like \n\nFile_size=requests.head(LINK).headers[&quot;Content-Length&quot;]\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/5909', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/394/']