title: HTTP Basic Authentication Doesn't Seem to Work with urllib2 in python
body: <p>I'm trying to download a page protected with basic authentication using urllib2. I'm using python 2.7 but I also tried it on another computer with python 2.5 and encountered the exact same behavior. I followed the example given in <a href="http://www.voidspace.org.uk/python/articles/authentication.shtml">this guide</a> as closely as I could and here is the code that I produced:</p>
  
  <pre><code>import urllib2
  
  passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
  passman.add_password(None, "http://authenticationsite.com/', "protected", "password")
  authhandler = urllib2.HTTPBasicAuthHandler(passman)
  opener = urllib2.build_opener(authhandler)
  
  f = opener.open("http://authenticationsite.com/content.html")
  print f.read()
  f.close()
  </code></pre>
  
  <p>Unfortunately the server isn't mine so I can't share the details; I swapped them out above and below. When I run it I get the following Traceback:</p>
  
  <pre><code>  File
  "/usr/lib/python2.7/urllib2.py", line
  397, in open
  response = meth(req, response)   File "/usr/lib/python2.7/urllib2.py",
  line 510, in http_response
  'http', request, response, code, msg, hdrs)   File
  "/usr/lib/python2.7/urllib2.py", line
  435, in error
  return self._call_chain(*args)   File "/usr/lib/python2.7/urllib2.py",
  line 369, in _call_chain
  result = func(*args)   File "/usr/lib/python2.7/urllib2.py", line
  518, in http_error_default
  raise HTTPError(req.get_full_url(), code,
  msg, hdrs, fp) urllib2.HTTPError: HTTP
  Error 401: Authorization Required
  </code></pre>
  
  <p>Now, the interesting part is when I monitor the tcp traffic on the computer using ngrep:</p>
  
  <pre><code>ngrep host 74.125.224.49 interface:
  wlan0 (192.168.1.0/255.255.255.0)
  filter: (ip) and ( host 74.125.224.49
  )
  #### T 192.168.1.74:34366 -74.125.224.49:80 [AP]   GET /content.html
  HTTP/1.1..Accept-Encoding:
  identity..Host:
  authenticationsite.com..Connection:
  close..User-Agent:
  Python-urllib/2.7.... 
  
  ## T 74.125.224.49:80 -192.168.1.74:34366 [AP]   HTTP/1.1 401 Authorization Required..Date: Sun, 27
  Feb 2011 03:39:31 GMT..Server:
  Apache/2.2.3 (Red
  Hat)..WWW-Authenticate: Digest
  realm="protected",
  nonce="6NSgTzudBAA=ac585d1f7ae0632c4b90324aff5e39e0f1fc25
  05", algorithm=MD5,
  qop="auth"..Content-Length:
  486..Connection: close..Content-Type: text/html;
  charset=iso-8859-1....&lt;!DOCTYPE HTML
  PUBLIC "-//IETF//DTD HTML
  2.0//EN"&gt;.&lt;html&gt;&lt;head&gt;.&lt;title&gt;401 Authorization   
  Required&lt;/title&gt;.&lt;/head&gt;&lt;body&gt;.&lt;h1&gt;Authorization
  Required&lt;/h1&gt;.&lt;p&gt;This server could not
  verify that you.are authorized to
  access the document.requested.  Either
  you supplied the wrong.credentials
  (e.g., badpassword), or
  your.browser doesn't understand how to
  supply.the credentials
  required.&lt;/p&gt;.&lt;hr&gt;.&lt;address&gt;Apache/2.2.3
  (Red Hat) Server at
  authenticationsite.com Port
  80&lt;/address&gt;.&lt;/body&gt;&lt;/html&gt;.  
  
  ####
  </code></pre>
  
  <p>It appears as though urllib2 is throwing that exception without even attempting to supply the credentials after getting the initial 401 error.</p>
  
  <p>For comparion, here is the output of ngrep when I do the authentication in a web browser instead:</p>
  
  <pre><code>ngrep host 74.125.224.49 interface:
  wlan0 (192.168.1.0/255.255.255.0)
  filter: (ip) and ( host 74.125.224.49
  )
  #### T 192.168.1.74:36102 -74.125.224.49:80 [AP]   GET /content.html HTTP/1.1..Host:
  authenticationsite.com..User-Agent:
  Mozilla/5.0 (X11; U; Linux i686;
  en-US; rv:1.9.2.12) Gecko/20101027
  Firefox/3.6.12..Accept: text  
  /html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..Accept-Language:
  en-us,en;q=0.5..Accept-Encoding:
  gzip,deflate..Accept-Charset:
  ISO-8859-1,utf-8;q=0.7,*;q=0.7..Keep-Alive:
  115..Connection: keep-   alive....  
  ## T 74.125.224.49:80 -192.168.1.74:36102 [AP]   HTTP/1.1 401 Authorization Required..Date: Sun, 27
  Feb 2011 03:43:42 GMT..Server:
  Apache/2.2.3 (Red
  Hat)..WWW-Authenticate: Digest
  realm="protected",
  nonce="rKCfXjudBAA=0c1111321169e30f689520321dbcce37a1876b
  be", algorithm=MD5,
  qop="auth"..Content-Length:
  486..Connection: close..Content-Type: text/html;
  charset=iso-8859-1....&lt;!DOCTYPE HTML
  PUBLIC "-//IETF//DTD HTML
  2.0//EN"&gt;.&lt;html&gt;&lt;head&gt;.&lt;title&gt;401 Authorization   
  Required&lt;/title&gt;.&lt;/head&gt;&lt;body&gt;.&lt;h1&gt;Authorization
  Required&lt;/h1&gt;.&lt;p&gt;This server could not
  verify that you.are authorized to
  access the document.requested.  Either
  you supplied the wrong.credentials
  (e.g., badpassword), or
  your.browser doesn't understand how to
  supply.the credentials
  required.&lt;/p&gt;.&lt;hr&gt;.&lt;address&gt;Apache/2.2.3
  (Red Hat) Server at
  authenticationsite.com Port
  80&lt;/address&gt;.&lt;/body&gt;&lt;/html&gt;.  
  
  ######### T 192.168.1.74:36103 -74.125.224.49:80 [AP]   GET /content.html HTTP/1.1..Host:
  authenticationsite.com..User-Agent:
  Mozilla/5.0 (X11; U; Linux i686;
  en-US; rv:1.9.2.12) Gecko/20101027
  Firefox/3.6.12..Accept: text  
  /html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..Accept-Language:
  en-us,en;q=0.5..Accept-Encoding:
  gzip,deflate..Accept-Charset:
  ISO-8859-1,utf-8;q=0.7,*;q=0.7..Keep-Alive:
  115..Connection: keep-   alive..Authorization: Digest
  username="protected",
  realm="protected",
  nonce="rKCfXjudBAA=0c1111199162342689520550dbcce37a1876bbe",
  uri="/content.html", algorithm=   MD5,
  response="3b65dadaa00e1d6a1892ffff49f9f325",
  qop=auth, nc=00000001,
  cnonce="7636125b7fde3d1b".... 
  
  ##
  </code></pre>
  
  <p>And then followed with the content of the site.</p>
  
  <p>I've been playing around with this for a while and am not able to figure out what I'm doing wrong. I would be very thankful if somebody can help me out!</p>