title: Fetching a URL from a basic-auth protected Jenkins server with urllib2
body: <p>I'm trying to fetch a URL from a Jekins server. Until somewhat recently I was able to use the pattern described on this page (<a href="http://docs.python.org/2/howto/urllib2.html" rel="nofollow">HOWTO Fetch Internet Resources Using urllib2</a>) to create a password-manager that correctly responded to BasicAuth challenges with the user-name &amp; password. All was fine until the Jenkins team <a href="https://wiki.jenkins-ci.org/display/JENKINS/Authenticating+scripted+clients" rel="nofollow">changed their security model</a>, and that code no longer worked.</p>
  
  <pre><code># DOES NOT WORK!
  import urllib2
  password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
  top_level_url = "http://localhost:8080"
  
  password_mgr.add_password(None, top_level_url, 'sal', 'foobar')
  handler = urllib2.HTTPBasicAuthHandler(password_mgr)
  opener = urllib2.build_opener(handler)
  
  a_url = 'http://localhost:8080/job/foo/4/api/python'
  print opener.open(a_url).read()
  </code></pre>
  
  <p>Stacktrace:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/sal/workspace/jenkinsapi/src/examples/password.py", line 11, in &lt;module&gt;
      print opener.open(a_url).read()
    File "/usr/lib/python2.7/urllib2.py", line 410, in open
      response = meth(req, response)
    File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
      'http', request, response, code, msg, hdrs)
    File "/usr/lib/python2.7/urllib2.py", line 448, in error
      return self._call_chain(*args)
    File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
      result = func(*args)
    File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
      raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
  urllib2.HTTPError: HTTP Error 403: Forbidden
  [Finished in 0.0s with exit code 1]
  </code></pre>
  
  <p>The problem appears to be that Jenkins returns not with the expected 401 code, but a 403 which urllib2 interprets as an end of conversation. It never actually sends the password. After some surfing around github found another developer's solution which works...</p>
  
  <pre><code># WORKS... SORTA
  def auth_headers(username, password):
     return 'Basic ' + base64.encodestring('%s:%s' % (username, password))[:-1]
  
  auth = auth_headers('sal', 'foobar')
  top_level_url = "http://localhost:8080"
  a_url = 'http://localhost:8080/job/foo/4/api/python'
  req = urllib2.Request(a_url)
  req.add_header('Authorization', auth)
  print urllib2.urlopen(req).read()
  </code></pre>
  
  <p>But that seems rather unsatisfying. It's not bothering to check whether the domain is relevant to the username and password... it's just sending my login details regardless!</p>
  
  <p>Can anybody suggest a way to make the original script work? I'd like to use a urllib2 password manager in such a way that I can login to Jenkins.</p>