title: urllib2.urlopen failing while urllib.urlopen working on same URL
body: <p>I am trying to use urllib and urllib2 to scrape some data from a particular website.</p>
  
  <p>Now the urllib was primarily for reading and processing the data while the code section with urllib2 was mainly for reading and storing the data. </p>
  
  <p>The external site experienced some changes and while the urllib code section kept working the urllib2 section simply keeled over.</p>
  
  <p>So I did some checks and noticed the urllib2.urlopen(URL) always returned a blank String while the urllib.urlopen(URL) always worked OK.</p>
  
  <p>I dug deeper and enable debug logging on both urllib and urllib modules:</p>
  
  <pre><code> &gt;&gt;&gt; response2 =urllib2.urlopen('http://www.xxxxxxxxltd.com/web/guest/attendancelist')
  send: 'GET /web/guest/attendancelist HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.xxxxxxxxltd.com\r\nConnection: close\r\nUser-Agent: Python-urllib/2.6\r\n\r\n'
  reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
  header: Server: nginx/0.7.67
  header: Date: Thu, 28 Nov 2013 19:12:28 GMT
  header: Transfer-Encoding: chunked
  header: Connection: close
  header: Location: http://www.xxxxxxxxplc.com/web/guest/attendancelist
  send: 'GET /web/guest/attendancelist HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.xxxxxxxxplc.com\r\nConnection: close\r\nUser-Agent: Python-urllib/2.6\r\n\r\n'
  reply: 'HTTP/1.1 301 Moved Permanently\r\n'
  header: Server: Apache-Coyote/1.1
  header: Location: /home/new/attendancelist.jsp
  header: Content-Length: 0
  header: Date: Thu, 28 Nov 2013 19:12:26 GMT
  header: Connection: close
  send: 'GET /home/new/attendancelist.jsp HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.xxxxxxxxplc.com\r\nConnection: close\r\nUser-Agent: Python-urllib/2.6\r\n\r\n'
  reply: 'HTTP/1.1 200 OK\r\n'
  header: Server: Apache-Coyote/1.1
  header: Set-Cookie: JSESSIONID=F02B1F76CCCF6F41BE48951F6E1A6205; Path=/home
  header: Content-Type: text/html;charset=utf-8
  header: Content-Length: 0
  header: Date: Thu, 28 Nov 2013 19:12:26 GMT
  header: Connection: close
  </code></pre>
  
  <p>And....</p>
  
  <pre><code>&gt;&gt;&gt; html3=urllib.urlopen('http://www.xxxxxxxxltd.com/web/guest/attendancelist')
  send: 'GET /web/guest/attendancelist HTTP/1.0\r\nHost: www.xxxxxxxxltd.com\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'
  reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
  header: Server: nginx/0.7.67
  header: Date: Thu, 28 Nov 2013 19:10:36 GMT
  header: Connection: close
  header: Location: http://www.xxxxxxxxplc.com/web/guest/attendancelist
  send: 'GET /web/guest/attendancelist HTTP/1.0\r\nHost: www.xxxxxxxxplc.com\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'
  reply: 'HTTP/1.1 301 Moved Permanently\r\n'
  header: Server: Apache-Coyote/1.1
  header: Location: /home/new/attendancelist.jsp
  header: Content-Length: 0
  header: Date: Thu, 28 Nov 2013 19:10:34 GMT
  header: Connection: close
  send: 'GET /home/new/attendancelist.jsp HTTP/1.0\r\nHost: www.xxxxxxxxplc.com\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'
  reply: 'HTTP/1.1 200 OK\r\n'
  header: Server: Apache-Coyote/1.1
  header: Set-Cookie: JSESSIONID=8CFB903B80C42CA3DA37EDF90D84FF99; Path=/home
  header: Content-Type: text/html;charset=utf-8
  header: Date: Thu, 28 Nov 2013 19:10:35 GMT
  header: Connection: close
  </code></pre>
  
  <p>As can be identified, the urllib2 connection flow has significantly more Connection headers ( one of which is the <code>Connection</code> header which has its value as <code>Close</code>).</p>
  
  <p>Can anyone assist in finding why the urllib2 fails to retrieve the data while urllib module works well.</p>
  
  <p>I am certain that it has something to do with the Connection headers but I want some sort of confirmation and thinking process explanation.</p>
  
  <p>Thanks.</p>