title: Urllib2 Python - Reconnecting and Splitting Response
body: <p>I am moving to Python from other language and I am not sure how to properly tackle this. Using the urllib2 library it is quite easy to set up a proxy and get a data from a site:</p>
  
  <pre><code>import urllib2
  
  req = urllib2.Request('http://www.voidspace.org.uk')
  response = urllib2.urlopen(req)
  the_page = response.read()
  </code></pre>
  
  <p>The problem I have is that the text file that is retrieved is very large (hundreds of MB) and the connection is often problematic. The code also need to catch connection, server and transfer errors (it will be a part of small extensively used pipeline).</p>
  
  <p>Could anyone suggest how to modify the code above to make sure the code automatically reconnects n times (for example 100 times) and perhaps split the response into chunks so the data will be downloaded faster and more reliably? </p>
  
  <p>I have already split the requests as much as I could so now have to make sure that the retrieve code is as good as it can be. Solutions based on core python libraries are ideal. </p>
  
  <p>Perhaps the library is already doing the above in which case is there any way to improve downloading large files? I am using UNIX and need to deal with a proxy. </p>
  
  <p>Thanks for your help. </p>