title: Dump all network requests and responses in Python
body: <p>How can I use python to dump all of the network requests and responses? What I'm looking to do would compare to the following (this example is in nodejs <a href="https://github.com/ariya/phantomjs/blob/master/examples/netlog.js" rel="nofollow">https://github.com/ariya/phantomjs/blob/master/examples/netlog.js</a>) </p>
  
  <p>I have been trying a tonne of different tools, including the following: </p>
  
  <p>Example:</p>
  
  <pre><code>import requests
  import logging
  
  logging.basicConfig(level=logging.DEBUG)
  r = requests.get('http://www.google.com')
  </code></pre>
  
  <p>Example: </p>
  
  <pre><code>import urllib2   
  
  request = urllib2.Request('http://jigsaw.w3.org/HTTP/300/302.html')
  response = urllib2.urlopen(request)
  print "Response code was: %d" % response.getcode()
  </code></pre>
  
  <p>Example: </p>
  
  <pre><code>import urllib2
  
  passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
  authhandler = urllib2.HTTPBasicAuthHandler(passman)
  handler=urllib2.HTTPHandler(debuglevel=1)
  opener = urllib2.build_opener(handler)
  opener=urllib2.build_opener(authhandler, urllib2.HTTPHandler(debuglevel=1))
  urllib2.install_opener(opener)
  response = urllib2.urlopen('http://groupon.com')
  print response
  </code></pre>
  
  <p>...there are more. </p>
  
  <p>An example of the type of information that I would like to capture is the following (I used fiddler2 to get this information. All of this and more came from visiting groupon.com): </p>
  
  <pre><code>#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
  6   200 HTTP    www.groupon.com /   23,236  private, max-age=0, no-cache, no-store, must-revalidate text/html; charset=utf-8    chrome:6080         
  7   200 HTTP    www.groupon.com /homepage-assets/styles-6fca4e9f48.css  6,766   public, max-age=31369910    text/css; charset=UTF-8 chrome:6080         
  8   200 HTTP    Tunnel to   img.grouponcdn.com:443  0           chrome:6080         
  9   200 HTTP    img.grouponcdn.com  /deal/gsPCLbbqioFVfvjT3qbBZo/The-Omni-Mount-Washington-Resort_01-960x582/v1/c550x332.jpg    94,555  public, max-age=315279127; Expires: Fri, 18 Oct 2024 22:20:20 GMT   image/jpeg  chrome:6080         
  10  200 HTTP    img.grouponcdn.com  /deal/d5YmjhxUBi2mgfCMoriV/pE-700x420/v1/c220x134.jpg   17,832  public, max-age=298601213; Expires: Mon, 08 Apr 2024 21:35:06 GMT   image/jpeg  chrome:6080         
  11  200 HTTP    www.groupon.com /homepage-assets/main-fcfaf867e3.js 9,604   public, max-age=31369913    application/javascript  chrome:6080         
  12  200 HTTP    www.groupon.com /homepage-assets/locale.js?locale=en_US&amp;country=US  1,507   public, max-age=994 application/javascript  chrome:6080         
  13  200 HTTP    www.groupon.com /tracky 3       application/octet-stream    chrome:6080         
  14  200 HTTP    www.groupon.com /cart/widget?consumerId=b577c9c2-4f07-11e4-8305-0025906127fe    17  private, max-age=0, no-cache, no-store, must-revalidate application/json; charset=utf-8 chrome:6080         
  15  200 HTTP    www.googletagmanager.com    /gtm.js?id=GTM-B76Z 39,061  private, max-age=911; Expires: Wed, 22 Oct 2014 20:48:14 GMT    text/javascript; charset=UTF-8  chrome:6080         
  </code></pre>