Dump all network requests and responses in Python
Question: How can I use python to dump all of the network requests and responses? What
I'm looking to do would compare to the following (this example is in nodejs
<https://github.com/ariya/phantomjs/blob/master/examples/netlog.js>)

I have been trying a tonne of different tools, including the following:

Example:

    
    
    import requests
    import logging
    
    logging.basicConfig(level=logging.DEBUG)
    r = requests.get('http://www.google.com')
    

Example:

    
    
    import urllib2   
    
    request = urllib2.Request('http://jigsaw.w3.org/HTTP/300/302.html')
    response = urllib2.urlopen(request)
    print "Response code was: %d" % response.getcode()
    

Example:

    
    
    import urllib2
    
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    authhandler = urllib2.HTTPBasicAuthHandler(passman)
    handler=urllib2.HTTPHandler(debuglevel=1)
    opener = urllib2.build_opener(handler)
    opener=urllib2.build_opener(authhandler, urllib2.HTTPHandler(debuglevel=1))
    urllib2.install_opener(opener)
    response = urllib2.urlopen('http://groupon.com')
    print response
    

...there are more.

An example of the type of information that I would like to capture is the
following (I used fiddler2 to get this information. All of this and more came
from visiting groupon.com):

    
    
    #   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
    6   200 HTTP    www.groupon.com /   23,236  private, max-age=0, no-cache, no-store, must-revalidate text/html; charset=utf-8    chrome:6080         
    7   200 HTTP    www.groupon.com /homepage-assets/styles-6fca4e9f48.css  6,766   public, max-age=31369910    text/css; charset=UTF-8 chrome:6080         
    8   200 HTTP    Tunnel to   img.grouponcdn.com:443  0           chrome:6080         
    9   200 HTTP    img.grouponcdn.com  /deal/gsPCLbbqioFVfvjT3qbBZo/The-Omni-Mount-Washington-Resort_01-960x582/v1/c550x332.jpg    94,555  public, max-age=315279127; Expires: Fri, 18 Oct 2024 22:20:20 GMT   image/jpeg  chrome:6080         
    10  200 HTTP    img.grouponcdn.com  /deal/d5YmjhxUBi2mgfCMoriV/pE-700x420/v1/c220x134.jpg   17,832  public, max-age=298601213; Expires: Mon, 08 Apr 2024 21:35:06 GMT   image/jpeg  chrome:6080         
    11  200 HTTP    www.groupon.com /homepage-assets/main-fcfaf867e3.js 9,604   public, max-age=31369913    application/javascript  chrome:6080         
    12  200 HTTP    www.groupon.com /homepage-assets/locale.js?locale=en_US&country=US  1,507   public, max-age=994 application/javascript  chrome:6080         
    13  200 HTTP    www.groupon.com /tracky 3       application/octet-stream    chrome:6080         
    14  200 HTTP    www.groupon.com /cart/widget?consumerId=b577c9c2-4f07-11e4-8305-0025906127fe    17  private, max-age=0, no-cache, no-store, must-revalidate application/json; charset=utf-8 chrome:6080         
    15  200 HTTP    www.googletagmanager.com    /gtm.js?id=GTM-B76Z 39,061  private, max-age=911; Expires: Wed, 22 Oct 2014 20:48:14 GMT    text/javascript; charset=UTF-8  chrome:6080         
    


Answer: This isn't exactly it, but it's close enough, and yes, it was urllib2:

    
    
    from bs4 import BeautifulSoup
    import requests
    import re
    import csv
    import json
    import time
    import fileinput
    import urllib2
    
    data = urllib2.urlopen("http://stackoverflow.com").read()
    soup = BeautifulSoup(data)
    

The `.read()` returns enough data to be scraped for all the http header's
urls.

