Obtaining loaded images from a given URL via Python
Question: Is there anyway to load a URL via Python and then retrieve a list of all of
the images that were loaded via that URL? I'm essentially looking to do
something similar to TamperData or Fiddler and retrieve a list of all images
that a given website loaded.


Answer: Interesting task. Here's one way of solving it, along the line suggested by
Jochen Ritzel.

It uses [pylibpcap](http://pylibpcap.sourceforge.net/) instead of
[pycap](http://pycap.sourceforge.net/). Personally, I find _pycap_ to be hard
to work with due to little amount of documentation available. For _pylibpcap_
, you can translate most code directly from the _libpcap_ examples (see for
example [this tutorial](http://www.tcpdump.org/pcap.html) for a nice
introduction). The man pages for [tcpdump](http://linux.die.net/man/8/tcpdump)
and [pcap](http://www.tcpdump.org/pcap3_man.html) are also great resources.

You may want to look at the standards for
[Ethernet](https://secure.wikimedia.org/wikipedia/en/wiki/Ethernet_frame),
[IPv4](https://secure.wikimedia.org/wikipedia/en/wiki/IPv4),
[TCP](https://secure.wikimedia.org/wikipedia/en/wiki/Tcp_protocol), and
[HTTP](https://secure.wikimedia.org/wikipedia/en/wiki/Http).

Note 1: The code below only prints out the HTTP _GET_ requests. Filtering out
the images and downloading them using the [urllib
module](http://docs.python.org/library/urllib.html) should pose no problem.

Note 2: This code works on Linux, not sure what device names you need to use
on Windows/MacOS. You'll also need root privileges.

    
    
    #!/usr/bin/env python
    
    import pcap
    import struct
    
    def parse_packet(data):
        """
        Parse Ethernet/IP/TCP packet.
        """
        # See the Ethernet, IP, and TCP standards for details.
    
        data = data[14:] # Strip Ethernet header
    
        header_length = 4 * (ord(data[0]) & 0x0f) # in bytes
        data = data[header_length:]  # Strip IP header
    
        dest_port = struct.unpack('!H', data[2:4])[0]
        if not dest_port == 80: # This is an outgoing package
            return
    
        header_length = 4 * ((ord(data[12]) & 0xf0) >> 4) # in bytes
        data = data[header_length:] # Strip TCP header
    
        return data
    
    
    def parse_get(data):
        """
        Parse a HTTP GET request, returning the request URI.
        """
        if data is None or not data.startswith('GET'):
            return
    
        fields = data.split('\n')
        uri = fields[0].split()[1]
    
        for field in fields[1:]:
            if field.lower().startswith('host:'):
                return field[5:].strip() + uri
    
    
    def packet_handler(length, data, timestamp):
        uri = parse_get(parse_packet(data))
        if not uri is None:
            print uri
    
    
    # Set up pcap sniffer
    INTERFACE = 'wlan0'
    FILTER = 'tcp port 80'
    p = pcap.pcapObject()
    p.open_live(INTERFACE, 1600, 0, 100)
    p.setfilter(FILTER, 0, 0)
    
    try:
        while True:
            p.dispatch(1, packet_handler)
    except KeyboardInterrupt:
        pass
    

