Python web scraping pauses
Question: I have the following code:

    
    
    #!/usr/bin/env python
    from mechanize import Browser
    from BeautifulSoup import BeautifulSoup
    
    mech = Browser()
    mech.set_handle_robots(False)
    url = "http://storage.googleapis.com/patents/retro/2011/ad20111231-02.zip"
    page = mech.open(url)
    html = page.read()
    
    soup = BeautifulSoup(html)
    print soup.prettify()
    

really simple web scraper trying to download a .zip file from a web page. When
I run this code, and bearing in mind this file is 4kb, the program just does
not finish, as if it is in an infinite while loop. what I have done here?


Answer: Try putting `print html` after you do your `page.read()`. You may not be
getting what you think you are, it sounds like you're receiving an error page
rather than the file itself. I'm not sure if you're even handling the file
correctly, you might find a better approach here: [Download all the
links(related documents) on a webpage using
Python](http://stackoverflow.com/questions/5974595/download-all-the-
linksrelated-documents-on-a-webpage-using-python).

The zip file isn't 4KB, btw, it's ~87MB and contains a _784MB_ XML file, which
you should be able to confirm by hitting that URL in a browser and downloading
it. It may not be an infinite loop that's the problem, it's just taking a long
time to load.

You're also trying to pass the data in as HTML when it's zip-archived XML. If
(once you actually _have_ the file) you store the response data in a
`StringIO`, you'll be able to unzip it in memory ([as outlined
here](http://stackoverflow.com/questions/10908877/extracting-a-zipfile-to-
memory)). You will then need to [explicitly tell
`BeautifulSoup`](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#id11)
that you're passing it XML.

    
    
    soup = BeautifulSoup(html, 'xml')
    

This will require you to [install
lxml](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#parser-
installation), but that will work out to your advantage, as it's possibly the
fastest XML parser under Python.

One last thing:

    
    
    mech.set_handle_robots(False)
    url = "http://storage.googleapis.com/patents/retro/2011/ad20111231-02.zip"
    

I was under the impression Google set up their `robots.txt` to disallow
scraping as much as possible. If you're still unable to even download a copy
of the file, I'd recommend trying `Selenium`; it's a lot like `mechanize` but
controls actual browsers, like Chrome & Firefox, so it will be a legitimate
browser request.

