Print code from web page with python and urllib
Question: I'm trying to use python and urllib to look at the code of a certain web page.
I've tried and succeeded this at other webpages using the code:

    
    
    from urllib import *
    url = 
    code = urlopen(url).read()
    print code
    

But it returns nothing at all. My guess is it's because the page has a lot of
javascripts? What to do?


Answer: ### Dynamic client side generated pages (JavaScript)

You can not use urllib alone to see code that been rendered dynamically client
side (JavaScript). The reason is that urllib only fetches the response from
the server which is headers and the body (the actual code). Because of that I
will not execute the client side code.

You can however use something like
[selenium](https://pypi.python.org/pypi/selenium) to remote control a web
browser (Chrome or Firefox). That will make it possible for you to scrap the
page even though it renders with javascript.

Here is a sample of scraping with selenium: [Using python with selenium to
scrape dynamic web pages](http://stackoverflow.com/questions/8650999/using-
python-with-selenium-to-scrape-dynamic-web-pages)

### But that is not your problem here

The problem with this site however seems to be that they don't want to be
scraped. They block clients with certain http user-agent headers.

You can however get the code anyway if you fake the http headers. Use urllib2
instead of urllib like this:

    
    
    import urllib2
    req = urllib2.Request(url)
    req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox')  # Add fake client
    response = urllib2.urlopen(req)
    print response.read()
    

But, they clearly don't want you to scrape their site, so you should consider
if this is a good idea.

