Why is requests.get() retrieving different HTML using Python than browser?
Question: I am attempting to extract data from an HTML table, but it appears that the
HTML isn't loading correctly when using `requests.get()`. Instead, a line in
the source reads:

> "JavaScript is not enabled and therefore this page may not function
> correctly."

When I navigate to the page in Google Chrome, the HTML appears as it should.

How do I get a Python script to load the proper HTML?


Answer: Welcome to the wonderful world of web-crawling. The problem you are
experiencing is that `requests.get()` would just get you the initial page that
the browser receives at the beginning of a page load. But, this is not the
page you see in the browser since there could be so much involved in forming
the web page: javascript function calls, AJAX calls etc.

If you want to programmatically get the HTML you see when you click "Show
source" in a web browser after the page was loaded - you would need a real
browser. This is there [`selenium`](http://selenium-python.readthedocs.org/)
could be a good option:

    
    
    from selenium import webdriver
    
    browser = webdriver.Firefox()
    browser.get(url)
    print browser.page_source
    

Note that `selenium` itself is very powerful in terms of [locating
elements](http://selenium-python.readthedocs.org/locating-elements.html) \-
you don't need a separate HTML parser for extracting the data out of the page.

Hope that helps.

