Web-scraping JavaScript page with Python
Question: I'm trying to develop a simple web scraper. I want to extract text without the
HTML code. In fact, I achieve this goal, but I have seen that in some pages
where JavaScript is loaded I didn't obtain good results.

For example, if some JavaScript code adds some text, I can't see it, because
when I call

    
    
    response = urllib2.urlopen(request)
    

I get the original text without the added one (because JavaScript is executed
in the client).

So, I'm looking for some ideas to solve this problem.


Answer: You can also use Python library
[dryscrape](https://github.com/niklasb/dryscrape) to scrape javascript driven
websites.

# Example

To give an example, I created a sample page with following HTML code.
([link](http://avi.im/stuff/js-or-no-js.html)):

    
    
    <!DOCTYPE html>
    <html>
    <head>
      <meta charset="utf-8">
      <title>Javascript scraping test</title>
    </head>
    <body>
      <p id='intro-text'>No javascript support</p>
      <script>
         document.getElementById('intro-text').innerHTML = 'Yay! Supports javascript';
      </script> 
    </body>
    </html>
    

without javascript it says: `No javascript support` and with javascript: `Yay!
Supports javascript`

# Scraping without JS support:

    
    
    >>> import requests
    >>> from bs4 import BeautifulSoup
    >>> response = requests.get(my_url)
    >>> soup = BeautifulSoup(response.text)
    >>> soup.find(id="intro-text")
    <p id="intro-text">No javascript support</p>
    

# Scraping with JS support:

    
    
    >>> import dryscrape
    >>> from bs4 import BeautifulSoup
    >>> session = dryscrape.Session()
    >>> session.visit(my_url)
    >>> response = session.body()
    >>> soup = BeautifulSoup(response)
    >>> soup.find(id="intro-text")
    <p id="intro-text">Yay! Supports javascript</p>
    

