Scrape a Google Chart script with Scraperwiki (Python)
Question: I'm just getting into scraping with Scraperwiki in Python. Already figured out
how to scrape tables from a page, run the scraper every month and save the
results on top of each other. Pretty cool.

Now I want to [scrape this
page](http://developer.android.com/about/dashboards/index.html "this page")
with information on Android versions and run the script monthly. In
particular, I want the table for the version, codename, API and distribution.
It's not easy.

The table is called with a wrapper div. Is there any way to scrape this
information? I can't find any solution.

Plan B is to scrape the visualisation. What I eventually need, is the codename
and the percentage, so that's sufficient. This information can be found in the
HTML in a Google Chart script.

![Google Chart API script](http://i.stack.imgur.com/UdEFd.jpg)

But I can't find this information with my 'souped' HTML. I have [a public
scraper over here](https://scraperwiki.com/scrapers/android_scraper/
"scraper"). You can edit it to make it work.

Can anyone explain how I can approach this problem? A working scraper with
comments on what's going on would be awesome.


Answer: As this is stored and rendered in JavaScript, the raw Python scraper is unable
to execute this code and view the visualisation or table.

ScraperWiki is great however I've always found, if you're doing a single page
each month, a python script + cron is much better and, if you need to have
this JavaScript parsing, using [Selenium](http://docs.seleniumhq.org/) and
it's [python driver](https://pypi.python.org/pypi/selenium) is a much more
powerful solution.

When you have the selenium server installed you can do _roughly_ the following
(in pseudocode)

    
    
    #!/bin/env python
    from selenium import webdriver
    
    browser = webdriver.Firefox() 
    # Load page with all Javascript rendered in the DOM for you.
    browser.get("http://developer.android.com/about/dashboards/index.html") 
    # Find the table
    table = browser.find_element_by_xpath("/html/body/div[3]/div[2]/div/div/div[2]/div/div/table") 
    # Do something with the table element
    # Save the data
    browser.close()
    

Then just have a cron job running the script on the first day of the month
like so:

    
    
    0 0 1 * * /path/to/python_script.py
    

