Parsing a website with a javascript call using Python
Question: Since I counldn't find an API function in common wikimedia to get alicense of
an image, the only thing left to do it to fetch the webpage and parse it
myself.

For each image, there is a nice popup in wikimedia that lists the
"Attribution" field which I need. For example, in the page
[http://commons.wikimedia.org/wiki/File:Brad_Pitt_Cannes_2011.jpg](http://commons.wikimedia.org/wiki/File%3aBrad_Pitt_Cannes_2011.jpg)
there is a link on the right saying `"Use this file on the web"`. When
clicking on it I can see the "Attribution" field which I need.

Using Python, how can I fetch the webpage and initiate a javascript call to
open that pop up in order to retrieve the text inside the "Attribution" field?

Thanks!

meir


Answer: using unutbu's answer, I converted it to use Selenium
[WebDriver](http://code.google.com/p/selenium/wiki/PythonBindings) (rather
than the older Selenium-RC).

    
    
    import codecs
    import lxml.html as lh
    from selenium import webdriver
    
    browser = webdriver.Firefox()
    browser.get('http://commons.wikimedia.org/wiki/File%3aBrad_Pitt_Cannes_2011.jpg')
    content = browser.page_source
    browser.quit()
    
    doc = lh.fromstring(content)
    for elt in doc.xpath('//span[a[contains(@title,"Use this file")]]/text()'):
        print elt
    

output:

    
    
    on the web
    on a wiki
    

