Fetching href of a link
Question: I am using lxml and python. I want to fetch the **href** for the link that
reads **More reviews (‎40)** on this
[page](http://maps.google.com/maps/place?cid=2860002122405830765). I am
basically scrapping this site and want to get the reviews.

Would appreciate the help. Thanx


Answer: The link is added using client-side javascript. So you cannot get the href
using normal HTML parsing. You can however look at the javascript code and get
the link from there:

    
    
    >>> import re
    >>> import urllib2
    >>> import lxml.html
    >>> page = urllib2.urlopen("http://maps.google.com/maps/place?cid=2860002122405830765").read()
    
    # have to search the page source since the link is added in javascript
    >>> mo = re.search(r'<div class="pp-more-reviews">.*?</div>', page)
    >>> div = lxml.html.fromstring(mo.group(0))
    >>> href = div.find("a").attrib["href"]
    

Other options are:

  1. Use [selenium](http://seleniumhq.org/) to control a real browser.
  2. Use the [phantomJS](http://www.phantomjs.org/) headless browser

