Getting number of Google hits for a larger list of words
Question: I saw some relevant questions for my problem, but no specific answer. In
brief, I have a larger list of words (more than 1000), and I would like to get
number of Google hits for each word. In particular, I read this tread at
[Stackoverflow: Google search to retrieve number of results for search
keywords](http://stackoverflow.com/questions/4785833/google-search-to-
retrieve-number-of-results-for-search-keywords). But the question of how to
handle large list is still open. Please, I would really appreciate if anyone
can trow some piece of Python code whit which I could play and build script.


Answer: You might be referring to this comment on the aforementioned question:

> If Google rejects your request, you could try scraping the search results
> page...

I would strongly recommend _not_ doing that, especially if you have huge
numbers of words to process. However, for _instructive_ purposes, this is the
code that would ordinarily work for you:

    
    
    import urllib2
    import re
    
    def results(word):
        text = urllib2.urlopen('http://www.google.com/search?q=%s'%word).read()
        m = re.search('About ([0-9,]+) results', text)
        if m is None:
            return None
        else:
            return int(m.group(1).replace(',', ''))   # remove commas and int-ify
    

I say "ordinarily" because, in my testing, instead of a search results page, I
received a polite request from Google:

> **403.** That’s an error.
>
> Your client does not have permission to get URL `/search?q=foo` from this
> server.

I determined that Google knows to reject my request by looking at the `User-
Agent` in the request header. It's simple to spoof the `User-Agent` and make
this code work, but again, please don't.

Another consideration is that Randall Munroe of xkcd fame has suggested the
hit counts on the page are wildly inaccurate:
<http://blog.xkcd.com/2011/02/04/trochee-chart/>

