Problems crawling wordreference
Question: I am trying to crawl `wordreference`, but I am not succeding.

The first problem I have encountered is, that a big part is loaded via
`JavaScript`, but that shouldn't be much problem because I can see what I need
in the source code.

So, for example, I want to extract for a given word, the first two meanings,
so in this url:
`http://www.wordreference.com/es/translation.asp?tranword=crane` I need to
extract `grulla` and `grúa`.

This is my code:

    
    
    import lxml.html as lh
    import urllib2
    
    url = 'http://www.wordreference.com/es/translation.asp?tranword=crane'
    doc = lh.parse((urllib2.urlopen(url)))
    trans = doc.xpath('//td[@class="ToWrd"]/text()')
    
    for i in trans:
    
        print i
    

The result is that I get an empty list.

I have tried to crawl it with scrapy too, no success. I am not sure what is
going on, the only way I have been able to crawl it is using `curl`, but that
is sloopy, I want to do it in an elegant way, with Python.

Thank you very much


Answer: It looks like you need a `User-Agent` header to be sent, see [Changing user
agent on urllib2.urlopen](http://stackoverflow.com/questions/802134/changing-
user-agent-on-urllib2-urlopen).

Also, just switching to [`requests`](http://docs.python-
requests.org/en/latest/) would do the trick (it automatically sends the
`python-requests/version` User Agent by default):

    
    
    import lxml.html as lh
    import requests
    
    url = 'http://www.wordreference.com/es/translation.asp?tranword=crane'
    
    response = requests.get("http://www.wordreference.com/es/translation.asp?tranword=crane")
    doc = lh.fromstring(response.content)
    
    trans = doc.xpath('//td[@class="ToWrd"]/text()')
    for i in trans:
        print(i)
    

Prints:

    
    
    grulla 
    grúa 
    plataforma 
    ...
    grulla blanca 
    grulla trompetera 
    

