Scrapy Xpath output empty
Question: I would like extract data on this website : <http://www.pokepedia.fr/Pikachu>
I'm learning the python and how to use Scrapy and my problem is : Why I can't
retrieve the data with Xpath ?

My Xpath look good when i test xpath in my browser, it return me the correct
value. (Google Chrome)

    
    
    import re
    from scrapy import Spider
    from scrapy.selector import Selector
    
    from stack.items import StackItem
    
    
    class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["pokepedia.fr"]
    start_urls = [
        "http://www.pokepedia.fr/Pikachu",
    ]
    
    def unicodize(seg):
        if re.match(r'\\u[0-9a-f]{4}', seg):
            return seg.decode('unicode-escape')
    
        return seg.decode('utf-8')
    
    def parse(self, response):
        pokemon = Selector(response).xpath('//*[@id="mw-content-text"]/table[2]')
    
        for question in pokemon:
            item = StackItem()
            item['title'] = question.xpath(
                '//*[@id="mw-content-text"]/table[2]/tbody/tr[1]/th[2]/text()').extract()[0]
            yield item
    

I want to extract the name of the pokemon in the page but when I use :

    
    
    scrapy crawl stack -o items.json -t json
    

My Json output :

    
    
    [
    

In my console i've this error :

    
    
    IndexError : list index out of range
    

I've followed this tuto : <https://realpython.com/blog/python/web-scraping-
with-scrapy-and-mongodb/>


Answer: Try this

    
    
    question.xpath( '//*[@id="mw-content-text"]/table[2]/tr[1]/th[2]/text()').extract()[0]
    

The browser adds the _tbody_ tags. They are not in the original HTML, that's
why scrapy returns an empty file.

PS: you might want to consider using

    
    
    scrapy shell URL
    

and then using

    
    
    response.xpath('...YOUR SELECTOR..')
    

for debugging and testing.

