scrapy - how to get text from 'div'
Question: I just started to get to know scrapy. Now I am trying to crawl by following
tutorials. But I have difficulty to crawl text from div.

This is items.py

    
    
    from scrapy.item import Item, Fied
    class DmozItem(Item):
        name = Field()
        title = Field()
        pass
    

This is dmoz_spider.py

    
    
    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector
    from scrapy.item import Item
    
    from dmoz.items import DmozItem
    
    class DmozSpider(BaseSpider):
        name = "dmoz"
        allowed_domains = ["roxie.com"]
        start_urls = ["http://www.roxie.com/events/details.cfm?eventID=4921702B-9E3D-8678-50D614177545A594"]
    
        def parse(self, response):
                hxs = HtmlXPathSelector(response)
                sites = hxs.select('//div[@id="eventdescription"]')
                items = []
                for site in sites:
                        item = DmozItem()
                        item['name'] = hxs.select("text()").extract()
                        items.append(item)
                return items
    

And now I am trying to crawl from top folder by commanding this:

    
    
    scrapy crawl dmoz -o scraped_data.json -t json
    

But the file was created with only '['.

It perfectly works in the console(by commanding each select), but somehow it
doesn't work as a script. I am really a starter of scrapy. Could you guys let
me know how can I get the data in 'div'? Thanks in advance.

_*_ In addition, this is what I get.

    
    
    2013-06-19 08:43:56-0700 [scrapy] INFO: Scrapy 0.16.5 started (bot: dmoz)
    /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/web/microdom.py:181: SyntaxWarning: assertion is always true, perhaps remove parentheses?
    assert (oldChild.parentNode is self,
    2013-06-19 08:43:56-0700 [scrapy] DEBUG: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
    2013-06-19 08:43:56-0700 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
    2013-06-19 08:43:56-0700 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
    2013-06-19 08:43:56-0700 [scrapy] DEBUG: Enabled item pipelines: 
    2013-06-19 08:43:56-0700 [dmoz] INFO: Spider opened
    2013-06-19 08:43:56-0700 [dmoz] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-06-19 08:43:56-0700 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
    2013-06-19 08:43:56-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
    2013-06-19 08:43:56-0700 [dmoz] DEBUG: Crawled (200) <GET http://www.roxie.com/events/details.cfm?eventID=4921702B-9E3D-8678-50D614177545A594> (referer: None)
    2013-06-19 08:43:56-0700 [dmoz] INFO: Closing spider (finished)
    2013-06-19 08:43:56-0700 [dmoz] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 281,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 27451,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2013, 6, 19, 15, 43, 56, 569164),
     'log_count/DEBUG': 7,
     'log_count/INFO': 4,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2013, 6, 19, 15, 43, 56, 417661)}
    2013-06-19 08:43:56-0700 [dmoz] INFO: Spider closed (finished)
    


Answer: I can get you the title of the film, but I'm somewhat crappy with XPath so the
description XPath will get you _everything_ within the `<div class="tabbertab"
title="Synopsis">` element. It's not ideal, but it's a starting point. Getting
the image URL is left as an exercise for the OP. :)

    
    
    from scrapy.item import Item, Fied
    class DmozItem(Item):
        title = Field()
        description = Field()
        pass
    
    class DmozSpider(BaseSpider):
        name = "test"
        allowed_domains = ["roxie.com"]
        start_urls = ["http://www.roxie.com/events/details.cfm?eventID=4921702B-9E3D-8678-50D614177545A594"]
    
        def parse(self, response):
            hxs = HtmlXPathSelector(response)
            item = DmozItem()
            item['title'] = hxs.select('//div[@style="width: 100%;"]/text()').extract()
            item['description'] = hxs.select('//div[@class="tabbertab"]').extract()
            return item
    

