python scrapy - output csv file empty
Question: My main Spider code:

    
    
        from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector
    from Belray_oil.items import BelrayOilItem
    
    class BelraySpider(BaseSpider):
        name = "Belray_oil"
        allowed_domains = ["mxdirtrider.com/"]
        start_urls = ["http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search"]
    
    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        name = hxs.select("//div[@id='product-title']/h1/span/text()").extract()
        MSRP = hxs.select("//div[@id='price']/span[1]/text()").extract()
        Sale = hxs.select("//div[@id='price']/span[2]/strong/text()").extract()
        print name, MSRP, Sale
    

my items file:

    
    
        from scrapy.item import Item, Field
    
    class BelrayOilItem(Item):
        name = Field()
        MSRP = Field()
        Sale = Field()
    

my terminal log output whene I run : scrapy crawl Belray_oil -o items.csv -t
csv

    
    
        2013-07-05 18:03:25-0400 [scrapy] INFO: Scrapy 0.14.4 started (bot: Belray_oil)
    2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
    2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
    2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
    2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled item pipelines: 
    2013-07-05 18:03:25-0400 [Belray_oil] INFO: Spider opened
    2013-07-05 18:03:25-0400 [Belray_oil] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-07-05 18:03:25-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
    2013-07-05 18:03:25-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
    2013-07-05 18:03:26-0400 [Belray_oil] DEBUG: Crawled (200) <GET http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search> (referer: None)
    2013-07-05 18:03:26-0400 [Belray_oil] ERROR: Spider error processing <GET http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search>
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1182, in mainLoop
            self.runUntilCurrent()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 805, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 381, in callback
            self._startRunCallbacks(result)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 576, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/scrapy/spider.py", line 62, in parse
            raise NotImplementedError
        exceptions.NotImplementedError: 
    
    2013-07-05 18:03:26-0400 [Belray_oil] INFO: Closing spider (finished)
    2013-07-05 18:03:26-0400 [Belray_oil] INFO: Dumping spider stats:
        {'downloader/request_bytes': 310,
         'downloader/request_count': 1,
         'downloader/request_method_count/GET': 1,
         'downloader/response_bytes': 13379,
         'downloader/response_count': 1,
         'downloader/response_status_count/200': 1,
         'finish_reason': 'finished',
         'finish_time': datetime.datetime(2013, 7, 5, 22, 3, 26, 204316),
         'scheduler/memory_enqueued': 1,
         'spider_exceptions/NotImplementedError': 1,
         'start_time': datetime.datetime(2013, 7, 5, 22, 3, 25, 970550)}
    2013-07-05 18:03:26-0400 [Belray_oil] INFO: Spider closed (finished)
    2013-07-05 18:03:26-0400 [scrapy] INFO: Dumping global stats:
        {'memusage/max': 116150272, 'memusage/startup': 116150272}
    

The CSV in the output is always empty, and I couldn't figure out what's the
problem exactly. Please little Help!


Answer: You should return an `Item` in your `parse` method:

    
    
    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector
    from Belray_oil.items import BelrayOilItem 
    
    
    class BelraySpider(BaseSpider):
        name = "Belray_oil"
        allowed_domains = ["mxdirtrider.com"]
        start_urls = [
            "http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search"]
    
        def parse(self, response):
            hxs = HtmlXPathSelector(response)
            item = BelrayOilItem()
            item['name'] = hxs.select("//div[@id='product-title']/h1/span/text()").extract()
            item['MSRP'] = hxs.select("//div[@id='price']/span[1]/text()").extract()
            item['Sale'] = hxs.select("//div[@id='price']/span[2]/strong/text()").extract()
            return item
    

Then, in `items.csv` you'll have:

    
    
    name,msrp,sale
    Bel-Ray 2T Mineral Engine 2-Stroke  ,MSRP $9.75,$8.13
    

Hope that helps.

