title: Getting an exception error when crawling with Scrapy
body: <p>I am starting to test Scrapy in order to crawl a site but when I test my code I get an error which I can't seem to understand how to solve.</p>
  
  <p>Here is the error output:</p>
  
  <pre><code>...
  2012-12-18 02:07:19+0000 [dmoz] DEBUG: Crawled (200) &lt;GET http://MYURL.COM&gt; (referer: None)
  2012-12-18 02:07:19+0000 [dmoz] ERROR: Spider error processing &lt;GET http://MYURL.COM&gt;
      Traceback (most recent call last):
        File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py", line 1178, in mainLoop
          self.runUntilCurrent()
        File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py", line 800, in runUntilCurrent
          call.func(*call.args, **call.kw)
        File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 368, in callback
          self._startRunCallbacks(result)
        File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
          self._runCallbacks()
      --- &lt;exception caught here&gt; ---
        File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 551, in _runCallbacks
          current.result = callback(current.result, *args, **kw)
        File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/scrapy/spider.py", line 57, in parse
          raise NotImplementedError
      exceptions.NotImplementedError: 
  
  2012-12-18 02:07:19+0000 [dmoz] INFO: Closing spider (finished)
  2012-12-18 02:07:19+0000 [dmoz] INFO: Dumping Scrapy stats:
      {'downloader/request_bytes': 357,
       'downloader/request_count': 1,
       'downloader/request_method_count/GET': 1,
       'downloader/response_bytes': 20704,
       'downloader/response_count': 1,
       'downloader/response_status_count/200': 1,
       'finish_reason': 'finished',
       'finish_time': datetime.datetime(2012, 12, 18, 2, 7, 19, 595977),
       'log_count/DEBUG': 7,
       'log_count/ERROR': 1,
       'log_count/INFO': 4,
       'response_received_count': 1,
       'scheduler/dequeued': 1,
       'scheduler/dequeued/memory': 1,
       'scheduler/enqueued': 1,
       'scheduler/enqueued/memory': 1,
       'spider_exceptions/NotImplementedError': 1,
       'start_time': datetime.datetime(2012, 12, 18, 2, 7, 18, 836322)}
  </code></pre>
  
  <p>It looks this might have to do with my <code>parse</code> function and the callback. I tried removing the <code>rule</code> and it worked but just for 1 single URL and what I need is to crawl the whole site.</p>
  
  <p>Here is my code</p>
  
  <pre><code>from scrapy.spider import BaseSpider
  from scrapy.selector import HtmlXPathSelector
  from tutorial.items import DmozItem
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from scrapy.item import Item
  
  
  class DmozSpider(BaseSpider):
      name = "dmoz"
      start_urls = ["http://MYURL.COM"]
      rules = (Rule(SgmlLinkExtractor(allow_domains=('http://MYURL.COM', )), callback='parse_l', follow=True),)
  
  
      def parse_l(self, response):
         hxs = HtmlXPathSelector(response)
         sites = hxs.select('//div[@class=\'content\']')
         items = []
         for site in sites:
             item = DmozItem()
             item['title'] = site.select('//div[@class=\'gig-title-g\']/h1').extract()
             item['link'] = site.select('//ul[@class=\'gig-stats prime\']/li[@class=\'queue \']/div[@class=\'big-txt\']').extract()
             item['desc'] = site.select('//li[@class=\'thumbs\'][1]/div[@class=\'gig-stats-numbers\']/span').extract()
             items.append(item)
         return items 
  </code></pre>
  
  <p>Any tip in the right direction will be appreciated.</p>
  
  <p>Thanks a lot!</p>