Scrapy key error
Question: I am having issues creating the Scrapy spider in the Scrapy tutorial:

<http://doc.scrapy.org/en/latest/intro/tutorial.html#our-first-spider>

Here is what I have in my spiders/dmoz_spider.py file:

    
    
    class DmozSpider(object):
      name = "dmoz"
      allowed_domains = ["dmoz.org"]
      start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
      ]
    
      @classmethod
      def from_crawler(cls, crawler):
        spider = crawler.spiders
        return cls(spider)
    
      def parse(self, response):
        filename = response.url.split("/")[-2]
        open(filename, 'wb').write(response.body)
    

The good news is I am pretty sure a spider is getting created. The bad news is
I get this error:

    
    
    (scrapestat)unknownc8e0eb148153:tutorial christopherspears$ scrapy crawl dmoz
    Traceback (most recent call last):
      File "/Users/christopherspears/.virtualenvs/scrapestat/bin/scrapy", line 4, in <module>
        execute()
      File "/Users/christopherspears/.virtualenvs/scrapestat/lib/python2.7/site-packages/scrapy/cmdline.py", line 143, in execute
        _run_print_help(parser, _run_command, cmd, args, opts)
      File "/Users/christopherspears/.virtualenvs/scrapestat/lib/python2.7/site-packages/scrapy/cmdline.py", line 89, in _run_print_help
        func(*a, **kw)
      File "/Users/christopherspears/.virtualenvs/scrapestat/lib/python2.7/site-packages/scrapy/cmdline.py", line 150, in _run_command
        cmd.run(args, opts)
      File "/Users/christopherspears/.virtualenvs/scrapestat/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 48, in run
        spider = crawler.spiders.create(spname, **opts.spargs)
      File "/Users/christopherspears/.virtualenvs/scrapestat/lib/python2.7/site-packages/scrapy/spidermanager.py", line 44, in create
        raise KeyError("Spider not found: %s" % spider_name)
    KeyError: 'Spider not found: dmoz'
    

Not sure what the issue is. Any hints?


Answer: DmozSpider should inherit from BaseSpider (or Spider, depends on your scrapy
version). So, make a following change in your code:

    
    
    from scrapy.spider import BaseSpider
    
    class DmozSpider(BaseSpider):
        ...
    

I tried that by myself and when spider class inherits from object that
KeyError is raised.

