Scrapy tutorial only crawling one page - tried everything in the current SO answer pages
Question: I'm having a problem with the Scrapy tutorial. No matter what solutions I
enter from others' questions about Scrapy only crawling one page that I've
found on the web (including all of them on SO), it only crawls one page. I'm
experienced enough with Python that I'm almost completely sure it's not an
indentation problem. Here is my current code, somewhat modified from the
tutorial to reflect others' solutions that I've found. Note that I've also
tried parse_item, and that didn't work either.

Basically, for my spider I am going to need to start with an .html page so I
did so for my own tutorial purposes. Is this the problem - does it need to be
a directory only?

    
    
    import scrapy
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
    
    class DmozSpider(CrawlSpider):
        name = "dmoz"
        allowed_domains = ["mollydesjardin.com"]
        start_urls = [
            "http://www.mollydesjardin.com/"
        ]
    
        rules = [Rule(SgmlLinkExtractor(allow=()), callback='parse_item')]
    
        def parse(self, response):
            filename = response.url.split("/")[-2]
            with open(filename, 'wb') as f:
                f.write(response.body)
    


Answer: It's a common trap for `CrawlSpider`: you should not override the built-in
`parse()` method (where all `CrawlSpider` magic happens), unless you know what
you're doing.

[The warning is in the
docs](http://doc.scrapy.org/en/latest/topics/spiders.html#crawling-rules) but
many users miss it.

Rename your `parse()` method to `parse_item()` (as you declare in your rules).

Depending on what you need, you may want to add a `follow=True` to your rule,
so that links found in grabbed pages (after the start URL) are also crawled.

