title: Scrapy does not crawl through data contained in start URLs
body: <p>I am trying to crawl an entire website using scrapy. As per scarpy's documentation </p>
  
  <blockquote>
    <p>start_urls - A list of URLs where the spider will begin to crawl from,
    when no particular URLs are specified. So, the first pages downloaded
    will be those listed here. The subsequent URLs will be generated
    successively from data contained in the start URLs.</p>
  </blockquote>
  
  <p>So according to this definition scrapy should spider through all the sub-urls on the pages mentioned under <code>start_urls</code> but it only crawls the url that I have specified. I did specify the rule as mentioned under <a href="https://stackoverflow.com/questions/15500281/scrapy-crawl-whole-website">Scrapy - Crawl whole website</a>, but it didn't help. It only scrapes and outputs pages that I specify in start_urls. </p>
  
  <p>Here is a snippet of my code:</p>
  
  <pre><code>class AcdivocaFirstSpider(scrapy.Spider):
      name = "example_sample"
      allowed_domains = ["example.org"]
      start_urls = ["http://www.example.org/site/id/home"]
      rules = rules = [Rule(SgmlLinkExtractor(), callback='parse_item', follow=True)]
  
      def parse(self, response):
          filename = response.url.split("/")[-1] #so eg it would name 'home'
          open(filename, 'wb').write(response.body)
  </code></pre>
  
  <p>This yields a single file with extracted HTML data for 'home' page. How do I get it to recursively crawl the entire website starting from the homepage?</p>
  
  <p>Any help is appreciated. Thank you. </p>