title: Scrapy start_urls
body: <p><a href="https://github.com/scrapy/dirbot/blob/master/dirbot/spiders/dmoz.py" rel="nofollow">The script</a> (below) from <a href="http://doc.scrapy.org/en/latest/intro/tutorial.html" rel="nofollow">this</a> tutorial contains two <code>start_urls</code>. </p>
  
  <pre><code>from scrapy.spider import Spider
  from scrapy.selector import Selector
  
  from dirbot.items import Website
  
  class DmozSpider(Spider):
      name = "dmoz"
      allowed_domains = ["dmoz.org"]
      start_urls = [
          "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
          "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
      ]
  
      def parse(self, response):
          """
          The lines below is a spider contract. For more info see:
          http://doc.scrapy.org/en/latest/topics/contracts.html
          @url http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/
          @scrapes name
          """
          sel = Selector(response)
          sites = sel.xpath('//ul[@class="directory-url"]/li')
          items = []
  
          for site in sites:
              item = Website()
              item['name'] = site.xpath('a/text()').extract()
              item['url'] = site.xpath('a/@href').extract()
              item['description'] = site.xpath('text()').re('-\s[^\n]*\\r')
              items.append(item)
  
          return items
  </code></pre>
  
  <p>But why does it scrape only these 2 web pages? I see  <code>allowed_domains = ["dmoz.org"]</code> but these two pages also contain links to other pages which are within <code>dmoz.org</code> domain! Why doesnt it scrape them too?</p>