title: Scrapy - parsing all sub-pages of a given domain
body: <p>I would like to parse kickstarter.com projects using scrapy, but can't figure out how to make the spider search projects that I don't explicitly specify under start_urls. I have the first part of the scrapy code figured out (I can extract the necessary information from one website), I just can't get it to do this for all projects under the domain kickstarter.com/projects.</p>
  
  <p>From what I've read, I believe that parsing is possible (1) using links on the starting page (kickstarter.com/projects), (2) using links from one project page to jump to another project, and (3) using a site map (which I don't think kickstarter.com has) to locate webpages to parse.</p>
  
  <p>I've spent hours trying each of these methods but and I am getting nowhere. </p>
  
  <p>I've used the scrapy tutorial code and built on it.</p>
  
  <p>Here is the part so far that works:</p>
  
  <pre><code>from scrapy import log
  from scrapy.contrib.spiders import CrawlSpider   
  from scrapy.selector import HtmlXPathSelector  
  
  from tutorial.items import kickstarteritem
  
  class kickstarter(CrawlSpider):
      name = 'kickstarter'
      allowed_domains = ['kickstarter.com']    
      start_urls = ["http://www.kickstarter.com/projects/brucegoldwell/dragon-keepers-book-iv-fantasy-mystery-magic"]
  
      def parse(self, response):
          x = HtmlXPathSelector(response)
  
          item = kickstarteritem()
          item['url'] = response.url
          item['name'] = x.select("//div[@class='NS-project_-running_board']/h2[@id='title']/a/text()").extract()
          item['launched'] = x.select("//li[@class='posted']/text()").extract()
          item['ended'] = x.select("//li[@class='ends']/text()").extract()
          item['backers'] = x.select("//span[@class='count']/data[@data-format='number']/@data-value").extract()
          item['pledge'] = x.select("//div[@class='num']/@data-pledged").extract()
          item['goal'] = x.select("//div[@class='num']/@data-goal").extract()
          return item
  </code></pre>