title: Scraping recursive page data with Scrapy
body: <p>I am trying to scrape the store location data from the <a href="http://www.subway.co.uk/business/storefinder/store-detail.aspx?id=453056039" rel="nofollow">Subway UK Restaurant Finder</a> using python and Scrapy. I have managed to scrape individual pages, but I would like to set it up to run through a list of, say, 1000 recursive id's at the end of the link. Any help would be appreciated.</p>
  
  <p>Disclaimer: I don't know what I'm doing</p>
  
  <pre><code>from scrapy.spider import BaseSpider
  from scrapy.selector import HtmlXPathSelector
  from subway.items import SubwayFinder
  
  class MySpider(BaseSpider):
  name = "subway"
  allowed_domains = ["http://www.subway.co.uk/"]
  start_urls = ["http://www.subway.co.uk/business/storefinder/store-detail.aspx?id=453056039"]
  
  def parse(self, response):
    hxs = HtmlXPathSelector(response)
    titles = hxs.select("//div[@class='mid']")
    items = []
    for titles in titles:
        item = SubwayFinder()
        item ["title"] = titles.select("p/span/text()").extract()
        items.append(item)
    return items
  </code></pre>