title: scraping multiple pages with scrapy
body: <p>I am trying to use scrapy to scrape a website that has several pages of information.</p>
  
  <p>my code is:</p>
  
  <pre><code>from scrapy.spider import BaseSpider
  from scrapy.selector import Selector
  from tcgplayer1.items import Tcgplayer1Item
  
  
  class MySpider(BaseSpider):
      name = "tcg"
      allowed_domains = ["http://www.tcgplayer.com/"]
      start_urls = ["http://store.tcgplayer.com/magic/journey-into-nyx?PageNumber=1"]
  
      def parse(self, response):
          hxs = Selector(response)
          titles = hxs.xpath("//div[@class='magicCard']")
          for title in titles:
              item = Tcgplayer1Item()
              item["cardname"] = title.xpath(".//li[@class='cardName']/a/text()").extract()[0]
  
              vendor = title.xpath(".//tr[@class='vendor ']")
              item["price"] = vendor.xpath("normalize-space(.//td[@class='price']/text())").extract()
              item["quantity"] = vendor.xpath("normalize-space(.//td[@class='quantity']/text())").extract()
              item["shipping"] = vendor.xpath("normalize-space(.//span[@class='shippingAmount']/text())").extract()
              item["condition"] = vendor.xpath("normalize-space(.//td[@class='condition']/a/text())").extract()
              item["vendors"] = vendor.xpath("normalize-space(.//td[@class='seller']/a/text())").extract()
              yield item
  </code></pre>
  
  <p>I am trying to scrape all the pages until it reaches the end of the pages ... sometimes there will be more pages than others so its hard to say exactly where the page numbers end.</p>