title: how to get scrapy go to the next page?
body: <p>I'm triyng to build scraper with Scapy.
  I don't undestand why Scrapy doesn't want to go to the next page. I thouht to extract link from pagination area..but, alas.
  My rule for extracting urls for going to the next page</p>
  
  <pre><code>Rule(LinkExtractor(restrict_xpaths='/html/body/div[19]/div[5]/div[2]/div[5]/div/div[3]/ul',allow=('page=[0-9]*')), follow=True)
  </code></pre>
  
  <p>Crawler</p>
  
  <pre><code>Class DmozSpider(CrawlSpider):
      name = "arbal"
      allowed_domains = ["bigbasket.com"]
      start_urls = [
          "http://bigbasket.com/pc/bread-dairy-eggs/bread-bakery/?nc=cs"
      ]
  
      rules = (
               Rule(LinkExtractor(restrict_xpaths='/html/body/div[19]/div[4]/ul',allow=('pc\/.*.\?nc=cs')), follow=True),
               Rule(LinkExtractor(restrict_xpaths='/html/body/div[19]/div[5]/div[2]/div[5]/div/div[3]/ul',allow=('page=[0-9]*')), follow=True),
               Rule(LinkExtractor(restrict_xpaths='//*[@id="products-container"]',allow=('pd\/*.+')), callback='parse_item', follow=True)
               )
  
      def parse_item(self, response):
          item = AlabaItem()
          hxs = HtmlXPathSelector(response)
          item['brand_name'] = hxs.select('.//*[contains(@id, "slidingProduct")]/div[2]/div[1]/a/text()').extract()
          item['prod_name'] =  hxs.select('//*[contains(@id, "slidingProduct")]/div[2]/div[2]/h1/text()').extract()
          yield item
  </code></pre>