title: How to Traverse a next pages using SgmlLinkextractor in Scrapy?
body: <p>I Cannot traverse next page, a bit stuck with Rule of SgmlLinkextractor.
  Please navigate me further, i don't know where i made mistake </p>
  
  <pre><code>import scrapy
  from product.items import ProductItem
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  
  
  class BurlingtoncoatfactoryDSpider(CrawlSpider):
      name = "burlingtoncoatfactory_d"
      allowed_domains = ["http://www.burlingtoncoatfactory.com"]
      start_urls = ['http://www.burlingtoncoatfactory.com/catalog/searchresults.aspx?filter=&amp;search=south+shore+furnitures&amp;viewMode=FortyEight#pg=1']
  
      rules = (
          Rule(SgmlLinkExtractor(allow=('catalog/searchresults\.aspx\?filter=&amp;search=south+shore+furnitures&amp;viewMode=FortyEight#pg=\d*',),),
               callback='parse', follow= True),
  
          )
  
  
      def parse(self, response):
          item=ProductItem()
          item['product_DetailUrl'] = [content for content in   response.xpath('//div[@id="hawkitemlist"]//div[@class="row clearfix"]//div[@class="product-image"]/a/@href').extract()]
          item['product_Rating'] = [content for content in response.xpath('//*//div[@class="product-inner clearfix"]/div[@class="review"]/span[1]/@class').extract()] 
          item['product_NumberOfReviews'] = [content for content in response.xpath('//*//div[@class="product-inner clearfix"]/div[@class="review"]/span[2]/text()').extract()] 
          yield item
  </code></pre>