title: Issues with my Crawl Spider and Pagination. Only Value from the First Page being Extracted
body: <p>My task is to take stock updates from one of my suppliers : www.sportsshoes.com .</p>
  
  <p>The issue I am facing is that despite the Crawl Spider visiting each page of a category page it only gives returns data from the first page. This also is the case if I try and scrape each page independently i.e even if I assign it to scrape the third page of the category it only returns results from the first page.</p>
  
  <p>My Code: </p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from scrapy.spider import BaseSpider
  from sportshoes.items import SportshoesItem
  import urlparse 
  from scrapy.http.request import Request
  
  
  class MySpider(CrawlSpider):
    name = "tennis"
    allowed_domains = ["sportsshoes.com"]
    start_urls = ["http://www.sportsshoes.com/products/shoe/tennis/",
                  "http://www.sportsshoes.com/products/shoe/tennis#page=2",
                  "http://www.sportsshoes.com/products/shoe/tennis#page=3"]
  
    rules = (Rule (SgmlLinkExtractor(allow=(),restrict_xpaths=('//div[@class="product-detail"]',))
      , callback="parse_items", follow= True,),)
  
    def parse_items(self, response):
      hxs = HtmlXPathSelector(response)
      titles = hxs.select("//html")
      items = []
      for titles in titles:
        item = SportshoesItem()
        item ["productname"] = titles.select("//h1[@id='product_title']/span/text()").extract()
        item ["Size"] = titles.select('//option[@class="sizeOption"]/text()').extract()
        item ["SKU"] = titles.select("//div[@id='product_ref']/strong/text()").extract()
  
        items.append(item)
        return(items)
  </code></pre>
  
  <p>PS : I had used this method too  :</p>
  
  <pre><code>  rules = (Rule (SgmlLinkExtractor(allow=(),restrict_xpaths=('//div[@class="paginator"]',)), follow= True),
      Rule (SgmlLinkExtractor(restrict_xpaths=('//div[@class="hproduct product"]',))
      , callback="parse_items", follow= True),)
  </code></pre>