title: Scrapy Ruel LinkExtractor Next Page
body: <p>I am building a Spider that traverses through several paginated pages and extracts data from the site:
  <a href="http://www.usnews.com/education/best-global-universities/neuroscience-behavior" rel="nofollow">http://www.usnews.com/education/best-global-universities/neuroscience-behavior</a></p>
  
  <p>This is the spider:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  from scrapy.contrib.spiders import Rule
  from scrapy.linkextractors import LinkExtractor
  from lxml import html
  from usnews.items import UsnewsItem
  
  
  class UniversitiesSpider(scrapy.Spider):
      name = "universities"
      allowed_domains = ["usnews.com"]
      start_urls = (
          'http://www.usnews.com/education/best-global-universities/neuroscience-behavior/',
          )
  
      #Rules = [
      #Rule(LinkExtractor(allow=(), restrict_xpaths=('.//a[@class="pager_link"]',)), callback="parse", follow= True)
      #]
  
      def parse(self, response):
          for sel in response.xpath('.//div[@class="sep"]'):
              item = UsnewsItem()
              item['name'] = sel.xpath('.//h2[@class="h-taut"]/a/text()').extract()
              item['location'] = sel.xpath('.//span[@class="t-dim t-small"]/text()').extract()
              item['ranking'] = sel.xpath('.//div[3]/div[2]/text()').extract()
              item['score'] = sel.xpath('.//div[@class="t-large t-strong t-constricted"]/text()').extract()
              #print(sel.xpath('.//text()').extract()
              yield item
  </code></pre>
  
  <p>The rule for traversing the pagination does not seem to do anything as the code just spits out the data for the first site. How can I implement the rule correctly so that the spider traverses through all 15 pages and extracts the 4 items (name, location, ranking, score) from the site?</p>