title: scrapy crawlspider is not following next links
body: <p>I am using scrapy to gather news releases from the Italian national police. The problem I have having is that the scraper is not following the "next" links, even though I have a rule set to find the "next" or in Italian "Successiva" button and follow that link. </p>
  
  <p>Here is my code.</p>
  
  <pre><code>from scrapy.spiders import Rule, CrawlSpider
  from scrapy.linkextractors import LinkExtractor
  from scrapy.selector import Selector
  from items import ScrapyCrimeScraperItem
  from string import replace
  
  class ItalyScraper(CrawlSpider):
  name = 'italy_crawler_test'
  allowed_domains = ['poliziadistato.it']
  
  start_urls = [
      'http://www.poliziadistato.it/archivio/category/1298/2015/',
      'http://www.poliziadistato.it/archivio/category/1298/2015/9/'
  
  ]
  
  rules = (Rule(LinkExtractor(allow=('http://.*/articolo/view/*.....')), callback='parse_article', follow=False),
  Rule(LinkExtractor(restrict_xpaths=("/html/body/div[@class='container'[1]/div[@class='row']/div[@class='col-md-6 col-md-push-3 padding0']/div[@class='trecolonne']/div[@class='center']/div[@class='bar']/ul[@class='paginazione']/li/a[contains(""@title,""'Successiva')]",)), follow=True))
  
  # def generate_article_links(self, response):
  #     for href in response.css('a'):
  #         url = href.extract()
  #         yield scrapy.Request(url, callback=self.parse_article)
  
  
  def parse_article(self, response):
      self.logger.info('Hi, this is an item page! %s', response.url)
      item = ScrapyCrimeScraperItem()
      item['city'] = response.selector.css('h1').extract()[0]
      item['country'] = 'italy'
      item['site_link'] = response.url
      item['article_link'] = response.url
      item['article_raw_text'] = self.remove_carriage_returns(response.selector.css('.resetfont '
                                                       'p').extract(
  
      )[0])
      item['article_raw_date'] = response.selector.css('.data').extract()[0]
      item['article_translated_text'] = ''
      item['article_translated_date'] = ''
      item['article_raw_markup'] = ''
      item['crimes'] = ''
      item['locations'] = ''
      item['dateformat'] = ''
      item['reserved1'] = ''
      item['reserved2'] = ''
  
      yield item
  
  
  def remove_carriage_returns(self,item):
  
      return(item.replace("\n", " "))
  </code></pre>
  
  <p>I looked through some other responses to a similar question, but I am using the explicit <code>follow=True</code> on the second rule. Do I need a callback to generate a new request--or should not the follow variable take care of generating that new request?</p>