title: Q: Scrapy: Next pages not crawled but crawler seems to be following links
body: <p>I'm trying to learn python and scrapy but I'm having problems with CrawlSpider.
  The code below works for me. It takes all the links in the start url that matches the xpath - <code>//div[@class="info"]/h3/a/@href</code> then passes those links to the function <em>parse_dir_contents</em>.</p>
  
  <p>What I need now, is to get the crawler to move to the next page. I tried to use rules and linkextractor but I can't seem to make it work properly. I also tried using <code>//a/@href</code> as the xpath for the <em>parse</em> function but it wouldn't pass the links to the <em>parse_dir_contents</em> function. I think I'm missing something REALLY simple. Any ideas?</p>
  
  <pre><code>class ypSpider(CrawlSpider):
  name = "ypTest"
  download_delay = 2
  allowed_domains = ["yellowpages.com"]
  start_urls = ["http://www.yellowpages.com/new-york-ny/restaurants?page=1"]
  
  rules = [
      Rule(LinkExtractor(allow=['restaurants?page=[1-2]']), callback="parse")
  ]
  
  def parse(self, response):
      for href in response.xpath('//div[@class="info"]/h3/a/@href'):
          url = response.urljoin(href.extract())
          if 'mip' in url:
              yield scrapy.Request(url, callback=self.parse_dir_contents)
  
  
  def parse_dir_contents(self, response):
      for sel in response.xpath('//div[@id="mip"]'):
          item = ypItem()
          item['url'] = response.url
          item['business'] = sel.xpath('//div/div/h1/text()').extract()
          ---extra items here---
          yield item
  </code></pre>
  
  <p><strong>Edit:</strong>
  Here's the updated code with three functions and is able to scrape 150 items. I think it's a problem with my rules but I've tried what I think might work, but still same output.</p>
  
  <pre><code>class ypSpider(CrawlSpider):
  name = "ypTest"
  download_delay = 2
  allowed_domains = ["yellowpages.com"]
  start_urls = ["http://www.yellowpages.com/new-york-ny/restaurants?page=1"]
  
  rules = [
      Rule(LinkExtractor(allow=[r'restaurants\?page\=[1-2]']), callback='parse')
  ]
  
  def parse(self, response):
      for href in response.xpath('//a/@href'):
          url = response.urljoin(href.extract())
          if 'restaurants?page=' in url:
              yield scrapy.Request(url, callback=self.parse_links)
  
  
  def parse_links(self, response):
      for href in response.xpath('//div[@class="info"]/h3/a/@href'):
          url = response.urljoin(href.extract())
          if 'mip' in url:
              yield scrapy.Request(url, callback=self.parse_page)
  
  
  def parse_page(self, response):
      for sel in response.xpath('//div[@id="mip"]'):
          item = ypItem()
          item['url'] = response.url
          item['business'] = sel.xpath('//div/div/h1/text()').extract()
          item['phone'] = sel.xpath('//div/div/section/div/div[2]/p[3]/text()').extract()
          item['street'] = sel.xpath('//div/div/section/div/div[2]/p[1]/text()').re(r'(.+)\,')
          item['city'] = sel.xpath('//div/div/section/div/div[2]/p[2]/text()').re(r'(.+)\,')
          item['state'] = sel.xpath('//div/div/section/div/div[2]/p[2]/text()').re(r'\,\s(.+)\s\d')
          item['zip'] = sel.xpath('//div/div/section/div/div[2]/p[2]/text()').re(r'(\d+)')
          item['category'] = sel.xpath('//dd[@class="categories"]/span/a/text()').extract()
          yield item
  </code></pre>