title: Scrapy pagination fails after page 2
body: <p>I'm creating a spider that would crawl every page here: <a href="http://web.archive.org/web/20141217173753/http://www.docstoc.com/documents/legal/" rel="nofollow noreferrer">http://web.archive.org/web/20141217173753/http://www.docstoc.com/documents/legal/</a> and return the card names only. As I expect, it should collect all items from the start page, then follow "Next" pagination link ('BookEnd' class) and repeat until there is no such a link.</p>
  
  <p>What do I need to change to make pagination work properly?</p>
  
  <p>I'm new to web scraping. I've already made this spider work by entering every page manually to <code>start_urls</code>, but I want to make it more automatic.</p>
  
  <pre><code>#!/usr/bin/env python3
  
  import scrapy
  from scrapy.http import Request
  
  class TypeSpider(scrapy.Spider):
      name = "types"
      start_urls = ["https://web.archive.org/web/20141217173745/http://www.docstoc.com/documents/legal"]
  
      def parse(self, response):
          for card1 in response.xpath("//*[@class='doc-title']"):
              text = card1.xpath(".//a/text()").extract_first()
              yield{"Title": text}
          for card2 in response.xpath("//*[@class='col-sm-10']"):
              text = card2.xpath(".//h3/text()").extract_first()
              yield{"Title": text}
          next_page = response.css("li.BookEnd &gt; a::attr(href)").extract_first()
          if next_page is not None:
              next_page = response.urljoin(next_page)
              yield scrapy.Request(url=next_page, callback=self.parse)
  </code></pre>
  
  <p>I expect the spider to crawl all 34 pages, but it quits after page 2 with this:</p>
  
  <p><code>DEBUG: Filtered duplicate request: &lt;GET https://web.archive.org/web/20141217173750/http://www.docstoc.com/documents/legal/2&gt; - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)</code></p>
  
  <p><code>dont_filter</code> doesn't work to me.</p>
  
  <p>P.S. I'm using both xPath and CSS here just because I can't extract the pagination link with xPath – can't figure out why.</p>