title: Scrapy doesn't crawl all pages
body: <p>This is my working code:</p>
  
  <pre><code>from scrapy.item import Item, Field
  
  class Test2Item(Item):
      title = Field()
  
  from scrapy.http import Request
  from scrapy.conf import settings
  from scrapy.selector import HtmlXPathSelector
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.contrib.spiders import CrawlSpider, Rule
  
  class Khmer24Spider(CrawlSpider):
      name = 'khmer24'
      allowed_domains = ['www.khmer24.com']
      start_urls = ['http://www.khmer24.com/']
      USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22 AlexaToolbar/alxg-3.1"
      DOWNLOAD_DELAY = 2
  
      rules = (
          Rule(SgmlLinkExtractor(allow=r'ad/.+/67-\d+\.html'), callback='parse_item', follow=True),
      )
  
      def parse_item(self, response):
          hxs = HtmlXPathSelector(response)
          i = Test2Item()
          i['title'] = (hxs.select(('//div[@class="innerbox"]/h1/text()')).extract()[0]).strip(' \t\n\r')
          return i
  </code></pre>
  
  <p>It can scrap only 10 or 15 records. Always random numbers! I can't manage to get all pages that has the pattern like <a href="http://www.khmer24.com/ad/any-words/67-anynumber.html" rel="nofollow">http://www.khmer24.com/ad/any-words/67-anynumber.html</a></p>
  
  <p>I really suspect that Scrapy finished crawling because of duplicate the request. They have suggested to use <code>dont_filter = True</code> however, I have no idea of where to put it in my code. <br><br>I'm a newbie to Scrapy and really need help.</p>