title: python/scrapy question: How to avoid endless loops
body: <p>I am using the web-scraping framework, scrapy, to data mine some sites. I am trying to use the CrawlSpider and the pages have a 'back' and 'next' button. The URLs are in the format</p>
  
  <pre><code>www.qwerty.com/###
  </code></pre>
  
  <p>where ### is a number that increments every time the next button is pressed. How do I format the rules so that an endless loop doesn't occur. </p>
  
  <p>Here is my rule:</p>
  
  <pre><code>rules = (
          Rule(SgmlLinkExtractor(allow='http://not-a-real-site.com/trunk-framework/791'),follow=True,callback='parse_item',
      ),
  )
  </code></pre>