title: Scrapy stopping pagination on condition?
body: <p>So I want to scrape articles from a site that has pagination. Basically, every page is a list of article links and the spider follows the links on the page in a <code>parse_article</code> method, as well as following the successive next page links. However, is there a way to make this stop after a given number of articles are scraped? For example, this is what I have so far using a <code>crawlspider</code>: </p>
  
  <pre><code>rules = (
  
      #next page rule: 
      Rule(LinkExtractor(restrict_xpaths="//a[@class='next']"),follow=True)
  
      #Extract all internal links which follows this regex:
      Rule(LinkExtractor(allow=('REGEXHERE',),deny=()),callback='parse_article'),
  )
  
  def parse_article(self, response):
      #do parsing stuff here 
  </code></pre>
  
  <p>I want to stop following the next page once I've parsed 150 articles. It doesn't matter if I scrape a little more than 150, I just want to stop going to the next page once I've hit that number. Is there any way to do that? Something like having a counter in the <code>parse_article</code> method? Just new to <code>scrapy</code> so I'm not sure what to try.... I looked into <a href="https://doc.scrapy.org/en/latest/topics/settings.html#depth-limit" rel="nofollow noreferrer">depth_limit</a>, but I'm not so sure that's what I am looking for. </p>
  
  <p>Any help would be greatly appreciated, thanks! </p>