title: Scrapy - Scraping links by date
body: <p>Is it possible to scrape links by the date associated with them? I'm trying to implement a daily run spider that saves article information to a database, but I don't want to re-scrape articles that I have already scraped before-- i.e yesterday's articles. I ran across <a href="https://stackoverflow.com/questions/13245910/scrapy-how-to-scrape-daily-for-new-pages">this</a> SO post asking the same thing and the <a href="https://github.com/scrapy-plugins/scrapy-deltafetch" rel="nofollow noreferrer">scrapy-deltafetch plugin</a> was suggested.</p>
  
  <p>However, this relies on checking new requests against previously saved <a href="https://github.com/scrapy/scrapy/blob/master/scrapy/utils/request.py#L19" rel="nofollow noreferrer">request fingerprints</a> stored in a database. I'm assuming that if the daily scraping went on for a while, there would be a need for significant memory overhead on the database to store request fingerprints that have already been scraped. </p>
  
  <p>So given a list of articles on  a site like <a href="http://www.cnn.com/specials/last-50-stories" rel="nofollow noreferrer">cnn.com</a>, I want to scrape all the articles that have been published today 6/14/17, but once the scraper hits later articles with a date listed as 6/13/17, I want to close the spider and stop scraping. Is this kind of approach possible with scrapy? Given a page of articles, will a <code>CrawlSpider</code> start at the top of the page and scrape articles in order? </p>
  
  <p>Just new to <code>Scrapy</code>, so not sure what to try. Any help would be greatly appreciated, thank you! </p>