title: Scrapy: Access CrawlSpider url list
body: <p>I am writing a script that will inventory all site urls.</p>
  
  <p>I am using CrawlSpider w/ rules handler to process scraped url's. Specifically, "filter_links" checks a table for existing url. If not found, writes new entry.</p>
  
  <pre><code>    rules = [
      Rule(SgmlLinkExtractor(unique=True), follow=True, callback="parse_item", process_links="filter_links")
  ]
  </code></pre>
  
  <p>I sense this is just a poor mans 'reinventing the wheel' where a better method surely exists. </p>
  
  <p>Is there a better way to dump the list of url's scrapy found vs. trying to parse this from response? Thanks</p>