title: Scrapy - Understanding CrawlSpider and LinkExtractor
body: <p>So I'm trying to use CrawlSpider and understand the following example in the <a href="https://scrapy.readthedocs.io/en/latest/topics/spiders.html#crawlspider-example" rel="noreferrer">Scrapy Docs</a>: </p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  
  class MySpider(CrawlSpider):
      name = 'example.com'
      allowed_domains = ['example.com']
      start_urls = ['http://www.example.com']
  
  rules = (
      # Extract links matching 'category.php' (but not matching 'subsection.php')
      # and follow links from them (since no callback means follow=True by default).
      Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
  
      # Extract links matching 'item.php' and parse them with the spider's method parse_item
      Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
  )
  
  def parse_item(self, response):
      self.logger.info('Hi, this is an item page! %s', response.url)
      item = scrapy.Item()
      item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
      item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
      item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
      return item
  </code></pre>
  
  <p>The description then given is: </p>
  
  <blockquote>
    <p>This spider would start crawling example.com’s home page, collecting category links, and item links, parsing the latter with the parse_item method. For each item response, some data will be extracted from the HTML using XPath, and an Item will be filled with it.</p>
  </blockquote>
  
  <p>I understand that for the second rule, it extracts links from <code>item.php</code> and then extracts the information using the <code>parse_item</code> method. However, what exactly is the purpose of the first rule? It just says that it "collects" the links. What does that mean and why is it useful if they are not extracting any data from it? </p>