title: Collecting text from links which scrapy "click"?
body: <p>I wanna collect text from links which <code>scrapy</code> "click" in the website.</p>
  
  <p>Consider the following example:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  from scrapy.linkextractors import LinkExtractor
  from scrapy.spiders import CrawlSpider, Rule
  
  
  class DnsDbSpider(CrawlSpider):
      name = 'dns_db'
      allowed_domains = ['www.iana.org']
      start_urls = ['http://www.iana.org/']
  
      rules = (
          Rule(LinkExtractor(
              allow_domains='www.iana.org',
              restrict_css=r'#home-panel-domains &gt; h2'),
              callback='parse_item',
              follow=True),
          Rule(LinkExtractor(
              allow_domains='www.iana.org',
              restrict_css=r'#main_right &gt; p:nth-child(3)'),
              callback='parse_item',
              follow=True),
          Rule(LinkExtractor(
              allow_domains='www.iana.org',
              restrict_css=r'#main_right &gt; ul:nth-child(4) &gt; li'),
              callback='parse_item',
              follow=True),
      )
  
  
      def parse_item(self, response):
          self.logger.info('## Parsing URL: %s', response.url)
          i = {}
          return i
  </code></pre>
  
  <p><code>scrapy</code> log:</p>
  
  <pre><code>$ scrapy crawl dns_db 2&gt;&amp;1 | grep 'Parsing URL'
  2017-01-17 22:14:01 [dns_db] INFO: ## Parsing URL: http://www.iana.org/domains
  2017-01-17 22:14:02 [dns_db] INFO: ## Parsing URL: http://www.iana.org/domains/root
  2017-01-17 22:14:02 [dns_db] INFO: ## Parsing URL: http://www.iana.org/domains/root/db
  </code></pre>
  
  <p>In this case <code>scrapy</code> has done the following:</p>
  
  <ol>
  <li>Open "<em>www.iana.org</em>"<br>
  <code>path = []</code></li>
  <li>Click "<em>Domain Names</em>" url.<br>
  <code>path = ['Domain Names']</code></li>
  <li>In the "<em>Domain Names</em>" page click "<em>The DNS Root Zone</em>" url.<br>
  <code>path = ['Domain Names', 'The DNS Root Zone']</code></li>
  <li>In the "<em>The DNS Root Zone</em>" page click "<em>Root Zone Database</em>" url.<br>
  <code>path = ['Domain Names', 'The DNS Root Zone', 'Root Zone Database']</code></li>
  <li>In "<em>Root Zone Database</em>" page I would start scrapping data, thus creating items. Final item would also have path attribute:<br>
  <code>path = ['Domain Names', 'The DNS Root Zone', 'Root Zone Database']</code></li>
  </ol>
  
  <p>A human being could navigate in the website just by looking at this path/list.</p>
  
  <p>How can I achieve that? </p>
  
  <p><strong><em>EDIT</em></strong></p>
  
  <p>Here's a working example:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  from scrapy.linkextractors import LinkExtractor
  
  
  class DnsDbSpider(scrapy.Spider):
      name = "dns_db"
      allowed_domains = ["www.iana.org"]
      start_urls = ['http://www.iana.org/']
  
      def parse(self, response):
          if 'req_path' not in response.meta:
              response.meta['req_path'] = []
          self.logger.warn('## Request path: %s', response.meta['req_path'])
          restrict_css = (
              r'#home-panel-domains &gt; h2',
              r'#main_right &gt; p:nth-child(3)',
              r'#main_right &gt; ul:nth-child(4) &gt; li'
          )
          links = [link for css in restrict_css for link in self.links(response, css)]
          for link in links:
              #self.logger.info('## Link: %s', link)
              request = scrapy.Request(
                  url=link.url,
                  callback=self.parse)
              request.meta['req_path'] = response.meta['req_path'].copy()
              request.meta['req_path'].append(dict(text=link.text, url=link.url))
              yield request
  
      def links(self, response, restrict_css=None):
          lex = LinkExtractor(
              allow_domains=self.allowed_domains,
              restrict_css=restrict_css)
          return lex.extract_links(response)
  </code></pre>
  
  <p>Command line output:</p>
  
  <pre><code>$ scrapy crawl -L WARN dns_db
  2017-02-12 00:13:50 [dns_db] WARNING: ## Request path: []
  2017-02-12 00:13:51 [dns_db] WARNING: ## Request path: [{'text': 'Domain Names', 'url': 'http://www.iana.org/domains'}]
  2017-02-12 00:13:51 [dns_db] WARNING: ## Request path: [{'text': 'Domain Names', 'url': 'http://www.iana.org/domains'}, {'text': 'The DNS Root Zone', 'url': 'http://www.iana.org/domains/root'}]
  2017-02-12 00:13:52 [dns_db] WARNING: ## Request path: [{'text': 'Domain Names', 'url': 'http://www.iana.org/domains'}, {'text': 'The DNS Root Zone', 'url': 'http://www.iana.org/domains/root'}, {'text': 'Root Zone Database', 'url': 'http://www.iana.org/domains/root/db/'}]
  </code></pre>