title: Scrapy CrawlSpider - add meta data to request
body: <p>I'm working on a <code>CrawlSpider</code> which gets multiple <code>Domain</code> objects (it contains <code>main_url</code> and <code>name</code> - domain name). Then it crawls whole page for every <code>Domain</code> object. </p>
  
  <p>So there is a <code>Domain</code> with <code>main_url = 'http://www.example.com'</code> and <code>name = 'example.com'</code>. I need to get this <code>domain</code> object for every response of this domain. So <code>http://www.example.com</code>, <code>http://www.example.com/x</code>, <code>http://www.example.com/x/x/y/</code> etc.</p>
  
  <p>So everytime I'm parsing response from this domain, I get particular <code>Domain</code> object.</p>
  
  <p>The problem is that it is not doable inside <code>rules = ()</code></p>
  
  <pre><code>class MainSpider(CrawlSpider):
      name = 'main_spider' 
      rules = (
          Rule(LinkExtractor(), callback='parse_item', follow=True,),
      )
  
      def start_requests(self):
          for d in Domain.objects.all():
              self.allowed_domains.append(d.name)
              yield scrapy.Request(d.main_url,callback=self.parse,meta={'domain':d})
  
  
      def parse_item(self, response):
          # I need to get `Domain` object for all responses
  </code></pre>
  
  <p>Do you know how to do that? Moreover, this has another problem. It doesn't send the first (root) url response to <code>parse_item</code> method.</p>