title: Scrapy: multi-level requests
body: <p>I´m trying to get data from a site that follow a hierarchical structure of 4 levels:</p>
  
  <pre><code>       root
     1         1
    2 2       2 2
   3 3 3     3 3 3
  4 4 4 4   4 4 4 4
  </code></pre>
  
  <p>I start at the root level, and each level contains links so the following level. After crawling all the pages, I would like to yield items of the first level containing the info parsed on all the levels bellow it.</p>
  
  <p>I tried something like this:</p>
  
  <pre><code>def parse(self,response):
     for link in links
         yield Request(link, callback = self.parse_1)
  
  def parse_1(self,response):
     item1 = Item1()
     #get level 1 info
     ...
     for link in level2_links
         yield Request(link, callback = self.parse_2, meta = {'item': item1})
     yield item1
  
  def parse_2(self,response):
     item1 = response.meta['item']
     item2 = Item2()
     #get level 2 info
     ...
     item1['items2'].append(item2)
     for link in level3_links
         yield Request(link, callback = self.parse_3, meta = {'item': item2})
     yield item 2
  
  def parse_3(self,response):
     item2 = response.meta['item']
     item3 = Item3()
     #get level 3 info
     ...
     item1['items3'].append(item3)
     for link in level4_links
         yield Request(link, callback = self.parse_4, meta = {'item': item3})
     yield item3
  
  def parse_4(self,response):
     item3 = response.meta['item']
     item4 = Item4()
     #get level 4 info
     item3['items4'].append(item4)
     yield item4
  </code></pre>
  
  <p>The problem that I´m having is that instead of sending the first item to the pipeline after all the data below it has been parsed, every  level is being sent to the pipeline as soon as it gets called so every item is sent with just the info of its current level.</p>
  
  <p>Maybe I´m just doing this horribly wrong and there´s an easier way</p>