title: How to recursively crawl subpages with Scrapy
body: <p>So basically I am trying to crawl a page with a set of categories, scrape the names of each category, follow a sublink associated with each category to a page with a set of subcategories, scrape their names, and then follow each subcategory to their associated page and retrieve text data. At the end I want to output a json file formatted somewhat like:</p>
  
  <ol>
  <li>Category 1 name
  
  <ul>
  <li>Subcategory 1 name
  
  <ul>
  <li>data from this subcategory's page</li>
  </ul></li>
  <li>Subcategory n name
  
  <ul>
  <li>data from this page</li>
  </ul></li>
  </ul></li>
  <li>Category n name
  
  <ul>
  <li>Subcategory 1 name
  
  <ul>
  <li>data from subcategory n's page</li>
  </ul></li>
  </ul></li>
  </ol>
  
  <p>etc.</p>
  
  <p>Eventually i want to be able to use this data with ElasticSearch</p>
  
  <p>I barely have any experience with Scrapy and this is what I have so far (just scrapes the category names from the first page, I have no idea what to do from here)... From my research I believe I need to use a CrawlSpider but am unsure of what that entails. I have also been suggested to use BeautifulSoup. Any help would be greatly appreciated.</p>
  
  <pre><code>class randomSpider(scrapy.Spider):
      name = "helpme"
      allowed_domains = ["example.com"]
      start_urls = ['http://example.com/categories',]
  
      def parse(self, response):
          for i in response.css('div.CategoryTreeSection'):
              yield {
                  'categories': i.css('a::text').extract_first()
              }
  </code></pre>