title: Find all cookies with Scrapy
body: <p>I making a web crawler with Scrapy which will visit a list of URLs and return all cookies from these domains including those set by third parties.</p>
  
  <p>This spider follows all links on the given URLs and writes each cookie in a separate text file:</p>
  
  <pre><code>from scrapy.linkextractors import LinkExtractor
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.item import Item, Field
  import requests
  
  class MyItem(Item):
      url= Field()
  
  
  class MySpider(CrawlSpider):
      name = "a"
      start_urls = ['http://www.dailymail.co.uk/home/index.html']
  
  rules = (Rule(LinkExtractor(), callback='parse_url', follow=False), )
  
  def parse_url(self, response):
      item = MyItem()
      item['url'] = response.url
      filename = '%s.txt'
      if response.headers.getlist('Set-Cookie'):
          page = response.url.split("/")[-2]       
          with open(filename, 'wb') as f:
              for cookie in response.headers.getlist('Set-Cookie'):
                  f.write(cookie)
  </code></pre>
  
  <p>This results in 11 different text files each containing a cookie. The result is inconsistent with that produced by the website cookie-checker.com.</p>
  
  <p>Is there a way to find all cookies set on a page using Scrapy?</p>