title: Scrapy: missing cookies in response
body: <p>I've created the basic scrapy project and enabled cookiemiddleware as in <a href="https://doc.scrapy.org/en/latest/topics/downloader-middleware.html" rel="nofollow noreferrer">documentation</a>.</p>
  
  <p><strong>settings.py</strong></p>
  
  <pre><code>COOKIES_ENABLED = True
  COOKIES_DEBUG = True
  
  DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700  
  }
  </code></pre>
  
  <p><strong>spiders/amazon_spider.py</strong></p>
  
  <pre><code>class AmazonSpider(Spider):
    name = "amazon_spider"
    start_urls = ['https://sellercentral.amazon.com/gp/sc-redirect']
  
    def parse(self, response):
        self.logger.info(response.headers.getlist('Set-Cookie'))
  </code></pre>
  
  <p>Though, for request with</p>
  
  <pre><code>COOKIES_ENABLED = True
  </code></pre>
  
  <p>the response is the same as for the request with</p>
  
  <pre><code>COOKIES_ENABLED = False
  </code></pre>
  
  <p>having</p>
  
  <blockquote>
    <p>Please Enable Cookies to Continue</p>
  </blockquote>
  
  <p>in it's body.</p>
  
  <ol>
  <li>Using Firefox &amp; Firebug</li>
  </ol>
  
  <blockquote>
    <p><strong>REQUEST</strong><br>
    GET /gp/sc-redirect HTTP/1.1<br>
    Host: sellercentral.amazon.com<br>
    User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101
    Firefox/50.0<br>
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,<em>/</em>;q=0.8<br>
    Accept-Language: en-US,en;q=0.5<br>
    Accept-Encoding: gzip, deflate, br<br>
    DNT: 1<br>
    Connection: keep-alive<br>
    Upgrade-Insecure-Requests: 1<br>
    <strong>RESPONSE</strong><br>
    HTTP/1.1 302 Found<br>
    Server: Server<br>
    Date: Mon, 30 Jan 2017 16:12:51 GMT<br>
    Content-Type: text/html;charset=UTF-8<br>
    Transfer-Encoding: chunked<br>
    Connection: keep-alive<br>
    Location: https:// sellercentral.amazon.com/ap/signin?...<br>
    Vary: Accept-Encoding,User-Agent<br>
    Cache-Control: no-cache, no-store, must-revalidate<br>
    Expires: 0<br>
    Pragma: no-cache<br>
    Content-Encoding: gzip<br>
    Set-Cookie: session-id-time=1486368000l; path=/; domain=.amazon.com; expires=Mon, 06-Feb-2017 16:12:51 GMT session-id=160-1127516-9252943; path=/; domain=.amazon.com; expires=Mon, 06-Feb-2017 16:12:51 GMT   </p>
  </blockquote>
  
  <ol start="2">
  <li>Using Scrapy</li>
  </ol>
  
  <blockquote>
    <p><strong>RESPONSE</strong><br>
    DEBUG: Crawled (200) https://sellercentral.amazon.com/robots.txt> (referer: None)<br>
    DEBUG: Redirecting (302) to https://sellercentral.amazon.com/ap/signin?...> from https://sellercentral.amazon.com/gp/sc-redirect/><br>
    DEBUG: Received cookies from: &lt;302 https:// sellercentral.amazon.com/ap/signin?...> 
    Set-Cookie: signin-sso-state-us=44538bf3-88d0-410b-9aa0-bc8da4b2d090; Domain=.amazon.com; Expires=Sun, 25-Jan-2037 16:09:14 GMT; Path=/ap/; Secure; HttpOnly<br>
    Set-Cookie: ap-fid=""; Domain=.amazon.com; Expires=Thu, 01-Jan-1970
    00:00:10 GMT; Path=/ap/; Secure</p>
  </blockquote>
  
  <p>Here are the full <a href="http://pastebin.com/yTjpwZsi" rel="nofollow noreferrer">log</a> and full final <a href="http://pastebin.com/28TZX1bq" rel="nofollow noreferrer">response body</a>.</p>
  
  <p>Why the Set-Cookie results are different and how to handle cookies with Scrapy for this particular case?</p>