prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Cookies from the Cookie request header are not processed
  I am new in scrapy, and I meet some problems which I can not get answer from google, so I post it here:
  
  1 Cookie not work even set in DEFAULT_REQUEST_HEADERS:
  
  ```
  DEFAULT_REQUEST_HEADERS = {
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'accept-encoding': 'gzip, deflate, sdch',
      'cache-control': 'no-cache',
      'cookie': 'xx=yy',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
  }
  ```
  
  ```
  class MySpider(scrapy.Spider):
      def make_requests_from_url(self, url):
          return scrapy.http.Request(url, headers=DEFAULT_REQUEST_HEADERS)
  ```
  
  I know the `make_requests_from_url` will only called once for the start_urls, and in my opinion, the first request will send the cookie I set in the `DEFAULT_REQUEST_HEADERS`, however it does not.
  
  2 Share settings between spiders.
  
  I have multiple spiders in the project which share most of the settings like `RandomAgentMiddleware` `RandomProxyMiddleware` `UserAgent` `DEFAULT_REQUEST_HEADERS` and etc, however they are configured inside the settings.py for each spider.
  
  Is it possible to share these settings?
  
  ---
  
  The 
  `COOKIES_ENABLED` is set to true.
  
  Double-encoded cookies
  When cookies are passed as UTF8 encoded bytes to the `Request` constructor, they end up being encoded twice and escaped in the `Cookie` header.
  
  ```
  $ scrapy shell
  (...)
  In [1]: fetch(scrapy.Request('https://httpbin.org/cookies', cookies={'a': u'á'.encode('utf8')}))
  
  In [2]: request.headers['Cookie']
  Out[2]: b"a=b'\\xc3\\xa1'"
  
  In [3]: print(response.text)
  {
    "cookies": {
      "a": "b'\\xc3\\xa1'"
    }
  }
  ```
  
  This seems to happen only in Python 3.
  ```
  $ scrapy version -v
  Scrapy       : 1.5.0
  lxml         : 4.2.6.0
  libxml2      : 2.9.8
  cssselect    : 1.0.3
  parsel       : 1.5.1
  w3lib        : 1.19.0
  Twisted      : 18.9.0
  Python       : 3.6.0 (default, Sep  1 2017, 10:59:37) - [GCC 4.8.4]
  pyOpenSSL    : 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018)
  cryptography : 2.4.2
  Platform     : Linux-4.4.0-134-generic-x86_64-with-debian-jessie-sid
  ```
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `scrapy/downloadermiddlewares/cookies.py`
  Content:
  ```
  1 import logging
  2 from collections import defaultdict
  3 
  4 from scrapy.exceptions import NotConfigured
  5 from scrapy.http import Response
  6 from scrapy.http.cookies import CookieJar
  7 from scrapy.utils.python import to_unicode
  8 
  9 
  10 logger = logging.getLogger(__name__)
  11 
  12 
  13 class CookiesMiddleware:
  14     """This middleware enables working with sites that need cookies"""
  15 
  16     def __init__(self, debug=False):
  17         self.jars = defaultdict(CookieJar)
  18         self.debug = debug
  19 
  20     @classmethod
  21     def from_crawler(cls, crawler):
  22         if not crawler.settings.getbool('COOKIES_ENABLED'):
  23             raise NotConfigured
  24         return cls(crawler.settings.getbool('COOKIES_DEBUG'))
  25 
  26     def process_request(self, request, spider):
  27         if request.meta.get('dont_merge_cookies', False):
  28             return
  29 
  30         cookiejarkey = request.meta.get("cookiejar")
  31         jar = self.jars[cookiejarkey]
  32         cookies = self._get_request_cookies(jar, request)
  33         for cookie in cookies:
  34             jar.set_cookie_if_ok(cookie, request)
  35 
  36         # set Cookie header
  37         request.headers.pop('Cookie', None)
  38         jar.add_cookie_header(request)
  39         self._debug_cookie(request, spider)
  40 
  41     def process_response(self, request, response, spider):
  42         if request.meta.get('dont_merge_cookies', False):
  43             return response
  44 
  45         # extract cookies from Set-Cookie and drop invalid/expired cookies
  46         cookiejarkey = request.meta.get("cookiejar")
  47         jar = self.jars[cookiejarkey]
  48         jar.extract_cookies(response, request)
  49         self._debug_set_cookie(response, spider)
  50 
  51         return response
  52 
  53     def _debug_cookie(self, request, spider):
  54         if self.debug:
  55             cl = [to_unicode(c, errors='replace')
  56                   for c in request.headers.getlist('Cookie')]
  57             if cl:
  58                 cookies = "\n".join("Cookie: {}\n".format(c) for c in cl)
  59                 msg = "Sending cookies to: {}\n{}".format(request, cookies)
  60                 logger.debug(msg, extra={'spider': spider})
  61 
  62     def _debug_set_cookie(self, response, spider):
  63         if self.debug:
  64             cl = [to_unicode(c, errors='replace')
  65                   for c in response.headers.getlist('Set-Cookie')]
  66             if cl:
  67                 cookies = "\n".join("Set-Cookie: {}\n".format(c) for c in cl)
  68                 msg = "Received cookies from: {}\n{}".format(response, cookies)
  69                 logger.debug(msg, extra={'spider': spider})
  70 
  71     def _format_cookie(self, cookie):
  72         # build cookie string
  73         cookie_str = '%s=%s' % (cookie['name'], cookie['value'])
  74 
  75         if cookie.get('path', None):
  76             cookie_str += '; Path=%s' % cookie['path']
  77         if cookie.get('domain', None):
  78             cookie_str += '; Domain=%s' % cookie['domain']
  79 
  80         return cookie_str
  81 
  82     def _get_request_cookies(self, jar, request):
  83         if isinstance(request.cookies, dict):
  84             cookie_list = [
  85                 {'name': k, 'value': v}
  86                 for k, v in request.cookies.items()
  87             ]
  88         else:
  89             cookie_list = request.cookies
  90 
  91         cookies = [self._format_cookie(x) for x in cookie_list]
  92         headers = {'Set-Cookie': cookies}
  93         response = Response(request.url, headers=headers)
  94 
  95         return jar.make_cookies(response, request)
  96
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/scrapy/downloadermiddlewares/cookies.py b/scrapy/downloadermiddlewares/cookies.py
  --- a/scrapy/downloadermiddlewares/cookies.py
  +++ b/scrapy/downloadermiddlewares/cookies.py
  @@ -29,8 +29,7 @@
   
           cookiejarkey = request.meta.get("cookiejar")
           jar = self.jars[cookiejarkey]
  -        cookies = self._get_request_cookies(jar, request)
  -        for cookie in cookies:
  +        for cookie in self._get_request_cookies(jar, request):
               jar.set_cookie_if_ok(cookie, request)
   
           # set Cookie header
  @@ -68,28 +67,65 @@
                   msg = "Received cookies from: {}\n{}".format(response, cookies)
                   logger.debug(msg, extra={'spider': spider})
   
  -    def _format_cookie(self, cookie):
  -        # build cookie string
  -        cookie_str = '%s=%s' % (cookie['name'], cookie['value'])
  -
  -        if cookie.get('path', None):
  -            cookie_str += '; Path=%s' % cookie['path']
  -        if cookie.get('domain', None):
  -            cookie_str += '; Domain=%s' % cookie['domain']
  -
  +    def _format_cookie(self, cookie, request):
  +        """
  +        Given a dict consisting of cookie components, return its string representation.
  +        Decode from bytes if necessary.
  +        """
  +        decoded = {}
  +        for key in ("name", "value", "path", "domain"):
  +            if not cookie.get(key):
  +                if key in ("name", "value"):
  +                    msg = "Invalid cookie found in request {}: {} ('{}' is missing)"
  +                    logger.warning(msg.format(request, cookie, key))
  +                    return
  +                continue
  +            if isinstance(cookie[key], str):
  +                decoded[key] = cookie[key]
  +            else:
  +                try:
  +                    decoded[key] = cookie[key].decode("utf8")
  +                except UnicodeDecodeError:
  +                    logger.warning("Non UTF-8 encoded cookie found in request %s: %s",
  +                                   request, cookie)
  +                    decoded[key] = cookie[key].decode("latin1", errors="replace")
  +
  +        cookie_str = "{}={}".format(decoded.pop("name"), decoded.pop("value"))
  +        for key, value in decoded.items():  # path, domain
  +            cookie_str += "; {}={}".format(key.capitalize(), value)
           return cookie_str
   
       def _get_request_cookies(self, jar, request):
  -        if isinstance(request.cookies, dict):
  -            cookie_list = [
  -                {'name': k, 'value': v}
  -                for k, v in request.cookies.items()
  -            ]
  -        else:
  -            cookie_list = request.cookies
  -
  -        cookies = [self._format_cookie(x) for x in cookie_list]
  -        headers = {'Set-Cookie': cookies}
  -        response = Response(request.url, headers=headers)
  -
  -        return jar.make_cookies(response, request)
  +        """
  +        Extract cookies from a Request. Values from the `Request.cookies` attribute
  +        take precedence over values from the `Cookie` request header.
  +        """
  +        def get_cookies_from_header(jar, request):
  +            cookie_header = request.headers.get("Cookie")
  +            if not cookie_header:
  +                return []
  +            cookie_gen_bytes = (s.strip() for s in cookie_header.split(b";"))
  +            cookie_list_unicode = []
  +            for cookie_bytes in cookie_gen_bytes:
  +                try:
  +                    cookie_unicode = cookie_bytes.decode("utf8")
  +                except UnicodeDecodeError:
  +                    logger.warning("Non UTF-8 encoded cookie found in request %s: %s",
  +                                   request, cookie_bytes)
  +                    cookie_unicode = cookie_bytes.decode("latin1", errors="replace")
  +                cookie_list_unicode.append(cookie_unicode)
  +            response = Response(request.url, headers={"Set-Cookie": cookie_list_unicode})
  +            return jar.make_cookies(response, request)
  +
  +        def get_cookies_from_attribute(jar, request):
  +            if not request.cookies:
  +                return []
  +            elif isinstance(request.cookies, dict):
  +                cookies = ({"name": k, "value": v} for k, v in request.cookies.items())
  +            else:
  +                cookies = request.cookies
  +            formatted = filter(None, (self._format_cookie(c, request) for c in cookies))
  +            response = Response(request.url, headers={"Set-Cookie": formatted})
  +            return jar.make_cookies(response, request)
  +
  +        return get_cookies_from_header(jar, request) + get_cookies_from_attribute(jar, request)
  
verification_info: {"golden_diff": "diff --git a/scrapy/downloadermiddlewares/cookies.py b/scrapy/downloadermiddlewares/cookies.py\n--- a/scrapy/downloadermiddlewares/cookies.py\n+++ b/scrapy/downloadermiddlewares/cookies.py\n@@ -29,8 +29,7 @@\n \n         cookiejarkey = request.meta.get(\"cookiejar\")\n         jar = self.jars[cookiejarkey]\n-        cookies = self._get_request_cookies(jar, request)\n-        for cookie in cookies:\n+        for cookie in self._get_request_cookies(jar, request):\n             jar.set_cookie_if_ok(cookie, request)\n \n         # set Cookie header\n@@ -68,28 +67,65 @@\n                 msg = \"Received cookies from: {}\\n{}\".format(response, cookies)\n                 logger.debug(msg, extra={'spider': spider})\n \n-    def _format_cookie(self, cookie):\n-        # build cookie string\n-        cookie_str = '%s=%s' % (cookie['name'], cookie['value'])\n-\n-        if cookie.get('path', None):\n-            cookie_str += '; Path=%s' % cookie['path']\n-        if cookie.get('domain', None):\n-            cookie_str += '; Domain=%s' % cookie['domain']\n-\n+    def _format_cookie(self, cookie, request):\n+        \"\"\"\n+        Given a dict consisting of cookie components, return its string representation.\n+        Decode from bytes if necessary.\n+        \"\"\"\n+        decoded = {}\n+        for key in (\"name\", \"value\", \"path\", \"domain\"):\n+            if not cookie.get(key):\n+                if key in (\"name\", \"value\"):\n+                    msg = \"Invalid cookie found in request {}: {} ('{}' is missing)\"\n+                    logger.warning(msg.format(request, cookie, key))\n+                    return\n+                continue\n+            if isinstance(cookie[key], str):\n+                decoded[key] = cookie[key]\n+            else:\n+                try:\n+                    decoded[key] = cookie[key].decode(\"utf8\")\n+                except UnicodeDecodeError:\n+                    logger.warning(\"Non UTF-8 encoded cookie found in request %s: %s\",\n+                                   request, cookie)\n+                    decoded[key] = cookie[key].decode(\"latin1\", errors=\"replace\")\n+\n+        cookie_str = \"{}={}\".format(decoded.pop(\"name\"), decoded.pop(\"value\"))\n+        for key, value in decoded.items():  # path, domain\n+            cookie_str += \"; {}={}\".format(key.capitalize(), value)\n         return cookie_str\n \n     def _get_request_cookies(self, jar, request):\n-        if isinstance(request.cookies, dict):\n-            cookie_list = [\n-                {'name': k, 'value': v}\n-                for k, v in request.cookies.items()\n-            ]\n-        else:\n-            cookie_list = request.cookies\n-\n-        cookies = [self._format_cookie(x) for x in cookie_list]\n-        headers = {'Set-Cookie': cookies}\n-        response = Response(request.url, headers=headers)\n-\n-        return jar.make_cookies(response, request)\n+        \"\"\"\n+        Extract cookies from a Request. Values from the `Request.cookies` attribute\n+        take precedence over values from the `Cookie` request header.\n+        \"\"\"\n+        def get_cookies_from_header(jar, request):\n+            cookie_header = request.headers.get(\"Cookie\")\n+            if not cookie_header:\n+                return []\n+            cookie_gen_bytes = (s.strip() for s in cookie_header.split(b\";\"))\n+            cookie_list_unicode = []\n+            for cookie_bytes in cookie_gen_bytes:\n+                try:\n+                    cookie_unicode = cookie_bytes.decode(\"utf8\")\n+                except UnicodeDecodeError:\n+                    logger.warning(\"Non UTF-8 encoded cookie found in request %s: %s\",\n+                                   request, cookie_bytes)\n+                    cookie_unicode = cookie_bytes.decode(\"latin1\", errors=\"replace\")\n+                cookie_list_unicode.append(cookie_unicode)\n+            response = Response(request.url, headers={\"Set-Cookie\": cookie_list_unicode})\n+            return jar.make_cookies(response, request)\n+\n+        def get_cookies_from_attribute(jar, request):\n+            if not request.cookies:\n+                return []\n+            elif isinstance(request.cookies, dict):\n+                cookies = ({\"name\": k, \"value\": v} for k, v in request.cookies.items())\n+            else:\n+                cookies = request.cookies\n+            formatted = filter(None, (self._format_cookie(c, request) for c in cookies))\n+            response = Response(request.url, headers={\"Set-Cookie\": formatted})\n+            return jar.make_cookies(response, request)\n+\n+        return get_cookies_from_header(jar, request) + get_cookies_from_attribute(jar, request)\n", "issue": "Cookies from the Cookie request header are not processed\nI am new in scrapy, and I meet some problems which I can not get answer from google, so I post it here:\n\n1 Cookie not work even set in DEFAULT_REQUEST_HEADERS:\n\n```\nDEFAULT_REQUEST_HEADERS = {\n    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',\n    'accept-encoding': 'gzip, deflate, sdch',\n    'cache-control': 'no-cache',\n    'cookie': 'xx=yy',\n    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'\n}\n```\n\n```\nclass MySpider(scrapy.Spider):\n    def make_requests_from_url(self, url):\n        return scrapy.http.Request(url, headers=DEFAULT_REQUEST_HEADERS)\n```\n\nI know the `make_requests_from_url` will only called once for the start_urls, and in my opinion, the first request will send the cookie I set in the `DEFAULT_REQUEST_HEADERS`, however it does not.\n\n2 Share settings between spiders.\n\nI have multiple spiders in the project which share most of the settings like `RandomAgentMiddleware` `RandomProxyMiddleware` `UserAgent` `DEFAULT_REQUEST_HEADERS` and etc, however they are configured inside the settings.py for each spider.\n\nIs it possible to share these settings?\n\n---\n\nThe \n`COOKIES_ENABLED` is set to true.\n\nDouble-encoded cookies\nWhen cookies are passed as UTF8 encoded bytes to the `Request` constructor, they end up being encoded twice and escaped in the `Cookie` header.\r\n\r\n```\r\n$ scrapy shell\r\n(...)\r\nIn [1]: fetch(scrapy.Request('https://httpbin.org/cookies', cookies={'a': u'\u00e1'.encode('utf8')}))\r\n\r\nIn [2]: request.headers['Cookie']\r\nOut[2]: b\"a=b'\\\\xc3\\\\xa1'\"\r\n\r\nIn [3]: print(response.text)\r\n{\r\n  \"cookies\": {\r\n    \"a\": \"b'\\\\xc3\\\\xa1'\"\r\n  }\r\n}\r\n```\r\n\r\nThis seems to happen only in Python 3.\r\n```\r\n$ scrapy version -v\r\nScrapy       : 1.5.0\r\nlxml         : 4.2.6.0\r\nlibxml2      : 2.9.8\r\ncssselect    : 1.0.3\r\nparsel       : 1.5.1\r\nw3lib        : 1.19.0\r\nTwisted      : 18.9.0\r\nPython       : 3.6.0 (default, Sep  1 2017, 10:59:37) - [GCC 4.8.4]\r\npyOpenSSL    : 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018)\r\ncryptography : 2.4.2\r\nPlatform     : Linux-4.4.0-134-generic-x86_64-with-debian-jessie-sid\r\n```\n", "before_files": [{"content": "import logging\nfrom collections import defaultdict\n\nfrom scrapy.exceptions import NotConfigured\nfrom scrapy.http import Response\nfrom scrapy.http.cookies import CookieJar\nfrom scrapy.utils.python import to_unicode\n\n\nlogger = logging.getLogger(__name__)\n\n\nclass CookiesMiddleware:\n    \"\"\"This middleware enables working with sites that need cookies\"\"\"\n\n    def __init__(self, debug=False):\n        self.jars = defaultdict(CookieJar)\n        self.debug = debug\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        if not crawler.settings.getbool('COOKIES_ENABLED'):\n            raise NotConfigured\n        return cls(crawler.settings.getbool('COOKIES_DEBUG'))\n\n    def process_request(self, request, spider):\n        if request.meta.get('dont_merge_cookies', False):\n            return\n\n        cookiejarkey = request.meta.get(\"cookiejar\")\n        jar = self.jars[cookiejarkey]\n        cookies = self._get_request_cookies(jar, request)\n        for cookie in cookies:\n            jar.set_cookie_if_ok(cookie, request)\n\n        # set Cookie header\n        request.headers.pop('Cookie', None)\n        jar.add_cookie_header(request)\n        self._debug_cookie(request, spider)\n\n    def process_response(self, request, response, spider):\n        if request.meta.get('dont_merge_cookies', False):\n            return response\n\n        # extract cookies from Set-Cookie and drop invalid/expired cookies\n        cookiejarkey = request.meta.get(\"cookiejar\")\n        jar = self.jars[cookiejarkey]\n        jar.extract_cookies(response, request)\n        self._debug_set_cookie(response, spider)\n\n        return response\n\n    def _debug_cookie(self, request, spider):\n        if self.debug:\n            cl = [to_unicode(c, errors='replace')\n                  for c in request.headers.getlist('Cookie')]\n            if cl:\n                cookies = \"\\n\".join(\"Cookie: {}\\n\".format(c) for c in cl)\n                msg = \"Sending cookies to: {}\\n{}\".format(request, cookies)\n                logger.debug(msg, extra={'spider': spider})\n\n    def _debug_set_cookie(self, response, spider):\n        if self.debug:\n            cl = [to_unicode(c, errors='replace')\n                  for c in response.headers.getlist('Set-Cookie')]\n            if cl:\n                cookies = \"\\n\".join(\"Set-Cookie: {}\\n\".format(c) for c in cl)\n                msg = \"Received cookies from: {}\\n{}\".format(response, cookies)\n                logger.debug(msg, extra={'spider': spider})\n\n    def _format_cookie(self, cookie):\n        # build cookie string\n        cookie_str = '%s=%s' % (cookie['name'], cookie['value'])\n\n        if cookie.get('path', None):\n            cookie_str += '; Path=%s' % cookie['path']\n        if cookie.get('domain', None):\n            cookie_str += '; Domain=%s' % cookie['domain']\n\n        return cookie_str\n\n    def _get_request_cookies(self, jar, request):\n        if isinstance(request.cookies, dict):\n            cookie_list = [\n                {'name': k, 'value': v}\n                for k, v in request.cookies.items()\n            ]\n        else:\n            cookie_list = request.cookies\n\n        cookies = [self._format_cookie(x) for x in cookie_list]\n        headers = {'Set-Cookie': cookies}\n        response = Response(request.url, headers=headers)\n\n        return jar.make_cookies(response, request)\n", "path": "scrapy/downloadermiddlewares/cookies.py"}], "after_files": [{"content": "import logging\nfrom collections import defaultdict\n\nfrom scrapy.exceptions import NotConfigured\nfrom scrapy.http import Response\nfrom scrapy.http.cookies import CookieJar\nfrom scrapy.utils.python import to_unicode\n\n\nlogger = logging.getLogger(__name__)\n\n\nclass CookiesMiddleware:\n    \"\"\"This middleware enables working with sites that need cookies\"\"\"\n\n    def __init__(self, debug=False):\n        self.jars = defaultdict(CookieJar)\n        self.debug = debug\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        if not crawler.settings.getbool('COOKIES_ENABLED'):\n            raise NotConfigured\n        return cls(crawler.settings.getbool('COOKIES_DEBUG'))\n\n    def process_request(self, request, spider):\n        if request.meta.get('dont_merge_cookies', False):\n            return\n\n        cookiejarkey = request.meta.get(\"cookiejar\")\n        jar = self.jars[cookiejarkey]\n        for cookie in self._get_request_cookies(jar, request):\n            jar.set_cookie_if_ok(cookie, request)\n\n        # set Cookie header\n        request.headers.pop('Cookie', None)\n        jar.add_cookie_header(request)\n        self._debug_cookie(request, spider)\n\n    def process_response(self, request, response, spider):\n        if request.meta.get('dont_merge_cookies', False):\n            return response\n\n        # extract cookies from Set-Cookie and drop invalid/expired cookies\n        cookiejarkey = request.meta.get(\"cookiejar\")\n        jar = self.jars[cookiejarkey]\n        jar.extract_cookies(response, request)\n        self._debug_set_cookie(response, spider)\n\n        return response\n\n    def _debug_cookie(self, request, spider):\n        if self.debug:\n            cl = [to_unicode(c, errors='replace')\n                  for c in request.headers.getlist('Cookie')]\n            if cl:\n                cookies = \"\\n\".join(\"Cookie: {}\\n\".format(c) for c in cl)\n                msg = \"Sending cookies to: {}\\n{}\".format(request, cookies)\n                logger.debug(msg, extra={'spider': spider})\n\n    def _debug_set_cookie(self, response, spider):\n        if self.debug:\n            cl = [to_unicode(c, errors='replace')\n                  for c in response.headers.getlist('Set-Cookie')]\n            if cl:\n                cookies = \"\\n\".join(\"Set-Cookie: {}\\n\".format(c) for c in cl)\n                msg = \"Received cookies from: {}\\n{}\".format(response, cookies)\n                logger.debug(msg, extra={'spider': spider})\n\n    def _format_cookie(self, cookie, request):\n        \"\"\"\n        Given a dict consisting of cookie components, return its string representation.\n        Decode from bytes if necessary.\n        \"\"\"\n        decoded = {}\n        for key in (\"name\", \"value\", \"path\", \"domain\"):\n            if not cookie.get(key):\n                if key in (\"name\", \"value\"):\n                    msg = \"Invalid cookie found in request {}: {} ('{}' is missing)\"\n                    logger.warning(msg.format(request, cookie, key))\n                    return\n                continue\n            if isinstance(cookie[key], str):\n                decoded[key] = cookie[key]\n            else:\n                try:\n                    decoded[key] = cookie[key].decode(\"utf8\")\n                except UnicodeDecodeError:\n                    logger.warning(\"Non UTF-8 encoded cookie found in request %s: %s\",\n                                   request, cookie)\n                    decoded[key] = cookie[key].decode(\"latin1\", errors=\"replace\")\n\n        cookie_str = \"{}={}\".format(decoded.pop(\"name\"), decoded.pop(\"value\"))\n        for key, value in decoded.items():  # path, domain\n            cookie_str += \"; {}={}\".format(key.capitalize(), value)\n        return cookie_str\n\n    def _get_request_cookies(self, jar, request):\n        \"\"\"\n        Extract cookies from a Request. Values from the `Request.cookies` attribute\n        take precedence over values from the `Cookie` request header.\n        \"\"\"\n        def get_cookies_from_header(jar, request):\n            cookie_header = request.headers.get(\"Cookie\")\n            if not cookie_header:\n                return []\n            cookie_gen_bytes = (s.strip() for s in cookie_header.split(b\";\"))\n            cookie_list_unicode = []\n            for cookie_bytes in cookie_gen_bytes:\n                try:\n                    cookie_unicode = cookie_bytes.decode(\"utf8\")\n                except UnicodeDecodeError:\n                    logger.warning(\"Non UTF-8 encoded cookie found in request %s: %s\",\n                                   request, cookie_bytes)\n                    cookie_unicode = cookie_bytes.decode(\"latin1\", errors=\"replace\")\n                cookie_list_unicode.append(cookie_unicode)\n            response = Response(request.url, headers={\"Set-Cookie\": cookie_list_unicode})\n            return jar.make_cookies(response, request)\n\n        def get_cookies_from_attribute(jar, request):\n            if not request.cookies:\n                return []\n            elif isinstance(request.cookies, dict):\n                cookies = ({\"name\": k, \"value\": v} for k, v in request.cookies.items())\n            else:\n                cookies = request.cookies\n            formatted = filter(None, (self._format_cookie(c, request) for c in cookies))\n            response = Response(request.url, headers={\"Set-Cookie\": formatted})\n            return jar.make_cookies(response, request)\n\n        return get_cookies_from_header(jar, request) + get_cookies_from_attribute(jar, request)\n", "path": "scrapy/downloadermiddlewares/cookies.py"}]}