We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
response_httprepr in DownloaderStats middleware causing scrapy to make unnecessary memory allocation.

### Description

Usage of [`scrapy.utils.response.response_httprepr` ](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/utils/response.py#L45)inside[ `DownloaderStats`](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/downloadermiddlewares/stats.py) middleware causing application to make unnecessary memory allocation.
`response_httprepr` used only one time - to calculate response sizes for `downloader/response_bytes` stats in that middleware.

In current implementation `response_httprepr` return `bytes` (immutable type) - in order to calculate `downloader/response_bytes` application will additionally allocate nearly the same memory amount as for original response (only for calculating `len` inside middleware).
https://github.com/scrapy/scrapy/blob/26836c4e1ae9588ee173c5977fc6611364ca7cc7/scrapy/utils/response.py#L45-L60


### Steps to Reproduce
In order to demonstrate influence of this i made this spider:
<details>
  <summary>spider code</summary>

```python
import sys
from importlib import import_module
import scrapy


class MemoryHttpreprSpider(scrapy.Spider):
    name = 'memory_httprepr'
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES':{
            'scrapy.downloadermiddlewares.stats.DownloaderStats': None
        }
    }
# the same as in MemoryUsage extension:
    def get_virtual_size(self):
        size = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss
        if sys.platform != 'darwin':
            # on macOS ru_maxrss is in bytes, on Linux it is in KB
            size *= 1024
        return size

    def start_requests(self):
        try:
            self.resource = import_module('resource')
        except ImportError:
            pass
        self.logger.info(f"used memory on start: {str(self.get_virtual_size())}")
        yield scrapy.Request(url='https://speed.hetzner.de/100MB.bin', callback=self.parse)
        #yield scrapy.Request(url='http://quotes.toscrape.com', callback=self.parse)

    def parse(self, response, **kwargs):
        self.logger.info(f"used memory after downloading response: {str(self.get_virtual_size())}")
```


</details>
It include:

- usage of `get_virtual_size` method - directly the same as on [`MemoryUsage`](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/extensions/memusage.py) extension
- url with 100Mb binary response https://speed.hetzner.de/100MB.bin
- disabled `DownloaderStats` middleware that uses `response_httprepr` and `request_httprepr`
- usual html url http://quotes.toscrape.com (for comparison)

| log output | 100Mb default settings | 100Mb no downloader stats | quotes default |
| --- | --- | --- | --- |
| `[memory_httprepr] used memory on start:` | `61 587 456` | `61 521 920` | `61 558 784` |
| `[memory_httprepr] used memory after downloading response:` | `375 910 400` | `271 179 776` | `61 558 784` |

### Versions

`[scrapy.utils.log] Scrapy 2.4.0 started (bot: httprepr)`
`[scrapy.utils.log] Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.2 (default, Apr 23 2020, 14:32:57) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020), cryptography 3.1.1, Platform Linux-4.15.0-76-generic-x86_64-with-glibc2.2.5`


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `scrapy/utils/response.py`
Content:
```
1 """
2 This module provides some useful functions for working with
3 scrapy.http.Response objects
4 """
5 import os
6 import webbrowser
7 import tempfile
8 from typing import Any, Callable, Iterable, Optional, Tuple, Union
9 from weakref import WeakKeyDictionary
10 
11 import scrapy
12 from scrapy.http.response import Response
13 
14 from twisted.web import http
15 from scrapy.utils.python import to_bytes, to_unicode
16 from w3lib import html
17 
18 
19 _baseurl_cache: "WeakKeyDictionary[Response, str]" = WeakKeyDictionary()
20 
21 
22 def get_base_url(response: "scrapy.http.response.text.TextResponse") -> str:
23     """Return the base url of the given response, joined with the response url"""
24     if response not in _baseurl_cache:
25         text = response.text[0:4096]
26         _baseurl_cache[response] = html.get_base_url(text, response.url, response.encoding)
27     return _baseurl_cache[response]
28 
29 
30 _metaref_cache: "WeakKeyDictionary[Response, Union[Tuple[None, None], Tuple[float, str]]]" = WeakKeyDictionary()
31 
32 
33 def get_meta_refresh(
34     response: "scrapy.http.response.text.TextResponse",
35     ignore_tags: Optional[Iterable[str]] = ('script', 'noscript'),
36 ) -> Union[Tuple[None, None], Tuple[float, str]]:
37     """Parse the http-equiv refrsh parameter from the given response"""
38     if response not in _metaref_cache:
39         text = response.text[0:4096]
40         _metaref_cache[response] = html.get_meta_refresh(
41             text, response.url, response.encoding, ignore_tags=ignore_tags)
42     return _metaref_cache[response]
43 
44 
45 def response_status_message(status: Union[bytes, float, int, str]) -> str:
46     """Return status code plus status text descriptive message
47     """
48     status_int = int(status)
49     message = http.RESPONSES.get(status_int, "Unknown Status")
50     return f'{status_int} {to_unicode(message)}'
51 
52 
53 def response_httprepr(response: Response) -> bytes:
54     """Return raw HTTP representation (as bytes) of the given response. This
55     is provided only for reference, since it's not the exact stream of bytes
56     that was received (that's not exposed by Twisted).
57     """
58     values = [
59         b"HTTP/1.1 ",
60         to_bytes(str(response.status)),
61         b" ",
62         to_bytes(http.RESPONSES.get(response.status, b'')),
63         b"\r\n",
64     ]
65     if response.headers:
66         values.extend([response.headers.to_string(), b"\r\n"])
67     values.extend([b"\r\n", response.body])
68     return b"".join(values)
69 
70 
71 def open_in_browser(
72     response: Union["scrapy.http.response.html.HtmlResponse", "scrapy.http.response.text.TextResponse"],
73     _openfunc: Callable[[str], Any] = webbrowser.open,
74 ) -> Any:
75     """Open the given response in a local web browser, populating the <base>
76     tag for external links to work
77     """
78     from scrapy.http import HtmlResponse, TextResponse
79     # XXX: this implementation is a bit dirty and could be improved
80     body = response.body
81     if isinstance(response, HtmlResponse):
82         if b'<base' not in body:
83             repl = f'<head><base href="{response.url}">'
84             body = body.replace(b'<head>', to_bytes(repl))
85         ext = '.html'
86     elif isinstance(response, TextResponse):
87         ext = '.txt'
88     else:
89         raise TypeError("Unsupported response type: "
90                         f"{response.__class__.__name__}")
91     fd, fname = tempfile.mkstemp(ext)
92     os.write(fd, body)
93     os.close(fd)
94     return _openfunc(f"file://{fname}")
95
```

Path: `scrapy/downloadermiddlewares/stats.py`
Content:
```
1 from scrapy.exceptions import NotConfigured
2 from scrapy.utils.request import request_httprepr
3 from scrapy.utils.response import response_httprepr
4 from scrapy.utils.python import global_object_name
5 
6 
7 class DownloaderStats:
8 
9     def __init__(self, stats):
10         self.stats = stats
11 
12     @classmethod
13     def from_crawler(cls, crawler):
14         if not crawler.settings.getbool('DOWNLOADER_STATS'):
15             raise NotConfigured
16         return cls(crawler.stats)
17 
18     def process_request(self, request, spider):
19         self.stats.inc_value('downloader/request_count', spider=spider)
20         self.stats.inc_value(f'downloader/request_method_count/{request.method}', spider=spider)
21         reqlen = len(request_httprepr(request))
22         self.stats.inc_value('downloader/request_bytes', reqlen, spider=spider)
23 
24     def process_response(self, request, response, spider):
25         self.stats.inc_value('downloader/response_count', spider=spider)
26         self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)
27         reslen = len(response_httprepr(response))
28         self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)
29         return response
30 
31     def process_exception(self, request, exception, spider):
32         ex_class = global_object_name(exception.__class__)
33         self.stats.inc_value('downloader/exception_count', spider=spider)
34         self.stats.inc_value(f'downloader/exception_type_count/{ex_class}', spider=spider)
35
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/scrapy/downloadermiddlewares/stats.py b/scrapy/downloadermiddlewares/stats.py
--- a/scrapy/downloadermiddlewares/stats.py
+++ b/scrapy/downloadermiddlewares/stats.py
@@ -1,7 +1,22 @@
 from scrapy.exceptions import NotConfigured
+from scrapy.utils.python import global_object_name, to_bytes
 from scrapy.utils.request import request_httprepr
-from scrapy.utils.response import response_httprepr
-from scrapy.utils.python import global_object_name
+
+from twisted.web import http
+
+
+def get_header_size(headers):
+    size = 0
+    for key, value in headers.items():
+        if isinstance(value, (list, tuple)):
+            for v in value:
+                size += len(b": ") + len(key) + len(v)
+    return size + len(b'\r\n') * (len(headers.keys()) - 1)
+
+
+def get_status_size(response_status):
+    return len(to_bytes(http.RESPONSES.get(response_status, b''))) + 15
+    # resp.status + b"\r\n" + b"HTTP/1.1 <100-599> "
 
 
 class DownloaderStats:
@@ -24,7 +39,8 @@
     def process_response(self, request, response, spider):
         self.stats.inc_value('downloader/response_count', spider=spider)
         self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)
-        reslen = len(response_httprepr(response))
+        reslen = len(response.body) + get_header_size(response.headers) + get_status_size(response.status) + 4
+        # response.body + b"\r\n"+ response.header + b"\r\n" + response.status
         self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)
         return response
 
diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py
--- a/scrapy/utils/response.py
+++ b/scrapy/utils/response.py
@@ -13,6 +13,7 @@
 
 from twisted.web import http
 from scrapy.utils.python import to_bytes, to_unicode
+from scrapy.utils.decorators import deprecated
 from w3lib import html
 
 
@@ -50,6 +51,7 @@
     return f'{status_int} {to_unicode(message)}'
 
 
+@deprecated
 def response_httprepr(response: Response) -> bytes:
     """Return raw HTTP representation (as bytes) of the given response. This
     is provided only for reference, since it's not the exact stream of bytes

{"golden_diff": "diff --git a/scrapy/downloadermiddlewares/stats.py b/scrapy/downloadermiddlewares/stats.py\n--- a/scrapy/downloadermiddlewares/stats.py\n+++ b/scrapy/downloadermiddlewares/stats.py\n@@ -1,7 +1,22 @@\n from scrapy.exceptions import NotConfigured\n+from scrapy.utils.python import global_object_name, to_bytes\n from scrapy.utils.request import request_httprepr\n-from scrapy.utils.response import response_httprepr\n-from scrapy.utils.python import global_object_name\n+\n+from twisted.web import http\n+\n+\n+def get_header_size(headers):\n+    size = 0\n+    for key, value in headers.items():\n+        if isinstance(value, (list, tuple)):\n+            for v in value:\n+                size += len(b\": \") + len(key) + len(v)\n+    return size + len(b'\\r\\n') * (len(headers.keys()) - 1)\n+\n+\n+def get_status_size(response_status):\n+    return len(to_bytes(http.RESPONSES.get(response_status, b''))) + 15\n+    # resp.status + b\"\\r\\n\" + b\"HTTP/1.1 <100-599> \"\n \n \n class DownloaderStats:\n@@ -24,7 +39,8 @@\n     def process_response(self, request, response, spider):\n         self.stats.inc_value('downloader/response_count', spider=spider)\n         self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)\n-        reslen = len(response_httprepr(response))\n+        reslen = len(response.body) + get_header_size(response.headers) + get_status_size(response.status) + 4\n+        # response.body + b\"\\r\\n\"+ response.header + b\"\\r\\n\" + response.status\n         self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)\n         return response\n \ndiff --git a/scrapy/utils/response.py b/scrapy/utils/response.py\n--- a/scrapy/utils/response.py\n+++ b/scrapy/utils/response.py\n@@ -13,6 +13,7 @@\n \n from twisted.web import http\n from scrapy.utils.python import to_bytes, to_unicode\n+from scrapy.utils.decorators import deprecated\n from w3lib import html\n \n \n@@ -50,6 +51,7 @@\n     return f'{status_int} {to_unicode(message)}'\n \n \n+@deprecated\n def response_httprepr(response: Response) -> bytes:\n     \"\"\"Return raw HTTP representation (as bytes) of the given response. This\n     is provided only for reference, since it's not the exact stream of bytes\n", "issue": "response_httprepr in DownloaderStats middleware causing scrapy to make unnecessary memory allocation.\n\r\n### Description\r\n\r\nUsage of [`scrapy.utils.response.response_httprepr` ](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/utils/response.py#L45)inside[ `DownloaderStats`](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/downloadermiddlewares/stats.py) middleware causing application to make unnecessary memory allocation.\r\n`response_httprepr` used only one time - to calculate response sizes for `downloader/response_bytes` stats in that middleware.\r\n\r\nIn current implementation `response_httprepr` return `bytes` (immutable type) - in order to calculate `downloader/response_bytes` application will additionally allocate nearly the same memory amount as for original response (only for calculating `len` inside middleware).\r\nhttps://github.com/scrapy/scrapy/blob/26836c4e1ae9588ee173c5977fc6611364ca7cc7/scrapy/utils/response.py#L45-L60\r\n\r\n\r\n### Steps to Reproduce\r\nIn order to demonstrate influence of this i made this spider:\r\n<details>\r\n  <summary>spider code</summary>\r\n\r\n```python\r\nimport sys\r\nfrom importlib import import_module\r\nimport scrapy\r\n\r\n\r\nclass MemoryHttpreprSpider(scrapy.Spider):\r\n    name = 'memory_httprepr'\r\n    custom_settings = {\r\n        'DOWNLOADER_MIDDLEWARES':{\r\n            'scrapy.downloadermiddlewares.stats.DownloaderStats': None\r\n        }\r\n    }\r\n# the same as in MemoryUsage extension:\r\n    def get_virtual_size(self):\r\n        size = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss\r\n        if sys.platform != 'darwin':\r\n            # on macOS ru_maxrss is in bytes, on Linux it is in KB\r\n            size *= 1024\r\n        return size\r\n\r\n    def start_requests(self):\r\n        try:\r\n            self.resource = import_module('resource')\r\n        except ImportError:\r\n            pass\r\n        self.logger.info(f\"used memory on start: {str(self.get_virtual_size())}\")\r\n        yield scrapy.Request(url='https://speed.hetzner.de/100MB.bin', callback=self.parse)\r\n        #yield scrapy.Request(url='http://quotes.toscrape.com', callback=self.parse)\r\n\r\n    def parse(self, response, **kwargs):\r\n        self.logger.info(f\"used memory after downloading response: {str(self.get_virtual_size())}\")\r\n```\r\n\r\n\r\n</details>\r\nIt include:\r\n\r\n- usage of `get_virtual_size` method - directly the same as on [`MemoryUsage`](https://github.com/scrapy/scrapy/blob/2.4.1/scrapy/extensions/memusage.py) extension\r\n- url with 100Mb binary response https://speed.hetzner.de/100MB.bin\r\n- disabled `DownloaderStats` middleware that uses `response_httprepr` and `request_httprepr`\r\n- usual html url http://quotes.toscrape.com (for comparison)\r\n\r\n| log output | 100Mb default settings | 100Mb no downloader stats | quotes default |\r\n| --- | --- | --- | --- |\r\n| `[memory_httprepr] used memory on start:` | `61 587 456` | `61 521 920` | `61 558 784` |\r\n| `[memory_httprepr] used memory after downloading response:` | `375 910 400` | `271 179 776` | `61 558 784` |\r\n\r\n### Versions\r\n\r\n`[scrapy.utils.log] Scrapy 2.4.0 started (bot: httprepr)`\r\n`[scrapy.utils.log] Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.2 (default, Apr 23 2020, 14:32:57) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020), cryptography 3.1.1, Platform Linux-4.15.0-76-generic-x86_64-with-glibc2.2.5`\r\n\n", "before_files": [{"content": "\"\"\"\nThis module provides some useful functions for working with\nscrapy.http.Response objects\n\"\"\"\nimport os\nimport webbrowser\nimport tempfile\nfrom typing import Any, Callable, Iterable, Optional, Tuple, Union\nfrom weakref import WeakKeyDictionary\n\nimport scrapy\nfrom scrapy.http.response import Response\n\nfrom twisted.web import http\nfrom scrapy.utils.python import to_bytes, to_unicode\nfrom w3lib import html\n\n\n_baseurl_cache: \"WeakKeyDictionary[Response, str]\" = WeakKeyDictionary()\n\n\ndef get_base_url(response: \"scrapy.http.response.text.TextResponse\") -> str:\n    \"\"\"Return the base url of the given response, joined with the response url\"\"\"\n    if response not in _baseurl_cache:\n        text = response.text[0:4096]\n        _baseurl_cache[response] = html.get_base_url(text, response.url, response.encoding)\n    return _baseurl_cache[response]\n\n\n_metaref_cache: \"WeakKeyDictionary[Response, Union[Tuple[None, None], Tuple[float, str]]]\" = WeakKeyDictionary()\n\n\ndef get_meta_refresh(\n    response: \"scrapy.http.response.text.TextResponse\",\n    ignore_tags: Optional[Iterable[str]] = ('script', 'noscript'),\n) -> Union[Tuple[None, None], Tuple[float, str]]:\n    \"\"\"Parse the http-equiv refrsh parameter from the given response\"\"\"\n    if response not in _metaref_cache:\n        text = response.text[0:4096]\n        _metaref_cache[response] = html.get_meta_refresh(\n            text, response.url, response.encoding, ignore_tags=ignore_tags)\n    return _metaref_cache[response]\n\n\ndef response_status_message(status: Union[bytes, float, int, str]) -> str:\n    \"\"\"Return status code plus status text descriptive message\n    \"\"\"\n    status_int = int(status)\n    message = http.RESPONSES.get(status_int, \"Unknown Status\")\n    return f'{status_int} {to_unicode(message)}'\n\n\ndef response_httprepr(response: Response) -> bytes:\n    \"\"\"Return raw HTTP representation (as bytes) of the given response. This\n    is provided only for reference, since it's not the exact stream of bytes\n    that was received (that's not exposed by Twisted).\n    \"\"\"\n    values = [\n        b\"HTTP/1.1 \",\n        to_bytes(str(response.status)),\n        b\" \",\n        to_bytes(http.RESPONSES.get(response.status, b'')),\n        b\"\\r\\n\",\n    ]\n    if response.headers:\n        values.extend([response.headers.to_string(), b\"\\r\\n\"])\n    values.extend([b\"\\r\\n\", response.body])\n    return b\"\".join(values)\n\n\ndef open_in_browser(\n    response: Union[\"scrapy.http.response.html.HtmlResponse\", \"scrapy.http.response.text.TextResponse\"],\n    _openfunc: Callable[[str], Any] = webbrowser.open,\n) -> Any:\n    \"\"\"Open the given response in a local web browser, populating the <base>\n    tag for external links to work\n    \"\"\"\n    from scrapy.http import HtmlResponse, TextResponse\n    # XXX: this implementation is a bit dirty and could be improved\n    body = response.body\n    if isinstance(response, HtmlResponse):\n        if b'<base' not in body:\n            repl = f'<head><base href=\"{response.url}\">'\n            body = body.replace(b'<head>', to_bytes(repl))\n        ext = '.html'\n    elif isinstance(response, TextResponse):\n        ext = '.txt'\n    else:\n        raise TypeError(\"Unsupported response type: \"\n                        f\"{response.__class__.__name__}\")\n    fd, fname = tempfile.mkstemp(ext)\n    os.write(fd, body)\n    os.close(fd)\n    return _openfunc(f\"file://{fname}\")\n", "path": "scrapy/utils/response.py"}, {"content": "from scrapy.exceptions import NotConfigured\nfrom scrapy.utils.request import request_httprepr\nfrom scrapy.utils.response import response_httprepr\nfrom scrapy.utils.python import global_object_name\n\n\nclass DownloaderStats:\n\n    def __init__(self, stats):\n        self.stats = stats\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        if not crawler.settings.getbool('DOWNLOADER_STATS'):\n            raise NotConfigured\n        return cls(crawler.stats)\n\n    def process_request(self, request, spider):\n        self.stats.inc_value('downloader/request_count', spider=spider)\n        self.stats.inc_value(f'downloader/request_method_count/{request.method}', spider=spider)\n        reqlen = len(request_httprepr(request))\n        self.stats.inc_value('downloader/request_bytes', reqlen, spider=spider)\n\n    def process_response(self, request, response, spider):\n        self.stats.inc_value('downloader/response_count', spider=spider)\n        self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)\n        reslen = len(response_httprepr(response))\n        self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)\n        return response\n\n    def process_exception(self, request, exception, spider):\n        ex_class = global_object_name(exception.__class__)\n        self.stats.inc_value('downloader/exception_count', spider=spider)\n        self.stats.inc_value(f'downloader/exception_type_count/{ex_class}', spider=spider)\n", "path": "scrapy/downloadermiddlewares/stats.py"}], "after_files": [{"content": "\"\"\"\nThis module provides some useful functions for working with\nscrapy.http.Response objects\n\"\"\"\nimport os\nimport webbrowser\nimport tempfile\nfrom typing import Any, Callable, Iterable, Optional, Tuple, Union\nfrom weakref import WeakKeyDictionary\n\nimport scrapy\nfrom scrapy.http.response import Response\n\nfrom twisted.web import http\nfrom scrapy.utils.python import to_bytes, to_unicode\nfrom scrapy.utils.decorators import deprecated\nfrom w3lib import html\n\n\n_baseurl_cache: \"WeakKeyDictionary[Response, str]\" = WeakKeyDictionary()\n\n\ndef get_base_url(response: \"scrapy.http.response.text.TextResponse\") -> str:\n    \"\"\"Return the base url of the given response, joined with the response url\"\"\"\n    if response not in _baseurl_cache:\n        text = response.text[0:4096]\n        _baseurl_cache[response] = html.get_base_url(text, response.url, response.encoding)\n    return _baseurl_cache[response]\n\n\n_metaref_cache: \"WeakKeyDictionary[Response, Union[Tuple[None, None], Tuple[float, str]]]\" = WeakKeyDictionary()\n\n\ndef get_meta_refresh(\n    response: \"scrapy.http.response.text.TextResponse\",\n    ignore_tags: Optional[Iterable[str]] = ('script', 'noscript'),\n) -> Union[Tuple[None, None], Tuple[float, str]]:\n    \"\"\"Parse the http-equiv refrsh parameter from the given response\"\"\"\n    if response not in _metaref_cache:\n        text = response.text[0:4096]\n        _metaref_cache[response] = html.get_meta_refresh(\n            text, response.url, response.encoding, ignore_tags=ignore_tags)\n    return _metaref_cache[response]\n\n\ndef response_status_message(status: Union[bytes, float, int, str]) -> str:\n    \"\"\"Return status code plus status text descriptive message\n    \"\"\"\n    status_int = int(status)\n    message = http.RESPONSES.get(status_int, \"Unknown Status\")\n    return f'{status_int} {to_unicode(message)}'\n\n\n@deprecated\ndef response_httprepr(response: Response) -> bytes:\n    \"\"\"Return raw HTTP representation (as bytes) of the given response. This\n    is provided only for reference, since it's not the exact stream of bytes\n    that was received (that's not exposed by Twisted).\n    \"\"\"\n    values = [\n        b\"HTTP/1.1 \",\n        to_bytes(str(response.status)),\n        b\" \",\n        to_bytes(http.RESPONSES.get(response.status, b'')),\n        b\"\\r\\n\",\n    ]\n    if response.headers:\n        values.extend([response.headers.to_string(), b\"\\r\\n\"])\n    values.extend([b\"\\r\\n\", response.body])\n    return b\"\".join(values)\n\n\ndef open_in_browser(\n    response: Union[\"scrapy.http.response.html.HtmlResponse\", \"scrapy.http.response.text.TextResponse\"],\n    _openfunc: Callable[[str], Any] = webbrowser.open,\n) -> Any:\n    \"\"\"Open the given response in a local web browser, populating the <base>\n    tag for external links to work\n    \"\"\"\n    from scrapy.http import HtmlResponse, TextResponse\n    # XXX: this implementation is a bit dirty and could be improved\n    body = response.body\n    if isinstance(response, HtmlResponse):\n        if b'<base' not in body:\n            repl = f'<head><base href=\"{response.url}\">'\n            body = body.replace(b'<head>', to_bytes(repl))\n        ext = '.html'\n    elif isinstance(response, TextResponse):\n        ext = '.txt'\n    else:\n        raise TypeError(\"Unsupported response type: \"\n                        f\"{response.__class__.__name__}\")\n    fd, fname = tempfile.mkstemp(ext)\n    os.write(fd, body)\n    os.close(fd)\n    return _openfunc(f\"file://{fname}\")\n", "path": "scrapy/utils/response.py"}, {"content": "from scrapy.exceptions import NotConfigured\nfrom scrapy.utils.python import global_object_name, to_bytes\nfrom scrapy.utils.request import request_httprepr\n\nfrom twisted.web import http\n\n\ndef get_header_size(headers):\n    size = 0\n    for key, value in headers.items():\n        if isinstance(value, (list, tuple)):\n            for v in value:\n                size += len(b\": \") + len(key) + len(v)\n    return size + len(b'\\r\\n') * (len(headers.keys()) - 1)\n\n\ndef get_status_size(response_status):\n    return len(to_bytes(http.RESPONSES.get(response_status, b''))) + 15\n    # resp.status + b\"\\r\\n\" + b\"HTTP/1.1 <100-599> \"\n\n\nclass DownloaderStats:\n\n    def __init__(self, stats):\n        self.stats = stats\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        if not crawler.settings.getbool('DOWNLOADER_STATS'):\n            raise NotConfigured\n        return cls(crawler.stats)\n\n    def process_request(self, request, spider):\n        self.stats.inc_value('downloader/request_count', spider=spider)\n        self.stats.inc_value(f'downloader/request_method_count/{request.method}', spider=spider)\n        reqlen = len(request_httprepr(request))\n        self.stats.inc_value('downloader/request_bytes', reqlen, spider=spider)\n\n    def process_response(self, request, response, spider):\n        self.stats.inc_value('downloader/response_count', spider=spider)\n        self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)\n        reslen = len(response.body) + get_header_size(response.headers) + get_status_size(response.status) + 4\n        # response.body + b\"\\r\\n\"+ response.header + b\"\\r\\n\" + response.status\n        self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)\n        return response\n\n    def process_exception(self, request, exception, spider):\n        ex_class = global_object_name(exception.__class__)\n        self.stats.inc_value('downloader/exception_count', spider=spider)\n        self.stats.inc_value(f'downloader/exception_type_count/{ex_class}', spider=spider)\n", "path": "scrapy/downloadermiddlewares/stats.py"}]}