prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Usage of HttpCompressionMiddleware needs to be reflected in Scrapy stats
  ## Summary
  Usage of `HttpCompressionMiddleware` needs to be relfected in Scrapy stats.
  ## Motivation
  In order to estimate scrapy memory usage efficiency and prevent.. memory leaks like [this](https://stackoverflow.com/q/63936759/10884791).
  I will need to know:
  1. number of request/response objects that can be active (can be achieved by using [`trackref`](https://docs.scrapy.org/en/latest/topics/leaks.html#debugging-memory-leaks-with-trackref) )
  2. size of memory required to store that number of request/response objects. 
  
  A lot of websites use compression to reduce traffic. In this case I would like to calculate average size of **decompressed** responses to estimate p.2.
  
  Decompression process means that at some point application will require to allocate memory to store both compressed and decompressed response body and I will need to know this sizes to have more complete vision of scrapy memory usage.
  
  Also size of  decompressed body will be several times more than size of compressed response and it will affect scrapy memory usage.
  
  ## Describe alternatives you've considered
  The easiest one - is to change priority of `DownloaderStats` middleware and check difference in `downloader/response_bytes` stats parameter.
  ```
      custom_settings = {"DOWNLOAD_DELAY":1,
                         "DOWNLOADER_MIDDLEWARES":{
                             'scrapy.downloadermiddlewares.stats.DownloaderStats':50
                         }
  ```
  Stats from quotes.toscrape.com spider (it uses `gzip` compression) with default settings:
  
  ```
  {'downloader/request_bytes': 2642,
   'downloader/request_count': 10,
   'downloader/request_method_count/GET': 10,
   'downloader/response_bytes': 24534,
  ```
   
  And with changed priority of `DownloaderStats` middleware:
  
  ```
  {'downloader/request_bytes': 912, # size reduced as it didn't count size of request headers populated by downloader middlewares
   'downloader/request_count': 10,
   'downloader/request_method_count/GET': 10,
   'downloader/response_bytes': 110191,   # it counted size of decompressed data 
  ```
  
  Average size of compressed response (by default) - 2453 bytes.
  Average size of decompressed response  - 11019 bytes (~4.5 times more).
  
  ## Additional context
  Potential solution is to add something like this:
  ` self.stats.inc_value('decompressed_bytes', spider=spider)`
  into `process_response` method of `HttpCompressionMiddleware`
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `scrapy/downloadermiddlewares/httpcompression.py`
  Content:
  ```
  1 import io
  2 import zlib
  3 
  4 from scrapy.utils.gz import gunzip
  5 from scrapy.http import Response, TextResponse
  6 from scrapy.responsetypes import responsetypes
  7 from scrapy.exceptions import NotConfigured
  8 
  9 
  10 ACCEPTED_ENCODINGS = [b'gzip', b'deflate']
  11 
  12 try:
  13     import brotli
  14     ACCEPTED_ENCODINGS.append(b'br')
  15 except ImportError:
  16     pass
  17 
  18 try:
  19     import zstandard
  20     ACCEPTED_ENCODINGS.append(b'zstd')
  21 except ImportError:
  22     pass
  23 
  24 
  25 class HttpCompressionMiddleware:
  26     """This middleware allows compressed (gzip, deflate) traffic to be
  27     sent/received from web sites"""
  28     @classmethod
  29     def from_crawler(cls, crawler):
  30         if not crawler.settings.getbool('COMPRESSION_ENABLED'):
  31             raise NotConfigured
  32         return cls()
  33 
  34     def process_request(self, request, spider):
  35         request.headers.setdefault('Accept-Encoding',
  36                                    b", ".join(ACCEPTED_ENCODINGS))
  37 
  38     def process_response(self, request, response, spider):
  39 
  40         if request.method == 'HEAD':
  41             return response
  42         if isinstance(response, Response):
  43             content_encoding = response.headers.getlist('Content-Encoding')
  44             if content_encoding:
  45                 encoding = content_encoding.pop()
  46                 decoded_body = self._decode(response.body, encoding.lower())
  47                 respcls = responsetypes.from_args(
  48                     headers=response.headers, url=response.url, body=decoded_body
  49                 )
  50                 kwargs = dict(cls=respcls, body=decoded_body)
  51                 if issubclass(respcls, TextResponse):
  52                     # force recalculating the encoding until we make sure the
  53                     # responsetypes guessing is reliable
  54                     kwargs['encoding'] = None
  55                 response = response.replace(**kwargs)
  56                 if not content_encoding:
  57                     del response.headers['Content-Encoding']
  58 
  59         return response
  60 
  61     def _decode(self, body, encoding):
  62         if encoding == b'gzip' or encoding == b'x-gzip':
  63             body = gunzip(body)
  64 
  65         if encoding == b'deflate':
  66             try:
  67                 body = zlib.decompress(body)
  68             except zlib.error:
  69                 # ugly hack to work with raw deflate content that may
  70                 # be sent by microsoft servers. For more information, see:
  71                 # http://carsten.codimi.de/gzip.yaws/
  72                 # http://www.port80software.com/200ok/archive/2005/10/31/868.aspx
  73                 # http://www.gzip.org/zlib/zlib_faq.html#faq38
  74                 body = zlib.decompress(body, -15)
  75         if encoding == b'br' and b'br' in ACCEPTED_ENCODINGS:
  76             body = brotli.decompress(body)
  77         if encoding == b'zstd' and b'zstd' in ACCEPTED_ENCODINGS:
  78             # Using its streaming API since its simple API could handle only cases
  79             # where there is content size data embedded in the frame
  80             reader = zstandard.ZstdDecompressor().stream_reader(io.BytesIO(body))
  81             body = reader.read()
  82         return body
  83
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py
  --- a/scrapy/downloadermiddlewares/httpcompression.py
  +++ b/scrapy/downloadermiddlewares/httpcompression.py
  @@ -1,10 +1,12 @@
   import io
  +import warnings
   import zlib
   
  -from scrapy.utils.gz import gunzip
  +from scrapy.exceptions import NotConfigured
   from scrapy.http import Response, TextResponse
   from scrapy.responsetypes import responsetypes
  -from scrapy.exceptions import NotConfigured
  +from scrapy.utils.deprecate import ScrapyDeprecationWarning
  +from scrapy.utils.gz import gunzip
   
   
   ACCEPTED_ENCODINGS = [b'gzip', b'deflate']
  @@ -25,11 +27,25 @@
   class HttpCompressionMiddleware:
       """This middleware allows compressed (gzip, deflate) traffic to be
       sent/received from web sites"""
  +    def __init__(self, stats=None):
  +        self.stats = stats
  +
       @classmethod
       def from_crawler(cls, crawler):
           if not crawler.settings.getbool('COMPRESSION_ENABLED'):
               raise NotConfigured
  -        return cls()
  +        try:
  +            return cls(stats=crawler.stats)
  +        except TypeError:
  +            warnings.warn(
  +                "HttpCompressionMiddleware subclasses must either modify "
  +                "their '__init__' method to support a 'stats' parameter or "
  +                "reimplement the 'from_crawler' method.",
  +                ScrapyDeprecationWarning,
  +            )
  +            result = cls()
  +            result.stats = crawler.stats
  +            return result
   
       def process_request(self, request, spider):
           request.headers.setdefault('Accept-Encoding',
  @@ -44,6 +60,9 @@
               if content_encoding:
                   encoding = content_encoding.pop()
                   decoded_body = self._decode(response.body, encoding.lower())
  +                if self.stats:
  +                    self.stats.inc_value('httpcompression/response_bytes', len(decoded_body), spider=spider)
  +                    self.stats.inc_value('httpcompression/response_count', spider=spider)
                   respcls = responsetypes.from_args(
                       headers=response.headers, url=response.url, body=decoded_body
                   )
  
verification_info: {"golden_diff": "diff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py\n--- a/scrapy/downloadermiddlewares/httpcompression.py\n+++ b/scrapy/downloadermiddlewares/httpcompression.py\n@@ -1,10 +1,12 @@\n import io\n+import warnings\n import zlib\n \n-from scrapy.utils.gz import gunzip\n+from scrapy.exceptions import NotConfigured\n from scrapy.http import Response, TextResponse\n from scrapy.responsetypes import responsetypes\n-from scrapy.exceptions import NotConfigured\n+from scrapy.utils.deprecate import ScrapyDeprecationWarning\n+from scrapy.utils.gz import gunzip\n \n \n ACCEPTED_ENCODINGS = [b'gzip', b'deflate']\n@@ -25,11 +27,25 @@\n class HttpCompressionMiddleware:\n     \"\"\"This middleware allows compressed (gzip, deflate) traffic to be\n     sent/received from web sites\"\"\"\n+    def __init__(self, stats=None):\n+        self.stats = stats\n+\n     @classmethod\n     def from_crawler(cls, crawler):\n         if not crawler.settings.getbool('COMPRESSION_ENABLED'):\n             raise NotConfigured\n-        return cls()\n+        try:\n+            return cls(stats=crawler.stats)\n+        except TypeError:\n+            warnings.warn(\n+                \"HttpCompressionMiddleware subclasses must either modify \"\n+                \"their '__init__' method to support a 'stats' parameter or \"\n+                \"reimplement the 'from_crawler' method.\",\n+                ScrapyDeprecationWarning,\n+            )\n+            result = cls()\n+            result.stats = crawler.stats\n+            return result\n \n     def process_request(self, request, spider):\n         request.headers.setdefault('Accept-Encoding',\n@@ -44,6 +60,9 @@\n             if content_encoding:\n                 encoding = content_encoding.pop()\n                 decoded_body = self._decode(response.body, encoding.lower())\n+                if self.stats:\n+                    self.stats.inc_value('httpcompression/response_bytes', len(decoded_body), spider=spider)\n+                    self.stats.inc_value('httpcompression/response_count', spider=spider)\n                 respcls = responsetypes.from_args(\n                     headers=response.headers, url=response.url, body=decoded_body\n                 )\n", "issue": "Usage of HttpCompressionMiddleware needs to be reflected in Scrapy stats\n## Summary\r\nUsage of `HttpCompressionMiddleware` needs to be relfected in Scrapy stats.\r\n## Motivation\r\nIn order to estimate scrapy memory usage efficiency and prevent.. memory leaks like [this](https://stackoverflow.com/q/63936759/10884791).\r\nI will need to know:\r\n1. number of request/response objects that can be active (can be achieved by using [`trackref`](https://docs.scrapy.org/en/latest/topics/leaks.html#debugging-memory-leaks-with-trackref) )\r\n2. size of memory required to store that number of request/response objects. \r\n\r\nA lot of websites use compression to reduce traffic. In this case I would like to calculate average size of **decompressed** responses to estimate p.2.\r\n\r\nDecompression process means that at some point application will require to allocate memory to store both compressed and decompressed response body and I will need to know this sizes to have more complete vision of scrapy memory usage.\r\n\r\nAlso size of  decompressed body will be several times more than size of compressed response and it will affect scrapy memory usage.\r\n\r\n## Describe alternatives you've considered\r\nThe easiest one - is to change priority of `DownloaderStats` middleware and check difference in `downloader/response_bytes` stats parameter.\r\n```\r\n    custom_settings = {\"DOWNLOAD_DELAY\":1,\r\n                       \"DOWNLOADER_MIDDLEWARES\":{\r\n                           'scrapy.downloadermiddlewares.stats.DownloaderStats':50\r\n                       }\r\n```\r\nStats from quotes.toscrape.com spider (it uses `gzip` compression) with default settings:\r\n\r\n```\r\n{'downloader/request_bytes': 2642,\r\n 'downloader/request_count': 10,\r\n 'downloader/request_method_count/GET': 10,\r\n 'downloader/response_bytes': 24534,\r\n```\r\n \r\nAnd with changed priority of `DownloaderStats` middleware:\r\n\r\n```\r\n{'downloader/request_bytes': 912, # size reduced as it didn't count size of request headers populated by downloader middlewares\r\n 'downloader/request_count': 10,\r\n 'downloader/request_method_count/GET': 10,\r\n 'downloader/response_bytes': 110191,   # it counted size of decompressed data \r\n```\r\n\r\nAverage size of compressed response (by default) - 2453 bytes.\r\nAverage size of decompressed response  - 11019 bytes (~4.5 times more).\r\n\r\n## Additional context\r\nPotential solution is to add something like this:\r\n` self.stats.inc_value('decompressed_bytes', spider=spider)`\r\ninto `process_response` method of `HttpCompressionMiddleware`\n", "before_files": [{"content": "import io\nimport zlib\n\nfrom scrapy.utils.gz import gunzip\nfrom scrapy.http import Response, TextResponse\nfrom scrapy.responsetypes import responsetypes\nfrom scrapy.exceptions import NotConfigured\n\n\nACCEPTED_ENCODINGS = [b'gzip', b'deflate']\n\ntry:\n    import brotli\n    ACCEPTED_ENCODINGS.append(b'br')\nexcept ImportError:\n    pass\n\ntry:\n    import zstandard\n    ACCEPTED_ENCODINGS.append(b'zstd')\nexcept ImportError:\n    pass\n\n\nclass HttpCompressionMiddleware:\n    \"\"\"This middleware allows compressed (gzip, deflate) traffic to be\n    sent/received from web sites\"\"\"\n    @classmethod\n    def from_crawler(cls, crawler):\n        if not crawler.settings.getbool('COMPRESSION_ENABLED'):\n            raise NotConfigured\n        return cls()\n\n    def process_request(self, request, spider):\n        request.headers.setdefault('Accept-Encoding',\n                                   b\", \".join(ACCEPTED_ENCODINGS))\n\n    def process_response(self, request, response, spider):\n\n        if request.method == 'HEAD':\n            return response\n        if isinstance(response, Response):\n            content_encoding = response.headers.getlist('Content-Encoding')\n            if content_encoding:\n                encoding = content_encoding.pop()\n                decoded_body = self._decode(response.body, encoding.lower())\n                respcls = responsetypes.from_args(\n                    headers=response.headers, url=response.url, body=decoded_body\n                )\n                kwargs = dict(cls=respcls, body=decoded_body)\n                if issubclass(respcls, TextResponse):\n                    # force recalculating the encoding until we make sure the\n                    # responsetypes guessing is reliable\n                    kwargs['encoding'] = None\n                response = response.replace(**kwargs)\n                if not content_encoding:\n                    del response.headers['Content-Encoding']\n\n        return response\n\n    def _decode(self, body, encoding):\n        if encoding == b'gzip' or encoding == b'x-gzip':\n            body = gunzip(body)\n\n        if encoding == b'deflate':\n            try:\n                body = zlib.decompress(body)\n            except zlib.error:\n                # ugly hack to work with raw deflate content that may\n                # be sent by microsoft servers. For more information, see:\n                # http://carsten.codimi.de/gzip.yaws/\n                # http://www.port80software.com/200ok/archive/2005/10/31/868.aspx\n                # http://www.gzip.org/zlib/zlib_faq.html#faq38\n                body = zlib.decompress(body, -15)\n        if encoding == b'br' and b'br' in ACCEPTED_ENCODINGS:\n            body = brotli.decompress(body)\n        if encoding == b'zstd' and b'zstd' in ACCEPTED_ENCODINGS:\n            # Using its streaming API since its simple API could handle only cases\n            # where there is content size data embedded in the frame\n            reader = zstandard.ZstdDecompressor().stream_reader(io.BytesIO(body))\n            body = reader.read()\n        return body\n", "path": "scrapy/downloadermiddlewares/httpcompression.py"}], "after_files": [{"content": "import io\nimport warnings\nimport zlib\n\nfrom scrapy.exceptions import NotConfigured\nfrom scrapy.http import Response, TextResponse\nfrom scrapy.responsetypes import responsetypes\nfrom scrapy.utils.deprecate import ScrapyDeprecationWarning\nfrom scrapy.utils.gz import gunzip\n\n\nACCEPTED_ENCODINGS = [b'gzip', b'deflate']\n\ntry:\n    import brotli\n    ACCEPTED_ENCODINGS.append(b'br')\nexcept ImportError:\n    pass\n\ntry:\n    import zstandard\n    ACCEPTED_ENCODINGS.append(b'zstd')\nexcept ImportError:\n    pass\n\n\nclass HttpCompressionMiddleware:\n    \"\"\"This middleware allows compressed (gzip, deflate) traffic to be\n    sent/received from web sites\"\"\"\n    def __init__(self, stats=None):\n        self.stats = stats\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        if not crawler.settings.getbool('COMPRESSION_ENABLED'):\n            raise NotConfigured\n        try:\n            return cls(stats=crawler.stats)\n        except TypeError:\n            warnings.warn(\n                \"HttpCompressionMiddleware subclasses must either modify \"\n                \"their '__init__' method to support a 'stats' parameter or \"\n                \"reimplement the 'from_crawler' method.\",\n                ScrapyDeprecationWarning,\n            )\n            result = cls()\n            result.stats = crawler.stats\n            return result\n\n    def process_request(self, request, spider):\n        request.headers.setdefault('Accept-Encoding',\n                                   b\", \".join(ACCEPTED_ENCODINGS))\n\n    def process_response(self, request, response, spider):\n\n        if request.method == 'HEAD':\n            return response\n        if isinstance(response, Response):\n            content_encoding = response.headers.getlist('Content-Encoding')\n            if content_encoding:\n                encoding = content_encoding.pop()\n                decoded_body = self._decode(response.body, encoding.lower())\n                if self.stats:\n                    self.stats.inc_value('httpcompression/response_bytes', len(decoded_body), spider=spider)\n                    self.stats.inc_value('httpcompression/response_count', spider=spider)\n                respcls = responsetypes.from_args(\n                    headers=response.headers, url=response.url, body=decoded_body\n                )\n                kwargs = dict(cls=respcls, body=decoded_body)\n                if issubclass(respcls, TextResponse):\n                    # force recalculating the encoding until we make sure the\n                    # responsetypes guessing is reliable\n                    kwargs['encoding'] = None\n                response = response.replace(**kwargs)\n                if not content_encoding:\n                    del response.headers['Content-Encoding']\n\n        return response\n\n    def _decode(self, body, encoding):\n        if encoding == b'gzip' or encoding == b'x-gzip':\n            body = gunzip(body)\n\n        if encoding == b'deflate':\n            try:\n                body = zlib.decompress(body)\n            except zlib.error:\n                # ugly hack to work with raw deflate content that may\n                # be sent by microsoft servers. For more information, see:\n                # http://carsten.codimi.de/gzip.yaws/\n                # http://www.port80software.com/200ok/archive/2005/10/31/868.aspx\n                # http://www.gzip.org/zlib/zlib_faq.html#faq38\n                body = zlib.decompress(body, -15)\n        if encoding == b'br' and b'br' in ACCEPTED_ENCODINGS:\n            body = brotli.decompress(body)\n        if encoding == b'zstd' and b'zstd' in ACCEPTED_ENCODINGS:\n            # Using its streaming API since its simple API could handle only cases\n            # where there is content size data embedded in the frame\n            reader = zstandard.ZstdDecompressor().stream_reader(io.BytesIO(body))\n            body = reader.read()\n        return body\n", "path": "scrapy/downloadermiddlewares/httpcompression.py"}]}