prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  IOError, 'Not a gzipped file'
  while trying to access sitemap from robots.txt , Scrapy fails with **IOError, 'Not a gzipped file'** error
  
  not sure if this issue is related to following issue(s)
  https://github.com/scrapy/scrapy/issues/193 -> closed issue
  https://github.com/scrapy/scrapy/pull/660 -> merged pull request to address issue 193
  https://github.com/scrapy/scrapy/issues/951 -> open issue
  
  > line where code fails in gzip.py at line # 197
  > 
  > ``` python
  > def _read_gzip_header(self):
  >         magic = self.fileobj.read(2)
  >         if magic != '\037\213':
  >             raise IOError, 'Not a gzipped file'
  > ```
  # Response Header
  
  ```
  Content-Encoding: gzip
  Accept-Ranges: bytes
  X-Amz-Request-Id: BFFF010DDE6268DA
  Vary: Accept-Encoding
  Server: AmazonS3
  Last-Modified: Wed, 15 Jun 2016 19:02:20 GMT
  Etag: "300bb71d6897cb2a22bba0bd07978c84"
  Cache-Control: no-transform
  Date: Sun, 19 Jun 2016 10:54:53 GMT
  Content-Type: binary/octet-stream
  ```
  
  Error Log:
  
  ``` log
   Traceback (most recent call last):
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
      yield next(it)
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
      for x in result:
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr>
      return (_set_referer(r) for r in result or ())
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
      return (r for r in result or () if _filter(r))
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
      return (r for r in result or () if _filter(r))
    File "D:\projects\sitemap_spider\sitemap_spider\spiders\mainspider.py", line 31, in _parse_sitemap
      body = self._get_sitemap_body(response)
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spiders\sitemap.py", line 67, in _get_sitemap_body
      return gunzip(response.body)
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\utils\gz.py", line 37, in gunzip
      chunk = read1(f, 8196)
    File "c:\venv\scrapy1.0\lib\site-packages\scrapy\utils\gz.py", line 21, in read1
      return gzf.read(size)
    File "c:\python27\Lib\gzip.py", line 268, in read
      self._read(readsize)
    File "c:\python27\Lib\gzip.py", line 303, in _read
      self._read_gzip_header()
    File "c:\python27\Lib\gzip.py", line 197, in _read_gzip_header
      raise IOError, 'Not a gzipped file'
  ```
  
  i did download file manually and was able to extract the content so it is not like file is corrupted 
  
  as an example sitemap url : you can follow amazon robots.txt
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `scrapy/utils/gz.py`
  Content:
  ```
  1 import struct
  2 
  3 try:
  4     from cStringIO import StringIO as BytesIO
  5 except ImportError:
  6     from io import BytesIO
  7 from gzip import GzipFile
  8 
  9 import six
  10 import re
  11 
  12 # - Python>=3.5 GzipFile's read() has issues returning leftover
  13 #   uncompressed data when input is corrupted
  14 #   (regression or bug-fix compared to Python 3.4)
  15 # - read1(), which fetches data before raising EOFError on next call
  16 #   works here but is only available from Python>=3.3
  17 # - scrapy does not support Python 3.2
  18 # - Python 2.7 GzipFile works fine with standard read() + extrabuf
  19 if six.PY2:
  20     def read1(gzf, size=-1):
  21         return gzf.read(size)
  22 else:
  23     def read1(gzf, size=-1):
  24         return gzf.read1(size)
  25 
  26 
  27 def gunzip(data):
  28     """Gunzip the given data and return as much data as possible.
  29 
  30     This is resilient to CRC checksum errors.
  31     """
  32     f = GzipFile(fileobj=BytesIO(data))
  33     output = b''
  34     chunk = b'.'
  35     while chunk:
  36         try:
  37             chunk = read1(f, 8196)
  38             output += chunk
  39         except (IOError, EOFError, struct.error):
  40             # complete only if there is some data, otherwise re-raise
  41             # see issue 87 about catching struct.error
  42             # some pages are quite small so output is '' and f.extrabuf
  43             # contains the whole page content
  44             if output or getattr(f, 'extrabuf', None):
  45                 try:
  46                     output += f.extrabuf
  47                 finally:
  48                     break
  49             else:
  50                 raise
  51     return output
  52 
  53 _is_gzipped_re = re.compile(br'^application/(x-)?gzip\b', re.I)
  54 
  55 def is_gzipped(response):
  56     """Return True if the response is gzipped, or False otherwise"""
  57     ctype = response.headers.get('Content-Type', b'')
  58     return _is_gzipped_re.search(ctype) is not None
  59
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py
  --- a/scrapy/utils/gz.py
  +++ b/scrapy/utils/gz.py
  @@ -50,9 +50,12 @@
                   raise
       return output
   
  -_is_gzipped_re = re.compile(br'^application/(x-)?gzip\b', re.I)
  +_is_gzipped = re.compile(br'^application/(x-)?gzip\b', re.I).search
  +_is_octetstream = re.compile(br'^(application|binary)/octet-stream\b', re.I).search
   
   def is_gzipped(response):
       """Return True if the response is gzipped, or False otherwise"""
       ctype = response.headers.get('Content-Type', b'')
  -    return _is_gzipped_re.search(ctype) is not None
  +    cenc = response.headers.get('Content-Encoding', b'').lower()
  +    return (_is_gzipped(ctype) or
  +            (_is_octetstream(ctype) and cenc in (b'gzip', b'x-gzip')))
  
verification_info: {"golden_diff": "diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py\n--- a/scrapy/utils/gz.py\n+++ b/scrapy/utils/gz.py\n@@ -50,9 +50,12 @@\n                 raise\n     return output\n \n-_is_gzipped_re = re.compile(br'^application/(x-)?gzip\\b', re.I)\n+_is_gzipped = re.compile(br'^application/(x-)?gzip\\b', re.I).search\n+_is_octetstream = re.compile(br'^(application|binary)/octet-stream\\b', re.I).search\n \n def is_gzipped(response):\n     \"\"\"Return True if the response is gzipped, or False otherwise\"\"\"\n     ctype = response.headers.get('Content-Type', b'')\n-    return _is_gzipped_re.search(ctype) is not None\n+    cenc = response.headers.get('Content-Encoding', b'').lower()\n+    return (_is_gzipped(ctype) or\n+            (_is_octetstream(ctype) and cenc in (b'gzip', b'x-gzip')))\n", "issue": "IOError, 'Not a gzipped file'\nwhile trying to access sitemap from robots.txt , Scrapy fails with **IOError, 'Not a gzipped file'** error\n\nnot sure if this issue is related to following issue(s)\nhttps://github.com/scrapy/scrapy/issues/193 -> closed issue\nhttps://github.com/scrapy/scrapy/pull/660 -> merged pull request to address issue 193\nhttps://github.com/scrapy/scrapy/issues/951 -> open issue\n\n> line where code fails in gzip.py at line # 197\n> \n> ``` python\n> def _read_gzip_header(self):\n>         magic = self.fileobj.read(2)\n>         if magic != '\\037\\213':\n>             raise IOError, 'Not a gzipped file'\n> ```\n# Response Header\n\n```\nContent-Encoding: gzip\nAccept-Ranges: bytes\nX-Amz-Request-Id: BFFF010DDE6268DA\nVary: Accept-Encoding\nServer: AmazonS3\nLast-Modified: Wed, 15 Jun 2016 19:02:20 GMT\nEtag: \"300bb71d6897cb2a22bba0bd07978c84\"\nCache-Control: no-transform\nDate: Sun, 19 Jun 2016 10:54:53 GMT\nContent-Type: binary/octet-stream\n```\n\nError Log:\n\n``` log\n Traceback (most recent call last):\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\utils\\defer.py\", line 102, in iter_errback\n    yield next(it)\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\spidermiddlewares\\offsite.py\", line 29, in process_spider_output\n    for x in result:\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\spidermiddlewares\\referer.py\", line 22, in <genexpr>\n    return (_set_referer(r) for r in result or ())\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\spidermiddlewares\\urllength.py\", line 37, in <genexpr>\n    return (r for r in result or () if _filter(r))\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\spidermiddlewares\\depth.py\", line 58, in <genexpr>\n    return (r for r in result or () if _filter(r))\n  File \"D:\\projects\\sitemap_spider\\sitemap_spider\\spiders\\mainspider.py\", line 31, in _parse_sitemap\n    body = self._get_sitemap_body(response)\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\spiders\\sitemap.py\", line 67, in _get_sitemap_body\n    return gunzip(response.body)\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\utils\\gz.py\", line 37, in gunzip\n    chunk = read1(f, 8196)\n  File \"c:\\venv\\scrapy1.0\\lib\\site-packages\\scrapy\\utils\\gz.py\", line 21, in read1\n    return gzf.read(size)\n  File \"c:\\python27\\Lib\\gzip.py\", line 268, in read\n    self._read(readsize)\n  File \"c:\\python27\\Lib\\gzip.py\", line 303, in _read\n    self._read_gzip_header()\n  File \"c:\\python27\\Lib\\gzip.py\", line 197, in _read_gzip_header\n    raise IOError, 'Not a gzipped file'\n```\n\ni did download file manually and was able to extract the content so it is not like file is corrupted \n\nas an example sitemap url : you can follow amazon robots.txt\n\n", "before_files": [{"content": "import struct\n\ntry:\n    from cStringIO import StringIO as BytesIO\nexcept ImportError:\n    from io import BytesIO\nfrom gzip import GzipFile\n\nimport six\nimport re\n\n# - Python>=3.5 GzipFile's read() has issues returning leftover\n#   uncompressed data when input is corrupted\n#   (regression or bug-fix compared to Python 3.4)\n# - read1(), which fetches data before raising EOFError on next call\n#   works here but is only available from Python>=3.3\n# - scrapy does not support Python 3.2\n# - Python 2.7 GzipFile works fine with standard read() + extrabuf\nif six.PY2:\n    def read1(gzf, size=-1):\n        return gzf.read(size)\nelse:\n    def read1(gzf, size=-1):\n        return gzf.read1(size)\n\n\ndef gunzip(data):\n    \"\"\"Gunzip the given data and return as much data as possible.\n\n    This is resilient to CRC checksum errors.\n    \"\"\"\n    f = GzipFile(fileobj=BytesIO(data))\n    output = b''\n    chunk = b'.'\n    while chunk:\n        try:\n            chunk = read1(f, 8196)\n            output += chunk\n        except (IOError, EOFError, struct.error):\n            # complete only if there is some data, otherwise re-raise\n            # see issue 87 about catching struct.error\n            # some pages are quite small so output is '' and f.extrabuf\n            # contains the whole page content\n            if output or getattr(f, 'extrabuf', None):\n                try:\n                    output += f.extrabuf\n                finally:\n                    break\n            else:\n                raise\n    return output\n\n_is_gzipped_re = re.compile(br'^application/(x-)?gzip\\b', re.I)\n\ndef is_gzipped(response):\n    \"\"\"Return True if the response is gzipped, or False otherwise\"\"\"\n    ctype = response.headers.get('Content-Type', b'')\n    return _is_gzipped_re.search(ctype) is not None\n", "path": "scrapy/utils/gz.py"}], "after_files": [{"content": "import struct\n\ntry:\n    from cStringIO import StringIO as BytesIO\nexcept ImportError:\n    from io import BytesIO\nfrom gzip import GzipFile\n\nimport six\nimport re\n\n# - Python>=3.5 GzipFile's read() has issues returning leftover\n#   uncompressed data when input is corrupted\n#   (regression or bug-fix compared to Python 3.4)\n# - read1(), which fetches data before raising EOFError on next call\n#   works here but is only available from Python>=3.3\n# - scrapy does not support Python 3.2\n# - Python 2.7 GzipFile works fine with standard read() + extrabuf\nif six.PY2:\n    def read1(gzf, size=-1):\n        return gzf.read(size)\nelse:\n    def read1(gzf, size=-1):\n        return gzf.read1(size)\n\n\ndef gunzip(data):\n    \"\"\"Gunzip the given data and return as much data as possible.\n\n    This is resilient to CRC checksum errors.\n    \"\"\"\n    f = GzipFile(fileobj=BytesIO(data))\n    output = b''\n    chunk = b'.'\n    while chunk:\n        try:\n            chunk = read1(f, 8196)\n            output += chunk\n        except (IOError, EOFError, struct.error):\n            # complete only if there is some data, otherwise re-raise\n            # see issue 87 about catching struct.error\n            # some pages are quite small so output is '' and f.extrabuf\n            # contains the whole page content\n            if output or getattr(f, 'extrabuf', None):\n                try:\n                    output += f.extrabuf\n                finally:\n                    break\n            else:\n                raise\n    return output\n\n_is_gzipped = re.compile(br'^application/(x-)?gzip\\b', re.I).search\n_is_octetstream = re.compile(br'^(application|binary)/octet-stream\\b', re.I).search\n\ndef is_gzipped(response):\n    \"\"\"Return True if the response is gzipped, or False otherwise\"\"\"\n    ctype = response.headers.get('Content-Type', b'')\n    cenc = response.headers.get('Content-Encoding', b'').lower()\n    return (_is_gzipped(ctype) or\n            (_is_octetstream(ctype) and cenc in (b'gzip', b'x-gzip')))\n", "path": "scrapy/utils/gz.py"}]}