Python3 urlopen read weirdness (gzip)
Question: I'm getting an URL from Schema.org. It's content-type="text/html"

Sometimes, read() functions as expected b'< !DOCTYPE html> ....'

Sometimes, read() returns something else b'\x1f\x8b\x08\x00\x00\x00\x00 ...'

    
    
    try:
        with urlopen("http://schema.org/docs/releases.html") as f:
            txt = f.read()
    except URLError:
        return
    

I've tried solving this with `txt = f.read().decode("utf-8").encode()` but
this results in an error... sometimes: UnicodeDecodeError: 'utf-8' codec can't
decode byte 0x8b in position 1: invalid start byte

The obvious work-around is to test if the first byte is hex and treat this
accordingly.

My question is: Is this a bug or something else?

[![enter image description
here](http://i.stack.imgur.com/mzWfR.png)](http://i.stack.imgur.com/mzWfR.png)
**Edit** Related [question](http://stackoverflow.com/questions/3746993/change-
python-byte-type-to-string). Apparently, sometimes I'm getting a gzipped
stream.

**Lastly** I solved this by adding the following code as [proposed
here](http://stackoverflow.com/questions/2695152/in-python-how-do-i-decode-
gzip-encoding)

    
    
    if 31 == txt[0]:
        txt = decompress(txt, 16+MAX_WBITS)
    

The question remains; why does this return text/html sometimes and zipped some
other times?


Answer: You are indeed receiving a gzipped response. You should be able to avoid it
by:

    
    
    from urllib import request
    try:
        req = request.Request("http://schema.org/docs/releases.html")
        req.add_header('Accept-Encoding', 'identity;q=1')
        with request.urlopen(req) as f:
            txt = f.read()
    except request.URLError:
        return
    

