Python decoding errors with BeautifulSoup, requests, and lxml
Question: I'm attempting to pull some data off a popular browser based game, but am
having trouble with some decoding errors:

    
    
    import requests
    from bs4 import BeautifulSoup
    
    r = requests.get("http://www.neopets.com/")
    p = BeautifulSoup(r.text)
    

This produces the following stack trace:

    
    
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "build/bdist.linux-x86_64/egg/bs4/__init__.py", line 172, in __init__
    
    File "build/bdist.linux-x86_64/egg/bs4/__init__.py", line 185, in _feed
    
    File "build/bdist.linux-x86_64/egg/bs4/builder/_lxml.py", line 195, in feed
    File "parser.pxi", line 1187, in lxml.etree._FeedParser.close    (src/lxml/lxml.etree.c:87912)
    File "parsertarget.pxi", line 130, in lxml.etree._TargetParserContext._handleParseResult (src/lxml/lxml.etree.c:97055)
    File "lxml.etree.pyx", line 294, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:8862)
    File "saxparser.pxi", line 274, in lxml.etree._handleSaxCData (src/lxml/lxml.etree.c:93385)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xb1 in position 476: invalid start byte
    

Doing the following:

    
    
    print repr(r.text[476 - 10: 476 + 10])
    

Produces:

    
    
    u'ttp-equiv="X-UA-Comp'
    

I'm really not sure what the issue here is. Any help is greatly appreciated.
Thank you.


Answer: `.text` on a response returns a decoded unicode value, but perhaps you should
let BeautifulSoup do the decoding for you:

    
    
    p = BeautifulSoup(r.content, from_encoding=r.encoding)
    

`r.content` returns the un-decoded raw bytestring, and `r.encoding` is the
encoding detected from the headers.

