title: BeautifulSoup character code error
body: <p>I am using BeautifulSoup for scraping website info. Specifically, I want to gather information on patents from a google search (title, inventors, abstract, etc).  I have a list of URLs for each patent, but BeautifulSoup is having trouble with certain sites, giving me the following error:</p>
  
  <p><strong>UnicodeDecodeError: 'utf8' codec can't decode byte 0xcc in position 531: invalid continuation byte</strong></p>
  
  <p>Below is the error traceback:</p>
  
  <pre><code>Traceback (most recent call last):
      soup = BeautifulSoup(the_page,from_encoding='utf-8')
    File "C:\Python27\lib\site-packages\bs4\__init__.py", line 172, in __init__
      self._feed()
    File "C:\Python27\lib\site-packages\bs4\__init__.py", line 185, in _feed
      self.builder.feed(self.markup)
    File "C:\Python27\lib\site-packages\bs4\builder\_lxml.py", line 195, in feed
      self.parser.close()
    File "parser.pxi", line 1209, in lxml.etree._FeedParser.close (src\lxml\lxml.etree.c:90597)
    File "parsertarget.pxi", line 142, in lxml.etree._TargetParserContext._handleParseResult (src\lxml\lxml.etree.c:99984)
    File "parsertarget.pxi", line 130, in lxml.etree._TargetParserContext._handleParseResult (src\lxml\lxml.etree.c:99807)
    File "lxml.etree.pyx", line 294, in lxml.etree._ExceptionContext._raise_if_stored (src\lxml\lxml.etree.c:9383)
    File "saxparser.pxi", line 259, in lxml.etree._handleSaxData (src\lxml\lxml.etree.c:95945)
  UnicodeDecodeError: 'utf8' codec can't decode byte 0xcc in position 531: invalid continuation byte
  </code></pre>
  
  <p>I checked the encoding of the site, and it claims to be 'utf-8'.  I specified this as an input to BeautifulSoup as well.  Below is my code:</p>
  
  <pre><code>import urllib, urllib2
  from bs4 import BeautifulSoup
  
  #url = 'https://www.google.com/patents/WO2001019016A1?cl=en'  # This one works
  url = 'https://www.google.com/patents/WO2006016929A2?cl=en' # This one doesn't work
  
  user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
  values = {'name' : 'Somebody',
            'location' : 'Somewhere',
            'language' : 'Python' }
  headers = { 'User-Agent' : user_agent }
  
  data = urllib.urlencode(values)
  req = urllib2.Request(url, data, headers)
  response = urllib2.urlopen(req)
  the_page = response.read()
  
  print response.headers['content-type']
  print response.headers.getencoding()
  
  soup = BeautifulSoup(the_page,from_encoding='utf-8')
  </code></pre>
  
  <p>I included two urls.  One results in an error, the other works fine (labeled as such in the comments). In both cases, I could print the html to the terminal fine, but BeautifulSoup consistently crashed.  </p>
  
  <p>Any recommendations? This is my first usage of BeautifulSoup.</p>