title: Python HTMLParser: UnicodeDecodeError
body: <p>I'm using HTMLParser to parse pages I pull down with urllib, and am coming across <code>UnicodeDecodeError</code> exceptions when passing some to <code>HTMLParser</code>.</p>
  
  <p>I tried using <code>chardet</code> to detect the encodings and to convert to <code>ascii</code>, or <code>utf-8</code> (the <a href="http://docs.python.org/library/htmlparser.html" rel="noreferrer">docs</a> don't seem to say what it should be). lossiness is acceptable, but while the decode/encode lines work just fine, I always get the error after self.feed().</p>
  
  <p>The information is there if I just <code>print</code> it out.</p>
  
  <pre><code>from HTMLParser import HTMLParser
  import urllib
  import chardet
  
  class search_youtube(HTMLParser):
  
      def __init__(self, search_terms):
          HTMLParser.__init__(self)
          self.track_ids = []
          for search in search_terms:
              self.__in_result = False
              search = urllib.quote_plus(search)
              query = 'http://youtube.com/results?search_query='
              page = urllib.urlopen(query + search).read()
              try:
                  self.feed(page)
              except UnicodeDecodeError:
                  encoding = chardet.detect(page)['encoding']
                  if encoding != 'unicode':
                      page = page.decode(encoding)
                      page = page.encode('ascii', 'ignore')
                  self.feed(page)
                  print 'success'
  
  searches = ['telepopmusik breathe']
  results = search_youtube(searches)
  print results.track_ids
  </code></pre>
  
  <p>here's the output:</p>
  
  <pre><code>Traceback (most recent call last):
    File "test.py", line 27, in &lt;module&gt;
      results = search_youtube(searches)
    File "test.py", line 23, in __init__
      self.feed(page)
    File "/usr/lib/python2.6/HTMLParser.py", line 108, in feed
      self.goahead(0)
    File "/usr/lib/python2.6/HTMLParser.py", line 148, in goahead
      k = self.parse_starttag(i)
    File "/usr/lib/python2.6/HTMLParser.py", line 252, in parse_starttag
      attrvalue = self.unescape(attrvalue)
    File "/usr/lib/python2.6/HTMLParser.py", line 390, in unescape
      return re.sub(r"&amp;(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s)
    File "/usr/lib/python2.6/re.py", line 151, in sub
      return _compile(pattern, 0).sub(repl, string, count)
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
  </code></pre>