<p>I'm doing some web scraping and sites frequently use HTML entities to represent non ascii characters. Does Python have a utility that takes a string with HTML entities and returns a unicode type?</p>

<p>For example:</p>

<p>I get back:</p>

<pre><code>&amp;#x01ce;
</code></pre>

<p>which represents an "ǎ" with a tone mark.  In binary, this is represented as the 16 bit 01ce.  I want to convert the html entity into the value  <code>u'\u01ce'</code></p>

[{'answer_id': 57745, 'author': 'jfs', 'author_id': 4279, 'author_profile': 'https://Stackoverflow.com/users/4279', 'pm_score': 3, 'selected': False, 'text': '<p>You could find an answer here -- <a href="https://stackoverflow.com/questions/53224/getting-international-characters-from-a-web-page#53246">Getting international characters from a web page?</a></p>\n\n<p><strong>EDIT</strong>: It seems like <code>BeautifulSoup</code> doesn\'t convert entities written in hexadecimal form. It can be fixed:</p>\n\n<pre><code>import copy, re\nfrom BeautifulSoup import BeautifulSoup\n\nhexentityMassage = copy.copy(BeautifulSoup.MARKUP_MASSAGE)\n# replace hexadecimal character reference by decimal one\nhexentityMassage += [(re.compile(\'&amp;#x([^;]+);\'), \n                     lambda m: \'&amp;#%d;\' % int(m.group(1), 16))]\n\ndef convert(html):\n    return BeautifulSoup(html,\n        convertEntities=BeautifulSoup.HTML_ENTITIES,\n        markupMassage=hexentityMassage).contents[0].string\n\nhtml = \'&lt;html&gt;&amp;#x01ce;&amp;#462;&lt;/html&gt;\'\nprint repr(convert(html))\n# u\'\\u01ce\\u01ce\'\n</code></pre>\n\n<p><strong>EDIT</strong>: </p>\n\n<p><a href="http://effbot.org/zone/re-sub.htm#unescape-html" rel="nofollow noreferrer"><code>unescape()</code></a> function mentioned by <a href="https://stackoverflow.com/questions/57708/convert-xml-html-entities-into-unicode-string-in-python/58125#58125">@dF</a> which uses  <code>htmlentitydefs</code> standard module and <code>unichr()</code> might be more appropriate in this case.</p>\n'}, {'answer_id': 57877, 'author': 'chryss', 'author_id': 5169, 'author_profile': 'https://Stackoverflow.com/users/5169', 'pm_score': 4, 'selected': False, 'text': "<p>Use the builtin <code>unichr</code> -- BeautifulSoup isn't necessary:</p>\n\n<pre><code>&gt;&gt;&gt; entity = '&amp;#x01ce'\n&gt;&gt;&gt; unichr(int(entity[3:],16))\nu'\\u01ce'\n</code></pre>\n"}, {'answer_id': 58125, 'author': 'dF.', 'author_id': 3002, 'author_profile': 'https://Stackoverflow.com/users/3002', 'pm_score': 6, 'selected': False, 'text': '<p>Python has the <a href="https://docs.python.org/2/library/htmllib.html#module-htmlentitydefs" rel="noreferrer">htmlentitydefs</a> module, but this doesn\'t include a function to unescape HTML entities.</p>\n\n<p>Python developer Fredrik Lundh (author of elementtree, among other things) has  such a function <a href="http://effbot.org/zone/re-sub.htm#unescape-html" rel="noreferrer">on his website</a>, which works with decimal, hex and named entities:</p>\n\n<pre><code>import re, htmlentitydefs\n\n##\n# Removes HTML or XML character references and entities from a text string.\n#\n# @param text The HTML (or XML) source text.\n# @return The plain text, as a Unicode string, if necessary.\n\ndef unescape(text):\n    def fixup(m):\n        text = m.group(0)\n        if text[:2] == "&amp;#":\n            # character reference\n            try:\n                if text[:3] == "&amp;#x":\n                    return unichr(int(text[3:-1], 16))\n                else:\n                    return unichr(int(text[2:-1]))\n            except ValueError:\n                pass\n        else:\n            # named entity\n            try:\n                text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])\n            except KeyError:\n                pass\n        return text # leave as is\n    return re.sub("&amp;#?\\w+;", fixup, text)\n</code></pre>\n'}, {'answer_id': 573629, 'author': 'karlcow', 'author_id': 62262, 'author_profile': 'https://Stackoverflow.com/users/62262', 'pm_score': 3, 'selected': False, 'text': '<p>This is a function which should help you to get it right and convert entities back to utf-8 characters.</p>\n\n<pre><code>def unescape(text):\n   """Removes HTML or XML character references \n      and entities from a text string.\n   @param text The HTML (or XML) source text.\n   @return The plain text, as a Unicode string, if necessary.\n   from Fredrik Lundh\n   2008-01-03: input only unicode characters string.\n   http://effbot.org/zone/re-sub.htm#unescape-html\n   """\n   def fixup(m):\n      text = m.group(0)\n      if text[:2] == "&amp;#":\n         # character reference\n         try:\n            if text[:3] == "&amp;#x":\n               return unichr(int(text[3:-1], 16))\n            else:\n               return unichr(int(text[2:-1]))\n         except ValueError:\n            print "Value Error"\n            pass\n      else:\n         # named entity\n         # reescape the reserved characters.\n         try:\n            if text[1:-1] == "amp":\n               text = "&amp;amp;amp;"\n            elif text[1:-1] == "gt":\n               text = "&amp;amp;gt;"\n            elif text[1:-1] == "lt":\n               text = "&amp;amp;lt;"\n            else:\n               print text[1:-1]\n               text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])\n         except KeyError:\n            print "keyerror"\n            pass\n      return text # leave as is\n   return re.sub("&amp;#?\\w+;", fixup, text)\n</code></pre>\n'}, {'answer_id': 4438857, 'author': 'rogerhu', 'author_id': 541895, 'author_profile': 'https://Stackoverflow.com/users/541895', 'pm_score': 2, 'selected': False, 'text': '<p>Not sure why the Stack Overflow thread does not include the \';\' in the search/replace (i.e. lambda m: \'&amp;#%d*<em>;</em>*\') If you don\'t, BeautifulSoup can barf because the adjacent character can be interpreted as part of the HTML code (i.e. &amp;#39B for &amp;#39Blackout).                                                                                                                                                                                                                                                                                     </p>\n\n<p>This worked better for me:</p>\n\n<pre><code>import re\nfrom BeautifulSoup import BeautifulSoup\n\nhtml_string=\'&lt;a href="/cgi-bin/article.cgi?f=/c/a/2010/12/13/BA3V1GQ1CI.DTL"title=""&gt;&amp;#x27;Blackout in a can; on some shelves despite ban&lt;/a&gt;\'\n\nhexentityMassage = [(re.compile(\'&amp;#x([^;]+);\'), \nlambda m: \'&amp;#%d;\' % int(m.group(1), 16))]\n\nsoup = BeautifulSoup(html_string, \nconvertEntities=BeautifulSoup.HTML_ENTITIES, \nmarkupMassage=hexentityMassage)\n</code></pre>\n\n<ol>\n<li>The int(m.group(1), 16) converts the number (specified in base-16) format back to an integer.                                                                             </li>\n<li>m.group(0) returns the entire match, m.group(1) returns the regexp capturing group                                                                                   </li>\n<li>Basically using markupMessage is the same as:<br>\nhtml_string = re.sub(\'&amp;#x([^;]+);\', lambda m: \'&amp;#%d;\' % int(m.group(1), 16), html_string)                                                                            </li>\n</ol>\n'}, {'answer_id': 9216990, 'author': 'pragmar', 'author_id': 1196188, 'author_profile': 'https://Stackoverflow.com/users/1196188', 'pm_score': 4, 'selected': False, 'text': "<p>An alternative, if you have lxml:</p>\n\n<pre><code>&gt;&gt;&gt; import lxml.html\n&gt;&gt;&gt; lxml.html.fromstring('&amp;#x01ce').text\nu'\\u01ce'\n</code></pre>\n"}, {'answer_id': 12614706, 'author': 'Vladislav', 'author_id': 1164730, 'author_profile': 'https://Stackoverflow.com/users/1164730', 'pm_score': 7, 'selected': True, 'text': "<p>The standard lib’s very own HTMLParser has an undocumented function unescape() which does exactly what you think it does:</p>\n\n<p>up to Python 3.4:\n</p>\n\n<pre><code>import HTMLParser\nh = HTMLParser.HTMLParser()\nh.unescape('&amp;copy; 2010') # u'\\xa9 2010'\nh.unescape('&amp;#169; 2010') # u'\\xa9 2010'\n</code></pre>\n\n<p>Python 3.4+:</p>\n\n<pre><code>import html\nhtml.unescape('&amp;copy; 2010') # u'\\xa9 2010'\nhtml.unescape('&amp;#169; 2010') # u'\\xa9 2010'\n</code></pre>\n"}, {'answer_id': 27424874, 'author': 'Markus Amalthea Magnuson', 'author_id': 11403, 'author_profile': 'https://Stackoverflow.com/users/11403', 'pm_score': 4, 'selected': False, 'text': '<p>If you are on Python 3.4 or newer, you can simply use the <a href="https://docs.python.org/3.4/library/html.html#html.unescape" rel="noreferrer"><code>html.unescape</code></a>:</p>\n\n<pre><code>import html\n\ns = html.unescape(s)\n</code></pre>\n'}, {'answer_id': 33486253, 'author': 'Community', 'author_id': -1, 'author_profile': 'https://Stackoverflow.com/users/-1', 'pm_score': 1, 'selected': False, 'text': '<p>Another solution is the builtin library xml.sax.saxutils (both for html and xml). However, it will convert only &amp;gt, &amp;amp and &amp;lt.</p>\n\n<pre><code>from xml.sax.saxutils import unescape\n\nescaped_text = unescape(text_to_escape)\n</code></pre>\n'}, {'answer_id': 34463462, 'author': 'Victor', 'author_id': 2086547, 'author_profile': 'https://Stackoverflow.com/users/2086547', 'pm_score': 0, 'selected': False, 'text': '<p>Here is the Python 3 version of <a href="https://stackoverflow.com/a/58125/2086547">dF\'s answer</a>:</p>\n\n<pre><code>import re\nimport html.entities\n\ndef unescape(text):\n    """\n    Removes HTML or XML character references and entities from a text string.\n\n    :param text:    The HTML (or XML) source text.\n    :return:        The plain text, as a Unicode string, if necessary.\n    """\n    def fixup(m):\n        text = m.group(0)\n        if text[:2] == "&amp;#":\n            # character reference\n            try:\n                if text[:3] == "&amp;#x":\n                    return chr(int(text[3:-1], 16))\n                else:\n                    return chr(int(text[2:-1]))\n            except ValueError:\n                pass\n        else:\n            # named entity\n            try:\n                text = chr(html.entities.name2codepoint[text[1:-1]])\n            except KeyError:\n                pass\n        return text # leave as is\n    return re.sub("&amp;#?\\w+;", fixup, text)\n</code></pre>\n\n<p>The main changes concern <code>htmlentitydefs</code> that is now <code>html.entities</code> and <code>unichr</code> that is now <code>chr</code>. See this <a href="http://docs.pythonsprints.com/python3_porting/py-porting.html#reorganization" rel="nofollow noreferrer">Python 3 porting guide</a>.</p>\n'}]
['https://Stackoverflow.com/questions/57708', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/680/']