<p>I want to scrape some information off a football (soccer) web page using simple python regexp's.  The problem is that players such as the first chap, ÄÄRITALO, comes out as &amp;#196;&amp;#196;RITALO!<br>
That is, html uses escaped markup for the special characters, such as &amp;#196;</p>

<p>Is there a simple way of reading the html into the correct python string? If it was XML/XHTML it would be easy, the parser would do it.</p>

[{'answer_id': 53226, 'author': 'Nick Fortescue', 'author_id': 5346, 'author_profile': 'https://Stackoverflow.com/users/5346', 'pm_score': 0, 'selected': False, 'text': '<p>I haven\'t tried it myself, but have you tried</p>\n\n<p><a href="http://zesty.ca/python/scrape.html" rel="nofollow noreferrer">http://zesty.ca/python/scrape.html</a> ?</p>\n\n<p>It seems to have a method htmldecode(text) which would do what you want.</p>\n'}, {'answer_id': 53245, 'author': 'Jacob Rigby', 'author_id': 5357, 'author_profile': 'https://Stackoverflow.com/users/5357', 'pm_score': 2, 'selected': False, 'text': '<p>Try using <a href="http://www.crummy.com/software/BeautifulSoup/" rel="nofollow noreferrer">BeautifulSoup</a>.  It should do the trick and give you a nicely formatted DOM to work with as well.</p>\n\n<p><a href="http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/" rel="nofollow noreferrer">This blog</a> entry seems to have had some success with it.</p>\n'}, {'answer_id': 53246, 'author': 'dF.', 'author_id': 3002, 'author_profile': 'https://Stackoverflow.com/users/3002', 'pm_score': 4, 'selected': True, 'text': '<p>I would recommend <a href="http://www.crummy.com/software/BeautifulSoup/" rel="nofollow noreferrer">BeautifulSoup</a> for HTML scraping. You also need to tell it to convert HTML entities to the corresponding Unicode characters, like so:</p>\n\n<pre><code>&gt;&gt;&gt; from BeautifulSoup import BeautifulSoup    \n&gt;&gt;&gt; html = "&lt;html&gt;&amp;#196;&amp;#196;RITALO!&lt;/html&gt;"\n&gt;&gt;&gt; soup = BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)\n&gt;&gt;&gt; print soup.contents[0].string\nÄÄRITALO!\n</code></pre>\n\n<p>(It would be nice if the standard <a href="http://docs.python.org/lib/module-codecs.html" rel="nofollow noreferrer">codecs</a> module included a codec for this, such that you could do   <code>"some_string".decode(\'html_entities\')</code> but unfortunately it doesn\'t!)</p>\n\n<p><strong>EDIT:</strong>\nAnother solution:\nPython developer Fredrik Lundh (author of elementtree, among other things) has  <a href="http://docs.python.org/lib/module-codecs.html" rel="nofollow noreferrer">a function to unsecape HTML entities</a> on his website, which works with decimal, hex and named entities (BeautifulSoup will not work with the hex ones).</p>\n'}]
['https://Stackoverflow.com/questions/53224', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/5346/']