title: Beautiful Soup and Unicode Problems
body: <p>I'm using BeautifulSoup to parse some web pages.</p>
  
  <p>Occasionally I run into a "unicode hell" error like the following :</p>
  
  <p>Looking at the source of this article on TheAtlantic.com [ <a href="http://www.theatlantic.com/education/archive/2013/10/why-are-hundreds-of-harvard-students-studying-ancient-chinese-philosophy/280356/" rel="nofollow noreferrer">http://www.theatlantic.com/education/archive/2013/10/why-are-hundreds-of-harvard-students-studying-ancient-chinese-philosophy/280356/</a> ]</p>
  
  <p>We see this in the og:description meta property :</p>
  
  <pre><code>&lt;meta property="og:description" content="The professor who teaches&amp;nbsp;Classical Chinese Ethical and Political Theory claims, &amp;quot;This course will change your life.&amp;quot;" /&gt;
  </code></pre>
  
  <p>When BeautifulSoup parses it, I see this:</p>
  
  <pre><code>&gt;&gt;&gt; print repr(description)
  u'The professor who teaches\xa0Classical Chinese Ethical and Political Theory claims, "This course will change your life."'
  </code></pre>
  
  <p>If I try encoding it to UTF-8 , like this SO comment suggests : <a href="https://stackoverflow.com/a/10996267/442650">https://stackoverflow.com/a/10996267/442650</a></p>
  
  <pre><code>&gt;&gt;&gt; print repr(description.encode('utf8'))
  'The professor who teaches\xc2\xa0Classical Chinese Ethical and Political Theory claims, "This course will change your life."'
  </code></pre>
  
  <p>Just when I thought I had all my unicode issues under control, I still don't quite understand what's going on, so I'm going to lay out a few questions:</p>
  
  <p>1- why would BeautifulSoup convert the <code>&amp;nbsp;</code> to <code>\xa0</code> [a latin charset space character]?  The charset and headers on this page are UTF-8, I thought BeautifulSoup pulls that data for the encoding ?  Why wasn't it replaced with a <code>&lt;space&gt;</code> ?</p>
  
  <p>2- Is there a common way to normalize whitespaces for conversion ?</p>
  
  <p>3- When I encoded to UTF8 , where did <code>\xa0</code> become the sequence of <code>\xc2\xa0</code> ?</p>
  
  <p>I can pipe everything through <code>unicodedata.normalize('NFKD',string)</code> to help get me where I want to be -- but I'd love to understand what's wrong and avoid problem like this in the future.</p>