title: Python: solving unicode hell with unidecode
body: <p>I have been working on ways to flatten text into ascii. So <strong>ā</strong> -> <strong>a</strong> and <strong>ñ</strong> -> <strong>n</strong>, etc.</p>
  
  <p><code>unidecode</code> has been fantastic for this.</p>
  
  <pre><code># -*- coding: utf-8 -*-
  from unidecode import unidecode
  print(unidecode(u"ā, ī, ū, ś, ñ"))
  print(unidecode(u"Estado de São Paulo"))
  </code></pre>
  
  <p>Produces:</p>
  
  <pre><code>a, i, u, s, n
  Estado de Sao Paulo
  </code></pre>
  
  <p><p>
  However, I can't duplicate this result with data from an input file.</p>
  
  <p>Content of test.txt file:</p>
  
  <pre><code>ā, ī, ū, ś, ñ
  Estado de São Paulo
  </code></pre>
  
  <p><p></p>
  
  <pre><code># -*- coding: utf-8 -*-
  from unidecode import unidecode
  with open("test.txt", 'r') as inf:
      for line in inf:
          print unidecode(line.strip())
  </code></pre>
  
  <p>Produces:</p>
  
  <pre><code>A, A&lt;&lt;, A&lt;&lt;, A, A+-
  Estado de SAPSo Paulo
  </code></pre>
  
  <p>And:</p>
  
  <blockquote>
    <p>RuntimeWarning: Argument  is not an unicode object.
    Passing an encoded string will likely have unexpected results.</p>
  </blockquote>
  
  <p><strong>Question</strong>: How can I read these lines in as unicode so that I can pass them to <code>unidecode</code>?</p>