title: Python: Sanitize a string for unicode?
body: <blockquote>
    <p><strong>Possible Duplicate:</strong><br>
    <a href="https://stackoverflow.com/questions/368805/python-unicodedecodeerror-am-i-misunderstanding-encode">Python UnicodeDecodeError - Am I misunderstanding encode?</a>  </p>
  </blockquote>
  
  
  
  <p>I have a string that I'm trying to make safe for the <code>unicode()</code> function:</p>
  
  <pre><code>&gt;&gt;&gt; s = " foo “bar bar ” weasel"
  &gt;&gt;&gt; s.encode('utf-8', 'ignore')
  
  Traceback (most recent call last):
    File "&lt;pyshell#8&gt;", line 1, in &lt;module&gt;
      s.encode('utf-8', 'ignore')
  UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 5: ordinal not in range(128)
  &gt;&gt;&gt; unicode(s)
  
  Traceback (most recent call last):
    File "&lt;pyshell#9&gt;", line 1, in &lt;module&gt;
      unicode(s)
  UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 5: ordinal not in range(128)
  </code></pre>
  
  <p>I'm mostly flailing around here. What do I need to do to remove the unsafe characters from the string?</p>
  
  <p>Somewhat related to this <a href="https://stackoverflow.com/questions/3224268/python-unicode-encode-error">question</a>, although I was unable to solve my problem from it.</p>
  
  <p>This also fails:</p>
  
  <pre><code>&gt;&gt;&gt; s
  ' foo \x93bar bar \x94 weasel'
  &gt;&gt;&gt; s.decode('utf-8')
  
  Traceback (most recent call last):
    File "&lt;pyshell#13&gt;", line 1, in &lt;module&gt;
      s.decode('utf-8')
    File "C:\Python25\254\lib\encodings\utf_8.py", line 16, in decode
      return codecs.utf_8_decode(input, errors, True)
  UnicodeDecodeError: 'utf8' codec can't decode byte 0x93 in position 5: unexpected code byte
  </code></pre>