Encoding issues with CLD
<p>After some <a href="http://stackoverflow.com/questions/13328180/installing-cld-libary-on-windows-and-bind-to-python">issues</a> getting Chrome Compact Language Detection library installed on Windows, I installed CLD from <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/#cld" rel="nofollow">this easy_install.</a></p>

<p>I can now use CLD, but getting some encoding issues.</p>

<h2>Background</h2>

<p>Pulling Tweets into a python script, and after stripping out the hashtags and links, passing them to CLD to detect the language.<br>
Following is a simplified version of my code:</p>

<pre><code>s = "I am a tweet from Twitter"
clean_s = s.encode('utf-8')
lan = cld.detect(clean_s, pickSummaryLanguage=True, removeWeakMatches=True)
</code></pre>

<h2>Problem</h2>

<p>4 out of 5 times, this works as expected (get returned a response about what language it is).</p>

<p>However, I keep getting this error popping up: </p>

<blockquote>
  <p>UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019'
  in position 15: character maps to undefined</p>
</blockquote>

<p>I <a href="http://blog.mikemccandless.com/2011/10/language-detection-with-googles-compact.html" rel="nofollow">did read</a> that: </p>

<blockquote>
  <p>"You must provide CLD clean (interchange-valid) UTF-8, so any encoding
  issues must be sorted out before-hand."</p>
</blockquote>

<p>However, I thought I had this covered with my statement to encode to UTF8?</p>

<p>I assume that I need to ensure that I pass a string to CLD that preserves fonts in languages such as arabic, asian, etc.</p>

<p>This is my first python project, so likely this is a rookie mistake.  Can anyone point out my mistake and how to rectify?</p>

<p>Let me know in comments if I need to gather more info, and I will edit my Q to provide more info.</p>

<p><em>EDIT</em>
If it helps, <a href="http://pastebin.com/3DU2BYp0" rel="nofollow">here is my rookie code</a> (cut down to replicate issue).
I am running Python 2.7 32bit.</p>

<p>Running this code, after awhile, I <a href="http://pastebin.com/dsYkJKHE" rel="nofollow">get this error</a>. 
Let me know if I have not correctly implemented the error reporting.</p>

<pre><code>Raw: Traceback (most recent call last):
  File "LanguageTesting.py", line 71, in &lt;module&gt;
    parse_tweet(tweet)
  File "LanguageTesting.py", line 43, in parse_tweet
    print "Raw:", raw
  File "C:\Python27\ArcGIS10.1\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 29-32: character maps to &lt;undefined&gt;
</code></pre>
