title: UnicodeDecodeError with word stemming in Python
body: <p>I'm so stumped.</p>
  
  <p>I have a list of a couple of thousand words</p>
  
  <pre><code>x = ['company', 'arriving', 'wednesday', 'and', 'then', 'beach', 'how', 'are', 'you', 'any', 'warmer', 'there', 'enjoy', 'your', 'day', 'follow', 'back', 'please', 'everyone', 'go', 'watch', 's', 'new', 'video', 'you', 'know', 'the', 'deal', 'make', 'sure', 'to', 'subscribe', 'and', 'like', '&lt;http&gt;', 'you', 'said', 'next', 'week', 'you', 'will', 'be', 'the', 'one', 'picking', 'me', 'up', 'lol', 'hindi', 'na', 'tl', 'huehue', 'that', 'works', 'you', 'said', 'everyone', 'of', 'us', 'my', 'little', 'cousin', 'keeps', 'asking', 'if', 'i', 'wanna', 'play', 'and', "i'm", 'like', 'yes', 'but', 'with', 'my', 'pals', 'not', 'you', "you're", 'welcome', 'pas', 'quand', 'tu', 'es', 'vers', '&lt;num&gt;', 'i', 'never', 'get', 'good', 'mornng', 'texts', 'sad', 'sad', 'moment', 'i', 'think', 'ima', 'go', 'get', 'a', 'glass', 'of', 'milk', 'ahah', 'for', 'the', 'first', 'time', 'i', 'actually', 'know', 'what', 'their', 'doing', 'd', 'thank', 'you', 'happy', 'birthday', 'hope', "you're"...........]
  </code></pre>
  
  <p>Now, I've confirmed the type of each element in this list to be a string</p>
  
  <pre><code>types = []
  for word in x:
      a.append(type(word))
  print set(a)
  
  &gt;&gt;&gt;set([&lt;type 'str'&gt;])
  </code></pre>
  
  <p>Now, I attempt to stem each word using NLTK's porter stemmer</p>
  
  <pre><code>import nltk
  porter = nltk.PorterStemmer()
  stemmed_x = [porter.stem(word) for word in x]
  </code></pre>
  
  <p>And I get this error that is clearly related to the stemming package and unicode somehow:</p>
  
  <pre><code>File "/Library/Python/2.7/site-packages/nltk-3.0.0b2-py2.7.egg/nltk/stem/porter.py", line 633, in stem
      stem = self.stem_word(word.lower(), 0, len(word) - 1)
    File "/Library/Python/2.7/site-packages/nltk-3.0.0b2-py2.7.egg/nltk/stem/porter.py", line 591, in stem_word
      word = self._step1ab(word)
    File "/Library/Python/2.7/site-packages/nltk-3.0.0b2-py2.7.egg/nltk/stem/porter.py", line 289, in _step1ab
      if word.endswith("ied"):
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)
  </code></pre>
  
  <p>I have tried everything, using <code>codecs.open</code>, trying to explicitly encode each word as <code>utf8</code> - still produce the same error.</p>
  
  <p>Please advise.</p>
  
  <p>EDIT:</p>
  
  <p>I should mention that this code worked perfect on my PC running Ubuntu. I recently got a macbook pro and I'm getting this error. I've checked the terminal settings on my mac and it is set to utf8 encoding.</p>
  
  <p>EDIT 2:</p>
  
  <p>Interesting, with this piece of code, I have isolated the problem words:</p>
  
  <pre><code>for w in x:
      try:
          porter.stem(w)
      except UnicodeDecodeError:
          print w 
  
  #sagittarius”
  #instadane…
  #bleedblue”
  #pr챕cieux
  #على_شرفة_الماضي
  #exploringsf…
  #fishing…
  #sindhubestfriend…
  #الإستعداد_لإنهيار_ال_سعود
  #jaredpreslar…
  #femalepains”
  #gobillings”
  #juicing…
  #instamood…
  </code></pre>
  
  <p>Seems like what they all have in common are extra punctuation at the end of the word, except for the word #pr챕cieux</p>