UnicodeDecodeError with word stemming in Python
Question: I'm so stumped.

I have a list of a couple of thousand words

    
    
    x = ['company', 'arriving', 'wednesday', 'and', 'then', 'beach', 'how', 'are', 'you', 'any', 'warmer', 'there', 'enjoy', 'your', 'day', 'follow', 'back', 'please', 'everyone', 'go', 'watch', 's', 'new', 'video', 'you', 'know', 'the', 'deal', 'make', 'sure', 'to', 'subscribe', 'and', 'like', '<http>', 'you', 'said', 'next', 'week', 'you', 'will', 'be', 'the', 'one', 'picking', 'me', 'up', 'lol', 'hindi', 'na', 'tl', 'huehue', 'that', 'works', 'you', 'said', 'everyone', 'of', 'us', 'my', 'little', 'cousin', 'keeps', 'asking', 'if', 'i', 'wanna', 'play', 'and', "i'm", 'like', 'yes', 'but', 'with', 'my', 'pals', 'not', 'you', "you're", 'welcome', 'pas', 'quand', 'tu', 'es', 'vers', '<num>', 'i', 'never', 'get', 'good', 'mornng', 'texts', 'sad', 'sad', 'moment', 'i', 'think', 'ima', 'go', 'get', 'a', 'glass', 'of', 'milk', 'ahah', 'for', 'the', 'first', 'time', 'i', 'actually', 'know', 'what', 'their', 'doing', 'd', 'thank', 'you', 'happy', 'birthday', 'hope', "you're"...........]
    

Now, I've confirmed the type of each element in this list to be a string

    
    
    types = []
    for word in x:
        a.append(type(word))
    print set(a)
    
    >>>set([<type 'str'>])
    

Now, I attempt to stem each word using NLTK's porter stemmer

    
    
    import nltk
    porter = nltk.PorterStemmer()
    stemmed_x = [porter.stem(word) for word in x]
    

And I get this error that is clearly related to the stemming package and
unicode somehow:

    
    
    File "/Library/Python/2.7/site-packages/nltk-3.0.0b2-py2.7.egg/nltk/stem/porter.py", line 633, in stem
        stem = self.stem_word(word.lower(), 0, len(word) - 1)
      File "/Library/Python/2.7/site-packages/nltk-3.0.0b2-py2.7.egg/nltk/stem/porter.py", line 591, in stem_word
        word = self._step1ab(word)
      File "/Library/Python/2.7/site-packages/nltk-3.0.0b2-py2.7.egg/nltk/stem/porter.py", line 289, in _step1ab
        if word.endswith("ied"):
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)
    

I have tried everything, using `codecs.open`, trying to explicitly encode each
word as `utf8` \- still produce the same error.

Please advise.

EDIT:

I should mention that this code worked perfect on my PC running Ubuntu. I
recently got a macbook pro and I'm getting this error. I've checked the
terminal settings on my mac and it is set to utf8 encoding.

EDIT 2:

Interesting, with this piece of code, I have isolated the problem words:

    
    
    for w in x:
        try:
            porter.stem(w)
        except UnicodeDecodeError:
            print w 
    
    #sagittarius”
    #instadane…
    #bleedblue”
    #pr챕cieux
    #على_شرفة_الماضي
    #exploringsf…
    #fishing…
    #sindhubestfriend…
    #الإستعداد_لإنهيار_ال_سعود
    #jaredpreslar…
    #femalepains”
    #gobillings”
    #juicing…
    #instamood…
    

Seems like what they all have in common are extra punctuation at the end of
the word, except for the word #pr챕cieux


Answer: You have probably a multi-byte UTF8 char lurking around as `0xe2` is one of
the first byte possible for an [16-bit codepoint encoded as
UTF-8](http://en.wikipedia.org/wiki/UTF-8#Description). As your program assume
ASCII chars, with valid encoded values from `0x00` to `0x7F`, this value is
rejected.

You might be able to identify the "bad" value by a simple comprehension, then
fix it by hand (as I assume from your data your want only deal with ASCII
chars):

    
    
    print [value for value in x if '\xe2' in x]
    

