python different result from IDLE and python script
Question: I have tried the following in Python 2.7 shell:

    
    
      >>> from nltk.stem.isri import ISRIStemmer
      >>> st = ISRIStemmer()
      >>> string = u'\u062D\u064E\u062F\u0651\u064E\u062B\u064E\u0646\u064E\u0627'
      >>> st.stem(string)
          u'\u062d\u062f\u062b'
    

So basically, I am trying to obtain:

    
    
    u'\u062d\u062f\u062b'
    

from

    
    
    u'\u062D\u064E\u062F\u0651\u064E\u062B\u064E\u0646\u064E\u0627'
    

using nltk's arabic stemmer, which works!

However, when I try to accomplish the exact thing through a python script, it
fails to stem any of the words in the list, `tokens` :

    
    
    #!/c/Python27/python
    # -*- coding: utf8 -*-
    
    import nltk
    import nltk.data
    from nltk.stem.isri import ISRIStemmer
    
    #In my script, I tokenize the following string
    commasection = '\u0623\u064E\u062E\u0652\u0628\u064E\u0631\u064E\u0646\u064E\u0627 \u0628\u0650\u0634\u0652\u0631\u064F \u0628\u0652\u0646\u064F \u0647\u0650\u0644\u0627\u064E\u0644\u064D'
    
    #The tokenizing works
    tokens = nltk.word_tokenize(commasection)
    st = ISRIStemmer()
    
    for word in tokens:
        #But the stemming of each word in tokens doesn't work????
        print st.stem(word)
    
        #Should display 
        #u'u0623\u062e\u0628\u0631'
        #u'\u0628\u0634\u0631'
        #u'\u0628\u0646'
        #u'\u0647\u0644\u0644'
    
        #But it just shows whatever is in commasection
    

I need my python code to stem all words in `tokens`. But I don't get how the
simpler example running in python shell works but not this script.

I have noticed that in the shell scenario, there is that 'u' in front of the
sequence of unicode, so I tried all sorts of encodings/decodings and read a
lot about it all night long (pulled an all-nighter on this one), but this
python script is just not stemming `word` from `tokens` like the python
shell!!!

If anyone can please help me make my script display the correct result I would
be super super appreciative


Answer: Unicode escapes only work in `unicode` literals.

    
    
    commasection = u'\u0623\u064E\u062E\u0652\u0628\u064E\u0631\u064E\u0646\u064E\u0627 \u0628\u0650\u0634\u0652\u0631\u064F \u0628\u0652\u0646\u064F \u0647\u0650\u0644\u0627\u064E\u0644\u064D'
    

