How to solve decoding while using stanford parser for Chinese text with python
Question: I want to use Stanford Parser to parse Chinese texts with Python interface. My
code is below:

    
    
    #!~/anaconda/bin/python
    # -*- coding: utf-8 -*-
    
    from nltk.parse import stanford
    
    parser = stanford.StanfordParser(path_to_jar='/home/stanford-parser/stanford-parser.jar', path_to_models_jar='/home/stanford-parser/stanford-parser-3.3.0-models.jar',model_path="/home/stanford-parser/chinesePCFG.ser.gz",encoding='utf8')
    
    sentences = parser.raw_parse_sents(("我 是 中国 人。", "他 来自 美国。"))
    print sentences 
    

However, when I try to run this code,an decoding error occurs

    
    
    Traceback (most recent call last):
      File "/home/test.py", line 8, in <module>
        sentences = parser.raw_parse_sents(("我 是 中国人。", "他 来自 美国。"))
      File "/home/anaconda/lib/python2.7/site-packages/nltk/parse/stanford.py", line 176, in raw_parse_sents
        return self._parse_trees_output(self._execute(cmd, '\n'.join(sentences), verbose))
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)
    

I have no idea about why such an error occurs,since my code is edited by
**utf-8** just as the second line. Could anyone please help me explain it and
solve it? I really need the help from stackoverflow community.


Answer: The parser expects a unicode object (you actually told it on creation that you
will be using data encoded in UTF-8. However, what you send to it as a
parameter are just plain string which are basically just sequences of bytes
(in Python 2.x). You can create [unicode
literals](https://docs.python.org/2/howto/unicode.html#unicode-literals-in-
python-source-code) by prepending a string with `u`, e.g. `u"我 是 中国 人"`

    
    
    >>> word = u"我 是 中国 人"
    >>> type(word)
    <type 'unicode'>
    >>> print word
    我 是 中国 人
    

And to convert an existing plain string into a unicode object:

    
    
    >>> word = "我 是 中国 人"
    >>> type(word)
    <type 'str'>
    >>> unicode_word = unicode(word, encoding='utf8')
    >>> type(unicode_word)
    <type 'unicode'>
    

If these kind of things cause you trouble, I strongly recommend reading the
[Unicode HOWTO](https://docs.python.org/2/howto/unicode.html) section of the
Python documentation, it will probably make the everything much more clear.

### Bonus

To convert a plain string representing a _Unicode escape_ sequence to a
Unicode string, use the [`'unicode_escape'`
encoding](https://docs.python.org/2/library/codecs.html#python-specific-
encodings).

    
    
    >>> type('\u6211')
    <type 'str'>
    >>> len('\u6211')
    6
    >>> converted = '\u6211'.decode('unicode_escape')
    >>> type(converted)
    <type 'unicode'>
    >>> len(converted)
    1
    >>> print converted
    我
    

