Encoding issue for Python tool Unidecode on CL
Question: I need to convert unicode files to ascii. In case, a letter doesn't exist in
ascii, it should be converted to it's closest ascii representation. I'm using
the Unidecode tool for it (<https://pypi.python.org/pypi/Unidecode>). It works
fine when I use it in the Python interpreter on the CL (thus, by invoking
`python` and then importing the libraries and then printing the decoded word
like this: `print unidecode(u'äèß')`)

Unfortunately, when I try to use this tool directly on the command line (thus,
by doing something like `python -c "from unidecode import *; print
unidecode(u'äèß')"`, it only prints gibberish (`A$?A"A` to be exact, even
though it should've printed (and did in the interpreter) `aess`). This is
annoying and I don't know how to solve that issue. I thought it might be due
to encoding errors with my Terminal, not being set correctly to utf-8 or
something. However, `locale` in my Terminal printed me the following output:

> LANG="de_DE.UTF-8"
>
> LC_COLLATE="de_DE.UTF-8"
>
> LC_CTYPE="de_DE.UTF-8"
>
> LC_MESSAGES="de_DE.UTF-8"
>
> LC_MONETARY="de_DE.UTF-8"
>
> LC_NUMERIC="de_DE.UTF-8"
>
> LC_TIME="de_DE.UTF-8"
>
> LC_ALL="de_DE.UTF-8"

Or, might it be due to Python that has problems with StdIn encoding on the
command line? It gave me correct output in the python interpreter, but when
invoking `python -c` not.

Do you guys have an idea?


Answer: If you try writing this in a file:

    
    
    #!/bin/python
    from unidecode import *
    print unidecode(u'äèß')
    
    [Wani@Linux tmp]$ python tmp.py 
    File "tmp.py", line 1
    SyntaxError: Non-ASCII character '\xc3' in file tmp.py on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
    [Wani@Linux tmp]$ 
    

To fix this, you do:

    
    
    #!/bin/python
    #coding: utf8
    from unidecode import *; print unidecode(u'äèß')
    
    [Wani@Linux tmp]$ python tmp.py
    aeess
    [Wani@Linux tmp]$
    

So, you need to call from command-line like this:

    
    
    [Wani@Linux tmp]$ python -c "#coding: utf8
    from unidecode import *; print unidecode(u'äèß')"
    aeess
    [Wani@Linux tmp]$ python -c "$(echo -e "#coding: utf8\nfrom unidecode import *; print unidecode(u'äèß')")"
    aeess
    [Wani@Linux tmp]
    

Further Reading: [Correct way to define Python source code
encoding](http://stackoverflow.com/questions/728891/correct-way-to-define-
python-source-code-encoding)

