Python encoding error? UnicodeDecodeError: 'ascii' codec can't decode byte ordinal not in range(128)
Question: I have an AWS autoscale instance. On the AMI for that instance, I have a file
`myfile.py` that contains the following string:

    
    
    X5ZŒ
    

In my AWS Cloudformation, LaunchConfiguration, I have UserData that executes
the following python instruction when the instance is spun up. It simply reads
`myfile.py` and attempts to replace all occurrences of the regexp "X\dZ" with
"XYZ":

    
    
    myString = "XYZ".join(re.compile('X\d\Z').split(open("myfile.py", "r").read()))
    

That produces this error:

    
    
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 4: ordinal not in range(128)
    

Okay. I understand that the character `Œ` is causing trouble because it is
non-ASCII. But when I manually run the same statement from the python shell,
it works fine without throwing any exception.

How can I force the script that is run by AWS LaunchConfig's UserData to have
the same behavior as when I run it manually myself? What encoding should I set
and how should I set it?


Answer: Your file is encoded, that means it represents unicode in a defined way. You
use some literal strings (e.g. `XYZ`), which shall be used to in oprations
together with the file content. These literal strings are encoded just like
the file, in which this code is defined.

Python tries to coerce both into the same to be able to operate on it. It is
best, if you convert both into unicode. For the string literals, just prepend
a `u`, like so: `u"XYZ"`. For the file you have to tell python the encoding,
if you don't, it by default assumes `ascii`. Try:

    
    
     myString = u"XYZ".join(re.compile(u'X\d\Z').split(
         open("myfile.py", "r").read().decode('utf-8')))
    

in this version I used `utf-8`, which has some probability of being right. If
you know it being different, you'd have to substitute it for the [right
one](https://docs.python.org/2/library/codecs.html#standard-encodings) (your
editor in which you saved the file can probably tell you).

EDIT: I removed the part about console and environment settings, which don't
apply, as @Martijn mentioned. He's also right about, that this error is only
reproducible if your string literals _are_ already implicitly unicode (`from
__future__ import unicode_literals`) - in this case there's no need to prepend
the string literals with a `u`.

