Some readble content, but impossible to JSON dump to file
Question: This [text file](http://gget.it/3hgi8uyu/temp.txt) (30 bytes only, the content
is `'(Ne pas r\xe9pondre a ce message)'`) can be opened and inserted in a
`dict` successfully :

    
    
    import json
    
    d = {}
    
    with open('temp.txt', 'r') as f:
       d['blah'] = f.read()
    
    with open('test.txt', 'w') as f:
        data = json.dumps(d)
        f.write(data)    
    

But it is impossible to dump the `dict` into a JSON file (see traceback
below). Why?

I tried lots of solutions provided by various SO questions. The closest
solution I could get was [this
answer](http://stackoverflow.com/a/25122834/1422096). When using this, I can
dump to file, but then the JSON file looks like this:

    
    
    # test.txt
    {"blah": "(Ne pas r\u00e9pondre a ce message)"}
    

instead of

    
    
    # test.txt
    {"blah": "(Ne pas répondre a ce message)"}
    

Traceback:

    
    
      File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
        return _iterencode(o, 0)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 9: invalid continuation byte
    [Finished in 0.1s with exit code 1]
    


Answer: Your file is not UTF-8 encoded. It uses a _Latin_ codec, like ISO-8859-1 or
Windows Codepage 1252. Reading the file gives you the _encoded_ text.

JSON however requires _Unicode_ text. Python has a separate Unicode type, and
byte strings (type `str`) need to be decoded using a suitable codec. The
`json.dumps()` function uses UTF-8 by default; UTF-8 is a widely used codec
for encoding Unicode text data that can handle all codepoints in the standard,
and is also the default codec for JSON strings to use.

You need to either decode the string manually or tell `json.dumps()` what
codec to use for the byte string:

    
    
    data = json.dumps(d, encoding='latin1')  # applies to all bytestrings in d
    

or

    
    
    d['blah'] = d['blah'].decode('latin1')
    data = json.dumps(d)
    

or using `io.open()` to decode as you read:

    
    
    import io
    
    with io.open('test.txt', 'w', encoding='latin1') as f:
        d['blah'] = f.read()
    

By default, the `json` library produces ASCII-safe JSON output by using the
`\uhhhh` escape syntax the JSON standard allows for. _This is entirely normal_
, the output is valid JSON and readable by any compliant JSON decoder.

If you _must_ produce UTF-8 encoded output without the `\uhhhh` escape
sequences, see [Saving utf-8 texts in json.dumps as UTF8, not as \u escape
sequence](https://stackoverflow.com/questions/18337407/saving-utf-8-texts-in-
json-dumps-as-utf8-not-as-u-escape-sequence)

