Get Python 2.7's 'json' to not throw an exception when it encounters random byte strings
Question: Trying to serialize a dict object into a json string using Python 2.7's `json`
(ie: `import json`).

    
    
    Example:
    json.dumps({
        'property1':        'A normal string',
        'pickled_property': \u0002]qu0000U\u0012
    })
    

The object has some byte strings in it that are "pickled" data using
`cPickle`, so for json's purposes, they are basically random byte strings. I
was using django.utils's `simplejson` and this worked fine. But I recently
switched to Python 2.7 on google app engine and they don't seem to have
simplejson available anymore.

Now that I am using `json`, it throws an exception when it encounters bytes
that aren't part of UTF-8. The error that I'm getting is:

    
    
    UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0: invalid start byte
    

It would be nice if it printed out a string of the character codes like the
debugging might do, ie: `\u0002]q\u0000U\u001201`. But I really don't much
care how it handles this data just as long as it doesn't throw an exception
and continues serializing the information that it does recognize.

How can I make this happen?

Thanks!


Answer: The [JSON spec](http://www.json.org/) defines strings in terms of unicode
characters. For this reason, the `json` module assumes that any `str` instance
it receives holds encoded unicode text. It will try UTF-8 as its default
encoding, which causes trouble when you have a string like the output of
`pickle.dumps` which may not be a valid UTF-8 sequence.

Fortunately, fixing the issue is easy. You simply need to tell the
`json.dumps` function what encoding to use instead of UTF-8. The following
will work, even though `my_bytestring` is not valid UTF-8 text:

    
    
    import json, cPickle as pickle
    
    my_data = ["some data", 1, 2, 3, 4]
    my_bytestring = pickle.dumps(my_data, pickle.HIGHEST_PROTOCOL)
    json_data = json.dumps(my_bytestring, encoding="latin-1")
    

I believe that any 8-bit encoding will work in place of the `latin-1` used
here (just be sure to use the same one for decoding later).

When you want to unpickle the JSON encoded data, you'll need to make a call to
`unicode.decode`, since `json.loads` always returns encoded strings as
`unicode` instances. So, to get the `my_data` list back out of `json_data`
above, you'd need this code:

    
    
    my_unicode_data = json.loads(json_data)
    my_new_bytestring = my_unicode_data.encode("latin-1")  # equal to my_bytestring
    my_new_data = pickle.loads(my_new_bytestring)          # equal to my_data
    

