Unpickling from converted string in python/numpy
Question: I have a ton of numpy ndarrays that are stored picked to strings. That may
have been a poor design choice but it's what I did, and now the picked strings
seem to have been converted or something along the way, when I try to unpickle
I notice they are of type `str` and I get the following error:

    
    
    TypeError: 'str' does not support the buffer interface
    

when I invoke

    
    
    numpy.loads(bin_str)
    

Where `bin_str` is the thing I'm trying to unpickle. If I print out
`bin_str`it looks like

    
    
    b'\x80\x02cnumpy.core.multiarray\n_reconstruct\nq\x00cnumpy\nndarray\nq\x01K\x00\x85q\x02c_codecs\nencode\nq\x03X\x01\x00\x00\ ...
    

continuing for some time, so the info seems to be there, I'm just not quite
sure how to convert it into whatever string format numpy/pickle need. On a
whim I tried

    
    
    numpy.loads( bytearray(bin_str, encoding='utf-8') )
    

and

    
    
    numpy.loads( bin_str.encode() )
    

which both throw an error `_pickle.UnpicklingError: unpickling stack
underflow`. Any ideas?

PS: I'm on python 3.3.2 and numpy 1.7.1

**Edit**

I discovered that if I do the following:

    
    
    open('temp.txt', 'wb').write(...)
    return numpy.load( 'temp.txt' )
    

I get back my array, and `...` denotes _copying and pasting_ the output of
`print(bin_str)` from another window. I've tried writing `bin_str` to a file
directly to unpickle but that doesn't work, it complains that `TypeError:
'str' does not support the buffer interface`. A few sane ways of converting
`bin_str` to something that _can_ be written directly to a binary file result
in pickle errors when trying to read it back.

**Edit 2** So I guess what's happened is that my binary pickle string ended up
encoded inside of a normal string, something like:

    
    
    "b'pickle'"
    

which is unfortunate and I haven't figured out how to deal with that, except
this ridiculous and convoluted way to get it back:

    
    
    open('temp.py', 'w').write('foo = ' + bin_str)
    from temp import foo
    numpy.loads( foo )
    

This seems like a very shameful solution to the problem, so please give me a
better one!


Answer: It sounds like your saved strings are the `repr`s of the original `bytes`
instances returned by your pickling code. That's a bit unfortunate, but not
too bad. `repr` is intended to return a "machine friendly" representation of
an object, and it can often be reversed by using `eval`:

    
    
    import numpy as np
    import pickle
    
    # this part has already happened
    orig_obj = np.array([1,2,3])
    orig_pickle = pickle.dumps(orig_obj)
    saved_str = repr(orig_pickle)     # this was a mistake, but it's already done
    
    # this is what you need to do to get something equivalent to orig_obj back
    reconstructed_pickle = eval(saved_str)
    reconstructed_obj = pickle.loads(reconstructed_pickle)
    
    # test
    if np.all(reconstructed_obj == orig_obj):
        print("It worked!")
    

Obligatory note that using `eval` can be dangerous: Be aware that `eval` can
run any Python code it wants, so don't call it with untrusted data. However,
pickle data has the same risks (a malicious Pickle string can run arbitrary
code upon unpickling), so you're not losing much safety in this situation. I'm
guessing that you trust your data in this case anyway.

