Why np.load() couldn't read my ndarray data in pickled file?
Question: I am trying to analyze a tensor data, but I could not read the data in picked
file by using np.load(). My python code is as follows:

    
    
    import pickle
    import numpy as np
    import sktensor as skt
    import numpy.random as rn
    
    data = np.ones((10, 8, 3), dtype='int32')  # 3-mode count tensor of size 10 x 8 x 3
    ##data = skt.dtensor(data)
    
    with open('data.dat', 'w+') as f:    # can be stored as a .dat using pickle 
        pickle.dump(data, f)
    
    with open('data.dat', 'r+') as f:  # can be loaded back in using pickle.load
        tmp = pickle.load(f)
        assert np.allclose(tmp, data)
    

But when I attempted to use np.load() to load the data in data.bat as follows:

    
    
    np.load('G:\data.dat')
    

Some error appears as"

    
    
    Traceback (most recent call last):
      File "<pyshell#34>", line 1, in <module>
        np.load('D:/GDELT_Tensor/data.dat', mmap_mode = 'r')
      File "C:\Python27\lib\site-packages\numpy\lib\npyio.py", line 416, in load
        "Failed to interpret file %s as a pickle" % repr(file))
    IOError: Failed to interpret file 'D:/data.dat' as a pickle.
    

Anyone can help me?


Answer: Don't use the pickle module to save NumPy arrays. Instead, use one of the
methods here: <http://docs.scipy.org/doc/numpy/reference/routines.io.html>

There's even one that uses pickle under the hood, for example:

    
    
    np.save('data.dat', data)
    tmp = np.load('data.dat')
    

Another format like CSV or HDF5 might be more suitable for most applications--
especially where you might want to interoperate with non-Python systems.

