32402


以下のサイトで提供されているカラー画像8000万枚のデータセットCIFAR-10を読み込みたいのですがいくつか他のサイトにあるものを試しているのですがどれもうまくいかず、ご存知の方いらしましたら教えて頂きたいです。

データ元  
<https://www.cs.toronto.edu/~kriz/cifar.html>

以下読み込むコード

[code]

    import pickle
    import numpy as np
    import os
    
    def unpickle(file):
        fo = open(file, 'rb')
        dict = pickle.load(fo)
        fo.close()
        return dict
    
    def conv_data2image(data):
        return np.rollaxis(data.reshape((3,32,32)),0,3)
    
    def get_cifar10(folder):
        tr_data = np.empty((0,32*32*3))
        tr_labels = np.empty(1)
        '''
        32x32x3
        '''
        for i in range(1,6):
            fname = os.path.join(folder, "%s%d" % ("data_batch_", i))
            data_dict = unpickle(fname)
            if i == 1:
                tr_data = data_dict['data']
                tr_labels = data_dict['labels']
            else:
                tr_data = np.vstack((tr_data, data_dict['data']))
                tr_labels = np.hstack((tr_labels, data_dict['labels']))
    
        data_dict = unpickle(os.path.join(folder, 'test_batch'))
        te_data = data_dict['data']
        te_labels = np.array(data_dict['labels'])
    
        bm = unpickle(os.path.join(folder, 'batches.meta'))
        label_names = bm['label_names']
        return tr_data, tr_labels, te_data, te_labels, label_names
    
    if __name__ == '__main__':
        datapath = "./data/cifar-10-batches-py"
    
        tr_data10, tr_labels10, te_data10, te_labels10, label_names10 = get_cifar10(datapath)
    
[/code]

以下エラー

> "C:\…\AppData\Local\Programs\Python\Python35\python.exe"  
>  "C:/…/PycharmProjects/Convolutional Neural Networks for Visual  
>  Recognition/input_cifar.py" Traceback (most recent call last): File  
>  "C:/…/PycharmProjects/Convolutional Neural Networks for Visual  
>  Recognition/input_cifar.py", line 41, in tr_data10,  
>  tr_labels10, te_data10, te_labels10, label_names10 =  
>  get_cifar10(datapath) File "C:/…/PycharmProjects/Convolutional Neural  
>  Networks for Visual Recognition/input_cifar.py", line 22, in  
>  get_cifar10 data_dict = unpickle(fname) File  
>  "C:/…/PycharmProjects/Convolutional Neural Networks for Visual  
>  Recognition/input_cifar.py", line 7, in unpickle dict =  
>  pickle.load(fo) UnicodeDecodeError: 'ascii' codec can't decode byte  
>  0x8b in position 6: ordinal not in range(128)
>
> Process finished with exit code 1


CC BY-SA 3.0
2017-02-05T13:31:20.400
32400
2017-02-05T18:03:28.650
2017-02-05T18:03:28.650
3068
20523
question
python
Python, CIFAR-10のデータを読み込む方法


使用のPythonVersionは3.x系でしょうか？  
7行目を以下にしてみてはいかがでしょうか。

[code]

    dict = pickle.load(fo, encoding='latin-1')
    
[/code]

または

[code]

    u = pickle._Unpickler(fo)
    u.encoding = 'latin1'
    dict = u.load()
    
[/code]

同様エラーのポスト  
<https://stackoverflow.com/questions/11305790/pickle-incompatability-of-numpy-
arrays-between-python-2-and-3>


CC BY-SA 3.0
2017-02-05T13:41:21.897
32402
2017-02-05T13:41:21.897
2017-05-23T12:38:56.083
-1
19716
32400
answer