61797


tfds.load()を使って，food101をロードしたいのですが，エラーに困っています．

具体的には，以下のようなコードを実行しています．

[code]

    import tensorflow_datasets as tfds
    dataset, info = tfds.load('food101', as_supervised = True, with_info = True, batch_size = -1)
    
[/code]

その際に，出力されるエラーが次の通りです．

[code]

    Downloading and preparing dataset food101 (4.65 GiB) to C:\Users\name\tensorflow_datasets\food101\1.0.0...
    Dl Completed...:   0%|                           C:\Users\name\py37\lib\site-packages\urllib3\connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning,0 file [00:00, ? file/s]
    Dl Completed...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [24:09<00:00, 734.24s/ url]
    Extraction completed...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [24:09<00:00, 1449.11s/ file]
    Extraction completed...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [24:09<00:00, 1449.11s/ file]
    Dl Size...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4764/4764 [24:09<00:00,  3.29 MiB/s]
    
    Dl Completed...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [24:09<00:00, 1449.11s/ url]
    Shuffling and writing examples to C:\Users\name\tensorflow_datasets\food101\1.0.0.incomplete00ZN8D\food101-train.tfrecord
    Computing statistics...:   0%|                                                                                                                                | 0/1 [00:00<?, ? split/s]2019-12-24 23:07:30.557351: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
                                        2019d to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.Faile
                                         2019-12-24 23:07:32.047151: W tensorflow/core/framework/model.cc:855] Failed to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.
                                          2019-12-24 23:07:32.390802: W tensorflow/core/framework/model.cc:855] Failed to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.
                                           2019-12-24 23:07:33.046748: W tensorflow/core/framework/model.cc:855] Failed to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.
                                            2019-12-24 23:07:34.343701: W tensorflow/core/framework/model.cc:855] Failed to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.
                                            2019-12-24 23:07:36.921180: W tensorflow/core/framework/model.cc:855] Failed to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.
    Computing statistics...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [04:13<00:00, 253.99s/ split]
    Dataset food101 downloaded and prepared to C:\Users\name\tensorflow_datasets\food101\1.0.0. Subsequent calls will reuse this data.
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Users\name\py37\lib\site-packages\wrapt\wrappers.py", line 564, in __call__
        args, kwargs)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\api_utils.py", line 52, in disallow_positional_args_dec
        return fn(*args, **kwargs)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\registered.py", line 314, in load
        ds = dbuilder.as_dataset(**as_dataset_kwargs)
      File "C:\Users\name\py37\lib\site-packages\wrapt\wrappers.py", line 603, in __call__
        args, kwargs)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\api_utils.py", line 52, in disallow_positional_args_dec
        return fn(*args, **kwargs)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\dataset_builder.py", line 455, in as_dataset
        datasets = utils.map_nested(build_single_dataset, split, map_tuple=True)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\utils\py_utils.py", line 145, in map_nested
        for k, v in data_struct.items()
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\utils\py_utils.py", line 145, in <dictcomp>
        for k, v in data_struct.items()
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\utils\py_utils.py", line 159, in map_nested
        return function(data_struct)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_datasets\core\dataset_builder.py", line 539, in _build_single_dataset
        return tf.data.experimental.get_single_element(dataset)
      File "C:\Users\name\py37\lib\site-packages\tensorflow_core\python\data\experimental\ops\get_single_element.py", line 70, in get_single_element
        dataset._variant_tensor, **dataset._flat_structure))  # pylint: disable=protected-access
      File "C:\Users\name\py37\lib\site-packages\tensorflow_core\python\ops\gen_dataset_ops.py", line 1065, in dataset_to_single_element
        _six.raise_from(_core._status_to_exception(e.code, message), None)
      File "<string>", line 3, in raise_from
    tensorflow.python.framework.errors_impl.ResourceExhaustedError: {{function_node __inference_Dataset_map_<class 'functools.partial'>_202275}} OOM when allocating tensor with shape[512,512,3] and type uint8 on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
             [[{{node decode_image/cond_jpeg/then/_0/DecodeJpeg}}]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
     [Op:DatasetToSingleElement]
    
[/code]

他のデータセット（MNISTやcifar10）では，上記のコードでロードできています．

データ容量が関係しているのかと思い，こちらの記事  
<https://qiita.com/sugaok/items/6d79a4ca9c0f2e902596>  
を参考に試してみましたが，同じエラーが出ます．

つきましては，改善策をご教授頂けると幸いです．

**実行環境**

  * Windows 10 Pro
  * Python 3.7.5
  * tensorflow 2.0.0
  * tensorflow-datasets 1.3.2


CC BY-SA 4.0
2019-12-26T03:57:16.673
61789
2019-12-26T05:05:15.233
2019-12-26T05:05:15.233
32986
question
tensorflow
Tensorflow Datasetsによるfood101のロード


ぱっと見の回答ですが...

> OOM when allocating tensor with shape[512,512,3] and type uint8 on
> /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

OOM(Out Of Memoryエラー)が発生しているので、メモリ不足になっているのではないでしょうか。`batch_size =
-1`の指定をすると、`tf.Tensors`としてフル・データセットを返すと書いてあるので、これを消すか、値を調整することでOOMが発生しなくなりませんかね？

<https://www.tensorflow.org/datasets/api_docs/python/tfds/load>

> batch_size: int, if set, add a batch dimension to examples. Note that
> variable length features will be 0-padded. If batch_size=-1, will return the
> full dataset as tf.Tensors.


CC BY-SA 4.0
2019-12-26T04:34:53.987
61797
2019-12-26T04:34:53.987
21092
61789
answer