CUDA_ERROR_OUT_OF_MEMORY I have a large network that is somewhat similar to Wavenet. Although (it seems) that my GPU has enough memory, I get an out of memory error on fitting (see logs below).

Any idea?

How can I troubleshoot these kind of CUDA driver issues?



2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.6575
pciBusID 0000:01:00.0
Total memory: 11.00GiB
Free memory: 10.71GiB

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-12-22 23:32:05.429386: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device
: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)

2017-12-22 23:32:06.131386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 10.17G (10922166272 bytes) fro
m device: CUDA_ERROR_OUT_OF_MEMORY
2017-12-22 23:32:06.599386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 9.15G (9829949440 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
2017-12-22 23:32:07.332586: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 8.24G (8846954496 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
built graph
Did you try to start your training with a smaller data set and that worked?

How about using generators to gradually input your data? For me that worked quite well (in Keras : model.fit_generator(...))