title: Model fits on a single GPU but script crashes when trying to fit on multiple GPUs
body: <p>I have a model that can train fine on a single GPU, but when I try to fit it using multi_gpu_model, I get this CUDA error before the script exits:</p>
  
  <pre><code>F tensorflow/stream_executor/cuda/cuda_dnn.cc:521] Check failed: cudnnSetTensorNdDescriptor(handle_.get(), elem_type, nd, dims.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)batch_descriptor: {count: 0 feature_map_count: 16 spatial: 128 128 128  value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX}
  </code></pre>
  
  <p>I tried to pass both the compiled and not-compiled version of the model instance to the multi_gpu_model function, but it did not change anything. I call it like this:</p>
  
  <pre><code>multi_model = multi_gpu_model(model, gpus=4)
  </code></pre>
  
  <p>The compilation is done this way and does not raise any errors:</p>
  
  <pre><code>multi_model.compile(
      optimizer=keras.optimizers.Adam(5e-4),
      loss=dice_coefficient_loss,
      metrics=[dice_coefficient]
              + get_label_wise_dice_coefficient_functions(n_labels))
  
  def dice_coefficient(y_true, y_pred, smooth=1.):
      y_true_f = K.flatten(y_true)
      y_pred_f = K.flatten(y_pred)
      intersection = K.sum(y_true_f * y_pred_f)
      return ((2. * intersection + smooth)
              / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth))
  
  
  def dice_coefficient_loss(y_true, y_pred):
      return -dice_coefficient(y_true, y_pred)
  
  
  def label_wise_dice_coefficient(y_true, y_pred, label_index):
      return dice_coefficient(y_true[:, label_index], y_pred[:, label_index])
  
  
  def get_label_dice_coefficient_function(label_index):
      f = functools.partial(label_wise_dice_coefficient, label_index=label_index)
      f.__setattr__('__name__', 'label_{0}_dice_coef'.format(label_index))
      return f
  
  
  def get_label_wise_dice_coefficient_functions(n_labels):
      return [get_label_dice_coefficient_function(i) for i in range(n_labels)]
  </code></pre>
  
  <p>(most of these functions and the model architecture have been stolen <a href="https://github.com/ellisdg/3DUnetCNN/blob/master/unet3d/metrics.py" rel="nofollow noreferrer">here</a>)</p>
  
  <p>I am using python 3.6.6, tensorflow-gpu 1.10.0, cudatoolkit 9.2, cudnn 7.2.1 from conda main repo and keras-contrib 2.0.8 installed with pip/git on top of a 64-bits CentOS 7.4.1708</p>
  
  <p>Looking at the previous log lines, it looks like the multiple GPUs are properly detected:</p>
  
  <pre><code>2018-10-09 16:30:19.977993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
  name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
  pciBusID: 0000:20:00.0
  totalMemory: 10.92GiB freeMemory: 10.74GiB
  2018-10-09 16:30:20.318137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties:
  name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
  pciBusID: 0000:21:00.0
  totalMemory: 10.92GiB freeMemory: 10.74GiB
  2018-10-09 16:30:20.595428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 2 with properties:
  name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
  pciBusID: 0000:22:00.0
  totalMemory: 10.92GiB freeMemory: 10.74GiB
  2018-10-09 16:30:20.953619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 3 with properties:
  name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
  pciBusID: 0000:23:00.0
  totalMemory: 10.92GiB freeMemory: 10.74GiB
  2018-10-09 16:30:20.967429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3
  2018-10-09 16:30:22.415906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
  2018-10-09 16:30:22.415957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 2 3
  2018-10-09 16:30:22.415965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y Y Y
  2018-10-09 16:30:22.415971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N Y Y
  2018-10-09 16:30:22.415982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 2:   Y Y N Y
  2018-10-09 16:30:22.415988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 3:   Y Y Y N
  2018-10-09 16:30:22.416681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10393 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:20:00.0, compute capability: 6.1)
  2018-10-09 16:30:22.536003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10393 MB memory) -&gt; physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:21:00.0, compute capability: 6.1)
  2018-10-09 16:30:22.637811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10393 MB memory) -&gt; physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:22:00.0, compute capability: 6.1)
  2018-10-09 16:30:22.747698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10393 MB memory) -&gt; physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:23:00.0, compute capability: 6.1)
  2018-10-09 16:30:25,557.557:__main__:INFO:Compiling model
  2018-10-09 16:30:25,634.634:__main__:INFO:Fitting model
  2018-10-09 16:31:31.773355: F tensorflow/stream_executor/cuda/cuda_dnn.cc:521] Check failed: cudnnSetTensorNdDescriptor(handle_.get(), elem_type, nd, dims.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)batch_descriptor: {count: 0 feature_map_count: 16 spatial: 128 128 128  value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX}
  /bin/bash: line 1: 160691 Aborted
  </code></pre>
  
  <p>Any help on what I did wrong would be greatly appreciated.</p>