CUDNN_STATUS_MAPPING_ERROR when training with pose2body
I'm trying to train https://github.com/NVIDIA/vid2vid. I'm...


...executing with pretty much the vanilla parametrization shown in the readme, I had to change the number of GPUs though and increased the number of threads for reading the dataset. Command:

python train.py \
--name pose2body_256p  \
--dataroot datasets/pose  \
--dataset_mode pose  \
--input_nc 6  \
--num_D 2  \ 
--resize_or_crop ScaleHeight_and_scaledCrop  \
--loadSize 384  \
--fineSize 256  \
--gpu_ids 0,1  \
--batchSize 1  \
--max_frames_per_gpu 3  \
--no_first_img  \
--n_frames_total 12  \
--max_t_step 4  \
--nThreads 6 
...training on the supplied example datasets.
...running a docker container built with the scripts in vid2vid/docker, e. g. with CUDA 9.0 and CUDNN 7.
...using two NVIDIA V100 GPUs.


Whenever I start training the script crashes after a couple of minutes with the message RuntimeError: CUDNN_STATUS_MAPPING_ERROR. Full error message:

Traceback (most recent call last):
  File "train.py", line 329, in &lt;module&gt;
    train()
  File "train.py", line 104, in train
    fake_B, fake_B_raw, flow, weight, real_A, real_Bp, fake_B_last = modelG(input_A, input_B, inst_A, fake_B_last)            
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/vid2vid/models/vid2vid_model_G.py", line 130, in forward
    fake_B, fake_B_raw, flow, weight = self.generate_frame_train(netG, real_A_all, fake_B_prev, start_gpu, is_first_frame)        
  File "/vid2vid/models/vid2vid_model_G.py", line 175, in generate_frame_train
    fake_B_feat, flow_feat, fake_B_fg_feat, use_raw_only)
  File "/vid2vid/models/networks.py", line 171, in forward
    downsample = self.model_down_seg(input) + self.model_down_img(img_prev)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR


From reading the issues in the vid2vid using two V100 should work with this setup. The error also occurs if CUDA 8/CUDNN 6 are used. I checked the flags but haven't found any indication of further necessary changes to the arguments supplied to train.py.

Any ideas on how to solve (or work around) this?

In case anybody deals the same issue: Training on P100 cards worked. Seems like the V100 architecture clashes with version of pytorch used in the supplied Dockerfile at some point. Not quite a solution, but a workaround.
