DataParallel multi-gpu RuntimeError: chunk expects at least a 1-dimensional tensor
I am trying to run my model on multiple gpus using DataParallel by setting model = nn.DataParallel(model).cuda(), but everytime getting this error - 


  RuntimeError: chunk expects at least a 1-dimensional tensor (chunk at
  /pytorch/aten/src/ATen/native/TensorShape.cpp:184).


My code is correct. Does anyone know what's wrong? 

I have tried setting device_ids=[0,1] parameter and also CUDA_VISIBLE_DEVICES on the terminal. Also tried different batch sizes.

To identify the problem, you should check the shape of your input data for each mini-batch. The documentation says, nn.DataParallel splits the input tensor in dim0 and sends each chunk to the specified GPUs. From the error message, it seems you are trying to pass a 0-dimensional tensor.

One possible reason can be if you have a mini-batch with n examples and you are running your program on more than n GPUs, then you will get this error.

Let's consider the following scenario.


Total training examples = 161, Batch size = 80, total mini-batches = 3
Number of GPUs specified for DataParallel = 3


Now, in the above scenario, in the 3rd mini-batch, there will be 1 example. So, it is not possible to send chunks to all the specific GPUs and you will receive the error message. So, please check if you are not a victim of this issue.
