Multiple GPU in MXNet C++ I am trying to make MXNet (C++ API) learn, with a common sample in C++, on multiple GPU.
According to this MXNet forum post, we need to aggregate manually the gradients that we fetch at the backpropagation time.
Now, if I separate the gradients of each GPU, both networks are training. If I concatenate the weights, it doesn't work. (Like this) :

gradValuesCombined.insert(gradValuesCombined.end(), gradValues1.begin(), gradValues1.end());
gradValuesCombined.insert(gradValuesCombined.end(), gradValues1.begin(), gradValues1.end());
NDArray combined1(gradValuesCombined.data(), combinedShape, ctx1);


However, if the gradients have the same batch_size, then summing the gradients works.

gradArray1[i] + gradArray2[i]


But, summing all the gradients like if it was a one-batch vector, doesn't work:

combinedOneDim[0] = 1;
int sizeOneBatch = combinedOneDim.Size();
gradValuesCombinedOneDim.insert(gradValuesCombinedOneDim.end(), gradValues1.begin(), gradValues1.begin() + sizeOneBatch);
for(int i = 1; i &lt; curGradArrayShape1[0]; i++) {
     std::transform(gradValuesCombinedOneDim.begin(), gradValuesCombinedOneDim.end(), gradValues1.begin() + i * sizeOneBatch, gradValuesCombinedOneDim.begin(), std::plus&lt;mx_float&gt;());
}
for(int i = 0; i &lt; curGradArrayShape2[0]; i++) {
     std::transform(gradValuesCombinedOneDim.begin(), gradValuesCombinedOneDim.end(), gradValues2.begin() + i * sizeOneBatch, gradValuesCombinedOneDim.begin(), std::plus&lt;mx_float&gt;());
}


At that stage, I figured that it was because I had to keep the initial shape of the gradient. But if I try to average the weights on the batch dimension, and copy the values to keep the same gradient shape, it should work. But it doesn't... And if I just sum everything on the first dimension and fill the rest with zeroes, it doesn't work either...

I verified, the synchronization of data is correct and working. So what do I miss?
Okay, I finally found the problem. It was that I was backpropagating the gradients on every layers. Including the input and output layers. And it seems that Mxnet expect strictly to have the same NDarray size than the one he computed (no bigger for instance). Otherwise, the array seems not to be copied at all (not even partially)and thus gives wrong results.

But to do that, I had to make sure the gradients size where equals, which is the case for every layers but the entry and output gradients. I incorrectly assumed that, if I increased the batch size of the input/output, all the layers of the network would have an increased batch size. However, the batch size of the input doesn't matter inside the network. So, what I thought I was doing by summing over the values for every batch size was clearly not doing what I thought - meaning averaging the gradients for every image. What I actually did with that was to level off every 3D kernels.

Thus, the simple solution in my case was just to sum the gradients for every layers but the first and last one.