pytorch compute pairwise difference: Incorrect result in NumPy vs PyTorch and different PyTorch versions
Suppose I have two arrays, and I want to calculate row-wise differences between every two rows of two matrices of the same shape as follows. This is how the procedure looks like in numpy, and I want to replicate the same thing in pytorch.

&gt;&gt;&gt; a = np.array([[1,2,3],[4,5,6]])
&gt;&gt;&gt; b = np.array([[3,4,5],[5,3,2]])
&gt;&gt;&gt; c = a[np.newaxis,:,:] - b[:,np.newaxis,:]
&gt;&gt;&gt; print(c)
[[[-2 -2 -2]
  [ 1  1  1]]

 [[-4 -1  1]
  [-1  2  4]]]


BTW, I tried the same thing using pytorch, but it does not work. Is there anyway we could accomplish the same thing in pytorch

&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.from_numpy(a)
&gt;&gt;&gt; b = torch.from_numpy(b)
&gt;&gt;&gt; c1 = a[None,:,:]
&gt;&gt;&gt; c2 = b[:,None,:]
&gt;&gt;&gt; diff = c1 - c2
&gt;&gt;&gt; print(diff.size())
torch.Size([1, 2, 3])


I was actually looking for torch.Size([2,2,3]). (P.S. I also tried unsqueeze from pytorch, but it doesn't work).

The issue arises because of using PyTorch 0.1. If using PyTorch 1.0.1, the same operation of NumPy generalize to PyTorch without any modifications and issues. Here is a snapshot of the run in Colab.



As we can see, we indeed get the same results.



Here is an attempt to reproduce the error you faced of getting incorrect result:

&gt;&gt;&gt; t1 = torch.from_numpy(a)
&gt;&gt;&gt; t2 = torch.from_numpy(b)
&gt;&gt;&gt; t1[np.newaxis, ...] - t2[:, np.newaxis, ...]

(0 ,.,.) = 
 -2 -2 -2
 -1  2  4
[torch.LongTensor of size 1x2x3]

&gt;&gt;&gt; torch.__version__
'0.1.12_1'


So, please upgrade your PyTorch version to 1.0.1!



Digging more into for details:

The main reason why it didn't work in PyTorch version 0.1 is that broadcasting was not completely implemented then. Basically, the tensor promotion to 3D, followed by a subtraction can be achieved in two steps as in (in version 1.0.1):

&gt;&gt;&gt; t1[:1, ] - t2
&gt;&gt;&gt; tensor([[-2, -2, -2],   # t1_r1
            [-4, -1,  1]])  # t1_r2

&gt;&gt;&gt; t1[1:, ] - t2
&gt;&gt;&gt; tensor([[ 1,  1,  1],   # t2_r1
            [-1,  2,  4]])  # t2_r2


The results of above two operations put together by stacking rows in the order (t1_r1, t2_r1, t1_r2, t2_r2), after each of the rows being a 2D would give us the shape (2, 2, 3).

Now, try doing the above two steps in version 0.1, it would throw the error:


  RuntimeError: inconsistent tensor size at /opt/conda/conda-bld/pytorch_1501971235237/work/pytorch-0.1.12/torch/lib/TH/generic/THTensorMath.c:831

