Do I need to match the representation of my output layer and the target labels in PyTorch Nllloss?
I'm trying to solve a classification problem with 3 possible outputs: 0, 1, or 2. 
My output layer finally outputs a vector of probabilities for each label, say [0.3,0.4,0.3]

My loss function is defined such:

loss = criterion(output_batch, label_batch) #criterion = nn.NLLLoss()


Now my question has to do with the outputs and labels not matching in the way the store data. The output is in the form of a size=3 probability vector (adding to 1 using soft max), and my target labels are simple scalars. 

I can convert my labels to vectors when the loss function is calculated but I'm not sure if this is necessary

0 ==&gt; [1,0,0]
1 ==&gt; [0,1,0]
2 ==&gt; [0,0,1]


Can someone please shed light on this issue? Thanks! 

Let's say your classes are: cat, dog and capibara.

You have so called softmax predictions.

[0.3,0.4,0.3]

The softmax function is pumping one result at a top. In this case if dog is under 0.4 our output is predicting the dog.

Note how the predictions sum to 1 = 0.3+0.4+0.3.

Now you need to calculate the log of that which is log softmax, and then NLL is just negative of that.



  I can convert my labels to vectors when the loss function is calculated but I'm not sure if this is necessary?


0 ==&gt; [1,0,0]
1 ==&gt; [0,1,0]
2 ==&gt; [0,0,1]


This is not necessary in your case. This means we had three different estimations (bs=3) while you showed just one.



Here is a little exercise:

batch_size, n_classes = 10, 5
x = torch.randn(batch_size, n_classes)
print("x:",x)

target = torch.randint(n_classes, size=(batch_size,), dtype=torch.long)
print("target:",target)


def log_softmax(x): 
    return x - x.exp().sum(-1).log().unsqueeze(-1)

def nll_loss(p, target):
    return -p[range(target.shape[0]), target].mean()

pred = log_softmax(x)
print ("pred:", pred)
ohe = torch.zeros(batch_size, n_classes)
ohe[range(ohe.shape[0]), target]=1
print("ohe:",ohe)

pe = pred[range(target.shape[0]), target]
print("pe:",pe)

mean = pred[range(target.shape[0]), target].mean()
print("mean:",mean)

negmean = -mean
print("negmean:", negmean)

loss = nll_loss(pred, target)
print("loss:",loss)


Out:

x: tensor([[ 1.5837, -1.3132,  1.5513,  1.4422,  0.8072],
        [ 1.1740,  1.9250,  0.4258, -1.0320, -0.4650],
        [-1.2447, -0.5360, -1.4950,  1.2020,  1.2724],
        [ 0.2300,  0.2587, -0.4463, -0.1397, -0.3617],
        [-0.7983,  0.7742,  0.0035,  0.9963, -0.7926],
        [ 0.7575, -0.8008,  0.7995,  0.0448,  0.6621],
        [-1.7153,  0.7672, -0.6841, -0.4826, -0.8614],
        [ 0.0263,  0.7244,  0.8751, -1.0226, -1.3762],
        [ 0.0192, -0.4368, -0.4010, -1.0660,  0.0364],
        [-0.5120, -1.4871,  0.6758,  1.2975,  0.2879]])
target: tensor([0, 4, 3, 0, 0, 4, 1, 2, 4, 2])
pred: tensor([[-1.2094, -4.1063, -1.2418, -1.3509, -1.9859],
        [-1.3601, -0.6091, -2.1083, -3.5661, -2.9991],
        [-3.3233, -2.6146, -3.5736, -0.8766, -0.8063],
        [-1.3302, -1.3015, -2.0065, -1.7000, -1.9220],
        [-2.7128, -1.1403, -1.9109, -0.9181, -2.7070],
        [-1.2955, -2.8538, -1.2535, -2.0081, -1.3909],
        [-3.0705, -0.5881, -2.0394, -1.8379, -2.2167],
        [-1.7823, -1.0841, -0.9334, -2.8311, -3.1847],
        [-1.2936, -1.7496, -1.7138, -2.3788, -1.2764],
        [-2.5641, -3.5393, -1.3764, -0.7546, -1.7643]])
ohe: tensor([[1., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 1., 0.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 0., 1.],
        [0., 0., 1., 0., 0.]])
pe: tensor([-1.2094, -2.9991, -0.8766, -1.3302, -2.7128, -1.3909, -0.5881, -0.9334,
        -1.2764, -1.3764])
mean: tensor(-1.4693)
negmean: tensor(1.4693)
loss: tensor(1.4693)

