Shape of target and predictions tensors in PyTorch loss functions
I am confused with the input shapes for tensors in nn.CrossEntropyLoss.
I am trying to implement a simple autoencoder for text sequences. The core of my problem can be illustrated by the following code

predictions = torch.rand(2, 3, 4)
target = torch.rand(2, 3)
print(predictions.shape)
print(target.shape)
nn.CrossEntropyLoss(predictions.transpose(1, 2), target)


In my case predictions has the shape (time_step, batch_size, vocabulary_size) while target has the shape (time_step, batch_size). Next I am transposing the predictions as per description which says that the second dimension of predictions should be the number of classes - vocabulary_size in my case. The code returns an error RuntimeError: bool value of Tensor with more than one value is ambiguous. Could someone please enlighten me how to use the damn thing? Thank you in advance!

You are not calling the loss function, but you are building it. The signature of the nn.CrossEntropyLoss constructor is:

nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean')


You are setting the predictions as the weight and the target as size_average, 
where weight is an optional rescaling of the classes and size_average is deprecated, but expects a boolean. The target is a tensor of size [2, 3], which cannot be converted to a boolean.

You need to create the loss function first, as you don't use any of the optional parameters of the constructor, you don't specify any of them.

# Create the loss function
cross_entropy = nn.CrossEntropyLoss()

# Call it to calculate the loss for your data
loss = cross_entropy(predictions.transpose(1, 2), target)


Alternatively, you can directly use the functional version nn.functional.cross_entropy:

import torch.nn.functional as F

loss = F.cross_entropy(predictions.transpose(1, 2), target)


The advantage of the class version, compared to the functional version, is that you only need to specify the extra parameters once (such as the weight) instead of having to supply them manually each time.

Regarding the dimensions of the tensors, the batch size must be the first dimension, because the losses are averaged per element in the batch, so you have tensor of losses with size [batch_size]. If you used reduction="none", you would get back theses losses per element in the batch, but by default (reduction="mean") the mean of these losses is returned. That result would be different if the mean is taken across time steps rather than batches.

Lastly, the targets need to be the class indices, which means they need to have type torch.long not torch.float. In this randomly chosen example, you could create the random classes with torch.randint.

predictions = torch.rand(2, 3, 4)
target = torch.randint(4, (2, 3))

# Reorder the dimensions
# From: [time_step, batch_size, vocabulary_size]
# To: [batch_size, vocabulary_size, time_step]
predictions = predictions.permute(1, 2, 0)
# From: [time_step, batch_size]
# To: [batch_size, time_step]
target = target.transpose(0, 1)

F.cross_entropy(predictions, target)

