Tensor sizes CrossEntropyLoss
In the snippet: 

criterion = nn.CrossEntropyLoss()
raw_loss = criterion(output.view(-1, ntokens), targets)


output size is torch.Size([5, 5, 8967]), targets size is torch.Size([25]), and ntokens is 8967

After modifying the code, my 

output size is torch.Size([5, 8967]) and targets size is torch.Size([25])

which rises dimensionality issues when computing the loss.

Is it sensible to increase the size of my Linear activation that produces the output by 5, so that I can resize the output later to be of the size torch.Size([5, 5, 8967])?

The problem with increasing the size of the tensor is that ntokens can become quite large and I can easily run out of memory because of that. Is there an alternative approach?

You should do something like this:

ntokens = 8000
output = Variable(torch.randn(5, 5, ntokens))
targets = Variable(torch.from_numpy(np.random.randint(0, ntokens, size=25)))
criterion = nn.CrossEntropyLoss()
loss = criterion(output.view(-1, ntokens), targets)
print(loss)


This prints:

Variable containing:
 9.4613
[torch.FloatTensor of size 1]


Here, I am assuming output contains predictions of next word for 5 sentences (minibatch size is 5) and each sentence is of length 5 (sequence length is 5). 8000 is the vocabulary size, so your model is predicting a probability distribution over the entire vocabulary.
Now, you can compute the loss of predicting each word as your target shape is 25 as required.

Please note, CrossEntropyLoss expects input to contain scores for each class. So, input has to be a 2D Tensor of size (minibatch, C) and the target has to be a class index (0 to C-1) for each value of a 1D tensor of size minibatch.
