How to convert binary classifier output/loss tensor in LSTM to multiclass
I am trying to train an LSTM model to predict what year a song was written given its lyrics using word-level association in Pytorch. There are 51 potential classes/labels (1965-2015) - however I was working off of a template that used a binary classifier for a different problem. I have been trying to figure out how to change the model to predict multiple classes (1965, 1966, etc). 

I understand that you are supposed to provide a tensor of size C=num_classes as the output. However, I did that by making output_size=51 but I am getting an error, which makes me think there is something related to defining or operating on the criterion class I am defining that I am not doing correctly.

Here is the model:

class LyricLSTM(nn.Module):
    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        super().__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        # embedding and LSTM layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers,
                            dropout=drop_prob, batch_first=True)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        self.sig = nn.Sigmoid()
        #self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x, hidden):
        batch_size = x.size(0)

        # embeddings and lstm_out
        embeds = self.embedding(x)
        lstm_out, hidden = self.lstm(embeds, hidden)

        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        # dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)
        # sigmoid function
        sig_out = self.sig(out)
        #sig_out = self.softmax(out)

        # reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1)
        sig_out = sig_out[:, -1]  # get last batch of labels

        # return last sigmoid output and hidden state
        return sig_out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())


        return hidden


And the training loop:

n_epochs = 10
batch_size = 16 #100  # 11 batches of size 337 so iters = 11 (11 * 337 = 3707)

# Split into training, validation, testing - train= 80% | valid = 10% | test = 10%
split_frac = 0.8
train_x = encoded_lyrics[0:int(split_frac * len(encoded_lyrics))] # 3707 training samples
train_y = encoded_years[0:int(split_frac * len(encoded_lyrics))]  # 3707 training samples

# Dataloaders and batching
# create Tensor datasets
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))

# make sure to SHUFFLE your data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, drop_last=True)

output_size = 51
embedding_dim = 400
hidden_dim = 128 #256
n_layers = 2
lstmc = lstm.LyricLSTM(vocab_len, output_size, embedding_dim, hidden_dim, n_layers)

# Loss function + accuracy reporting
current_loss = 0
losses = np.zeros(n_epochs)  # For plotting
accuracy = np.zeros(n_epochs)

lr = 0.001
criterion = nn.CrossEntropyLoss() #nn.BCELoss()
optimizer = torch.optim.Adam(lstmc.parameters(), lr=lr)
counter = 0
print_every = 1
clip = 5  # gradient clipping

# Main training loop
start = time.time()
lstmc.train()
for epoch in range(0, n_epochs):
    # initialize hidden state
    h = lstmc.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        lstmc.zero_grad()

        # get the output from the model
        inputs = inputs.type(torch.LongTensor)
        output, h = lstmc(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.float())
        loss.backward()

        nn.utils.clip_grad_norm_(lstmc.parameters(), clip)
        optimizer.step()


I am getting this error when I run the code

File "main.py", line 182, in main
    loss = criterion(output.squeeze(), labels.float())
/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
/venv/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
/venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
/venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1295, in log_softmax
    ret = input.log_softmax(dim)
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)


This is the output I am getting and the labels (for batch size 16):

Output: tensor([0.4962, 0.5025, 0.4963, 0.4936, 0.5058, 0.4872, 0.4995, 0.4852, 0.4840,
        0.4791, 0.4984, 0.5034, 0.4796, 0.4826, 0.4811, 0.4859],
       grad_fn=&lt;SqueezeBackward0&gt;)

Labels: tensor([1994., 1965., 1981., 1986., 1973., 1981., 1975., 1968., 1981., 1968.,
        1989., 1981., 1988., 1991., 1983., 1982.])


I was expecting for the output to be a tensor of length 51 where each element contained the likelihood that that year was the correct answer (ex: output[0] = first year / 1965, output[1] = 1966, etc).

You must provide input as (N, C) and target as (N) to CrossEntropyLoss. I suspect in your model's foward() method, the following code segment is wrong.

sig_out = self.sig(out) # shape: batch_size*seq_len x output_size

# reshape to be batch_size first 
sig_out = sig_out.view(batch_size, -1) # shape: batch_size x seq_len*output_size
sig_out = sig_out[:, -1] # shape: batch_size


What you are trying to do with your last statement? Also, what you want to do with the seq_len dimension of the LSTM output?

Try to think about what you are doing here.

Although I think the shape of the output tensor is wrong, make sure output is a 2d tensor of shape (N, C) and labels is a 1d tensor of shape (N).

Also, I see a few problems in your code.


Usually, it is a good practice to apply zero_grad to the optimizer, not on the model. Don't do the following.


# zero accumulated gradients
lstmc.zero_grad()


Instead, do: optimizer.zero_grad()


You shouldn't use Sigmoid with 51 classes. Rather use fully connected layers followed by a softmax layer. And you don't need view() operations before the fc and softmax layers.


self.fc = nn.Linear(hidden_dim, output_size)
self.softmax = nn.LogSoftmax(dim=-1) # use -1 to apply in the last axis

...

out = self.dropout(lstm_out)
out = self.softmax(self.fc(out))


So, don't use the following code segment.

# stack up lstm outputs
lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim) # DON'T DO THIS

