BERT model loss function from one hot encoded labels
For the line: loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
I have labels hot encoded such that it is a tensor of 32x17, since the batch size is 32 and there are 17 classes for the text categories. However, BERT model only takes for the label with a single dimension vector.
Hence, I get the error:
Expected input batch_size (32) to match target batch_size (544)
The 544 is the product of 32x17. However, my question is how could I use one hot encoded labels to  get the loss value in each iteration? I could use just label encoded labels, but that would not really be suitable for unordered labels.
# BERT training loop
for _ in trange(epochs, desc=&quot;Epoch&quot;):  
  
  ## TRAINING
  
  # Set our model to training mode
  model.train()  
  # Tracking variables
  tr_loss = 0
  nb_tr_examples, nb_tr_steps = 0, 0
  # Train the data for one epoch
  for step, batch in enumerate(train_dataloader):
    # Add batch to GPU
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_input_mask, b_labels = batch
    # Clear out the gradients (by default they accumulate)
    optimizer.zero_grad()
    # Forward pass
    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
    train_loss_set.append(loss.item())    
    # Backward pass
    loss.backward()
    # Update parameters and take a step using the computed gradient
    optimizer.step()
    # Update tracking variables
    tr_loss += loss.item()
    nb_tr_examples += b_input_ids.size(0)
    nb_tr_steps += 1
  print(&quot;Train loss: {}&quot;.format(tr_loss/nb_tr_steps))


As stated in the comment, Bert for sequence classification expects the target tensor as a [batch] sized tensors with values spanning the range [0, num_labels). A one-hot encoded tensor can be converted by argmaxing it over the label dim, i.e. labels=b_labels.argmax(dim=1).
