RuntimeError: Found dtype Long but expected Float when fine-tuning using Trainer API
I'm trying to fine-tune BERT model for sentiment analysis (classifying text as positive/negative) with Huggingface Trainer API. My dataset has two columns, Text and Sentiment, it looks like this.
Text                     Sentiment
This was good place          1
This was bad place           0

Here is my code:
from datasets import load_dataset
from datasets import load_dataset_builder
from datasets import Dataset
import datasets
import transformers
from transformers import TrainingArguments
from transformers import Trainer

dataset = load_dataset('csv', data_files='./train/test.csv', sep=';')
tokenizer = transformers.BertTokenizer.from_pretrained(&quot;TurkuNLP/bert-base-finnish-cased-v1&quot;)
model = transformers.BertForSequenceClassification.from_pretrained(&quot;TurkuNLP/bert-base-finnish-cased-v1&quot;, num_labels=1) 
def tokenize_function(examples):
    return tokenizer(examples[&quot;Text&quot;], truncation=True, padding='max_length')

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.rename_column('Sentiment', 'label')
tokenized_datasets = tokenized_datasets.remove_columns('Text')
training_args = TrainingArguments(&quot;test_trainer&quot;)
trainer = Trainer(
    model=model, args=training_args, train_dataset=tokenized_datasets['train']
)
trainer.train()

Running this throws error:
Variable._execution_engine.run_backward(
RuntimeError: Found dtype Long but expected Float

The error may come from dataset itself, but can I fix it with my code somehow? I searched the Internet and this error seems to have been previously solved by &quot;converting tensors to float&quot; but how would I do it with Trainer API? Any advise is very highly appreciated.
Some reference:
https://discuss.pytorch.org/t/run-backward-expected-dtype-float-but-got-dtype-long/61650/10

Most likely, the problem is with loss function. This can be fixed if you set up the model correctly, mainly by specifying the correct loss to use. Refer to this code to see the logic for deciding the proper loss.
Your problem has binary labels and thus should be framed as a single-label classification problem. As such, the code you have shared will be inferred as a regression problem, which explains the error that it expected float but found long type for target labels.
You need to pass the correct problem type.
model = transformers.BertForSequenceClassification.from_pretrained(
    &quot;TurkuNLP/bert-base-finnish-cased-v1&quot;, 
    num_labels=1, 
    problem_type = &quot;single_label_classification&quot;
) 

This will make use of BCE loss. For BCE loss, you need the target to float, so you also have to cast the labels to float. I think you can do that with the dataset API. See this.
The other way would be to use a multi-class classifier or CE loss. For that, just fixing num_labels should be fine.
model = transformers.BertForSequenceClassification.from_pretrained(
    &quot;TurkuNLP/bert-base-finnish-cased-v1&quot;, 
    num_labels=2,
) 

