"ValueError: You have to specify either input_ids or inputs_embeds" when training AutoModelWithLMHead Model (GPT-2)
I want to fine-tune the AutoModelWithLMHead model from this repository, which is a German GPT-2 model. I have followed the tutorials for pre-processing and fine-tuning. I have prepocessed a bunch of text passages for the fine-tuning, but when beginning training, I receive the following error:
File &quot;GPT\lib\site-packages\torch\nn\modules\module.py&quot;, line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;GPT\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py&quot;, line 774, in forward
    raise ValueError(&quot;You have to specify either input_ids or inputs_embeds&quot;)
ValueError: You have to specify either input_ids or inputs_embeds

Here is my code for reference:
# Load data
with open(&quot;Fine-Tuning Dataset/train.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as train_file:
    train_data = train_file.read().split(&quot;--&quot;)

with open(&quot;Fine-Tuning Dataset/test.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as test_file:
    test_data = test_file.read().split(&quot;--&quot;)

# Load pre-trained tokenizer and prepare input
tokenizer = AutoTokenizer.from_pretrained('dbmdz/german-gpt2')

tokenizer.pad_token = tokenizer.eos_token
train_input = tokenizer(train_data, padding=&quot;longest&quot;)
test_input = tokenizer(test_data, padding=&quot;longest&quot;)

# Define model

model = AutoModelWithLMHead.from_pretrained(&quot;dbmdz/german-gpt2&quot;)
training_args = TrainingArguments(&quot;test_trainer&quot;)


# Evaluation

metric = load_metric(&quot;accuracy&quot;)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = numpy.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_input,
    eval_dataset=test_input,
    compute_metrics=compute_metrics,
)
trainer.train()
trainer.evaluate()


Does anyone know the reason for this? Any help is welcome!

I didn't find the concrete answer to this question, but a workaround. For anyone looking for examples on how to fine-tune the GPT models from HuggingFace, you may have a look into this repo. They listed a couple of examples on how to fine-tune different Transformer models, complemented by documented code examples. I used the run_clm.py script and it achieved what I wanted.
