Tokenizer can add padding without error, but data collator cannot
I'm trying to fine tune a GPT2-based model on my data using the run_clm.py example script from HuggingFace.
I have a .json data file that looks like this:
...
{&quot;text&quot;: &quot;some text&quot;}
{&quot;text&quot;: &quot;more text&quot;}
...

I had to change the default behavior of the script that used to concatenate input text, because all my examples are separate demonstrations that should not be concatenated:
def add_labels(example):
    example['labels'] = example['input_ids'].copy()
    return example

with training_args.main_process_first(desc=&quot;grouping texts together&quot;):
    lm_datasets = tokenized_datasets.map(
        add_labels,
        batched=False,
        # batch_size=1,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
        desc=f&quot;Grouping texts in chunks of {block_size}&quot;,
    )

This essentially only adds the appropriate 'labels' field required by CLM.
However since GPT2 has a 1024-sized context-window, the examples should be padded to that length.
I can achieve this by modifying the tokenization procedure like this:
def tokenize_function(examples):
    with CaptureLogger(tok_logger) as cl:
        output = tokenizer(
            examples[text_column_name], padding='max_length') # added: padding='max_length'
        # ...

The training runs correctly.
However, I believe this should not be done by the tokenizer, but by the data collator instead. When I remove padding='max_length' from the tokenizer, I get the following error:
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

And also, above that:
Traceback (most recent call last):
  File &quot;/home/jan/repos/text2task/.venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py&quot;, line 716, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: expected sequence of length 9 at dim 1 (got 33)

During handling of the above exception, another exception occurred:

To fix this, I have created a data collator that should do the padding:
data_collator = DataCollatorWithPadding(tokenizer, padding='max_length')

This is what is passed to the trainer. However, the above error remains.
What's going on?

I managed to fix the error but I'm really unsure about my solution, details below. Will accept a better answer.
This seems to solve it:
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model, padding=True)

Found in the documentation
It seems like DataCollatorWithPadding doesn't pad the labels?
My problem is about generating an output sequence from an input sequence, so I'm guessing that using DataCollatorForSeq2Seq is what I actually want to do. However, my data does not have separate input and target columns, but a single text column (that contains a string input =&gt; target). I'm not really that this collator is intended to be used for GPT2...
