title: HuggingFace T5 transformer model - how to prep a custom dataset for fine-tuning?
body: <p>I am trying to use the HuggingFace library to fine-tune the T5 transformer model using a custom dataset. HF provide an example of <a href="https://huggingface.co/transformers/master/custom_datasets.html" rel="nofollow noreferrer">fine-tuning with custom data</a> but this is for distilbert model, not the T5 model I want to use. From their example it says I need to implement <strong>len</strong> and <strong>getitem</strong> methods in my dataset subclass, but there doesn't seem to be much documentation about what to change when using t5 instead of distilbert. Here is the tokenizer code followed by my attempt at changing <strong>getitem</strong></p>
  <p><a href="https://i.stack.imgur.com/sqiNI.png" rel="nofollow noreferrer"><strong>getitem</strong> method code</a></p>
  <p>and the resulting error from trainer.train() which says &quot; KeyError: 'labels' &quot;</p>
  <p><a href="https://i.stack.imgur.com/TBMid.png" rel="nofollow noreferrer">trainer.train() error message</a></p>
  <p>I have seen the following <a href="https://stackoverflow.com/questions/67691530/key-error-while-fine-tunning-t5-for-summarization-with-huggingface">discussion</a> which seems to relate to this problem, but the answer offered still produces an error in trainer.train() which I can also post if useful.</p>
  <p>Using the original example code from the &quot;fine-tuning with custom data&quot; then the dataset class is:</p>
  <p><a href="https://i.stack.imgur.com/t9ibj.png" rel="nofollow noreferrer">original code from hf distilbert example applied to T5</a></p>
  <p>but then the error with the trainer changes:</p>
  <p><a href="https://i.stack.imgur.com/PrGLK.png" rel="nofollow noreferrer">trainer error using hf distilbert example applied to T5</a></p>
  <p>which is what originally got me looking around for solutions. So using &quot;fine-tuning with custom data&quot; doesn't seem to be as simple as changing the model and the tokenizer (and the input/output data you are training on) when switching from say distilbert to a text to text model like T5. distilbert doesn't have any output text to train on, so I would have thought (but what do I know?) it would be different to T5 but I can't find documentation on how? At the bottom of this <a href="https://stackoverflow.com/questions/67691530/key-error-while-fine-tunning-t5-for-summarization-with-huggingface">question</a> seems to point to a direction to follow but once again I don't know (much!)</p>
  <p>I think I may have solved the problem (at least the trainer runs and completes). The distilbert model doesn't have output text, it has flags that are provided to the dataset class as a list of integers. The T5 model has output text, so you assign the output encodings and rely upon DataCollatorForSeq2Seq() to prepare the data/featurs that the T5 model expects. See changes (for T5) with commented out HF code (for distilbert) below:</p>
  <p><a href="https://i.stack.imgur.com/JSl2f.png" rel="nofollow noreferrer">Changes for T5 - commented out distilbert code</a></p>
  <p>Raised an issue to HuggingFace and they advised that the fine-tuning with custom datasets example on their website was out of date and that I needed to work off their maintained <a href="https://github.com/huggingface/transformers/tree/master/examples" rel="nofollow noreferrer">examples</a>.</p>