title: huggingface transformer models: KeyError: 'input_ids' message at beginning of BERT model training
body: <p>Using the Huggingface transformer library, I am encountering a bug in the final step when I go to fine tune the BERT language model for masked language modelling task.  I am looking to fine tune it for an in domain finance corpus that model has not been trained on yet.  However, I get the following error message when I call the model to train:  KeyError: 'input_ids'.  Provided below are the steps and code I took.  Any insights are appreciated!</p>
  <p>First, I created a dataset object from a pandas dataframe that was in turn created from a csv file with 1 column of many rows of text:</p>
  <pre><code>unlabelled_dataset = Dataset.from_pandas(unlabelled)    
  </code></pre>
  <p>Second, I then tokenized the dataset with the following code:</p>
  <pre><code>tokenizerBERT = BertTokenizerFast.from_pretrained('bert-base-uncased')  #BERT model tokenization &amp; check
  tokenizerBERT(unlabelled_dataset['paragraphs'], padding=True, truncation=True)
  tokenizerBERT.save_pretrained('tokenizers/pytorch/labelled/BERT/')
  </code></pre>
  <p>Third, I created a data collator as instructed:</p>
  <pre><code>data_collator_BERT = DataCollatorForLanguageModeling(tokenizer=tokenizerBERT, mlm=True, mlm_probability=0.15)
  </code></pre>
  <p>Next, I select my model from_pretrained to get the benefits of transfer learning:</p>
  <pre><code>model_BERT = BertForMaskedLM.from_pretrained(&quot;bert-base-uncased&quot;)
  </code></pre>
  <p>Next, I passed my training args to the transformer trainer and initialize:</p>
  <pre><code>training_args_BERT = TrainingArguments(
      output_dir=&quot;./BERT&quot;,
      num_train_epochs=10,
      evaluation_strategy='steps',
      warmup_steps=10000,
      weight_decay=0.01,
      per_gpu_train_batch_size=64,    
  )
  
  trainer_BERT = Trainer(
      model=model_BERT,
      args=training_args_BERT,
      data_collator=data_collator_BERT,
      train_dataset=unlabelled_dataset,
  )
  </code></pre>
  <p>Last, I call the model to train and get the error KeyError: 'input_ids'</p>
  <pre><code>trainer_BERT.train()
  </code></pre>
  <p>Any insights on how to debug this approach to training the model?</p>
  <p>Provided below is the exact error message received:</p>
  <pre><code>---------------------------------------------------------------------------
  KeyError                                  Traceback (most recent call last)
  &lt;ipython-input-9-83b7063dea0b&gt; in &lt;module&gt;
  ----&gt; 1 trainer_BERT.train()
        2 trainer.save_model(&quot;./models/royalBERT&quot;)
  
  ~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path, trial)
      755             self.control = self.callback_handler.on_epoch_begin(self.args, self.state, self.control)
      756 
  --&gt; 757             for step, inputs in enumerate(epoch_iterator):
      758 
      759                 # Skip past any already trained steps if resuming training
  
  ~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
      361 
      362     def __next__(self):
  --&gt; 363         data = self._next_data()
      364         self._num_yielded += 1
      365         if self._dataset_kind == _DatasetKind.Iterable and \
  
  ~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
      401     def _next_data(self):
      402         index = self._next_index()  # may raise StopIteration
  --&gt; 403         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
      404         if self._pin_memory:
      405             data = _utils.pin_memory.pin_memory(data)
  
  ~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
       45         else:
       46             data = self.dataset[possibly_batched_index]
  ---&gt; 47         return self.collate_fn(data)
  
  ~/anaconda3/lib/python3.7/site-packages/transformers/data/data_collator.py in __call__(self, examples)
      193     ) -&gt; Dict[str, torch.Tensor]:
      194         if isinstance(examples[0], (dict, BatchEncoding)):
  --&gt; 195             examples = [e[&quot;input_ids&quot;] for e in examples]
      196         batch = self._tensorize_batch(examples)
      197         if self.mlm:
  
  ~/anaconda3/lib/python3.7/site-packages/transformers/data/data_collator.py in &lt;listcomp&gt;(.0)
      193     ) -&gt; Dict[str, torch.Tensor]:
      194         if isinstance(examples[0], (dict, BatchEncoding)):
  --&gt; 195             examples = [e[&quot;input_ids&quot;] for e in examples]
      196         batch = self._tensorize_batch(examples)
      197         if self.mlm:
  
  KeyError: 'input_ids'
  </code></pre>