title: Getting KeyErrors when training Hugging Face Transformer
body: <p>I am generally following this tutorial (<a href="https://huggingface.co/docs/transformers/training#:%7E:text=%F0%9F%A4%97%20Transformers%20provides%20access%20to,an%20incredibly%20powerful%20training%20technique." rel="nofollow noreferrer">https://huggingface.co/docs/transformers/training#:~:text=%F0%9F%A4%97%20Transformers%20provides%20access%20to,an%20incredibly%20powerful%20training%20technique.</a>) to implement fine-tuning on a pretrained transformer. The main difference is I am using my own custom dataset that is being sourced from a JSON file that has a document's text and the label it should belong to. To be able to do this I needed to create my own class which is based off of the Dataset class from pytorch. This is what that class looks like:</p>
  <pre><code>class PDFsDataset(torch.utils.data.Dataset):
      def __init__(self, encodings, labels):
          self.encodings = encodings
          self.labels = labels
  
      def __getitem__(self, idx):
          
          print(&quot;\n\n\n\nindex&quot;,idx)
          item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
          item['labels'] = torch.tensor(self.labels[idx])
          return item
  
      def __len__(self):
          return len(self.labels)
  
  </code></pre>
  <p>I am getting an error when training the transformer that says</p>
  <pre><code>Traceback (most recent call last):
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\pandas\core\indexes\base.py&quot;, line 3621, in get_loc
      return self._engine.get_loc(casted_key)
    File &quot;pandas\_libs\index.pyx&quot;, line 136, in pandas._libs.index.IndexEngine.get_loc
    File &quot;pandas\_libs\index.pyx&quot;, line 163, in pandas._libs.index.IndexEngine.get_loc
    File &quot;pandas\_libs\hashtable_class_helper.pxi&quot;, line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
    File &quot;pandas\_libs\hashtable_class_helper.pxi&quot;, line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
  KeyError: 19
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File &quot;c:\Users\e417922\Downloads\enwiki-20220601-pages-meta-history1.xml-p1p857\HF_Transformer.py&quot;, line 147, in &lt;module&gt;
      transformer.train_transformer()
    File &quot;c:\Users\e417922\Downloads\enwiki-20220601-pages-meta-history1.xml-p1p857\HF_Transformer.py&quot;, line 135, in train_transformer
      trainer.train()
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\transformers\trainer.py&quot;, line 1409, in train
      return inner_training_loop(
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\transformers\trainer.py&quot;, line 1625, in _inner_training_loop
      for step, inputs in enumerate(epoch_iterator):
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\torch\utils\data\dataloader.py&quot;, line 530, in __next__
      data = self._next_data()
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\torch\utils\data\dataloader.py&quot;, line 570, in _next_data
      data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\torch\utils\data\_utils\fetch.py&quot;, line 49, in fetch
      data = [self.dataset[idx] for idx in possibly_batched_index]
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\torch\utils\data\_utils\fetch.py&quot;, line 49, in &lt;listcomp&gt;
      data = [self.dataset[idx] for idx in possibly_batched_index]
    File &quot;c:\Users\e417922\Downloads\enwiki-20220601-pages-meta-history1.xml-p1p857\HF_Transformer.py&quot;, line 42, in __getitem__
      for key in self.encodings[idx]:
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\pandas\core\series.py&quot;, line 958, in __getitem__
      return self._get_value(key)
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\pandas\core\series.py&quot;, line 1069, in _get_value
      loc = self.index.get_loc(label)
    File &quot;C:\Users\e417922\AppData\Roaming\Python\Python39\site-packages\pandas\core\indexes\base.py&quot;, line 3623, in get_loc
      raise KeyError(key) from err
  KeyError: 19
  </code></pre>
  <p>The KeyError that it fails on changes each time I run it. I'm a beginner with Transformers and HuggingFace so I have no clue what's causing this problem.</p>
  <p>Edit:
  Sample Input is a JSON File where elements would look like this:
  {
  &quot;text_clean&quot;: [
  &quot;article with a few hundred words&quot;,
  another article with a lot of words&quot;,
  &quot;yet another article&quot;
  ],
  &quot;most_similar_label&quot;:[
  &quot;Quantum&quot;
  &quot;Artificial intelligence&quot;
  &quot;Materials&quot;
  ]
  }</p>
  <p>Full Code:</p>
  <pre><code>import tkinter as tk
  from tkinter import filedialog
  import json
  import pandas as pd
  from transformers import AutoTokenizer
  from transformers import AutoModelForSequenceClassification
  from transformers import TrainingArguments
  from transformers import TrainingArguments, Trainer
  import numpy as np
  from datasets import load_metric
  from sklearn.model_selection import train_test_split
  import torch
  
  class PDFsDataset(torch.utils.data.Dataset):
      def __init__(self, encodings, labels):
          self.encodings = encodings
          self.labels = labels
  
      def __getitem__(self, idx):
          
          print(&quot;\n\n\n\nindex&quot;,idx)
          item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
          item['labels'] = torch.tensor(self.labels[idx])
          return item
  
      def __len__(self):
          return len(self.labels)
  
  class HFTransformer:
      def __init__ (self):
          pass
  
      def import_from_json(self):
          #Prompts user to select json file
          root = tk.Tk()
          root.withdraw()
          self.json_file_path = filedialog.askopenfile().name
          #opens json file and loads data
          with open(self.json_file_path, &quot;r&quot;) as json_file:
                  try:
                      json_load = json.load(json_file)
                  except:
                      raise ValueError(&quot;No PDFs to convert to JSON&quot;)
          self.pdfs = json_load
          #converts json file data to dataframe for easier manipulation
          self.pdfs = pd.DataFrame.from_dict(self.pdfs)
  
          for index in range(len(self.pdfs[&quot;new_tags&quot;])):
              if self.pdfs[&quot;new_tags&quot;][index] == &quot;&quot;:
                  self.pdfs[&quot;new_tags&quot;][index] = self.pdfs[&quot;most_similar_label&quot;][index]
  
          self.pdfs[&quot;labels&quot;] = self.pdfs[&quot;new_tags&quot;].apply(lambda val: self.change_tag_to_num(val))
          # for label in self.data[&quot;labels&quot;]:
       
      def change_tag_to_num(self, value):
          if value == &quot;Quantum&quot;:
              return 0
          elif value == &quot;Artificial intelligence&quot;:
              return 1
          elif value == &quot;Materials&quot;:
              return 2
          elif value == &quot;Energy&quot;:
              return 3
          elif value == &quot;Defense&quot;:
              return 4
          elif value == &quot;Satellite&quot;:
              return 5
          elif value == &quot;Other&quot;:
              return 6
  
      def tokenize_dataset(self):
          tokenizer = AutoTokenizer.from_pretrained(&quot;bert-base-cased&quot;)
  
          X_train, X_test, y_train, y_test = train_test_split(self.pdfs[&quot;text_clean&quot;], self.pdfs[&quot;labels&quot;],test_size=0.2)
  
          train_encodings = X_train.apply(lambda string: tokenizer(string, truncation=True, padding=True,max_length=10))
          test_encodings = X_test.apply(lambda string: tokenizer(string, truncation=True, padding=True,max_length=10))
      
          
          self.train_dataset = PDFsDataset(train_encodings, y_train)
      
          data_to_add = {&quot;input_ids&quot;: [], &quot;token_type_ids&quot;: [], &quot;attention_mask&quot;: []}
  
          for i in self.train_dataset.encodings:
              data_to_add[&quot;input_ids&quot;].append(i[&quot;input_ids&quot;])
              data_to_add[&quot;token_type_ids&quot;].append(i[&quot;token_type_ids&quot;])
              data_to_add[&quot;attention_mask&quot;].append(i[&quot;attention_mask&quot;])
  
          self.train_dataset.encodings = data_to_add
  
          self.eval_dataset = PDFsDataset(test_encodings,y_test)
          data_to_add = {&quot;input_ids&quot;: [], &quot;token_type_ids&quot;: [], &quot;attention_mask&quot;: []}
  
          for i in self.eval_dataset.encodings:
              data_to_add[&quot;input_ids&quot;].append(i[&quot;input_ids&quot;])
              data_to_add[&quot;token_type_ids&quot;].append(i[&quot;token_type_ids&quot;])
              data_to_add[&quot;attention_mask&quot;].append(i[&quot;attention_mask&quot;])
          
          self.eval_dataset.encodings = data_to_add
  
      def train_transformer(self):
          model = AutoModelForSequenceClassification.from_pretrained(&quot;bert-base-cased&quot;, num_labels=7)
          training_args = TrainingArguments(output_dir=&quot;test_trainer&quot;)
          self.metric = load_metric(&quot;accuracy&quot;)
          training_args = TrainingArguments(output_dir=&quot;test_trainer&quot;, evaluation_strategy=&quot;epoch&quot;)
      
  
          trainer = Trainer(
              model=model,
              args=training_args,
              train_dataset=self.train_dataset,
              eval_dataset=self.eval_dataset,
              compute_metrics=self.compute_metrics
          )
          trainer.train()
      def compute_metrics(self, eval_pred):
          logits, labels = eval_pred
          predictions = np.argmax(logits, axis=-1)
          return self.metric.compute(predictions=predictions, references=labels)
  
  
  if __name__  == &quot;__main__&quot;:
      transformer = HFTransformer()
      transformer.import_from_json()
      transformer.tokenize_dataset()
      transformer.train_transformer()
  </code></pre>