title: Using function objects to speed up the training
body: <pre><code>from datasets import load_dataset #Huggingface
  from transformers import BertTokenizer #Huggingface:
  
  def tokenized_dataset(dataset):
      &quot;&quot;&quot; Method that tokenizes each document in the train, test and validation dataset
  
      Args:
          dataset (DatasetDict): dataset that will be tokenized (train, test, validation)
      
      Returns:
          dict: dataset once tokenized
      &quot;&quot;&quot;
  
      tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
  
      encode = lambda document: tokenizer(document, return_tensors='pt', padding=True, truncation=True)
      train_articles = [encode(document) for document in dataset[&quot;train&quot;][&quot;article&quot;]]
      test_articles = [encode(document) for document in dataset[&quot;test&quot;][&quot;article&quot;]]
      val_articles = [encode(document) for document in dataset[&quot;val&quot;][&quot;article&quot;]]
      train_abstracts = [encode(document) for document in dataset[&quot;train&quot;][&quot;abstract&quot;]]
      test_abstracts = [encode(document) for document in dataset[&quot;test&quot;][&quot;abstract&quot;]]
      val_abstracts = [encode(document) for document in dataset[&quot;val&quot;][&quot;abstract&quot;]]
  
      return {&quot;train&quot;: (train_articles, train_abstracts),
              &quot;test&quot;: (test_articles, test_abstracts),
              &quot;val&quot;: (val_articles, val_abstracts)}
  
  if __name__ == &quot;__main__&quot;:
      dataset = load_data(&quot;./train/&quot;, &quot;./test/&quot;, &quot;./val/&quot;, &quot;./.cache_dir&quot;)
      tokenized_data = tokenized_dataset(dataset)
  </code></pre>
  <p>I would like to modify the function <code>tokenized_dataset</code> because it creates a very heavy dictionary. The dataset created by that function will be reused for a ML training. However, dragging that dictionary during the training will slow down the training a lot.</p>
  <p>Be aware that <code>document</code> is similar to</p>
  <pre><code>[['eleven politicians from 7 parties made comments in letter to a newspaper .',
    &quot;said dpp alison saunders had ` damaged public confidence ' in justice .&quot;,
    'ms saunders ruled lord janner unfit to stand trial over child abuse claims .',
    'the cps has pursued at least 19 suspected paedophiles with dementia .'],
   ['an increasing number of surveys claim to reveal what makes us happiest .',
    'but are these generic lists really of any use to us ?',
    'janet street-porter makes her own list - of things making her unhappy !'],
   [&quot;author of ` into the wild ' spoke to five rape victims in missoula , montana .&quot;,
    &quot;` missoula : rape and the justice system in a college town ' was released april 21 .&quot;,
    &quot;three of five victims profiled in the book sat down with abc 's nightline wednesday night .&quot;,
    'kelsey belnap , allison huguet and hillary mclaughlin said they had been raped by university of montana football '
    'players .',
    &quot;huguet and mclaughlin 's attacker , beau donaldson , pleaded guilty to rape in 2012 and was sentenced to 10 years .&quot;,
    'belnap claimed four players gang-raped her in 2010 , but prosecutors never charged them citing lack of probable '
    'cause .',
    'mr krakauer wrote book after realizing close friend was a rape victim .'],
   ['tesco announced a record annual loss of £ 6.38 billion yesterday .',
    'drop in sales , one-off costs and pensions blamed for financial loss .',
    'supermarket giant now under pressure to close 200 stores nationwide .',
    'here , retail industry veterans , plus mail writers , identify what went wrong .'],
   ...,
   ['snp leader said alex salmond did not field questions over his family .',
    &quot;said she was not ` moaning ' but also attacked criticism of women 's looks .&quot;,
    'she made the remarks in latest programme profiling the main party leaders .',
    'ms sturgeon also revealed her tv habits and recent image makeover .',
    'she said she relaxed by eating steak and chips on a saturday night .']]
  </code></pre>
  <p>So in the dictionary, the keys are just strings, but the values are all list of lists of strings. Instead of having value = list of list of strings, I think it is better to create sort of a list of object function instead of having list of lists of strings. It will make the dictionary much lighter. How can I do that?</p>
  <p><strong>EDIT</strong></p>
  <p>For me, there is a difference between copying a list of lists of strings and copying a list of objects. Copying an object will simply copy the reference while copying a list of strings will copy everything. So it is much faster to copy the reference instead. This is the point of this question.</p>