title: Speed up a program
body: <p>Here is the implementation of <code>encode</code></p>
  <pre><code>from transformers import BertTokenizer
  
  def encode(self, document): 
      tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
      return tokenizer(document, 
                      max_length=50,
                      return_tensors='pt',
                      padding=True,
                      truncation=True)
  </code></pre>
  <p>When I ran</p>
  <pre><code>start = time.process_time()
  train_articles = [self.encode(document) for document in dataset[&quot;train&quot;][&quot;article&quot;][:100]]
  print(&quot;Time:&quot;, time.process_time() - start)
  </code></pre>
  <p>the output was <code>Time: 34.859681388</code>. Be aware that each element in <code>dataset[&quot;train&quot;][&quot;article&quot;]</code> is a list of approx. 47 sentences with in mean 15 words per sentence. The process is quite long to process only 100 documents. I am very new in programming. Is there a way to speed up the process? I have to run <code>self.encode</code> on 100000 documents, so I have to find a way to speed up the process.</p>