title: Batch size during training vs batch size during evaluation
body: <p>I am confused about the difference between batch size during training versus batch size during evaluation. I am trying to measure how batch size influences the inference time(speed of prediction) of different NLP models after they have  been trained using the Huggingface Trainer API. The code I used is below</p>
  <pre><code>def print_summary(result):
      print(f&quot;Time: {result.metrics['test_runtime']:.2f}&quot;)
      print(f&quot;Samples/second: {result.metrics['test_samples_per_second']:.2f}&quot;)
      print(f&quot;Test Loss: {result.metrics['test_loss']:.2f}&quot;)
      print(f&quot;Test Accuracy: {result.metrics['test_accuracy']:.2f}&quot;)
     
      print(result.metrics)
  </code></pre>
  <pre><code>logging.set_verbosity_error()
  training_args = TrainingArguments( output_dir=&quot;cat&quot;, per_device_train_batch_size=1,per_device_eval_batch_size=1,do_train = False,
          do_predict = True)
  model = AutoModelForSequenceClassification.from_pretrained(&quot;distilrob&quot;)
  trainer = Trainer(model=model, args=training_args,compute_metrics=compute_metrics)
  result=trainer.predict(tokenized_datasets_test_distilrob)
  print_summary(result)
  </code></pre>
  <p>My initial idea was that the <code>per_device_train_batch_size</code> would have no effect, since the training is actually done and I am now looking at the performance of the trained models, but its value does actually change the inference time. Why would it affect the inference time after the training is complete and what would be the correct set up if I  want to  measure the inference time (speed of prediction) as (academically) precisely as possible?</p>