title: Pytorch NLP model doesn’t use GPU when making inference
body: <p>I have a NLP model trained on Pytorch to be run in Jetson Xavier. I installed Jetson stats to monitor usage of CPU and GPU. When I run the Python script, only CPU cores work on-load, GPU bar does not increase. I have searched on Google about that with keywords of &quot; How to check if pytorch is using the GPU?&quot; and checked results on stackoverflow.com etc. According to their advices to someone else facing similar issue, cuda is available and there is cuda device in my Jetson Xavier. However, I don’t understand why GPU bar does not change, CPU core bars go to the ends.</p>
  <p>I don’t want to use CPU, it takes so long to compute. In my opinion, it uses CPU, not GPU. How can I be sure and if it uses CPU, how can I change it to GPU?</p>
  <p><strong>Note:</strong> Model is taken from huggingface transformers library. I have tried to use cuda() method on the model. (model.cuda()) In this scenario, GPU is used but I can not get an output from model and raises exception.</p>
  <p>Here is the code:</p>
  <pre><code>from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
  import torch
  
  BERT_DIR = &quot;savasy/bert-base-turkish-squad&quot;    
  
  tokenizer = AutoTokenizer.from_pretrained(BERT_DIR)
  model = AutoModelForQuestionAnswering.from_pretrained(BERT_DIR)
  nlp=pipeline(&quot;question-answering&quot;, model=model, tokenizer=tokenizer)
  
  
  def infer(question,corpus):
      try:
          ans = nlp(question=question, context=corpus)
          return ans[&quot;answer&quot;], ans[&quot;score&quot;]
      except:
          ans = None
          pass
  
      return None, 0
  </code></pre>