<p dir="auto">Hi,</p>
<p dir="auto">Is there a way to run batches with QuestionAnsweringPipeline rather than just one example?</p>
<p dir="auto">Thanks.</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="desktop_computer" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f5a5.png">🖥</g-emoji> Benchmarking <code class="notranslate">transformers</code></h1>
<h2 dir="auto">Benchmark</h2>
<p dir="auto">I'm trying to benchmark QA model with <code class="notranslate">bert-large-uncased-whole-word-masking-finetuned-squad</code>. But it seems it is extremely slow e.g. 3-4 sec for 1 question with 2 contexts.</p>
<p dir="auto">I feel there is something I'm missing in pipeline.</p>
<h2 dir="auto">Sample Code:</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def answer(self, contexts:List[str], question:str, **kwargs):
   ## tokenizer, model, pipeline all are cached in actual implementation 
  ## via [reify](https://docs.pylonsproject.org/projects/pyramid/en/latest/api/decorator.html) 
  ## so model loading is not a problem.
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_len=500)
    model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking- 
        finetuned-squad')
    pipeline('question-answering', model=model, tokenizer=tokenizer)
    pipeline_input = []
    for c in contexts:
        pipeline_input.append({
            'question' : question,
            'context' : c
        })
    answers = pipeline(pipeline_input)"><pre class="notranslate"><code class="notranslate">def answer(self, contexts:List[str], question:str, **kwargs):
   ## tokenizer, model, pipeline all are cached in actual implementation 
  ## via [reify](https://docs.pylonsproject.org/projects/pyramid/en/latest/api/decorator.html) 
  ## so model loading is not a problem.
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_len=500)
    model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking- 
        finetuned-squad')
    pipeline('question-answering', model=model, tokenizer=tokenizer)
    pipeline_input = []
    for c in contexts:
        pipeline_input.append({
            'question' : question,
            'context' : c
        })
    answers = pipeline(pipeline_input)
</code></pre></div>
<h2 dir="auto">Set-up</h2>
<p dir="auto">CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz<br>
memory: 16GB</p>