title: Seq2seq pytorch Inference slow
body: <p>I tried the seq2seq pytorch implementation available here <a href="http://github.com/MaximumEntropy/Seq2Seq-PyTorch" rel="nofollow noreferrer">seq2seq</a> . After profiling the evaluation(evaluate.py) code, the piece of code taking longer time was the <a href="http://github.com/MaximumEntropy/Seq2Seq-PyTorch/blob/master/evaluate.py#L74" rel="nofollow noreferrer">decode_minibatch</a> method</p>
  
  <pre><code>def decode_minibatch(
      config,
      model,
      input_lines_src,
      input_lines_trg,
      output_lines_trg_gold
  ):
      """Decode a minibatch."""
      for i in xrange(config['data']['max_trg_length']):
  
          decoder_logit = model(input_lines_src, input_lines_trg)
          word_probs = model.decode(decoder_logit)
          decoder_argmax = word_probs.data.cpu().numpy().argmax(axis=-1)
          next_preds = Variable(
              torch.from_numpy(decoder_argmax[:, -1])
          ).cuda()
  
          input_lines_trg = torch.cat(
              (input_lines_trg, next_preds.unsqueeze(1)),
              1
          )
  
  return input_lines_trg
  </code></pre>
  
  <p>Trained the model on GPU and loaded the model in CPU mode to make inference. But unfortunately, every sentence seems to take ~10sec. Is slow prediction expected on pytorch? </p>
  
  <p>Any fixes, suggestions to speed up would be much appreciated. Thanks.</p>