We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
small bug with truncated decoding and tgt feature concat.
I'm running https://github.com/Henry-E/opennmt-gen and getting an error with matrix multiplication.
` File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/blas.py", line 26, in forward
    matrix1, matrix2, out=output)` 
The full stack trace is below

The problem is in a multiplication inside the decoder RNN. It's trying to multiply matrix 1 by matrix 2. At a guess it's something to do with the extra dimensions of the features not being accounted for by the decoder. I'm not super familiar with the equations being used for the features though so it's hard to tell where the model is being initialised incorrectly. 
```
ipdb> matrix1.size()
torch.Size([5, 2400])
ipdb> matrix2.size()
torch.Size([600, 2400])
```

The model
```
NMTModel (
  (encoder): MeanEncoder (
    (embeddings): Embeddings (
      (make_embedding): Sequential (
        (emb_luts): Elementwise (
          (0): Embedding(10818, 600, padding_idx=1)
          (1): Embedding(704, 600, padding_idx=1)
          (2): Embedding(40, 600, padding_idx=1)
          (3): Embedding(4, 600, padding_idx=1)
        )
      )
    )
  )
  (decoder): InputFeedRNNDecoder (
    (embeddings): Embeddings (
      (make_embedding): Sequential (
        (emb_luts): Elementwise (
          (0): Embedding(10818, 600, padding_idx=1)
        )
      )
    )
    (dropout): Dropout (p = 0.3)
    (rnn): StackedLSTM (
      (dropout): Dropout (p = 0.3)
      (layers): ModuleList (
        (0): LSTMCell(1200, 600)
        (1): LSTMCell(600, 600)
      )
    )
    (attn): GlobalAttention (
      (linear_out): Linear (1200 -> 600)
      (sm): Softmax ()
      (tanh): Tanh ()
    )
    (copy_attn): GlobalAttention (
      (linear_out): Linear (1200 -> 600)
      (sm): Softmax ()
      (tanh): Tanh ()
    )
  )
  (generator): CopyGenerator (
    (linear): Linear (600 -> 10818)
    (linear_copy): Linear (600 -> 1)
  )
)
```



Stack trace

```
Traceback (most recent call last):
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/ipdb/__main__.py", line 198, in main
    pdb._runscript(mainpyfile)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/pdb.py", line 1548, in _runscript
    self.run(statement)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 1, in <module>
    from __future__ import division
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 284, in main
    train_model(model, train, valid, fields, optim)
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 144, in train_model
    train_stats = trainer.train(epoch, report_func)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Trainer.py", line 119, in train
    self.model(src, tgt, src_lengths, dec_state)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Models.py", line 421, in forward
    else dec_state)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Models.py", line 175, in forward
    self._run_forward_pass(input, context, state)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Models.py", line 337, in _run_forward_pass
    rnn_output, hidden = self.rnn(emb_t, hidden)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/downloads/OpenNMT-py/onmt/modules/StackedRNN.py", line 24, in forward
    h_1_i, c_1_i = layer(input, (h_0[i], c_0[i]))
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 608, in forward
    self.bias_ih, self.bias_hh,
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 26, in LSTMCell
    hgates = F.linear(hidden[0], w_hh)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 555, in linear
    output = input.matmul(weight.t())
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 560, in matmul
    return torch.matmul(self, other)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/functional.py", line 173, in matmul
    return torch.mm(tensor1, tensor2)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 579, in mm
    return Addmm.apply(output, self, matrix, 0, 1, True)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/blas.py", line 26, in forward
    matrix1, matrix2, out=output)
RuntimeError: size mismatch at /opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCTensorMathBlas.cu:243
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `onmt/Trainer.py`
Content:
```
1 from __future__ import division
2 """
3 This is the loadable seq2seq trainer library that is
4 in charge of training details, loss compute, and statistics.
5 See train.py for a use case of this library.
6 
7 Note!!! To make this a general library, we implement *only*
8 mechanism things here(i.e. what to do), and leave the strategy
9 things to users(i.e. how to do it). Also see train.py(one of the
10 users of this library) for the strategy things we do.
11 """
12 import time
13 import sys
14 import math
15 import torch
16 import torch.nn as nn
17 
18 import onmt
19 import onmt.modules
20 
21 
22 class Statistics(object):
23     """
24     Train/validate loss statistics.
25     """
26     def __init__(self, loss=0, n_words=0, n_correct=0):
27         self.loss = loss
28         self.n_words = n_words
29         self.n_correct = n_correct
30         self.n_src_words = 0
31         self.start_time = time.time()
32 
33     def update(self, stat):
34         self.loss += stat.loss
35         self.n_words += stat.n_words
36         self.n_correct += stat.n_correct
37 
38     def accuracy(self):
39         return 100 * (self.n_correct / self.n_words)
40 
41     def ppl(self):
42         return math.exp(min(self.loss / self.n_words, 100))
43 
44     def elapsed_time(self):
45         return time.time() - self.start_time
46 
47     def output(self, epoch, batch, n_batches, start):
48         t = self.elapsed_time()
49         print(("Epoch %2d, %5d/%5d; acc: %6.2f; ppl: %6.2f; " +
50                "%3.0f src tok/s; %3.0f tgt tok/s; %6.0f s elapsed") %
51               (epoch, batch,  n_batches,
52                self.accuracy(),
53                self.ppl(),
54                self.n_src_words / (t + 1e-5),
55                self.n_words / (t + 1e-5),
56                time.time() - start))
57         sys.stdout.flush()
58 
59     def log(self, prefix, experiment, optim):
60         t = self.elapsed_time()
61         experiment.add_scalar_value(prefix + "_ppl", self.ppl())
62         experiment.add_scalar_value(prefix + "_accuracy", self.accuracy())
63         experiment.add_scalar_value(prefix + "_tgtper",  self.n_words / t)
64         experiment.add_scalar_value(prefix + "_lr", optim.lr)
65 
66 
67 class Trainer(object):
68     def __init__(self, model, train_iter, valid_iter,
69                  train_loss, valid_loss, optim,
70                  trunc_size, shard_size):
71         """
72         Args:
73             model: the seq2seq model.
74             train_iter: the train data iterator.
75             valid_iter: the validate data iterator.
76             train_loss: the train side LossCompute object for computing loss.
77             valid_loss: the valid side LossCompute object for computing loss.
78             optim: the optimizer responsible for lr update.
79             trunc_size: a batch is divided by several truncs of this size.
80             shard_size: compute loss in shards of this size for efficiency.
81         """
82         # Basic attributes.
83         self.model = model
84         self.train_iter = train_iter
85         self.valid_iter = valid_iter
86         self.train_loss = train_loss
87         self.valid_loss = valid_loss
88         self.optim = optim
89         self.trunc_size = trunc_size
90         self.shard_size = shard_size
91 
92         # Set model in training mode.
93         self.model.train()
94 
95     def train(self, epoch, report_func=None):
96         """ Called for each epoch to train. """
97         total_stats = Statistics()
98         report_stats = Statistics()
99 
100         for i, batch in enumerate(self.train_iter):
101             target_size = batch.tgt.size(0)
102             # Truncated BPTT
103             trunc_size = self.trunc_size if self.trunc_size else target_size
104 
105             dec_state = None
106             _, src_lengths = batch.src
107 
108             src = onmt.IO.make_features(batch, 'src')
109             tgt = onmt.IO.make_features(batch, 'tgt')
110             report_stats.n_src_words += src_lengths.sum()
111 
112             for j in range(0, target_size-1, trunc_size):
113                 # 1. Create truncated target.
114                 tgt = tgt[j: j + trunc_size]
115 
116                 # 2. F-prop all but generator.
117                 self.model.zero_grad()
118                 outputs, attns, dec_state = \
119                     self.model(src, tgt, src_lengths, dec_state)
120 
121                 # 3. Compute loss in shards for memory efficiency.
122                 batch_stats = self.train_loss.sharded_compute_loss(
123                         batch, outputs, attns, j,
124                         trunc_size, self.shard_size)
125 
126                 # 4. Update the parameters and statistics.
127                 self.optim.step()
128                 total_stats.update(batch_stats)
129                 report_stats.update(batch_stats)
130 
131                 # If truncated, don't backprop fully.
132                 if dec_state is not None:
133                     dec_state.detach()
134 
135             if report_func is not None:
136                 report_func(epoch, i, len(self.train_iter),
137                             total_stats.start_time, self.optim.lr,
138                             report_stats)
139                 report_stats = Statistics()
140 
141         return total_stats
142 
143     def validate(self):
144         """ Called for each epoch to validate. """
145         # Set model in validating mode.
146         self.model.eval()
147 
148         stats = Statistics()
149 
150         for batch in self.valid_iter:
151             _, src_lengths = batch.src
152             src = onmt.IO.make_features(batch, 'src')
153             tgt = onmt.IO.make_features(batch, 'tgt')
154 
155             # F-prop through the model.
156             outputs, attns, _ = self.model(src, tgt, src_lengths)
157 
158             # Compute loss.
159             gen_state = onmt.Loss.make_gen_state(
160                 outputs, batch, attns, (0, batch.tgt.size(0)))
161             _, batch_stats = self.valid_loss(batch, **gen_state)
162 
163             # Update statistics.
164             stats.update(batch_stats)
165 
166         # Set model back to training mode.
167         self.model.train()
168 
169         return stats
170 
171     def epoch_step(self, ppl, epoch):
172         """ Called for each epoch to update learning rate. """
173         return self.optim.updateLearningRate(ppl, epoch)
174 
175     def drop_checkpoint(self, opt, epoch, fields, valid_stats):
176         """ Called conditionally each epoch to save a snapshot. """
177         real_model = (self.model.module
178                       if isinstance(self.model, nn.DataParallel)
179                       else self.model)
180         real_generator = (real_model.generator.module
181                           if isinstance(real_model.generator, nn.DataParallel)
182                           else real_model.generator)
183 
184         model_state_dict = real_model.state_dict()
185         model_state_dict = {k: v for k, v in model_state_dict.items()
186                             if 'generator' not in k}
187         generator_state_dict = real_generator.state_dict()
188         checkpoint = {
189             'model': model_state_dict,
190             'generator': generator_state_dict,
191             'vocab': onmt.IO.ONMTDataset.save_vocab(fields),
192             'opt': opt,
193             'epoch': epoch,
194             'optim': self.optim
195         }
196         torch.save(checkpoint,
197                    '%s_acc_%.2f_ppl_%.2f_e%d.pt'
198                    % (opt.save_model, valid_stats.accuracy(),
199                       valid_stats.ppl(), epoch))
200
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/onmt/Trainer.py b/onmt/Trainer.py
--- a/onmt/Trainer.py
+++ b/onmt/Trainer.py
@@ -106,12 +106,12 @@
             _, src_lengths = batch.src
 
             src = onmt.IO.make_features(batch, 'src')
-            tgt = onmt.IO.make_features(batch, 'tgt')
+            tgt_outer = onmt.IO.make_features(batch, 'tgt')
             report_stats.n_src_words += src_lengths.sum()
 
             for j in range(0, target_size-1, trunc_size):
                 # 1. Create truncated target.
-                tgt = tgt[j: j + trunc_size]
+                tgt = tgt_outer[j: j + trunc_size]
 
                 # 2. F-prop all but generator.
                 self.model.zero_grad()

{"golden_diff": "diff --git a/onmt/Trainer.py b/onmt/Trainer.py\n--- a/onmt/Trainer.py\n+++ b/onmt/Trainer.py\n@@ -106,12 +106,12 @@\n             _, src_lengths = batch.src\n \n             src = onmt.IO.make_features(batch, 'src')\n-            tgt = onmt.IO.make_features(batch, 'tgt')\n+            tgt_outer = onmt.IO.make_features(batch, 'tgt')\n             report_stats.n_src_words += src_lengths.sum()\n \n             for j in range(0, target_size-1, trunc_size):\n                 # 1. Create truncated target.\n-                tgt = tgt[j: j + trunc_size]\n+                tgt = tgt_outer[j: j + trunc_size]\n \n                 # 2. F-prop all but generator.\n                 self.model.zero_grad()\n", "issue": "small bug with truncated decoding and tgt feature concat.\nI'm running https://github.com/Henry-E/opennmt-gen and getting an error with matrix multiplication.\r\n` File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/blas.py\", line 26, in forward\r\n    matrix1, matrix2, out=output)` \r\nThe full stack trace is below\r\n\r\nThe problem is in a multiplication inside the decoder RNN. It's trying to multiply matrix 1 by matrix 2. At a guess it's something to do with the extra dimensions of the features not being accounted for by the decoder. I'm not super familiar with the equations being used for the features though so it's hard to tell where the model is being initialised incorrectly. \r\n```\r\nipdb> matrix1.size()\r\ntorch.Size([5, 2400])\r\nipdb> matrix2.size()\r\ntorch.Size([600, 2400])\r\n```\r\n\r\nThe model\r\n```\r\nNMTModel (\r\n  (encoder): MeanEncoder (\r\n    (embeddings): Embeddings (\r\n      (make_embedding): Sequential (\r\n        (emb_luts): Elementwise (\r\n          (0): Embedding(10818, 600, padding_idx=1)\r\n          (1): Embedding(704, 600, padding_idx=1)\r\n          (2): Embedding(40, 600, padding_idx=1)\r\n          (3): Embedding(4, 600, padding_idx=1)\r\n        )\r\n      )\r\n    )\r\n  )\r\n  (decoder): InputFeedRNNDecoder (\r\n    (embeddings): Embeddings (\r\n      (make_embedding): Sequential (\r\n        (emb_luts): Elementwise (\r\n          (0): Embedding(10818, 600, padding_idx=1)\r\n        )\r\n      )\r\n    )\r\n    (dropout): Dropout (p = 0.3)\r\n    (rnn): StackedLSTM (\r\n      (dropout): Dropout (p = 0.3)\r\n      (layers): ModuleList (\r\n        (0): LSTMCell(1200, 600)\r\n        (1): LSTMCell(600, 600)\r\n      )\r\n    )\r\n    (attn): GlobalAttention (\r\n      (linear_out): Linear (1200 -> 600)\r\n      (sm): Softmax ()\r\n      (tanh): Tanh ()\r\n    )\r\n    (copy_attn): GlobalAttention (\r\n      (linear_out): Linear (1200 -> 600)\r\n      (sm): Softmax ()\r\n      (tanh): Tanh ()\r\n    )\r\n  )\r\n  (generator): CopyGenerator (\r\n    (linear): Linear (600 -> 10818)\r\n    (linear_copy): Linear (600 -> 1)\r\n  )\r\n)\r\n```\r\n\r\n\r\n\r\nStack trace\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/ipdb/__main__.py\", line 198, in main\r\n    pdb._runscript(mainpyfile)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/pdb.py\", line 1548, in _runscript\r\n    self.run(statement)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/bdb.py\", line 431, in run\r\n    exec(cmd, globals, locals)\r\n  File \"<string>\", line 1, in <module>\r\n  File \"/home/henrye/downloads/OpenNMT-py/train.py\", line 1, in <module>\r\n    from __future__ import division\r\n  File \"/home/henrye/downloads/OpenNMT-py/train.py\", line 284, in main\r\n    train_model(model, train, valid, fields, optim)\r\n  File \"/home/henrye/downloads/OpenNMT-py/train.py\", line 144, in train_model\r\n    train_stats = trainer.train(epoch, report_func)\r\n  File \"/home/henrye/downloads/OpenNMT-py/onmt/Trainer.py\", line 119, in train\r\n    self.model(src, tgt, src_lengths, dec_state)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/henrye/downloads/OpenNMT-py/onmt/Models.py\", line 421, in forward\r\n    else dec_state)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/henrye/downloads/OpenNMT-py/onmt/Models.py\", line 175, in forward\r\n    self._run_forward_pass(input, context, state)\r\n  File \"/home/henrye/downloads/OpenNMT-py/onmt/Models.py\", line 337, in _run_forward_pass\r\n    rnn_output, hidden = self.rnn(emb_t, hidden)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/henrye/downloads/OpenNMT-py/onmt/modules/StackedRNN.py\", line 24, in forward\r\n    h_1_i, c_1_i = layer(input, (h_0[i], c_0[i]))\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/rnn.py\", line 608, in forward\r\n    self.bias_ih, self.bias_hh,\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/_functions/rnn.py\", line 26, in LSTMCell\r\n    hgates = F.linear(hidden[0], w_hh)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py\", line 555, in linear\r\n    output = input.matmul(weight.t())\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py\", line 560, in matmul\r\n    return torch.matmul(self, other)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/functional.py\", line 173, in matmul\r\n    return torch.mm(tensor1, tensor2)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py\", line 579, in mm\r\n    return Addmm.apply(output, self, matrix, 0, 1, True)\r\n  File \"/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/blas.py\", line 26, in forward\r\n    matrix1, matrix2, out=output)\r\nRuntimeError: size mismatch at /opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCTensorMathBlas.cu:243\r\n```\n", "before_files": [{"content": "from __future__ import division\n\"\"\"\nThis is the loadable seq2seq trainer library that is\nin charge of training details, loss compute, and statistics.\nSee train.py for a use case of this library.\n\nNote!!! To make this a general library, we implement *only*\nmechanism things here(i.e. what to do), and leave the strategy\nthings to users(i.e. how to do it). Also see train.py(one of the\nusers of this library) for the strategy things we do.\n\"\"\"\nimport time\nimport sys\nimport math\nimport torch\nimport torch.nn as nn\n\nimport onmt\nimport onmt.modules\n\n\nclass Statistics(object):\n    \"\"\"\n    Train/validate loss statistics.\n    \"\"\"\n    def __init__(self, loss=0, n_words=0, n_correct=0):\n        self.loss = loss\n        self.n_words = n_words\n        self.n_correct = n_correct\n        self.n_src_words = 0\n        self.start_time = time.time()\n\n    def update(self, stat):\n        self.loss += stat.loss\n        self.n_words += stat.n_words\n        self.n_correct += stat.n_correct\n\n    def accuracy(self):\n        return 100 * (self.n_correct / self.n_words)\n\n    def ppl(self):\n        return math.exp(min(self.loss / self.n_words, 100))\n\n    def elapsed_time(self):\n        return time.time() - self.start_time\n\n    def output(self, epoch, batch, n_batches, start):\n        t = self.elapsed_time()\n        print((\"Epoch %2d, %5d/%5d; acc: %6.2f; ppl: %6.2f; \" +\n               \"%3.0f src tok/s; %3.0f tgt tok/s; %6.0f s elapsed\") %\n              (epoch, batch,  n_batches,\n               self.accuracy(),\n               self.ppl(),\n               self.n_src_words / (t + 1e-5),\n               self.n_words / (t + 1e-5),\n               time.time() - start))\n        sys.stdout.flush()\n\n    def log(self, prefix, experiment, optim):\n        t = self.elapsed_time()\n        experiment.add_scalar_value(prefix + \"_ppl\", self.ppl())\n        experiment.add_scalar_value(prefix + \"_accuracy\", self.accuracy())\n        experiment.add_scalar_value(prefix + \"_tgtper\",  self.n_words / t)\n        experiment.add_scalar_value(prefix + \"_lr\", optim.lr)\n\n\nclass Trainer(object):\n    def __init__(self, model, train_iter, valid_iter,\n                 train_loss, valid_loss, optim,\n                 trunc_size, shard_size):\n        \"\"\"\n        Args:\n            model: the seq2seq model.\n            train_iter: the train data iterator.\n            valid_iter: the validate data iterator.\n            train_loss: the train side LossCompute object for computing loss.\n            valid_loss: the valid side LossCompute object for computing loss.\n            optim: the optimizer responsible for lr update.\n            trunc_size: a batch is divided by several truncs of this size.\n            shard_size: compute loss in shards of this size for efficiency.\n        \"\"\"\n        # Basic attributes.\n        self.model = model\n        self.train_iter = train_iter\n        self.valid_iter = valid_iter\n        self.train_loss = train_loss\n        self.valid_loss = valid_loss\n        self.optim = optim\n        self.trunc_size = trunc_size\n        self.shard_size = shard_size\n\n        # Set model in training mode.\n        self.model.train()\n\n    def train(self, epoch, report_func=None):\n        \"\"\" Called for each epoch to train. \"\"\"\n        total_stats = Statistics()\n        report_stats = Statistics()\n\n        for i, batch in enumerate(self.train_iter):\n            target_size = batch.tgt.size(0)\n            # Truncated BPTT\n            trunc_size = self.trunc_size if self.trunc_size else target_size\n\n            dec_state = None\n            _, src_lengths = batch.src\n\n            src = onmt.IO.make_features(batch, 'src')\n            tgt = onmt.IO.make_features(batch, 'tgt')\n            report_stats.n_src_words += src_lengths.sum()\n\n            for j in range(0, target_size-1, trunc_size):\n                # 1. Create truncated target.\n                tgt = tgt[j: j + trunc_size]\n\n                # 2. F-prop all but generator.\n                self.model.zero_grad()\n                outputs, attns, dec_state = \\\n                    self.model(src, tgt, src_lengths, dec_state)\n\n                # 3. Compute loss in shards for memory efficiency.\n                batch_stats = self.train_loss.sharded_compute_loss(\n                        batch, outputs, attns, j,\n                        trunc_size, self.shard_size)\n\n                # 4. Update the parameters and statistics.\n                self.optim.step()\n                total_stats.update(batch_stats)\n                report_stats.update(batch_stats)\n\n                # If truncated, don't backprop fully.\n                if dec_state is not None:\n                    dec_state.detach()\n\n            if report_func is not None:\n                report_func(epoch, i, len(self.train_iter),\n                            total_stats.start_time, self.optim.lr,\n                            report_stats)\n                report_stats = Statistics()\n\n        return total_stats\n\n    def validate(self):\n        \"\"\" Called for each epoch to validate. \"\"\"\n        # Set model in validating mode.\n        self.model.eval()\n\n        stats = Statistics()\n\n        for batch in self.valid_iter:\n            _, src_lengths = batch.src\n            src = onmt.IO.make_features(batch, 'src')\n            tgt = onmt.IO.make_features(batch, 'tgt')\n\n            # F-prop through the model.\n            outputs, attns, _ = self.model(src, tgt, src_lengths)\n\n            # Compute loss.\n            gen_state = onmt.Loss.make_gen_state(\n                outputs, batch, attns, (0, batch.tgt.size(0)))\n            _, batch_stats = self.valid_loss(batch, **gen_state)\n\n            # Update statistics.\n            stats.update(batch_stats)\n\n        # Set model back to training mode.\n        self.model.train()\n\n        return stats\n\n    def epoch_step(self, ppl, epoch):\n        \"\"\" Called for each epoch to update learning rate. \"\"\"\n        return self.optim.updateLearningRate(ppl, epoch)\n\n    def drop_checkpoint(self, opt, epoch, fields, valid_stats):\n        \"\"\" Called conditionally each epoch to save a snapshot. \"\"\"\n        real_model = (self.model.module\n                      if isinstance(self.model, nn.DataParallel)\n                      else self.model)\n        real_generator = (real_model.generator.module\n                          if isinstance(real_model.generator, nn.DataParallel)\n                          else real_model.generator)\n\n        model_state_dict = real_model.state_dict()\n        model_state_dict = {k: v for k, v in model_state_dict.items()\n                            if 'generator' not in k}\n        generator_state_dict = real_generator.state_dict()\n        checkpoint = {\n            'model': model_state_dict,\n            'generator': generator_state_dict,\n            'vocab': onmt.IO.ONMTDataset.save_vocab(fields),\n            'opt': opt,\n            'epoch': epoch,\n            'optim': self.optim\n        }\n        torch.save(checkpoint,\n                   '%s_acc_%.2f_ppl_%.2f_e%d.pt'\n                   % (opt.save_model, valid_stats.accuracy(),\n                      valid_stats.ppl(), epoch))\n", "path": "onmt/Trainer.py"}], "after_files": [{"content": "from __future__ import division\n\"\"\"\nThis is the loadable seq2seq trainer library that is\nin charge of training details, loss compute, and statistics.\nSee train.py for a use case of this library.\n\nNote!!! To make this a general library, we implement *only*\nmechanism things here(i.e. what to do), and leave the strategy\nthings to users(i.e. how to do it). Also see train.py(one of the\nusers of this library) for the strategy things we do.\n\"\"\"\nimport time\nimport sys\nimport math\nimport torch\nimport torch.nn as nn\n\nimport onmt\nimport onmt.modules\n\n\nclass Statistics(object):\n    \"\"\"\n    Train/validate loss statistics.\n    \"\"\"\n    def __init__(self, loss=0, n_words=0, n_correct=0):\n        self.loss = loss\n        self.n_words = n_words\n        self.n_correct = n_correct\n        self.n_src_words = 0\n        self.start_time = time.time()\n\n    def update(self, stat):\n        self.loss += stat.loss\n        self.n_words += stat.n_words\n        self.n_correct += stat.n_correct\n\n    def accuracy(self):\n        return 100 * (self.n_correct / self.n_words)\n\n    def ppl(self):\n        return math.exp(min(self.loss / self.n_words, 100))\n\n    def elapsed_time(self):\n        return time.time() - self.start_time\n\n    def output(self, epoch, batch, n_batches, start):\n        t = self.elapsed_time()\n        print((\"Epoch %2d, %5d/%5d; acc: %6.2f; ppl: %6.2f; \" +\n               \"%3.0f src tok/s; %3.0f tgt tok/s; %6.0f s elapsed\") %\n              (epoch, batch,  n_batches,\n               self.accuracy(),\n               self.ppl(),\n               self.n_src_words / (t + 1e-5),\n               self.n_words / (t + 1e-5),\n               time.time() - start))\n        sys.stdout.flush()\n\n    def log(self, prefix, experiment, optim):\n        t = self.elapsed_time()\n        experiment.add_scalar_value(prefix + \"_ppl\", self.ppl())\n        experiment.add_scalar_value(prefix + \"_accuracy\", self.accuracy())\n        experiment.add_scalar_value(prefix + \"_tgtper\",  self.n_words / t)\n        experiment.add_scalar_value(prefix + \"_lr\", optim.lr)\n\n\nclass Trainer(object):\n    def __init__(self, model, train_iter, valid_iter,\n                 train_loss, valid_loss, optim,\n                 trunc_size, shard_size):\n        \"\"\"\n        Args:\n            model: the seq2seq model.\n            train_iter: the train data iterator.\n            valid_iter: the validate data iterator.\n            train_loss: the train side LossCompute object for computing loss.\n            valid_loss: the valid side LossCompute object for computing loss.\n            optim: the optimizer responsible for lr update.\n            trunc_size: a batch is divided by several truncs of this size.\n            shard_size: compute loss in shards of this size for efficiency.\n        \"\"\"\n        # Basic attributes.\n        self.model = model\n        self.train_iter = train_iter\n        self.valid_iter = valid_iter\n        self.train_loss = train_loss\n        self.valid_loss = valid_loss\n        self.optim = optim\n        self.trunc_size = trunc_size\n        self.shard_size = shard_size\n\n        # Set model in training mode.\n        self.model.train()\n\n    def train(self, epoch, report_func=None):\n        \"\"\" Called for each epoch to train. \"\"\"\n        total_stats = Statistics()\n        report_stats = Statistics()\n\n        for i, batch in enumerate(self.train_iter):\n            target_size = batch.tgt.size(0)\n            # Truncated BPTT\n            trunc_size = self.trunc_size if self.trunc_size else target_size\n\n            dec_state = None\n            _, src_lengths = batch.src\n\n            src = onmt.IO.make_features(batch, 'src')\n            tgt_outer = onmt.IO.make_features(batch, 'tgt')\n            report_stats.n_src_words += src_lengths.sum()\n\n            for j in range(0, target_size-1, trunc_size):\n                # 1. Create truncated target.\n                tgt = tgt_outer[j: j + trunc_size]\n\n                # 2. F-prop all but generator.\n                self.model.zero_grad()\n                outputs, attns, dec_state = \\\n                    self.model(src, tgt, src_lengths, dec_state)\n\n                # 3. Compute loss in shards for memory efficiency.\n                batch_stats = self.train_loss.sharded_compute_loss(\n                        batch, outputs, attns, j,\n                        trunc_size, self.shard_size)\n\n                # 4. Update the parameters and statistics.\n                self.optim.step()\n                total_stats.update(batch_stats)\n                report_stats.update(batch_stats)\n\n                # If truncated, don't backprop fully.\n                if dec_state is not None:\n                    dec_state.detach()\n\n            if report_func is not None:\n                report_func(epoch, i, len(self.train_iter),\n                            total_stats.start_time, self.optim.lr,\n                            report_stats)\n                report_stats = Statistics()\n\n        return total_stats\n\n    def validate(self):\n        \"\"\" Called for each epoch to validate. \"\"\"\n        # Set model in validating mode.\n        self.model.eval()\n\n        stats = Statistics()\n\n        for batch in self.valid_iter:\n            _, src_lengths = batch.src\n            src = onmt.IO.make_features(batch, 'src')\n            tgt = onmt.IO.make_features(batch, 'tgt')\n\n            # F-prop through the model.\n            outputs, attns, _ = self.model(src, tgt, src_lengths)\n\n            # Compute loss.\n            gen_state = onmt.Loss.make_gen_state(\n                outputs, batch, attns, (0, batch.tgt.size(0)))\n            _, batch_stats = self.valid_loss(batch, **gen_state)\n\n            # Update statistics.\n            stats.update(batch_stats)\n\n        # Set model back to training mode.\n        self.model.train()\n\n        return stats\n\n    def epoch_step(self, ppl, epoch):\n        \"\"\" Called for each epoch to update learning rate. \"\"\"\n        return self.optim.updateLearningRate(ppl, epoch)\n\n    def drop_checkpoint(self, opt, epoch, fields, valid_stats):\n        \"\"\" Called conditionally each epoch to save a snapshot. \"\"\"\n        real_model = (self.model.module\n                      if isinstance(self.model, nn.DataParallel)\n                      else self.model)\n        real_generator = (real_model.generator.module\n                          if isinstance(real_model.generator, nn.DataParallel)\n                          else real_model.generator)\n\n        model_state_dict = real_model.state_dict()\n        model_state_dict = {k: v for k, v in model_state_dict.items()\n                            if 'generator' not in k}\n        generator_state_dict = real_generator.state_dict()\n        checkpoint = {\n            'model': model_state_dict,\n            'generator': generator_state_dict,\n            'vocab': onmt.IO.ONMTDataset.save_vocab(fields),\n            'opt': opt,\n            'epoch': epoch,\n            'optim': self.optim\n        }\n        torch.save(checkpoint,\n                   '%s_acc_%.2f_ppl_%.2f_e%d.pt'\n                   % (opt.save_model, valid_stats.accuracy(),\n                      valid_stats.ppl(), epoch))\n", "path": "onmt/Trainer.py"}]}