---
abstract: 'OpenNMT is an open-source toolkit for neural machine translation (NMT). The system prioritizes efficiency, modularity, and extensibility with the goal of supporting NMT research into model architectures, feature representations, and source modalities, while maintaining competitive performance and reasonable training requirements. The toolkit consists of modeling and translation support, as well as detailed pedagogical documentation about the underlying techniques. OpenNMT has been used in several production MT systems, modified for numerous research papers, and is implemented across several deep learning frameworks.'
author:
- 
bibliography:
- 'mail.bib'
title: '**OpenNMT: Neural Machine Translation Toolkit**'
---

Introduction
============

Neural machine translation (NMT) is a new methodology for machine translation that has led to remarkable improvements, particularly in terms of human evaluation, compared to rule-based and statistical machine translation (SMT) systems [@wu2016google; @systran]. Originally developed using pure sequence-to-sequence models [@sutskever14sequence; @Cho2014] and improved upon using attention-based variants [@Bahdanau2015; @Luong2015], NMT has now become a widely-applied technique for machine translation, as well as an effective approach for other related NLP tasks such as dialogue, parsing, and summarization.

As NMT approaches are standardized, it becomes more important for the machine translation and NLP community to develop open implementations for researchers to benchmark against, learn from, and extend upon. Just as the SMT community benefited greatly from toolkits like Moses [@koehn2007moses] for phrase-based SMT and CDec [@dyer2010cdec] for syntax-based SMT, NMT toolkits can provide a foundation to build upon. A toolkit should aim to provide a shared framework for developing and comparing open-source systems, while at the same time being efficient and accurate enough to be used in production contexts.

\[t\]

With these goals in mind, in this work we present an open-source toolkit for developing neural machine translation systems, known as *OpenNMT* (<http://opennmt.net>). Since its launch in December 2016, OpenNMT has become a collection of implementations targeting both academia and industry. The system is designed to be simple to use and easy to extend, while maintaining efficiency and state-of-the-art accuracy. In addition to providing code for the core translation tasks, OpenNMT was designed with two aims: (a) prioritize training and test efficiency, (b) maintain model modularity and readability hence research extensibility.

During this time, many other stellar open-source NMT implementations have also been released, including *GroundHog*, *Blocks*, *Nematus*, *tensorflow-seq2seq*, *GNMT*, *fair-seq*, *Tensor2Tensor*, *Sockeye*, *Neural Monkey*, *lamtram*, *XNMT*, *SGNMT*, and *Marian*. These projects mostly implement variants of the same underlying systems, and differ in their prioritization of features. The open-source community around this area is flourishing, and is providing the NLP community a useful variety of open-source NMT frameworks. In the ongoing development of OpenNMT, we aim to build upon the strengths of those systems, while supporting a framework with high-accuracy translation, multiple options and clear documentation.

This engineering report describes how the system targets our design goals. We begin by briefly surveying the background for NMT, and then describing the high-level implementation details. We end by showing benchmarks of the system in terms of accuracy, speed, and memory usage for several translation and natural language generation tasks.

Background
==========

NMT has now been extensively described in many excellent tutorials (see for instance <https://sites.google.com/site/acl16nmt/home>). We give only a condensed overview.

NMT takes a conditional language modeling view of translation by modeling the probability of a target sentence $w_{1:T}$ given a source sentence $x_{1:S}$ as $p(w_{1:T}| x) = \prod_{1}^T p(w_t| w_{1:t-1}, x; \theta)$ where the distribution is parameterized with $\theta$. This distribution is estimated using an attention-based encoder-decoder architecture [@Bahdanau2015]. A source encoder recurrent neural network (RNN) maps each source word to a word vector, and processes these to a sequence of hidden vectors $\mathbf{h}_1, \ldots, \mathbf{h}_S$. The target decoder combines an RNN hidden representation of previously generated words ($w_1, ... w_{t-1}$) with source hidden vectors to predict scores for each possible next word. A softmax layer is then used to produce a next-word distribution $ p(w_t| w_{1:t-1}, x; \theta)$. The source hidden vectors influence the distribution through an attention pooling layer that weights each source word relative to its expected contribution to the target prediction. The complete model is trained end-to-end to minimize the negative log-likelihood of the training corpus. An unfolded network diagram is shown in Figure \[fig:rnn\].

In practice, there are also many other important aspects that improve the effectiveness of the base model. Here we briefly mention four areas: (a) It is important to use a gated RNN such as an LSTM [@hochreiter1997long] or GRU [@chung2014empirical] which help the model learn long-term features. (b) Translation requires relatively large, stacked RNNs, which consist of several vertical layers (2-16) of RNNs at each time step [@sutskever14sequence]. (c) Input feeding, where the previous attention vector is fed back into the input as well as the predicted word, has been shown to be quite helpful for machine translation [@Luong2015]. (d) Test-time decoding is done through *beam search* where multiple hypothesis target predictions are considered at each time step. Implementing these correctly can be difficult, which motivates their inclusion in a NMT framework.

Implementation
==============

OpenNMT is a community of projects supporting easy adoption neural machine translation. At the heart of the project are libraries for training, using, and deploying neural machine translation models. The system was based originally on *seq2seq-attn*, which was rewritten for ease of efficiency, readability, and generalizability. The project supports vanilla NMT models along with support for attention, gating, stacking, input feeding, regularization, copy models, beam search and all other options necessary for state-of-the-art performance.

OpenNMT has currently three main implementations. All of them are actively maintained:

-   *OpenNMT-lua* The original project developed in Torch 7. Full-featured, optimized, and stable code ready for quick experiments and production.

-   *OpenNMT-py* An OpenNMT-lua clone using PyTorch. Initially created by by Adam Lerer and the Facebook AI research team as an example, this implementation is easy to extend and particularly suited for research.

-   *OpenNMT-tf* An implementation following the style of TensorFlow. This is a newer project focusing on large scale experiments and high performance model serving using the latest TensorFlow features.

OpenNMT is developed completely in the open on GitHub at (<http://github.com/opennmt>) and is MIT licensed. The initial release has primarily contributions from SYSTRAN Paris, the Harvard NLP group and Facebook AI research. Since official beta release, the project (OpenNMT-lua, OpenNMT-py and OpenNMT-tf) has been starred by over 2500 users in total, and there have been over 100 outside contributors. The project has an active forum for community feedback with over five hundred posts in the last two months. There is also a live demonstration available of the system in use (Figure \[fig:live\]).

One often overlooked benefit of NMT compared to SMT is its relative compactness. OpenNMT-lua including preprocessing and model variants is roughly 16K lines of code, the PyTorch version is less than 4K lines and Tensorflow version has around 7K lines. For comparison the Moses SMT framework including language modeling is over 100K lines. This makes our system easy to completely understand for newcomers. Each project is fully self-contained depending on minimal number of external libraries and also includes some preprocessing, visualization and analysis tools.

Design Goals
============

![\[fig:codestruct\]Schematic overview of OpenNMT-py code](OpenNMTDiagram.png)

System Efficiency
-----------------

As NMT systems can take from days to weeks to train, training efficiency is a paramount concern. Slightly faster training can make the difference between plausible and impossible experiments.

#### Memory Sharing & Sharding

When training GPU-based NMT models, memory size restrictions are the most common limiter of batch size, and thus directly impact training time. Neural network toolkits, such as Torch, are often designed to trade-off extra memory allocations for speed and declarative simplicity. For OpenNMT, we wanted to have it both ways, and so we implemented an external memory sharing system that exploits the known time-series control flow of NMT systems and aggressively shares the internal buffers between clones. The potential shared buffers are dynamically calculated by exploration of the network graph before starting training. In practical use, aggressive memory reuse provides a saving of 70% of GPU memory with the default model size. For OpenNMT-py, we implemented a sharding mechanism both for data loading to enable training on extremely large datasets that cannot fit into memory, and for back-propagation to reduce memory footprints during training.

#### Multi-GPU

OpenNMT additionally supports multi-GPU training using data parallelism. Each GPU has a replica of the master parameters and processes independent batches during training phase. Two modes are available: synchronous and asynchronous training [@dean2012large]. Experiments with 8 GPUs show a 6$\times$ speed up in per epoch, but a slight loss in training efficiency. When training to similar loss, it gives a 3.5$\times$ total speed-up to training.

#### C/Mobile/GPU Translation

Training NMT systems requires significant code complexity to facilitate fast back-propagation-through-time. At deployment, the system is much less complex, and only requires (i) forwarding values through the network and (ii) running a beam search that is much simplified compared to SMT. OpenNMT includes several different translation deployments specialized for different run-time environments: a batched CPU/GPU implementation for very quickly translating a large set of sentences, a simple single-instance implementation for use on mobile devices, and a specialized C implementation suited for industrial use.

Modularity for Research
-----------------------

A secondary goal was a desire for code readability and extensibility. We targeted this goal by explicitly separating training, optimization and different components of the model, and by including tutorial documentation within the code. A schematic overview of our data structures in OpenNMT-py is shown in Figure \[fig:codestruct\]. We provide users with simple interfaces *preprocess*, *train* and *translate*, which only require source/target files as input, while we provide a highly modularized library for advanced users. Each module in the library is highly customizable and configurable with multiple ready-for-use features. Advanced users can access the modules directly through a library interface to construct and train variant of the standard NMT setup.

  <span> System</span>             <span> BLEU-cased</span>
  ------------------------------ --------------------------
  uedin-nmt-ensemble                                   28.3
  LMU-nmt-reranked-wmt17-en-de                         27.1
  SYSTRAN-single (OpenNMT)                             26.7

  : \[tab:wmt\]Top 3 on English-German *newstest2017* WMT17.

#### Extensible Data, Models, and Search

In addition to plain text, OpenNMT also supports different input types including models with discrete features [@sennrich2016linguistic], models with non-sequential input such as tables, continuous data such as speech signals, and multi-dimensional data such as images. To support these different input modalities the library implements image encoder [@xu2015show; @deng2017image] and audio encoders [@DBLP:journals/corr/ChanJLV15]. OpenNMT implements various attention types including general, dot product, and concatenation [@Luong2015; @britz2017massive]. This also includes recent extensions to these standard modules such as the copy mechanism [@vinyals2015pointer; @gu2016incorporating], which is widely used in summarization and generation applications.

The newer implementations of OpenNMT have also been updated to include support for recent innovations in non-recurrent translation models. In particular recent support has been added for convolution translation [@gehring2017convolutional] and the attention-only transformer network [@vaswani2017attention].

Finally, the translation code allows for user customization. In addition to out-of-vocabulary (OOV) handling [@luong2015b], OpenNMT also allows beam search with various normalizations including length and attention coverage normalization [@wu2016google], and dynamic dictionary support for copy/pointer networks. We also provide an interface for customized hypothesis filtering, enabling beam search under various constraints such as maximum number of OOV's and lexical constraints.

#### Modularity

Due to the deliberate modularity of our code, OpenNMT is readily extensible for novel feature development. As one example, by substituting the attention module, we can implement local attention [@Luong2015], sparse-max attention [@martins2016softmax] and structured attention [@kim2017structured] with minimal change of code. As another example, in order to get feature-based factored neural translation [@sennrich2016linguistic] we simply need to modify the input network to process the feature-based representation, and the output network to produce multiple conditionally independent predictions.

We have seen instances of this use in published research. In addition to machine translation [@levin2017toward; @ha2017effective; @ma2017osu], researchers have employed OpenNMT for parsing [@van2017neural], document summarization [@ling2017coarse], data-to-document [@wiseman2017challenges; @gardent2017webnlg], and transliteration [@ameur2017arabic], to name a few of many applications.

#### Additional Tools

OpenNMT packages several additional tools, including: 1) reversible tokenizer, which can also perform Byte Pair Encoding (BPE) [@DBLP:journals/corr/SennrichHB15]; 2) loading and exporting word embeddings; 3) translation server which enables showcase results remotely; and 4) visualization tools for debugging or understanding, such as beam search visualization, profiler and TensorBoard logging.

Experiments
===========

OpenNMT achieves competitive results against other systems, e.g. in the recent WMT 2017 translation task, it won third place in English-German translation with a single model as shown in Table \[tab:wmt\]. The system is also competitive in speed as shown in Table \[tab:res\]. Here we compare training and test speed to the publicly available *Nematus* system[^1] on English-to-German (EN$\rightarrow$DE) using the WMT2015[^2] dataset.

We have found that OpenNMT's default setting is useful for experiments, but not optimal for large-scale NMT. This has been a cause of poor reported performance in other default comparisons by @britz2017massive and @hieber2017sockeye. We trained models with our best effort to conform to their settings and report our results in Table \[tab:results-updated\], which shows comparable performance with other systems. We suspect that the reported poor performance is due to the fact that our default setting discards sequences of length greater than 50, which is too short for BPE. Moreover, while the reported poor performance was obtained by training with ADAM, we find that training with (the default) SGD with learning rate decay is generally better.

We also compare OpenNMT with the *GNMT* [@wu2016google] model in Table \[tab:results-google\]. @vaswani2017attention have established a new state-of-the-art with the Transformer model. We have also implemented this in our framework, and compare it with *Tensor2Tensor* (T2T) in Table \[tab:results-transformer\]. (These experiments are run on a modified version of WMT 2017, namely News Comm v11 instead of v12, and no Rapid 2016.)

Additionally we have found interest from the community in using OpenNMT for language geneation tasks like sentence document summarization and dialogue response generation, among others. Using OpenNMT, we were able to replicate the sentence summarization results of @chopra2016abstractive, reaching a ROUGE-1 score of 35.51 on the Gigaword data. We have also trained a model on 14 million sentences of the OpenSubtitles data set based on the work @vinyals2015neural, achieving comparable perplexity. Many other models are at <http://opennmt.net/Models-py> and <http://opennmt.net/Models>.

[^1]: <https://github.com/rsennrich/nematus> Comparison with OpenNMT/Nematus github revisions <span>907824</span>/<span>75c6ab1</span>

[^2]: <http://statmt.org/wmt15>

{'source': 'arXiv_out/1805'}