Auto-Encoding Variational Neural Machine Translation
Bryan Eikema Wilker Aziz


Fast and accurate functionalization of opaque conductive samples with single nano particles

    Oliver Benson^1,2
    December 30, 2023
=============================================================================================




We present a deep generative model of bilingual sentence pairs for machine translation. The model generates source and target sentences jointly from a shared latent representation and is parameterised by neural networks. We perform efficient training using amortised variational inference and reparameterised gradients. Additionally, we discuss the statistical implications of joint modelling and propose an efficient approximation to maximum a posteriori decoding for fast test-time predictions. We demonstrate the effectiveness of our model in three machine translation scenarios: in-domain training, mixed-domain training, and learning from a mix of gold-standard and synthetic data. Our experiments show consistently that our joint formulation outperforms conditional modelling (i.e. standard neural machine translation) in all such scenarios. 







§ INTRODUCTION




Neural machine translation (NMT) systems <cit.> require vast amounts of  labelled data, i.e. bilingual sentence pairs, to be trained effectively. 
Oftentimes, the data we use to train these systems are a byproduct of mixing different sources of data. 
For example, labelled data are sometimes obtained by putting together corpora from different domains <cit.>. 
Even for a single domain, parallel data often result from the combination of documents independently translated from different languages by different people or agencies, possibly following different guidelines. 
When resources are scarce, it is not uncommon to mix in some synthetic data, e.g. bilingual data artificially obtained by having a model translate target monolingual data to the source language <cit.>.

Translation direction, original language, and quality of translation are some of the many factors that we typically choose not to control for (due to lack of information or simply for convenience).[Also note that this list is by no means exhaustive. For example,  <cit.> show influence of factors such as personal traits and demographics in translation.  Another clear case is presented by <cit.>, who combine parallel resources for multiple languages to train a single encoder-decoder architecture.]
All those arguably contribute to making our labelled data a mixture of samples from various data distributions.





Regular NMT systems do not explicitly account for latent factors of variation, 

instead, 
given a source sentence, NMT models a single conditional distribution over target sentences as a fully supervised problem.




In this work, 
we introduce a deep generative model 
that generates source and target sentences jointly from a shared latent representation. 
The model has the potential to use the latent representation to capture global aspects of the observations, such as some of the latent factors of variation just discussed.

The result is a model that accommodates members of a more complex class of marginal distributions. 

Due to the presence of latent variables, this model requires posterior inference, in particular, we employ the framework of amortised variational inference <cit.>. 
Additionally, we propose an efficient approximation to maximum a posteriori (MAP) decoding for fast test-time predictions.








  
Contributions We introduce a deep generative model for NMT (<ref>) and discuss  theoretical advantages of joint modelling over conditional modelling (<ref>). 
We also derive an efficient approximation to MAP decoding that requires only a single forward pass through the network for prediction (<ref>).
Finally, we show in <ref> that our proposed model improves translation performance in at least three practical scenarios: i) in-domain training on little data, where test data are expected to follow the training data distribution closely; ii) mixed-domain training, where we train a single model but test independently on each domain; and iii) learning from large noisy synthetic data.





§ NEURAL MACHINE TRANSLATION


In machine translation our observations are pairs of random sequences, a source sentence x = ⟨ x_1, …, x_m ⟩ and a target sentence y = ⟨ y_1, …, y_n ⟩, whose lengths m and n we denote by |x| and |y|, respectively. 
In NMT, the likelihood of the target given the source 


    P(|, θ)    = ∏_j=1^(y_j|f_θ(, ))

factorises without Markov assumptions <cit.>.
We have a fixed parameterised function f_θ, i.e. a neural network architecture, compute categorical parameters for varying inputs, namely, the source sentence and target prefix (denoted ). 

Given a dataset 𝒟 of i.i.d. observations, the parameters θ of the model 
are point-estimated to attain a local maximum of the log-likelihood function, ℒ(θ|𝒟) = ∑_(, ) ∈𝒟log P(y|x, θ),





via stochastic gradient-based optimisation <cit.>. 









  
Predictions For a trained model, predictions are performed by searching for the target sentence  that maximises the conditional P(|), or equivalently its logarithm, with a greedy algorithm 

    _  P(|, θ) ≈_ log P(|, θ)

such as beam-search <cit.>, possibly aided by a manually tuned length penalty. 
This decision rule is often referred to as MAP decoding <cit.>.



§ AUTO-ENCODING VARIATIONAL NMT




To account for a latent space where global features of observations can be captured, we introduce a random sentence embedding z ∈ℝ^d and model the joint distribution over observations as a marginal of 
p(z, , |θ).[We use uppercase P(·) for probability mass functions and lowercase p(·) for probability density functions.] 
 

That is, (, ) ∈𝒟 is assumed to be sampled from the distribution

    P(, |θ)    = ∫ p(z) P(, |z, θ) z   .

where we impose a standard Gaussian prior on the latent variable, i.e. Z ∼𝒩(0, I), and assume X ⊥ Y | Z. 
That is, given a sentence embedding z, 





we first generate the source conditioned on z, 

    P(|z, θ) 
    =   ∏_i=1^(x_i|g_θ(z, ))  ,

then generate the target conditioned on  and z, 

    P(|, z, θ) 
    =   ∏_j=1^(y_j|f_θ(z, , ))  .

Note that the source sentence is generated without Markov assumptions by drawing one word at a time from a categorical distribution parameterised by a recurrent neural network g_θ. The target sentence is generated similarly by drawing target words in context from a categorical distribution parameterised by a sequence-to-sequence architecture f_θ. 
This essentially combines a neural language model <cit.> and a neural translation model (<ref>), each extended to condition on an additional stochastic input, namely, z.




 §.§ Statistical considerations
 

Modelling the conditional directly, as in standard NMT, corresponds to the statistical assumption that the distribution over source sentences can provide no information about the distribution over target sentences given a source. 
That is, conditional NMT assumes independence of β determining P(y|x, β) and α determining P(x|α). 
Scenarios where this assumption is unlikely to hold are common: where  is noisy (e.g. synthetic or crowdsourced), poor quality  should be assigned low probability P(|α) which in turn should inform the conditional. Implications of this assumption extend to parameter estimation:  updates to the conditional are not sensitive to how exotic  is.  

Let us be more explicit about how we parameterise our model by identifying 3 sets of parameters θ = {θ_emb-x, θ_LM, θ_TM}, where θ_emb-x parameterises an embedding layer for the source language. 
The embedding layer is shared  between the two model components

    P(x, y|z, θ) = 
    
    P(x|z, θ_emb-x, θ_LM_α) P(y|x, z, θ_emb-x, θ_TM_β)

and it is then clear by inspection that α∩β = {z, θ_emb-x}. In words, 
we break the independence assumption in two ways, namely, by having the two distributions share parameters and by having them depend on a shared latent sentence representation z.
Note that while the embedding layer is deterministic and global to all sentence pairs in the training data, the latent representation is stochastic and local to each sentence pair.

Now let us turn to considerations about latent variable modelling. 

Consider a model P(|θ_emb-x, θ_LM)P(|, θ_emb-x, θ_TM) of the joint distribution over observations that does not employ latent variables.
This alternative, which we discuss further in experiments, models each component directly, whereas our proposed model (<ref>) requires marginalisation of latent embeddings z. 
Marginalisation turns our directed graphical model into an undirected one inducing further structure in the marginal. 
See Appendix <ref>, and Figure <ref> in particular, for an extended discussion.




 §.§ Parameter estimation
 
The marginal in Equation (<ref>) is clearly intractable, thus precluding maximum likelihood estimation.  
Instead, we resort to variational inference <cit.>  and introduce a variational approximation q(z|, , λ)
to the intractable posterior p(z|, , θ). 
We let the approximate posterior be a diagonal Gaussian 

    Z|λ, ,    ∼𝒩(𝐮, (𝐬⊙𝐬)) 
    𝐮   = μ_λ(, ) 
    𝐬   = σ_λ(, )

and predict its parameters (i.e. 𝐮∈ℝ^d, 𝐬∈ℝ^d_>0) with neural networks whose parameters we denote by λ. This makes the model an instance of a variational auto-encoder <cit.>. See Figure <ref> in Appendix <ref> for a graphical depiction of the generative and inference models.


We can then jointly estimate the parameters of both models (generative  θ and inference λ) by maximising the ELBO <cit.>, a lowerbound on the marginal log-likelihood, 







    log P(, |θ) ≥ℰ(θ, λ|, ) =
    𝔼_ϵ∼𝒩(0, I)[ log P(, |z=𝐮 + ϵ⊙𝐬, θ) ] 
     
        -(𝒩(z|𝐮, (𝐬⊙𝐬)) || 𝒩(z|0, I))  ,

where we have expressed the expectation with respect to a fixed distribution—a reparameterisation available to location-scale families such as the Gaussian <cit.>.  
Due to this reparameterisation, we can compute a Monte Carlo estimate of the gradient of the first term via back-propagation <cit.>.
The  term, on the other hand, is available in closed form <cit.>.





 §.§ Predictions
 

In a latent variable model, MAP decoding (<ref>) requires searching for  that maximises the marginal  P(|, θ) ∝ P(, |θ), or equivalently its logarithm. 
In addition to approximating exact search with a greedy algorithm, other approximations are necessary in order to achieve fast prediction. 
First, rather than searching through the true marginal, we search through the evidence lowerbound. 
Second, we  replace the approximate posterior q(z|, ) by an auxiliary distribution r(z|). 
As we are searching through the space of target sentences, not conditioning on  circumvents combinatorial explosion and allows us to drop terms that depend on  alone (<ref>).
Finally, instead of approximating the expectation via MC sampling, we condition on the expected latent representation and search greedily (<ref>).





    _ log P(|) 
       ≈_ 𝔼_r(z|)[log P(|z, )]  
       ≈_ log P(|𝔼_r(z|)[z], )


Together, these approximations enable prediction with a single call to an  solver, in our case a standard greedy search algorithm, which leads to prediction times that are very close to that of the conditional model. 
This strategy, and (<ref>) in particular,  suggests that a good auxiliary distribution r(z|) should approximate q(z|, ) closely.


We parameterise this prediction model using a neural network and investigate different options to estimate its parameters.
As a first option, we restrict the approximate posterior to conditioning on  alone, i.e. we approach posterior inference with q_λ(z|) rather than q_λ(z|, ), and thus,  we can use r(z|) = q_λ(z|) for prediction.[Note that this does not stand in contrast to our motivation for joint modelling, as we still tie source and target through z in the generative model, but it does limit the context available for posterior inference.] 



As a second option, we make r_ϕ(z|) a diagonal Gaussian 
and estimate parameters ϕ to make r_ϕ(z|)  close to the approximate posterior q_λ(z|, ) as measured by (r_ϕ, q_λ).
For as long as (r_ϕ, q_λ) ∈ℝ_≥ 0 for every choice of ϕ and λ, we can estimate ϕ jointly with θ and λ by maximising a modified 

    log P(, |θ) ≥ℰ(θ, λ|, ) - (r_ϕ, q_λ)

which is loosened by the gap between r_ϕ and q_λ.




In experiments we investigate a few options for (r_ϕ, q_λ), all available in closed form for Gaussians, such as (r_ϕ || q_λ), (q_λ||r_ϕ), as well as  

the Jensen-Shannon () divergence. 


Note that r_ϕ is used only for prediction as a decoding heuristic and as such need not be stochastic. We can, for example, design r_ϕ() to be a point estimate of the posterior mean and optimise

    ℰ(θ, λ|, ) - r_ϕ() - 𝔼_q_λ(z|, )[z]_2^2

which remains a lowerbound on log-likelihood.










 §.§ Computational Complexity (I don't like this title)

In order to come to the joint formulation for neural machine translation that we propose we added additional parameters to the model in the form of a source language model, an inference network for z, and a network parameterizing the auxiliary distribution r_ϕ. However, note that most of these can trivially be parallelized on a GPU as we do not add much sequential computation to the translation model. Whereas P(|, z) does depend on z, we can in practice run the inference network simultaneously with the translation model encoder. After that, the translation model decoder and source language model can also run in parallel.



§ EXPERIMENTS



We investigate two translation tasks, namely, WMT's  translation of news <cit.> and IWSLT's translation of transcripts of TED talks <cit.>, 
and concentrate on translations for German (De) and English (En) in either direction.
In this section we aim to investigate scenarios where we expect observations to be representative of various data distributions. 
As a sanity check, we start where training conditions can be considered in-domain with respect to test conditions. Though note that this does not preclude the potential for appreciable variability in observations as various other latent factors still likely play a role (see <ref>). 
We then mix datasets from these two remarkably different translation tasks and investigate whether  performance can be improved across tasks with a single model. 
Finally, we investigate the case where we learn from synthetic data in addition to gold-standard data. 
For this investigation we derive synthetic data from observations that are close to the domain of the test set in an attempt to avoid further confounders.




  
Data 
For bilingual data we use News Commentary (NC) v12 <cit.> and IWSLT 2014 <cit.>, where we assume NC to be representative of the test domain of the WMT News task. 

The datasets consist of 255,591 training sentences and 153,326 training sentences respectively. 
In experiments with synthetic data, we subsample 10^6  sentences from the News Crawl 2016 articles <cit.> for either German or English depending on the target language. 
For the WMT task, we concatenate  and  for validation/development (5,172 sentence pairs) and report test results on  (2,999 sentence pairs).
For IWSLT, we use the split proposed by <cit.> who separated 6,969 training instances for validation/development and reported test results on a concatenation of ,  and  (6,750 sentence pairs).




  
Pre-processing We tokenized and truecased all data using standard scripts from the Moses toolkit <cit.>, and removed sentences longer than 50 tokens. For computational efficiency and to avoid problems with closed vocabularies, we segment the data using BPE <cit.> with 32,000 merge operations independently for each language. 
For training the truecaser and the BPEs we used a concatenation of all the available bilingual and monolingual data for German and all bilingual data for English. 





  
Systems We develop all of our models on top of Tensorflow NMT <cit.>.
Our baseline system is a standard implementation of conditional NMT (Cond) <cit.>.
To illustrate the importance of latent variable modelling, we also include in the comparison a simpler attempt at Joint modelling where we do not induce a shared latent space. Instead, the model is trained in a fully-supervised manner to maximise what is essentially a combination of two nearly independent objectives, 

    ℒ(θ|𝒟) = ∑_(, ) ∈𝒟∑_i=1^log P(x_i|, θ_emb-x, θ_LM) 
    + ∑_j=1^log P(y_j|, , θ_emb-x, θ_TM)  ,

namely, a language model and a conditional translation model. 

Note that the two components of the model share very little, i.e. an embedding layer for the source language.
Finally, we aim at investigating the effectiveness of our auto-encoding variational NMT (AEVNMT).[Code available from <github.com/Roxot/AEVNMT>.] 
Appendix <ref> contains a detailed description of the architectures that parameterise our systems.[In comparison to Cond, AEVNMT requires additional components: a source language model, an inference network, and possibly a prediction network. However, this does not add much sequential computation: the inference network can run in parallel with the source encoder, and the source language model runs in parallel with the target decoder.]





  
Hyperparameters Our recurrent cells are 256-dimensional GRU units <cit.>.  
We train on batches of 64 sentence pairs with Adam <cit.>, learning rate 3× 10^-4, 
for at least T updates. We then perform convergence checks every 500 batches and stop after 20 checks without any improvement measured by BLEU <cit.>. 
For in-domain training we set T=140,000, and for mixed-domain training, as well as training with synthetic data, we set T=280,000. For decoding we use a beam width of 10 and a length penalty of 1.0. 
We investigate the use of dropout <cit.> for the conditional baseline with rates from 10% to 60% in increments of 10%. 
Best validation performance on WMT required a rate of 40% for En-De and 50% for De-En, while on IWSLT it required 50% for either translation direction.

To spare resources, we also use these rates for training the simple Joint model. 








  
Avoiding collapsing to prior Many have noticed that VAEs whose observation models are parameterised by strong generators, such as recurrent neural networks, learn to ignore the latent representation <cit.>. In such cases, the approximate posterior “collapses” to the prior, and where one has a fixed prior, such as our standard Gaussian, this means that the posterior becomes independent of the data, which is obviously not desirable. <cit.> proposed two techniques to counter this effect, namely, “ annealing”, and target word dropout. 
 annealing consists in incorporating the  term of Equation (<ref>) into the objective gradually, thus allowing the posterior to move away from the prior more freely at early stages of training. After a number of annealing steps, the  term is incorporated in full and training continues with the actual ELBO. 
In our search we considered annealing for 20,000 to 80,000 training steps.  
Word dropout consists in randomly masking words in observed target prefixes at a given rate. 
The idea is to harm the potential of the decoder to capitalise on correlations internal to the structure of the observation in the hope that it will rely more on the latent representation instead. We considered rates from 20% to 40% in increments of 10%. 


Table <ref> shows the configurations that achieve best validation results on En-De.
To spare resources, we reuse these hyperparameters for De-En experiments.


With these settings, we attain a non-negligible validation  (see, last row of Table <ref>), which indicates that the approximate posterior is different from the prior at the end of training. 

















  
ELBO variants We investigate the effect of conditioning on target observations for posterior inference during training against a simpler variant that conditions on the source alone. 




Table <ref> suggests that conditioning on  is sufficient and thus we opt to continue with this simpler version.
Do note that when we use both observations for posterior inference, i.e. q_λ(z|, ), and thus train an approximation r_ϕ for prediction, we have additional parameters to estimate (e.g. due to the need to encode  for q_λ and  for r_ϕ), thus it may be the case that for these variants to show their potential we need larger data and/or prolonged training.











 §.§ Results
 





In this section we report test results in terms of BLEU <cit.> and BEER <cit.>, but in Appendix <ref> we additionally report METEOR <cit.> and TER <cit.>.

We de-truecase and de-tokenize our system's predictions and compute BLEU scores using SacreBLEU <cit.>.[Version string: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.2.12] 
For BEER, METEOR and TER, we tokenize the results and test sets using the same tokenizer as used by SacreBLEU. We make use of BEER 2.0, and for METEOR and TER use MultEval <cit.>.
In Appendix <ref> we report validation results, in this case in terms of BLEU alone as that is what we used for model selection. 
Finally, to give an indication of the degree to which results are sensitive to initial conditions (e.g. random initialisation of parameters), and to avoid possibly misleading signifiance testing, we report the average and standard deviation of 5 independently trained models. 
To spare resources we do not report multiple runs for Joint, but our experience is that its performance varies similarly to that of the conditional baseline.







We start with the case where we can reasonably assume training data to be in-domain with respect to test data. 
Table <ref> shows in-domain training performance. 
First, we remark that our conditional baseline for the IWSLT14 task (IWSLT training) is very close to an external baseline trained on the same data <cit.>.[<cit.> report 27.56 on the same test set for De-En, though note that they train on words rather than BPEs and use a different implementation of BLEU.]  
The results on IWSLT show benefits from joint modelling and in particular from learning a shared latent space. For the WMT16 task (NC training), BLEU shows a similar trend, namely, joint modelling with a shared latent space (AEVNMT) outperforms both conditional modelling and the simple joint model. 



We now consider the scenario where we know for a fact that observations come from two different data distributions, which we realise by training our models on a concatenation of IWSLT and NC.

In this case, we perform model selection once on the concatenation of both development sets and evaluate the same model on each domain separately. 

We can see in Table <ref> 
that conditional modelling is never preferred,  Joint performs reasonably well, especially for De-En, and that in every comparison our AEVNMT outperforms the conditional baseline both in terms of BLEU and BEER.





Another common scenario where two very distinct data distributions are mixed is when we capitalise on the abundance of monolingual data and train on a concatenation of gold-standard bilingual data (we use NC) and synthetic bilingual data derived from target monolingual corpora via back-translation <cit.> (we use News Crawl). In such a scenario the latent variable might be able to inform the translation model of the amount of noise present in the source sentence.


Table <ref> shows results for both baselines and AEVNMT. First, note that synthetic data greatly improves the conditional baseline, in particular translating into English. 
Once again AEVNMT consistently outperforms conditional modelling and joint modelling without latent variables.







By mixing different sources of data we are trying to diagnose whether the generative model we propose is  robust to unknown and diverse sources of variation mixed together in one training set (e.g. NC + IWSLT or gold-standard + synthetic data). 
However, note that a point we are certainly not trying to make is that the model has been designed to perform domain adaptation.


Nonetheless, in Appendix <ref> we try to shed light on what happens when we use the model to translate genres it has never seen.
On a dataset covering various unseen genres, we observe that both Cond and AEVNMT perform considerably worse showing that without taking domain adaptation seriously both models are inadequate. In terms of BLEU, differences range from -0.3 to 0.8 (En-De) and 0.3 to 0.7 (De-En) and are mostly in favour of AEVNMT (17/20 comparisons).





  
Remarks It is intuitive to expect latent variable modelling to be most useful in settings containing high variability in the data, i.e. mixed-domain and synthetic data settings, though in our experiments AEVNMT shows larger improvements in the in-domain setting. We speculate two reasons for this: i) it is conceivable that variation in the mixed-domain and synthetic data settings are too large to be well accounted by a diagonal Gaussian; and ii) the benefits of latent variable modelling may diminish as the amount of available data grows.






  






 §.§ Probing latent space


To investigate what information the latent space encodes we explore the idea of training simple linear probes or diagnostic classifiers <cit.>. 
With simple Bayesian logistic regression we have managed to predict from Z ∼ q(z|) domain indicators (i.e. newswire vs transcripts) and gold-standard vs synthetic data at performance above 90% accuracy on development set. 
However, a similar performance is achieved from the deterministic average state of the bidirectional encoder of the conditional baseline. 
We have also been able to predict from Z ∼ q(z|) the level of noise in back-translated data measured on the development set at the sentence level by an automatic metric, i.e. METEOR, with performance above what can be done with random features. 
Though again, the performance is not much better than what can be done with a conditional baseline. 
Still, it is worth highlighting that these aspects are rather coarse, and it is possible that the performance gains we report in <ref> are due to far more nuanced variations in the data. 
At this point, however, we do not have a good qualitative assessment of this conjecture. 





§ RELATED WORK





  
Joint modelling

In similar work, <cit.> propose a joint generative model whose probabilistic formulation is essentially identical to ours. 
Besides some small differences in architecture, our work differs in two regards: motivation and strategy for predictions. 
Their goal is to jointly learn from multiple language pairs by sharing a single polyglot architecture <cit.>. 

Their strategy for prediction is based on a form of stochastic hill-climbing, where they sample an initial z from the standard Gaussian prior and decode via beam search in order to obtain a draft translation = _ P(|z, ).This translation is then iteratively refined by encoding the pair ⟨, ⟩, re-sampling z, though this time from q(z|, ), and re-decoding with beam search. 
Unlike our approach, this requires multiple calls to the inference network and to beam search. Moreover, the inference model, which is trained on gold-standard observations, is used on noisy target sentences.

<cit.> interpret back-translation as a single iteration of a wake-sleep algorithm <cit.> for a joint  model of bitext P(, |θ) = P(y|x, θ) P_⋆().


They sample directly from the data distribution P_⋆() and learn two NMT models, a generative P(|, θ) and an auxiliary model Q(|, ϕ), each trained on a separate objective. 





<cit.> propose a joint model of bitext trained to incorporate the back-translation heuristic as a trainable component in a  formulation similar to that of <cit.>.

In both cases, joint modelling is done without a shared latent space and without a source language model. 



  
Multi-task learning An alternative to joint learning is to turn to multi-task learning and explore parameter sharing across models trained on different, though related, data with different objectives. 
For example, <cit.> incorporate both source and target monolingual data by multi-tasking with a non-differentiable auto-encoding objective. They jointly train a source-to-target and target-to-source system that act as encoder and decoder respectively.
<cit.> combine a source language model objective with a source-to-target conditional NMT objective and shared the source encoder in a multi-task learning fashion.














  
Variational LMs and NMT
<cit.> first proposed to augment a neural language model with a prior over latent space. Our source component is an instance of their model. More recently, <cit.> proposed to use a hyperspherical uniform prior rather than a Gaussian and showed the former leads to better representations.
<cit.> proposed the first VAE for NMT. They augment the conditional with a Gaussian sentence embedding 
and model observations as draws from the marginal P(y|x, θ) = ∫ p(z|x, θ) P(y|x, z, θ)  z. 
Their formulation is a conditional deep generative model <cit.> that does not model the source side of the data, where, rather than a fixed standard Gaussian, the latent model is itself parameterised and depends on the data. 
<cit.> extend the model of  <cit.> with a Markov chain of latent variables, one per timestep, allowing the model to capture greater variability.  










  
Latent domains In the context of statistical MT, <cit.> estimate a joint distribution over sentence pairs while marginalising discrete latent domain indicators. 
Their model factorises over word alignments and is not used directly for translation, but rather to improve word and phrase alignments, or to perform data selection <cit.>, prior to training.  
There is a vast literature on domain adaptation for statistical machine translation <cit.>, as well as for NMT <cit.>, but a full characterisation of this exciting field is beyond the scope of this paper.







§ DISCUSSION AND FUTURE WORK



We have presented a joint generative model of translation data that generates both observations conditioned on a shared latent  representation.
Our formulation leads to questions such as why joint learning? and why latent variable modelling? to which we give an answer based on statistical facts about conditional modelling and marginalisation as well as empirical evidence of improved performance. Our model shows moderate but consistent improvements across various settings and over multiple independent runs. 

In future work, we shall investigate datasets annotated with demographics and personal traits in an attempt to assess how far we can go in capturing fine grained variation.
Though note that if such factors of variation vary widely in distribution, it may be naïve to expect we can model them well with a simple Gaussian prior. If that turns out to be the case, we will investigate mixing Gaussian components <cit.> and/or employing a hierarchical prior <cit.>. 






§ ACKNOWLEDGEMENTS


[4]l0.26



    < g r a p h i c s >


This project has received funding from the Dutch Organization for Scientific Research VICI Grant No 277-89-002 and from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825299 (GoURMET). We also thank Philip Schulz, Khalil Sima'an, and Joost Bastings for comments and helpful discussions.
A Titan Xp card used for this research was donated by the NVIDIA Corporation. 





acl_natbib







§ ARCHITECTURES


Here we describe parameterisation of the different models presented in <ref>. Rather than completely specifying standard blocks, we use the notation (inputs; parameters), where we give an indication of the relevant parameter set. This makes it easier to visually track  which model a component belongs to.



 §.§ Source Language Model


The source language model consists of a sequence of categorical draws for i=1, …,

    X_i|z,    ∼(g_θ(z, ))

parameterised by a single-layer recurrent neural network using GRU units:


    𝐟_i    = (x_i; θ_emb-x)  
    𝐡_0    = tanh((z; θ_init-lm)) 
    𝐡_i    = (𝐡_i-1, 𝐟_i-1; θ_gru-lm) 
    
        g_θ(z, )    = ((𝐡_i; θ_out-x))  .


We initialise the GRU cell with a transformation (<ref>) of the stochastic encoding z. 

For the simple joint model baseline we initialise the GRU with a vector of zeros as there is no stochastic encoding we can condition on in that case.



 §.§ Translation Model

The translation model consists of a sequence of categorical draws for j = 1, …,

    Y_j|z, , ∼(f_θ(z, , ))

parameterised by an architecture that roughly follows <cit.>. 
The encoder is a bidirectional GRU encoder (<ref>) that shares source embeddings with the language model (<ref>) and is initialised with its own projection of the latent representation put through a tanh activation. The decoder, also initialised with its own projection of the latent representation (<ref>), is a single-layer recurrent neural network with GRU units (<ref>). 
At any timestep the decoder is a function of the previous state, previous output word embedding, and a context vector. 
This context vector (<ref>) is a weighted average of the bidirectional source encodings, of which the weights are computed by a Bahdanau-style attention mechanism. 
The output of the GRU decoder is projected to the target vocabulary size and mapped to the simplex using a  activation (<ref>) to obtain the categorical parameters:


    𝐬_0    = tanh((z; θ_init-enc))
    𝐬_1^m    = (𝐟_1^m, 𝐬_0; θ_bigru-x)  
    𝐞_j    = (y_j; θ_emb-y) 
    𝐭_0    = tanh((z; θ_init-dec)) 
    𝐜_j    = (𝐬_1^m, 𝐭_j-1; θ_bahd) 
    𝐭_j    = (𝐭_j-1, [𝐜_j, 𝐞_j-1]; θ_gru-dec)   ,


and 

    f_θ(z, , ) = (([𝐭_j, 𝐞_j-1, 𝐜_j]; θ_out-y))  .

In baseline models, recurrent cells are initialised with a vector of zeros as there is no stochastic encoding we can condition on.



 §.§ Inference Network


The inference model q(z|, , λ) is a diagonal Gaussian

    Z|, ∼𝒩(𝐮, (𝐬⊙𝐬))

whose parameters are computed by an inference network. 
We use two bidirectional GRU encoders to encode the source and target sentences separately. To spare memory, we reuse embeddings from the generative model (<ref>-<ref>), but we prevent updates to those parameters based on gradients of the inference network, which we indicate with the function . 
To obtain fixed-size representations for the sentences, GRU encodings are averaged (<ref>-<ref>) .


    𝐟_1^m    = ((x_1^m; θ_emb-x)) 
    𝐞_1^n    = ((y_1^n; θ_emb-y)) 
    𝐡_x    = ((𝐟_1^m; λ_gru-x))  
    𝐡_y    = ((𝐞_1^n; λ_gru-y)) 
    𝐡_xy   = (𝐡_x,  𝐡_y) 
    𝐡_u   = ((𝐡_xy; λ_u-hid)) 
    𝐡_s    = ((𝐡_xy; λ_s-hid) 
    𝐮   = (𝐡_u; λ_u-out) 
    𝐬   = ((𝐡_s; λ_s-out))


We use a concatenation 𝐡_xy of the average source and target encodings  (<ref>) as inputs to compute the parameters of the Gaussian approximate posterior, namely, d-dimensional location and scale vectors. Both transformations use ReLU hidden activations <cit.>, but locations live in ℝ^d and therefore call for linear output activations (<ref>), whereas scales live in ℝ^d_>0 and call for strictly positive outputs (<ref>), we follow <cit.> and use .
The complete set of parameters used for inference is thus λ = {λ_gru-x, λ_gru-y, λ_u-hid, λ_u-out, λ_s-hid, λ_s-out}.



 §.§ Prediction Network


The prediction network parameterises our prediction model r(z|, ϕ), a variant of the inference model that conditions on the source sentence alone. 
In <ref> we explore several variants of the ELBO using different parameterisations of r_ϕ. 
In the simplest case we do not condition on the target sentence during training, thus we can use the same network both for training and prediction. The network is similar to the one described in <ref>, except that there is a single bidirectional GRU and we use the average source encoding  (<ref>) as input to the predictors for 𝐮 and 𝐬 (<ref>-<ref>).


    𝐡_u    = ((𝐡_x; λ_u-hid)) 
    𝐡_s    = ((𝐡_x; λ_s-hid)) 
    𝐮   = (𝐡_u; λ_u-out) 
    𝐬   = ((𝐡_s; λ_s-out))



In all other cases we use q(z|, , λ) parameterised as discussed in <ref> for training, and design a separate network to parameterise r_ϕ for prediction. 
Much like the inference model, the prediction model is a diagonal Gaussian

    Z|∼𝒩(û, (ŝ⊙ŝ))

also  parameterised by d-dimensional location and scale vectors, however in predicting û and ŝ (<ref>-<ref>) it can only access an encoding of the source (<ref>). 


    𝐡_x    = ((𝐟_1^m; ϕ_gru-x))  
    𝐡_u    = ((𝐡_x; ϕ_u-hid)) 
    𝐡_s    = ((𝐡_x; ϕ_s-hid)) 
    û   = (𝐡_u; ϕ_u-out) 
    ŝ   = ((𝐡_s; ϕ_s-out))


The complete set of parameters is then ϕ = {ϕ_gru-x, ϕ_u-hid, ϕ_u-out, ϕ_s-hid, ϕ_s-out}.
For the deterministic variant, we use û (<ref>) alone to approximate 𝐮 (<ref>), i.e. the posterior mean of Z.





§ GRAPHICAL MODELS


Figure <ref> is a graphical depiction of our AEVNMT model. Circled nodes denote random variables while uncircled nodes denote deterministic quantities. Shaded random variables correspond to observations and unshaded random variables are latent. The plate denotes a dataset of |𝒟| observations.




In Figure <ref>, we illustrate the precise statistical assumptions of AEVNMT. Here plates iterate over words in either the source or the target sentence. Note that the arrow from x_i to y_j states that the jth target word depends on all of the source sentence, not on the ith source word alone, and that is the case because x_i is within the source plate. 
In Figure <ref>, we illustrate the statistical dependencies induced in the marginal distribution upon marginalisation of latent variables. Recall that the marginal is the distribution which by assumption produced the observed data. Now compare that to the distribution modelled by the simple Joint model (Figure <ref>). 
Marginalisation induces undirected dependencies amongst random variables creating more structure in the marginal distribution. In graphical models literature this is known as moralisation <cit.>.











§ ROBUSTNESS TO OUT-OF-DOMAIN DATA



We use our stronger models, those trained on gold-standard NC bilingual data and synthetic News data, to translate test sets in various unseen genres. These data sets are collected and distributed by TAUS,[TAUS Hardware, TAUS Software, TAUS Industrial Electronics, TAUS Professional & Business Services, and TAUS Legal available from TAUS data cloud <http://tausdata.org/>.] and have been used in scenarios of adapation to all domains at once <cit.>. 
Table <ref> shows the performance of AEVNMT and the conditional baseline. The first thing to note is the remarkable drop in performance showing that without taking domain adaptation seriously both models are inadequate. In terms of BLEU, differences range from -0.3 to 0.8 (En-De) and 0.3 to 0.7 (De-En) and are mostly in favour of AEVNMT, though note the increased standard deviations.


    
            4cEn-De     4cDe-En

         (r2pt)3-6 (r2pt)7-10
        Model     Domain     BLEU ↑     METEOR ↑     TER ↓     BEER ↑     BLEU ↑     METEOR ↑     TER ↓     BEER ↑  


    Cond
            Computer Hardware     8.7 (0.3)     23.6 (0.2)     82.3 (4.0)     44.2 (0.3)     13.0 (0.4)     20.8 (0.1)     73.5 (3.6)     49.2 (0.1)

            Computer Software     7.9 (0.2)     22.9 (0.1)     83.6 (3.2)     44.1 (0.3)     11.8 (0.8)     20.4 (0.3)     80.8 (7.0)     48.6 (0.3)

            Industrial Electronics     7.6 (0.3)     21.8 (0.2)     89.6 (5.5)     42.7 (0.3)     10.1 (1.0)     17.6 (0.4)     85.7 (9.7)     46.0 (0.4)

            Professional & Business Services     8.1 (0.4)     23.3 (0.4)     75.6 (0.4)     43.1 (0.4)     11.1 (0.2)     20.5 (0.1)     73.8 (1.4)     44.9 (0.1)

            Legal     12.3 (0.1)     29.7 (0.3)     75.3 (0.5)     49.6 (0.1)     14.2 (0.3)     23.0 (0.3)     67.3 (2.1)     50.8 (0.3)

    AEVNMT
            Computer Hardware     9.1 (0.3)     23.9 (0.4)     79.4 (1.3)     44.7 (0.2)     13.6 (0.3)     21.0 (0.2)     70.6 (1.4)     49.3 (0.2)

            Computer Software     8.3 (0.1)     23.2 (0.2)     81.4 (0.7)     44.5 (0.2)     12.2 (0.5)     20.7 (0.3)     77.0 (3.7)     48.6 (0.3)

            Industrial Electronics     8.1 (0.2)     22.2 (0.3)     85.2 (1.0)     43.1 (0.2)     10.8 (0.6)     17.9 (0.3)     79.9 (4.9)     46.3 (0.2)

            Professional & Business Services     7.8 (0.7)     23.0 (0.7)     75.7 (1.0)     43.0 (0.5)     11.5 (0.2)     20.6 (0.2)     73.1 (0.7)     45.1 (0.1)

            Legal     13.1 (0.2)     30.5 (0.3)     74.4 (0.6)     49.9 (0.2)     14.5 (0.2)     23.2 (0.2)     66.1 (1.3)     50.9 (0.3)


    
    Performance of models trained with NC and back-translated News on various TAUS test sets: we report average (1std) across 5 independent runs.




§ VALIDATION RESULTS


    






§ ADDITIONAL METRICS



1.4 
    
 
    
     
     
    IWSLT14           4cEn-De     4cDe-En 

                (r2pt)2-5 (l2pt)6-9
                     BLEU ↑      METEOR ↑     TER ↓     BEER ↑     BLEU ↑     METEOR ↑     TER ↓     BEER ↑  
   Cond      23.0 (0.1)     42.4 (0.1)     56.0 (0.1)     58.6 (0.1)     27.3 (0.2)     30.3 (0.1)     52.4 (0.5)     59.8 (0.1)

    Joint       23.2(0.0)     42.8(0.0)     56.1(0.0)     58.7(0.0)     27.5(0.0)     30.3(0.0)     52.7(0.0)     59.8(0.0)

    AEVNMT     23.4 (0.1)     42.8 (0.2)     55.5 (0.3)     58.8 (0.1)     28.0 (0.1)    
    30.6 (0.1)     51.2 (0.6)     60.1 (0.1)
 
    
    tableTest results for in-domain training on IWSLT: we report average (1std) across 5 independent runs for Cond and AEVNMT, but a single run of Joint.
    
    

    
     
     
    WMT16           4cEn-De     4cDe-En 

                (r2pt)2-5 (l2pt)6-9
                     BLEU ↑      METEOR ↑     TER ↓     BEER ↑     BLEU ↑     METEOR ↑     TER ↓     BEER ↑ 
   Cond      17.8 (0.2)     35.9 (0.2)     65.2 (0.4)     53.1 (0.1)     20.1 (0.1)      26.0 (0.1)     62.0 (0.3)     53.7 (0.1)

    Joint      17.9(0.0)     36.2(0.0)     64.1(0.0)     53.4(0.0)     20.1(0.0)     26.0(0.0)     62.7(0.0)     53.7(0.0)

    AEVNMT     18.4 (0.2)      36.6 (0.2)     64.0 (0.4)     53.5 (0.1)     20.6 (0.2)     26.0 (0.1)     60.7 (0.8)     53.6 (0.1)

    
    tableTest results for in-domain training on NC: we report average (1std) across 5 independent runs for Cond and AEVNMT, but a single run of Joint. 
    
    

    
    
    WMT16     4cEn-De     4cDe-En

    (r2pt)2-5 (l2pt)6-9
        BLEU ↑     METEOR ↑     TER ↓     BEER ↑     BLEU ↑     METEOR ↑     TER ↓     BEER ↑  

    Cond     17.8 (0.2)     35.9 (0.2)     65.2 (0.4)     53.1 (0.1)     20.1 (0.1)     26.0 (0.1)     62.0 (0.3)     53.7 (0.1)


      + synthetic data      22.3 (0.3)      40.9 (0.2)     58.5 (0.5)     57.0 (0.2)     26.9 (0.2)      30.4 (0.1)     53.0 (0.5)     58.5 (0.1) 

    
    Joint + synthetic data      22.2(0.0)      40.8(0.0)     58.1(0.0)     57.0(0.0)     26.7(0.0)      30.2(0.0)     52.1(0.0)     58.6(0.0) 

    
    AEVNMT + synthetic data     22.5 (0.2)     41.0 (0.1)     58.1 (0.2)     57.0 (0.1)     27.4 (0.2)     30.6 (0.1)     52.0 (0.1)     58.8 (0.1) 

    
    
    tableTest results for training on NC plus  synthetic data (back-translated News Crawl): we report average (1std) across 5 independent runs for Cond and AEVNMT, but a single run of Joint.





    
    En-De         4cWMT16     4cIWSLT14

    (r2pt)3-6 (l2pt)7-10
    Training     Model     BLEU ↑     METEOR ↑     TER ↓     BEER ↑     BLEU ↑     METEOR ↑     TER ↓     BEER ↑  

    En-De     Cond     17.6 (0.4)     35.7 (0.3)     61.9 (0.7)     53.9 (0.2)     23.9 (0.3)     43.1 (0.3)     54.7 (0.2)     59.3 (0.1)

    
        Joint     18.1(0.0)     36.3(0.0)     61.4(0.0)     54.3(0.0)     24.2(0.0)     43.4(0.0)     54.5(0.0)     59.5(0.0)


        AEVNMT     18.4 (0.2)     36.6 (0.3)     60.9 (0.4)     54.5 (0.2)     24.1 (0.3)     43.5 (0.3)     54.0 (0.3)     59.5 (0.2)

    
    De-En     Cond     21.6 (0.2)     27.4 (0.1)     59.9 (0.7)     55.5 (0.2)     29.1 (0.2)     31.5 (0.1)     50.9 (0.5)     60.9 (0.1)

    
        Joint     22.3(0.0)     27.4(0.0)     58.8(0.0)     55.6(0.0)     29.2(0.0)     31.5(0.0)     49.2(0.0)     61.2(0.0)

    
        AEVNMT     22.3 (0.1)     27.5 (0.1)     57.8 (0.6)     55.6 (0.1)     29.2 (0.1)     31.5 (0.0)     49.2 (0.4)     61.1 (0.1)


    
    
    Test results for mixed-domain training: we report average (1std) across 5 independent runs for Cond and AEVNMT, but a single run of Joint.




