Generative Bridging Network for Neural Sequence Prediction
Wenhu Chen,1 Guanlin Li,2 Shuo Ren,5 Shujie Liu,3 Zhirui Zhang,4 Mu Li,3 Ming Zhou3
University of California, Santa Barbara1
Harbin Institute of Technology2
Microsoft Research Asia3
University of Science and Technology of China4
Beijing University of Aeronautics and Astronautics5
wenhuchen@cs.ucsb.edu epsilonlee.green@gmail.com {v-shure, shujliu, v-zhirzh, muli, mingzhou}@microsoft.com

arXiv:1706.09152v5 [] 17 Mar 2018

Abstract
In order to alleviate data sparsity and overfitting problems in maximum likelihood estimation (MLE) for sequence prediction tasks,
we propose the Generative Bridging Network
(GBN), in which a novel bridge module is introduced to assist the training of the sequence
prediction model (the generator network). Unlike MLE directly maximizing the conditional
likelihood, the bridge extends the point-wise
ground truth to a bridge distribution conditioned on it, and the generator is optimized to
minimize their KL-divergence. Three different
GBNs, namely uniform GBN, language-model
GBN and coaching GBN, are proposed to penalize confidence, enhance language smoothness and relieve learning burden. Experiments
conducted on two recognized sequence prediction tasks (machine translation and abstractive
text summarization) show that our proposed
GBNs can yield significant improvements over
strong baselines. Furthermore, by analyzing samples drawn from different bridges, expected influences on the generator are verified.

1

Introduction

Sequence prediction has been widely used in tasks
where the outputs are sequentially structured and
mutually dependent. Recently, massive explorations in this area have been made to solve practical problems, such as machine translation (Bahdanau et al., 2014; Ma et al., 2017; Norouzi et al.,
2016), syntactic parsing (Vinyals et al., 2015),
spelling correction (Bahdanau et al., 2014), image
captioning (Xu et al., 2015) and speech recognition (Chorowski et al., 2015). Armed with modern computation power, deep LSTM (Hochreiter
and Schmidhuber, 1997) or GRU (Chung et al.,
2014) based neural sequence prediction models
have achieved the state-of-the-art performance.
The typical training algorithm for sequence
prediction is Maximum Likelihood Estimation

𝐵𝑟𝑖𝑑𝑔𝑒 𝑚𝑜𝑑𝑢𝑙𝑒

𝑃𝜂 (𝑌|𝑌 ∗ )

𝑦1

𝑗

𝑦2

𝑗

…

𝑦𝑇 𝑗

∗
𝒀∗ 𝑦1

𝑦2∗

…

𝑦𝑇∗ ′

𝐵𝑟𝑖𝑑𝑔𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑠

𝑗

𝒀𝟏… 𝒀𝒋 … 𝒀𝑲

𝑗

𝑦1

𝑗

𝑦2

𝑠
…
𝑥1

𝑥2

…

𝑗

𝑦𝑇 𝑗

𝒀𝒋

…

𝐴𝑡𝑡

𝑿

…

𝑗

𝑦1

…

𝑗

𝑦𝑇 𝑗 −1

𝐺𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟 𝑛𝑒𝑡𝑤𝑜𝑟𝑘 𝑃𝜃 (𝑌|𝑋)

𝑥𝑇

Figure 1: The overall architecture of our novel Generative Bridging Network (GBN). Two main components, namely the generator network and the bridge
module, are connected through samples (Y 1 . . . Y K in
red) from the bridge module during training time. (We
sometimes call them generator and bridge in brief respectively in the following discussion.) The generator
is implemented through an attentive encoder-decoder,
where in the figure Att represents the attention module.

(MLE), which maximizes the likelihood of the target sequences conditioned on the source ones:
θ∗ = argmax
θ

E

(X,Y ∗ )∼D

log pθ (Y ∗ |X)

(1)

Despite the popularity of MLE or teacher forcing (Doya, 1992) in neural sequence prediction
tasks, two general issues are always haunting: 1).
data sparsity and 2). tendency for overfitting, with
which can both harm model generalization.
To combat data sparsity, different strategies
have been proposed. Most of them try to take
advantage of monolingual data (Sennrich et al.,
2015; Zhang and Zong, 2016; Cheng et al., 2016).
Others try to modify the ground truth target based
on derived rules to get more similar examples for
training (Norouzi et al., 2016; Ma et al., 2017).
To alleviate overfitting, regularization techniques,

such as confidence penalization (Pereyra et al.,
2017) and posterior regularization (Zhang et al.,
2017), are proposed recently.
As shown in Figure 1, we propose a novel learning architecture, titled Generative Bridging Network (GBN), to combine both of the benefits from
synthetic data and regularization. Within the architecture, the bridge module (bridge) first transforms the point-wise ground truth into a bridge
distribution, which can be viewed as a target proposer from whom more target examples are drawn
to train the generator. By introducing different
constraints, the bridge can be set or trained to possess specific property, with which the drawn samples can augment target-side data (alleviate data
sparsity) while regularizing the training (avoid
overfitting) of the generator network (generator).
In this paper, we introduce three different constraints to build three bridge modules. Together
with the generator network, three GBN systems
are constructed: 1). a uniform GBN, instantiating
the constraint as a uniform distribution to penalize confidence; 2). a language-model GBN, instantiating the constraint as a pre-trained neural
language model to increase language smoothness;
3). a coaching GBN, instantiating the constraint as
the generator’s output distribution to seek a closeto-generator distribution, which enables the bridge
to draw easy-to-learn samples for the generator to
learn. Without any constraint, our GBN degrades
to MLE. The uniform GBN is proved to minimize
KL-divergence with a so-called payoff distribution
as in reward augmented maximum likelihood or
RAML (Norouzi et al., 2016).
Experiments are conducted on two sequence
prediction tasks, namely machine translation and
abstractive text summarization. On both of them,
our proposed GBNs can significantly improve
task performance, compared with strong baselines. Among them, the coaching GBN achieves
the best. Samples from these three different
bridges are demonstrated to confirm the expected
impacts they have on the training of the generator.
In summary, our contributions are:
• A novel GBN architecture is proposed for sequence prediction to alleviate the data sparsity and overfitting problems, where the
bridge module and the generator network are
integrated and jointly trained.
• Different constraints are introduced to build
GBN variants: uniform GBN, language-

model GBN and coaching GBN. Our GBN
architecture is proved to be a generalized
form of both MLE and RAML.
• All proposed GBN variants outperform the
MLE baselines on machine translation and
abstractive text summarization. Similar relative improvements are achieved compared to
recent state-of-the-art methods in the translation task. We also demonstrate the advantage of our GBNs qualitatively by comparing
ground truth and samples from bridges.

𝑌∗
𝑝𝐶 (𝑌)
𝐶𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡

𝑆(𝑌, 𝑌 ∗ )

𝑃𝜂 (𝑌|𝑌 ∗ )
𝑌∗

𝐶(𝑝𝜂 , 𝑝𝑐 )
Knowledge
injection

𝐵𝑟𝑖𝑑𝑔𝑒

𝑝𝐶 (𝑌)

𝐾𝐿(𝑝𝜂 , 𝑝𝜃 )
Regularization-bysynthetic-samples

𝑝𝜃 (𝑌|𝑋)
𝐺𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟

Figure 2: Conceptual interpretation of our Generative
Bridging Network (GBN). See detailed discussion in
the beginning of Sec. 2.

2

Generative Bridging Network

In this section, we first give a conceptual interpretation of our novel learning architecture which is
sketched in Figure 2. Since data augmentation and
regularization are two golden solutions for tackling data sparsity and overfitting issues. We are
willing to design an architecture which can integrate both of their benefits. The basic idea is to use
a so-called bridge which transforms Y ∗ to an easyto-sample distribution, and then use this distribution (samples) to train and meanwhile regularize
the sequence prediction model (the generator).
The bridge is viewed as a conditional distribution1 pη (Y |Y ∗ ) to get more target Y s given Y ∗
so as to construct more training pairs (X, Y ). In
the meantime, we could inject (empirical) prior
knowledge into the bridge through its optimization objective which is inspired by the design of
the payoff distribution in RAML. We formulate
the optimization objective with two parts in Equation (2): a) an expected similarity score computed through a similarity score function S(·, Y ∗ )
interpolated with b) a knowledge injection constraint2 C(pη (Y |Y ∗ ), pc (Y )) where α controls the
1
η should be treated as an index of the bridge distribution,
so it is not necessarily the parameters to be learned.
2
Note that, in our paper, we specify C to be KL-divergence
between the bridge distribution pη and certain constraint dis-

strength of the regularization, formally, we write
the objective function LB (η) as follows:
LB (η) =
E

[−S(Y, Y ∗ )] + αC(pη (Y |Y ∗ ), pc (Y ))

Y ∼pη (Y |Y ∗ )

(2)
Minimizing it empowers the bridge distribution
not only to concentrate its mass around the ground
truth Y ∗ but also to adopt certain hope property
from pc (Y ). With the constructed bridge distribution, we optimize the generator network Pθ (Y |X)
to match its output distribution towards the bridge
distribution by minimizing their KL-divergence:
LG (θ) = KL(pη (Y |Y ∗ )||pθ (Y |X))

(3)

In practice, the KL-divergence is approximated
through sampling process detailed in Sec. 2.3.
As a matter of fact, the bridge is the crux of the
integration: it synthesizes new targets to alleviate data sparsity and then uses the synthetic data
as regularization to overcome overfitting. Thus
a regularization-by-synthetic-example approach,
which is very similar to the prior-incorporationby-virtual-example method (Niyogi et al., 1998).
2.1

Generator Network

Our generator network is parameterized with
the commonly used encoder-decoder architecture (Bahdanau et al., 2014; Cho et al., 2014). The
encoder is used to encode the input sequence X
to a sequence of hidden states, based on which
an attention mechanism is leveraged to compute
context vectors at the decoding stage. The context vector together with previous decoder’s hidden state and previously predicted label are used,
at each time step, to compute the next hidden state
and predict an output label.
As claimed in Equation (3), the generator network is not trained to maximize the likelihood of
the ground truth but tries best to match the bridge
distribution, which is a delegate of the ground
truth. We use gradient descent to optimize the KLdivergence with respect to the generator:
∇LG (θ) =

E

Y ∼pη (Y |Y ∗ )

log ∇pθ (Y |X)

(4)

The optimization process can be viewed as the
generator maximizing the likelihood of samples
tribution pc , however, we believe mathematical form of C is
not restricted, which could motivate further development.

drawn from the bridge. This may alleviate data
sparsity and overfitting by posing more unseen
scenarios to the generator and may help the generator generalize better in test time.
2.2

Bridge Module3

Our bridge module is designed to transform a
single target example Y ∗ to a bridge distribution pη (Y |Y ∗ ). We design its optimization target in Equation (2) to consist of two terms,
namely, a concentration requirement and a constraint. The constraint is instantiated as KLdivergence between the bridge and a contraint distribution pc (Y ). We transform Equation (2) as follows, which is convenient for mathematical manipulation later:
LB (η) =
E

Y ∼pη (Y |Y ∗ )

[−

S(Y, Y ∗ )
] + KL(pη (Y |Y ∗ )||pc (Y ))
τ
(5)

S(Y, Y ∗ ) is a predefined score function which
measures similarity between Y and Y ∗ and peaks
when Y = Y ∗ , while pc (Y ) reshapes the bridge
distribution. More specifically, the first term ensures that the bridge should concentrate around the
ground truth Y ∗ , and the second introduces willing
property which can help regularize the generator.
The hyperparameter τ can be interpreted as a temperature which scales the score function. In the
following bridge specifications, the score function
S(Y, Y ∗ ) is instantiated according to Sec. 3.1.
1. Delta Bridge The delta bridge can be seen
as the simplest case where α = 0 or no constraint is imposed. The bridge seeks to minimize
∗)
E
[− S(Y,Y
]. The optimal solution is
τ
Y ∼pη (Y |Y ∗ )

when the bridge only samples Y ∗ , thus the Dirac
delta distribution is described as follows:
pη (Y |Y ∗ ) = δY ∗ (Y )

(6)

This exactly corresponds to MLE, where only examples in the dataset are used to train the generator. We regard this case as our baseline.
2. Uniform Bridge The uniform bridge adopts
a uniform distribution U (Y ) as constraint. This
3
Although we name it bridge module, we explicitly learn
it with the generator when a closed-form static solution exists in terms of Equation (5). Otherwise, we will adopt an
encoder-decoder to construct a dynamic bridge network.

bridge motivates to include noise into target example, which is similar to label smoothing (Szegedy
et al., 2016). The loss function can be written as:
LB (η) =
E

[−

Y ∼pη (Y |Y ∗ )

S(Y, Y ∗ )
] + KL(pη (Y |Y ∗ )||U (Y ))
τ
(7)

We can re-write it as follows by adding a constant
to not change the optimization result:
exp S(Y,Y
τ
LB (η) + C = KL(pη (Y |Y )||
Z
∗

∗)

)

4. Coaching Bridge The coaching bridge utilizes the generator’s output distribution as constraint, which motivates to generate training samples which are easy to be understood by the
generator, so as to relieve its learning burden.
The coaching bridge follows the same spirit as
the coach proposed in Imitation-via-Coaching (He
et al., 2012), which, in reinforcement learning vocabulary, advocates to guide the policy (generator) with easy-to-learn action trajectories and let
it gradually approach the oracle when the optimal
action is hard to achieve.
LB (η) =

(8)
E

This bridge is static for having a closed-form solution:
exp S(Y,Y
τ
pη (Y |Y ) =
Z

∗)

∗

(9)

where Z is the partition function. Note that our
uniform bridge corresponds to the payoff distribution described in RAML (Norouzi et al., 2016).
3. Language-model (LM) Bridge The LM
bridge utilizes a pretrained neural language model
pLM (Y ) as constraint, which motivates to propose
target examples conforming to language fluency.

Y ∼pη (Y |Y ∗ )

[−

S(Y, Y ∗ )
] + KL(pθ (Y |X)||pη (Y |Y ∗ ))
τ
(13)

Since the KL constraint is a moving target when
the generator is updated, the coaching bridge
should not remain static. Therefore, we perform
iterative optimization to train the bridge and the
generator jointly. Formally, the derivatives for the
coaching bridge are written as follows:
∇LB (η) =

[−

E

Y ∼pη (Y |Y ∗ )

+

E

Y ∼pθ (Y |X)

S(Y, Y ∗ )
∇ log pη (Y |Y ∗ )]
τ

∇ log pη (Y |Y ∗ )
(14)

LB (η) =
E

[−

Y ∼pη (Y |Y ∗ )

S(Y, Y ∗ )
] + KL(pη (Y |Y ∗ )||pLM )
τ
(10)

Similar to the uniform bridge case, we can re-write
the loss function to a KL-divergence:
LB (η) + C
pLM (Y ) · exp S(Y,Y
τ
=KL(pη (Y |Y )||
Z
∗

∗)

(11)
)

Thus, the LM bridge is also static and can be seen
as an extension of the uniform bridge, where the
exponentiated similarity score is re-weighted by a
pretrained LM score, and renormalized:
pLM (Y ) exp S(Y,Y
τ
p(Y |Y ) =
Z
∗

The first term corresponds to the policy gradient
algorithm described in REINFORCE (Williams,
1992), where the coefficient −S(Y, Y ∗ )/τ corresponds to reward function. Due to the mutual dependence between bridge module and generator
network, we design an iterative training strategy,
i.e. the two networks take turns to update their
own parameters treating the other as fixed.
2.3

Training

The training of the above three variants is illustrated in Figure 3. Since the proposed bridges can
be divided into static ones, which only require pretraining, and dynamic ones, which require continual training with the generator, we describe their
training process in details respectively.

∗)

(12)

where Z is the partition function. The above equation looks just like the payoff distribution, whereas
an additional factor is considered.

2.3.1 Stratified-Sampled Training
Since closed-formed optimal distributions can be
found for uniform/LM GBNs, we only need to
draw samples from the static bridge distributions
to train our sequence generator. Unfortunately,

𝜹(𝒀)

𝑝𝑑𝑎𝑡𝑎 (𝑌 ∗ )

𝑝𝐿𝑀 (𝑌)

𝑷𝜼

𝑷𝜽

Pre-trained
𝑈(𝑌)

LM Bridge

𝑝𝜂 (𝑌|𝑌 ∗ )

Uniform Bridge

𝑝𝜂 (𝑌|𝑌 ∗ )

Stratified-sampled
Training

Generator
𝑝𝜃 (𝑌|𝑋)

Coach Bridge

𝑝𝜂 (𝑌|𝑌 ∗ )

1. Update coach 𝑃𝜂

Iterative
Training

2. Update learner 𝑃𝜃

Figure 3: The training processes of the three different
variants of our GBN architecture (Sec. 2.3).
3. Update coach 𝑃𝜂

due to the intractability of these bridge distributions, direct sampling is infeasible. Therefore, we
follow Norouzi et al. (2016); Ma et al. (2017) and
adopt stratified sampling to approximate the direct
sampling process. Given a sentence Y ∗ , we first
sample an edit distance m, and then randomly select m positions to replace the original tokens. The
difference between the uniform and the LM bridge
lies in that the uniform bridge replaces labels by
drawing substitutions from a uniform distribution,
while LM bridge takes the history as condition and
draws substitutions from its step-wise distribution.
2.3.2

Iterative Training

Since the KL-constraint is a moving target for the
coaching bridge, an iterative training strategy is
designed to alternately update both the generator
and the bridge (Algorithm 1). We first pre-train
both the generator and the bridge and then start to
alternately update their parameters. Figure 4 intuitively demonstrates the intertwined optimization
effects over the coaching bridge and the generator.
We hypothesize that iterative training with easyto-learn guidance could benefit gradient update,
thus result in better local minimum.

3

Experiment

We select machine translation and abstractive text
summarization as benchmarks to verify our GBN
framework.
3.1

Similarity Score Function

In our experiments, instead of directly using
BLEU or ROUGE as reward to guide the bridge
network’s policy search, we design a simple sur-

4. Update learner 𝑃𝜃

𝑌

𝑌

𝑌

𝑌

Figure 4: Four iterative updates of the coaching bridge
and the generator. In an early stage, the pre-trained
generator Pθ may not put mass on some ground truth
target points within the output space, shown by δ(Y ).
The coaching bridge is first updated with Equation (14)
to locate in between the Dirac delta distribution and
the generator’s output distribution. Then, by sampling
from the coaching bridge for approximating Equation
(4), target samples which demonstrate easy-to-learn sequence segments facilitate the generator to be optimized to achieve closeness with the coaching bridge.
Then this process repeats until the generator converges.

rogate n-gram matching reward as follows:
S(Y, Y ∗ ) = 0.4∗N4 +0.3∗N3 +0.2∗N2 +0.1∗N1
(15)
Nn represents the n-gram matching score between
Y and Y ∗ . In order to alleviate reward sparsity at
sequence level, we further decompose the global
reward S(Y, Y ∗ ) as a series of local rewards at every time step. Formally, we write the step-wise
reward s(yt |y1:t−1 , Y ∗ ) as follows:


1.0; N (y1:t , yt−3:t ) ≤ N (Y ∗ , yt−3:t )


0.6; N (y , y
∗

1:t
t−2:t ) ≤ N (Y , yt−2:t )

∗
∗
s(yt |y1:t−1 , Y ) = 0.3; N (y1:t , yt−1:t ) ≤ N (Y , yt−1:t )



0.1; N (y1:t , yt ) ≤ N (Y ∗ , yt )



0.0; otherwise
(16)

where N (Y, Ỹ ) represents the occurrence of subsequence Ỹ in whole sequence Y . Specifically, if

Methods
MIXER
BSO
AC
Softmax-Q
Uniform GBN
(τ = 0.8)
LM GBN
(τ = 0.8)
Coaching GBN
(τ = 0.8)
Coaching GBN
(τ = 1.2)
Coaching GBN
(τ = 1.0)

Baseline
20.10
24.03
27.56
27.66

Model
21.81 +1.71
26.36 +2.33
28.53 +0.97
28.77 +1.11
29.80 +0.70
29.90 +0.80

29.10

29.98 +0.88
30.15 +1.05
30.18 +1.08

Table 1: Comparison with existing works on IWSLT2014 German-English Machine Translation Task.
Coaching GBN Learning Curve
100
95
90

BLEU

Algorithm 1 Training Coaching GBN
procedure P RE - TRAINING
Initialize pθ (Y |X) and pη (Y |Y ∗ ) with random weights θ and η
Pre-train pθ (Y |X) to predict Y ∗ given X
Use pre-trained pθ (Y |X) to generate Ŷ
given X
Pre-train pη (Y |Y ∗ ) to predict Ŷ given Y ∗
end procedure
procedure I TERATIVE -T RAINING
while Not Converged do
Receive a random example (X, Y ∗ )
if Bridge-step then
Draw samples Y from pθ (Y |X)
Update bridge via Equation (14)
else if Generator-step then
Draw samples Y from pη (Y |Y ∗ )
Update generator via Equation (4)
end if
end while
end procedure

85
80
75
70

a certain sub-sequence yt−n+1:t from Y appears
less times than in the reference Y ∗ , yt receives reward. Formally, we rewrite the step-level gradient
for each sampled Y as follows:

(17)
3.2

1

2

3

4

5

6

7

8

9

10

11

12

13

14

9

10

11

12

13

14

Epoch (Bridge)
32.2
32.1
32

BLEU

S(Y, Y ∗ )
−
∇ log pη (Y |Y ∗ )
τ
X s(yt |y1:t−1 , Y ∗ )
=
−
· ∇ log pη (yt |y1:t−1 , Y ∗ )
τ
t

0

31.9
31.8
31.7
31.6
31.5
0

1

2

3

4

5

6

7

8

Epoch (Generator)

Figure 5: Coaching GBN’s learning curve on IWSLT
German-English Dev set.

Machine Translation

Dataset We follow Ranzato et al. (2015); Bahdanau et al. (2016) and select German-English machine translation track of the IWSLT 2014 evaluation campaign. The corpus contains sentencewise aligned subtitles of TED and TEDx talks. We
use Moses toolkit (Koehn et al., 2007) and remove
sentences longer than 50 words as well as lowercasing. The evaluation metric is BLEU (Papineni
et al., 2002) computed via the multi-bleu.perl.
System Setting We use a unified GRU-based
RNN (Chung et al., 2014) for both the generator
and the coaching bridge. In order to compare with
existing papers, we use a similar system setting
with 512 RNN hidden units and 256 as embedding size. We use attentive encoder-decoder to
build our system (Bahdanau et al., 2014). During training, we apply ADADELTA (Zeiler, 2012)

with  = 10−6 and ρ = 0.95 to optimize parameters of the generator and the coaching bridge.
During decoding, a beam size of 8 is used to approximate the full search space. An important
hyper-parameter for our experiments is the temperature τ . For the uniform/LM bridge, we follow Norouzi et al. (2016) to adopt an optimal temperature τ = 0.8. And for the coaching bridge,
we test hyper-parameters from τ ∈ {0.8, 1.0, 1.2}.
Besides comparing with our fine-tuned baseline,
other systems for comparison of relative BLEU
improvement are: MIXER (Ranzato et al., 2015),
BSO (Wiseman and Rush, 2016), AC (Bahdanau
et al., 2016), Softmax-Q (Ma et al., 2017).
Results The experimental results are summarized in Table 1. We can observe that our
fine-tuned MLE baseline (29.10) is already over-

RG-2
11.32
11.88
14.45
17.54
15.95

RG-L
26.42
26.96
30.71
33.63
31.68

34.10

16.70

31.75

82
81.5
81

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

9

10

11

12

13

14

Epoch (Bridge)
22.7

22.3
22.1

21.9

34.32

16.88

31.89

21.7
0

1

2

3

4

5

6

7

8

Epoch (Generator)

34.49

16.70

31.95

34.83

16.83

32.25

35.26

17.22

32.67

competing other systems and our proposed GBN
can yield a further improvement. We also observe that LM GBN and coaching GBN have both
achieved better performance than Uniform GBN,
which confirms that better regularization effects
are achieved, and the generators become more robust and generalize better. We draw the learning
curve of both the bridge and the generator in Figure 5 to demonstrate how they cooperate during
training. We can easily observe the interaction
between them: as the generator makes progress,
the coaching bridge also improves itself to propose
harsher targets for the generator to learn.
Abstractive Text Summarization

Dataset We follow the previous works by Rush
et al. (2015); Zhou et al. (2017) and use the
same corpus from Annotated English Gigaword
dataset (Napoles et al., 2012). In order to be comparable, we use the same script 4 released by Rush
et al. (2015) to pre-process and extract the training and validation sets. For the test set, we use the
English Gigaword, released by Rush et al. (2015),
and evaluate our system through ROUGE (Lin,
2004). Following previous works, we employ
ROUGE-1, ROUGE-2, and ROUGE-L as the evaluation metrics in the reported experimental results.
4

83
82.5

22.5

Table 2: Full length ROUGE F1 evaluation results on
the English Gigaword test set used by (Rush et al.,
2015). RG in the Table denotes ROUGE. Results
for comparison are taken from SAEASS (Zhou et al.,
2017).

3.3

Coaching GBN Learning Curve
83.5

ROUGE-2

RG-1
29.55
29.76
33.10
36.15
34.04

ROUGE-2

Methods
ABS
ABS+
Luong-NMT
SAEASS
seq2seq+att
Uniform GBN
(τ = 0.8)
LM GBN
(τ = 0.8)
Coaching GBN
(τ = 0.8)
Coaching GBN
(τ = 1.2)
Coaching GBN
(τ = 1.0)

https://github.com/facebookarchive/NAMAS

Figure 6: Coaching GBN’s learning curve on Abstractive Text Summarization Dev set.

System Setting We follow Zhou et al. (2017);
Rush et al. (2015) to set input and output vocabularies to 119,504 and 68,883 respectively,
and we also set the word embedding size to
300 and all GRU hidden state size to 512.
Then we adopt dropout (Srivastava et al., 2014)
with probability p = 0.5 strategy in our output layer. We use attention-based sequence-tosequence model (Bahdanau et al., 2014; Cho et al.,
2014) as our baseline and reproduce the results of
the baseline reported in Zhou et al. (2017). As
stated, the attentive encoder-decode architecture
can already outperform existing ABS/ABS+ systems (Rush et al., 2015). In coaching GBN, due to
the fact that the input of abstractive summarization
X contains more information than the summary
target Y ∗ , directly training the bridge pη (Y |Y ∗ )
to understand the generator pθ (Y |X) is infeasible.
Therefore, we re-design the coaching bridge to receive both source and target input X, Y and we
enlarge its vocabulary size to 88,883 to encompass more information about the source side. In
Uniform/LM GBN experiments, we also fix the
hyper-parameter τ = 0.8 as the optimal setting.

Results The experimental results are summarized in Table 2. We can observe a significant
improvement via our GBN systems. Similarly,
the coaching GBN system achieves the strongest
performance among all, which again reflects our
assumption that more sophisticated regularization
can benefit generator’s training. We draw the
learning curve of the coaching GBN in Figure 6
to demonstrate how the bridge and the generator
promote each other.

4

System
Property
Reference
Bridge
System
Property
Reference
Bridge
System
Property
Reference
Bridge
Property

Analysis

By introducing different constraints into the bridge
module, the bridge distribution will propose different training samples for the generator to learn.
From Table 3, we can observe that most samples
still reserve their original meaning. The uniform
bridge simply performs random replacement without considering any linguistic constraint. The LM
bridge strives to smooth reference sentence with
high-frequent words. And the coaching bridge
simplifies difficult expressions to relieve generator’s learning burden. From our experimental results, the more rational and aggressive diversification from the coaching GBN clearly benefits generator the most and helps the generator generalize
to more unseen scenarios.

5
5.1

Related Literature
Data Augmentation and Self-training

In order to resolve the data sparsity problem in
Neural Machine Translation (NMT), many works
have been conducted to augment the dataset. The
most popular strategy is via self-learning, which
incorporates the self-generated data directly into
training. Zhang and Zong (2016) and Sennrich
et al. (2015) both use self-learning to leverage
massive monolingual data for NMT training. Our
bridge can take advantage of the parallel training
data only, instead of external monolingual ones to
synthesize new training data.
5.2

Reward Augmented Maximum
Likelihood

Reward augmented maximum likelihood or
RAML (Norouzi et al., 2016) proposes to integrate task-level reward into MLE training by
using an exponentiated payoff distribution. KL
divergence between the payoff distribution and the
generator’s output distribution are minimized to
achieve an optimal task-level reward. Following
this work, Ma et al. (2017) introduces softmax
Q-Distribution to interpret RAML and reveals its
relation with Bayesian decision theory. These
two works both alleviate data sparsity problem by
augmenting target examples based on the ground
truth. Our method draws inspiration from them
but seeks to propose the more general Generative
Bridging Network, which can transform the
ground truth into different bridge distributions,
from where samples are drawn will account for
different interpretable factors.

Reference
Bridge

Uniform GBN
Random Replacement
the question is , is it worth it ?
the question lemon , was it worth it ?

Language-model GBN
Word Replacement
now how can this help us ?
so how can this help us ?

Coaching GBN
Reordering
i need to have a health care lexicon .
i need a lexicon for health care .

Simplification
this is the way that most of us were taught
to tie our shoes .
most of us learned to bind our shoes .

Table 3: Qualitative analysis for three different bridge
distributions.

5.3

Coaching

Our coaching GBN system is inspired by imitation learning by coaching (He et al., 2012). Instead of directly behavior cloning the oracle, they
advocate learning hope actions as targets from a
coach which is interpolated between learner’s policy and the environment loss. As the learner makes
progress, the targets provided by the coach will
become harsher to gradually improve the learner.
Similarly, our proposed coaching GBN is motivated to construct an easy-to-learn bridge distribution which lies in between the ground truth and
the generator. Our experimental results confirm its
effectiveness to relieve the learning burden.

6

Conclusion

In this paper, we present the Generative Bridging Network (GBN) to overcome data sparsity and
overfitting issues with Maximum Likelihood Estimation in neural sequence prediction. Our implemented systems prove to significantly improve the
performance, compared with strong baselines. We
believe the concept of bridge distribution can be
applicable to a wide range of distribution matching
tasks in probabilistic learning. In the future, we intend to explore more about GBN’s applications as
well as its provable computational and statistical
guarantees.

References
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu,
Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron
Courville, and Yoshua Bengio. 2016. An actor-critic
algorithm for sequence prediction. arXiv preprint
arXiv:1607.07086 .

Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham
Neubig, and Eduard Hovy. 2017. Softmax qdistribution estimation for structured prediction: A
theoretical interpretation for raml. arXiv preprint
arXiv:1705.07136 .

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint
arXiv:1409.0473 .

Courtney Napoles, Matthew Gormley, and Benjamin
Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge
Extraction. Association for Computational Linguistics, pages 95–100.

Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua
Wu, Maosong Sun, and Yang Liu. 2016. Semisupervised learning for neural machine translation.
arXiv preprint arXiv:1606.04596 .

Partha Niyogi, Federico Girosi, and Tomaso Poggio.
1998. Incorporating prior information in machine
learning by creating virtual examples. Proceedings
of the IEEE 86(11):2196–2209.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. 2014. Learning
phrase representations using rnn encoder-decoder
for statistical machine translation. arXiv preprint
arXiv:1406.1078 .
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy
Serdyuk, Kyunghyun Cho, and Yoshua Bengio.
2015. Attention-based models for speech recognition. In Advances in Neural Information Processing
Systems. pages 577–585.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho,
and Yoshua Bengio. 2014. Empirical evaluation of
gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 .
Kenji Doya. 1992. Bifurcations in the learning of recurrent neural networks. In Circuits and Systems,
1992. ISCAS’92. Proceedings., 1992 IEEE International Symposium on. IEEE, volume 6, pages 2777–
2780.
He He, Jason Eisner, and Hal Daume. 2012. Imitation
learning by coaching. In Advances in Neural Information Processing Systems. pages 3149–3157.
Sepp Hochreiter and Jürgen Schmidhuber. 1997.
Long short-term memory. Neural computation
9(8):1735–1780.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, et al. 2007. Moses: Open source
toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on
interactive poster and demonstration sessions. Association for Computational Linguistics, pages 177–
180.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop. Barcelona, Spain, volume 8.

Mohammad Norouzi, Samy Bengio, Navdeep Jaitly,
Mike Schuster, Yonghui Wu, Dale Schuurmans,
et al. 2016. Reward augmented maximum likelihood for neural structured prediction. In Advances
In Neural Information Processing Systems. pages
1723–1731.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of
the 40th annual meeting on association for computational linguistics. Association for Computational
Linguistics, pages 311–318.
Gabriel Pereyra, George Tucker, Jan Chorowski,
Łukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output
distributions. arXiv preprint arXiv:1701.06548 .
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,
and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint
arXiv:1511.06732 .
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint
arXiv:1509.00685 .
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2015.
Improving neural machine translation
models with monolingual data. arXiv preprint
arXiv:1511.06709 .
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky,
Ilya Sutskever, and Ruslan Salakhutdinov. 2014.
Dropout: a simple way to prevent neural networks
from overfitting. Journal of Machine Learning Research 15(1):1929–1958.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,
Jon Shlens, and Zbigniew Wojna. 2016. Rethinking
the inception architecture for computer vision. In
Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition. pages 2818–2826.

Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov,
Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural
Information Processing Systems. pages 2773–2781.
Ronald J Williams. 1992. Simple statistical gradientfollowing algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256.
Sam Wiseman and Alexander M Rush. 2016.
Sequence-to-sequence learning as beam-search optimization. arXiv preprint arXiv:1606.02960 .
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,
Aaron Courville, Ruslan Salakhudinov, Rich Zemel,
and Yoshua Bengio. 2015. Show, attend and tell:
Neural image caption generation with visual attention. In International Conference on Machine
Learning. pages 2048–2057.
Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 .
Jiacheng Zhang, Yang Liu, Huanbo Luan, Jingfang Xu,
and Maosong Sun. 2017. Prior knowledge integration for neural machine translation using posterior
regularization. In Proceedings of the 55th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages
1514–1523.
Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine
translation. In EMNLP. pages 1535–1545.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou.
2017. Selective encoding for abstractive sentence
summarization. arXiv preprint arXiv:1704.07073 .

A

Supplemental Material

Derivation of Equation (11)
LB (η)

This part first provides detailed derivation of
Equation (8) and (11) from Equation (7) and
(10), since our uniform bridge distribution and
language-model bridge distribution have closedform solutions given a fixed uniform distribution
and a language model as constraints. Then, we
give explanation of Equation (13), the objective
function of coaching bridge, where the constraint
is the inverse KL compared with previous two
bridges and then give detailed derivation of the
gradient update Equation (14).

S(Y, Y ∗ )
= E −
+ KL(pη (Y |Y ∗ )||pLM (Y ))
Y ∼pη
τ
Z
S(Y, Y ∗ )
−pη (Y |Y ∗ ) log exp(
)
=
τ
Y
Z
pη (Y |Y ∗ )
pη (Y |Y ∗ ) log
+
pLM (Y )
Z Y
pη (Y |Y ∗ )
pη (Y |Y ∗ ) log
=
∗)
Y
exp( S(Y,Y
) · pLM (Y )
τ
Z
∗
pη (Y |Y )
pη (Y |Y ∗ ) log
=
+ Const
S(Y,Y ∗ )
exp

Y

Derivation of Equation (8)

=KL(pη (Y |Y ∗ )||

exp

·PLM (Y )

τ

Z
S(Y,Y ∗ )
τ

· PLM (Y )

Z

) + Const0
(20)

LB (η)
∗)

S(Y, Y
= E −
+ KL(pη (Y |Y ∗ )||U (Y ))
Y ∼pη
τ
Z
S(Y, Y ∗ )
=
−pη (Y |Y ∗ ) log exp(
)
τ
Y
Z
pη (Y |Y ∗ )
+
pη (Y |Y ∗ ) log
U (Y )
Z Y
pη (Y |Y ∗ )
=
pη (Y |Y ∗ ) log
∗)
Y
) · U (Y )
exp( S(Y,Y
τ
Z
∗
pη (Y |Y )
=
pη (Y |Y ∗ ) log
∗)
1
Y
exp( S(Y,Y
) · |Y|
τ
Z
pη (Y |Y ∗ )
=
pη (Y |Y ∗ ) log
∗)
Y
exp S(Y,Y
τ
Z
+ log |Y|
pη (Y |Y ∗ )
Y
Z
pη (Y |Y ∗ )
=
pη (Y |Y ∗ ) log
∗ ) + Const
Y
exp S(Y,Y
τ
Z
pη (Y |Y ∗ )
=
pη (Y |Y ∗ ) log
+ Const0
S(Y,Y ∗ )
exp

Y

τ

exp S(Y,Y
τ
= KL(pη (Y |Y )||
Z

Explanation of Equation (13) This equation
is the objective function of our coaching bridge,
which uses an inverse KL term5 as part of its objective. The use of inverse KL is out of the consideration of computational stability. The reasons are
two-fold: 1). the inverse KL will do not change
the effect of the constraint; 2). the inverse KL requires sampling from the generator and uses those
samples as the target to train the bridge, which has
the same gradient update ad MLE, so we do not
need to consider baseline tricks in Reinforcement
Learning implementation.
Gradient derivation of Equation (13)
∇LB (η)
S(Y, Y ∗ )
+ ∇η KL(pθ (Y |X)||pη (Y |Y ∗ ))
τ
Y ∼pη (Y |Y ∗ )
S(Y, Y ∗ )
=
E
−
∇η log pη (Y |Y ∗ )
τ
Y ∼pη (Y |Y ∗ )
=∇η

Z
∗

Here, the Y ∗ related constant Z is needed
to transform a unnormalized weighted similarity
score to a probability:
Z
S(Y, Y ∗ )
∗
Z(Y ) =
exp
· PLM (Y ) (21)
τ
Y

∗)

) + Const0
(18)

+ ∇η
Here, the Y ∗ related constant Z is needed to
transform a unnormalized similarity score to a
probability:

=

−

E

E

Y ∼pθ (Y |X)

E

Y ∼pη (Y |Y ∗ )

+

E

−

log pη (Y |Y ∗ )

S(Y, Y ∗ )
∇ log pη (Y |Y ∗ )
τ

∇ log pη (Y |Y ∗ )

Y ∼pθ (Y |X)

(22)
Z(Y ∗ ) =

Z
Y

S(Y, Y ∗ )
exp
τ

(19)

5

That is the use of KL(pθ ||pη ) instead of KL(pη ||pθ ).

