SEQUENCE TRAINING AND ADAPTATION OF HIGHWAY DEEP NEURAL NETWORKS
Liang Lu
Toyota Technological Institute at Chicago, USA

arXiv:1607.01963v5 [cs.CL] 22 Mar 2017

llu@ttic.edu

ABSTRACT
Highway deep neural network (HDNN) is a type of depthgated feedforward neural network, which has shown to be
easier to train with more hidden layers and also generalise
better compared to conventional plain deep neural networks
(DNNs). Previously, we investigated a structured HDNN architecture for speech recognition, in which the two gate functions were tied across all the hidden layers, and we were able
to train a much smaller model without sacrificing the recognition accuracy. In this paper, we carry on the study of this architecture with sequence-discriminative training criterion and
speaker adaptation techniques on the AMI meeting speech
recognition corpus. We show that these two techniques improve speech recognition accuracy on top of the model trained
with the cross entropy criterion. Furthermore, we demonstrate that the two gate functions that are tied across all the
hidden layers are able to control the information flow over
the whole network, and we can achieve considerable improvements by only updating these gate functions in both sequence
training and adaptation experiments.
Index Terms— Highway deep neural networks, speech
recognition, sequence training, adaptation
1. INTRODUCTION
Although they have been tremendously successful in the field
of speech processing, neural network models are usually criticised to be lack of structure, less interpretable and less adaptable. Furthermore, most neural network acoustic models
are much larger than conventional models using Gaussian
mixtures, which make it challenging to deploy these models in resource constrained platforms such as embedded devices. Recently, there have been some works to overcome
these limitations. For example, Tan et al. in [1] investigated
the stimulated learning of deep feedforward neural networks
to make them more interpretable and to gain insight about
the behaviour of those networks. On the other hand, in order to address the size of neural network acoustic models,
small-footprint neural network models have received considerable research efforts such as using low-rank matrices [2, 3],
teacher-student style training [4, 5, 6] and structured linear
This work was done at The University of Edinburgh.

layers [7, 8, 9]. Small-footprint models are superior in several aspects. Apart from requiring lower computational cost
and taking less memory, they may be more applicable for
low-resource languages, where the amount of training data
are usually much smaller. Furthermore, with smaller number
of model parameters, these models may be more adaptable to
the target domains, environments or speakers, when they are
different from the training condition.
Previously, we proposed a small-footprint acoustic model
using highway deep neural network (HDNN) [10]. HDNN
is a type of network with shortcut connections between hidden layers [11]. Compared to the plain networks with skip
connections, HDNNs are equipped with two gate functions
– transform and carry gate – to control and facilitate the information flow over all the whole network. In particular, the
transform gate is used to scale the output of a hidden layer
and the carry gate is used to pass through the input directly
after elementwise rescaling. The gate functions are the key
to train very deep networks [11] and to speed up convergence
as experimentally validated in [10]. Furthermore, for speech
recognition, the recognition accuracy can be retained by simply increasing the depth of the network, while the the number
of hidden units in each hidden layer can be significantly reduced. As a result, the networks became much thinner and
deeper with much smaller number of model parameters. In
contrast to training plain feedforward neural networks of the
same depth and width, we did not encounter any difficulty
to train these highway networks using the standard stochastic
gradient decent algorithm without pretraining [10].
However, our study was focused on the cross entropy (CE)
training of the networks previously, while in this paper, we investigate if our previous observations still hold in the case of
sequence training. To further understand the effect of the gate
functions in HDNNs, we performed the ablation experiments,
in which we disabled the update of the model parameters in
the hidden layers and/or classification layer during sequence
training. Based on the experiments using the AMI meeting
transcription corpus, we observed that by only updating the
parameters in the gate functions, we were able to retain most
of the improvement by sequence training, which supports our
argument that the gate functions can manipulate the behaviour
of all the hidden layers in the nonlinear feature extractor.
Since the number of model parameters in the gate functions is

relatively small, we then study the speaker adaptation techniques in the unsupervised fashion, in which we only fine
tune the gate functions using the speaker dependent data. Using the seed models from both CE and sequence training, we
were able to obtain consistent improvement by speaker adaptation. Overall, the small-footprint HDNN acoustic model
with 5 million model parameters achieved slightly better results compared to the DNN baseline with 30 million parameters, while the HDNN model with 2 million parameters obtained only slightly lower accuracy compared to the baseline.
2. HIGHWAY DEEP NEURAL NETWORKS

(1)

and the classifier predicts the label using a softmax function
as
ŷt = g(x̂t , θc ).

(2)

In this paper, we focus on the feedforward neural network, in
which f (·) is composed by multiple layers nonlinear transformations.
Highway deep neural networks [11] augment the feature
extractor with gate functions, in which the hidden layer may
be represented as
hl = σ(hl−1 , θl ) ◦ T (hl−1 , WT )
+ hl−1 ◦ C(hl−1 , Wc )

where Wl> is the weight matrix in the l-th layer, and we then
compute W̃l hl−1 once for all. By this trick, we can leverage on the power of GPUs on computing large matrix-matrix
multiplications efficiently in the minibatch mode, which can
speed up the training significantly.
2.1. Sequence Training

To facilitate our discussion, we divide the parameters in a
standard neural network acoustic model into two sets – θc represents the model parameters in the classifier, and θh denotes
the model parameters of the hidden layers in the feature extractor. Given an input acoustic frame xt at the time step t,
the feature extractor transforms the input into another feature
representation as
x̂t = f (xt , θh ),

functions, the training speed can still be improved if the size
of the weight matrix is smaller. Furthermore, the matrices can
be packed together as

>
W̃l = Wl> , WT> , Wc> ,
(4)

(3)

where hl denotes the hidden activations of l-th layer parameterised by θl , and σ denotes the activation function such as
sigmoid or tanh; T (·) is the transform gate that scales
the original hidden activations; C(·) is the carry gate, which
scales the input before passing it directly to the next hidden
layer; ◦ denotes elementwise multiplication; The outputs of
T (·) and C(·) are constrained to be within [0, 1], and we use
the sigmoid function for both gates that are parameterised by
WT and Wc respectively. Following our previous work [11],
we tie the parameters in the gate functions across all the hidden layers, which can significantly save model parameters.
In this work, we do not use any bias vector in the two gate
functions. Since the parameters in T (·) and C(·) are layerindependent, we denote θg = (WT , Wc ), and we will look
into the specific roles of these model parameters in sequence
training and model adaptation experiments.
Note that, although there are more computational steps for
each hidden layer compared to plain DNNs due to the gate

Our previous results of HDNNs are obtained with the CE
training criterion, where the loss function is defined as
X
L(CE) (θ) = −
yjt log ŷjt ,
(5)
j

where j is the index of the hidden Markov model (HMM)
state, and yt denotes the ground truth label (the loss function
is defined with one training utterance here for the simplicity of notation). However, state-of-the-art speech recognition
systems are usually built with sequence-training techniques,
where the loss function is defined as the sequence level. These
approaches have been well understood for neural network
acoustic models [12, 13, 14, 15]. For instance, if we denote
X as the sequence of acoustic frames X = {x1 , . . . , xT }
and Y as the sequence of labels, where T is the length of the
signal, the loss function from the scalable minimum Bayesian
risk criterion(sMBR) [16, 12] is defined as
P
p(X | W)k P (W)A(Y , Ŷ )
P
, (6)
L(sM BR) (θ) = W∈Φ
k
W∈Φ p(X | W) P (W)
where A(Y , Ŷ ) measures the state level distance between
the ground truth and predicted labels; Φ denotes the hypothesis space represented by a denominator lattice, and W is the
word-level transcription; k is the acoustic score scaling parameter. However, only applying the sequence training criterion without regularisation may lead to overfitting as observed
in [14, 15]. To address this problem, we interpolate the sMBR
loss function with the CE loss as used in [15]:
L(θ) = L(sM BR) (θ) + pL(CE) (θ),

(7)

where p ∈ R+ is the smoothing parameter1 . In this paper, we
only focus on the sMBR criterion since it can achieve comparable or slightly better results compared to the maximum mutual information (MMI) or minimum pone error (MPE) criterion [14]. In the experimental section, we also study the effect
of the regularisation term for different model parameter sets
in the highway neural network acoustic models.
1 The sequence training recipe used in this paper is adapted from the one
developed by Y. Zhang et al. at the JSALT 2015 workshop [17].

Table 1. Comparison of DNN and HDNN system with CE
and sMBR training. The DNN systems were built using Kaldi
toolkit, where the networks were pre-trained using restricted
Bolzman machines. Results are shown in terms of word error
rates (WERs). We use H to denote the size of hidden units,
and L the number of layers. M indicates million model parameters.
Model
DNN-H2048 L6
DNN-H512 L10
DNN-H256 L10
DNN-H128 L10
HDNN-H512 L10
HDNN-H256 L10
HDNN-H128 L10
HDNN-H512 L15
HDNN-H256 L15

Size
30M
4.6M
1.7M
0.71M
5.1M
1.8M
0.74M
6.4M
2.1M

eval
CE
sMBR
26.8
24.6
28.0
25.6
30.4
27.5
34.1
30.8
27.2
24.9
28.6
26.0
32.0
29.4
27.1
24.7
28.4
25.9

CE
26.0
26.8
28.4
31.5
26.0
27.2
29.9
25.8
26.9

dev
sMBR
24.3
25.1
26.5
29.3
24.5
25.2
28.1
24.3
25.2

Table 2. Results of switching off the update of different
model parameters in sequence training. θh denotes all the
model parameters of the hidden layers, θg denotes the parameters in the two gate functions, and θc is the parameters in the
softmax layer.
Model
HDNN-H512 L10

HDNN-H256 L10

HDNN-H512 L15

2.2. Adaptation
HDNN-H256 L15

Adaption of standard feedforward neural networks is challenging due to the large number of unstructured model parameters, while the amount of adaptation data is usually
much smaller. Traditional approaches include input or output layer adaptation, while recently, researchers incorporate
speaker dependent model parameters into the model space
that can manipulate the behaviour or transform the output of
the network. Techniques belong to this category may include
speaker code [18], LHUC [19] and multiple basis neural networks [20]. The HDNN architecture studied in this paper is
more structured in the sense that the parameters in the gate
functions are layer-independent, and as will be demonstrated
in the experimental section, they are able to control the behaviour of all the hidden layers. This motivates us to investigate the adaptation of highway gates by only fining tune these
model parameters. Although the number of parameters in the
gate functions are still much larger compared to the amount
of adaptation data at the per-speaker level, the size of the gate
functions is more controllable, as we can reduce the number
of hidden units without sacrificing the accuracy by increasing
the depth of the neural network [10]. Another adaptation approach is the input feature augmentation method such as using
i-vectors [21], which may be complimentary to our study, but
it is not investigated in this paper.
3. EXPERIMENTS
3.1. System Setup
Our experiments were performed on the individual headset
microphone (IHM) subset of the AMI meeting speech transcription corpus [22]. The amount of training data is around
80 hours, corresponding to roughly 28 million frames. We
used 40-dimensional fMLLR adapted features vectors nor-

sMBR Update
θh θg
θc
×
×
×
√ √
√
√
√
×
√
×
×
×
×
×
√ √
√
√
√
×
√
×
×
×
×
×
√ √
√
√
√
×
√
×
×
×
×
×
√ √
√
√
√
×
√
×
×

WER
(eval)
27.2
24.9
25.2
25.8
28.6
26.0
26.6
27.0
27.1
24.7
25.2
25.6
28.4
25.9
26.4
26.6

malised on the per-speaker level, which were then spliced by
a context window of 15 frames (i.e. ±7). The number of tied
HMM states is 3927. The HDNN models were trained using
the CNTK toolkit [23], while the results were obtained using
the Kaldi decoder [24]. We also used the Kaldi toolkit to compute the alignment and lattices for sequence training, as well
as feature transforms. We set the momentum to be 0.9 after
the 1st epoch for CE training, and we used the sigmoid activation for all the networks. The weights in each hidden layer
of HDNNs were randomly initialised with a uniform distribution in the range of [−0.5, 0.5] and the bias parameters were
initialised to be 0. We used a trigram language model for decoding. In order to make the experimental results comparable,
we used the same training and cross-validation sets split and
alignment for both HDNN and plain DNN systems. All the
systems used the same decision tree for state tying.
3.2. Sequence Training
In [10], we showed that a smaller HDNN acoustic model was
comparable to a much larger plain DNN model in terms of
the accuracy when both were trained with the CE criterion,
and it performed much better compared to DNNs of similar
size. In this experiment, we investigate if this observation still
holds after sequence training. We performed the sMBR update for 4 iterations, and we set the learning rate per frame
to be 1 × 10−5 for both CNTK and Kaldi systems. We set
the regularisation parameter p to be 0.2 in Eq. (7) to avoid
overfitting for CNTK systems, while we followed the recipe
in [14] for Kaldi systems using a slightly different regularisation technique. The baseline DNN systems were reproduced

Word Error Rate (%)

28

28
Update all, p=0.2
Update all, p=0

Update gates, p=0.2
Update gates, p=0

27

27

26

26

25

25

24

24
0

1

2

3

4

0

1

Word Error Rate (%)

H512L10 system

2

3

4

H512L10 system

30

30

28

28

26

26
Update all, p=0.2
Update all, p=0

Update gates, p=0.2
Update gates, p=0

24

24
0

1

2

3

4

H256L10 system

0

1

2

3

4

H256L10 system

Fig. 1. Convergence curves of sMBR training with and without the CE regularisation. The regularisation term can stabilise the
convergence when updating all the model parameter, while its role is diminishing when updating the gate functions only.
Table 3. Results of sMBR training with and without regularisation.
Model
HDNN-H512 L10
HDNN-H512 L10
HDNN-H256 L10
HDNN-H256 L10

sMBR Update
{θh , θg , θc }
θg
{θh , θg , θc }
θg

WER (eval)
p = 0.2 p = 0
24.9
25.0
25.8
25.3
26.0
29.1
27.0
26.8

using the up-to-date Kaldi toolkit, and we obtained slightly
better results compared to those in [10]. We did not use the
CNTK to train plain DNN models because it did not converge
with random initialisation for thin and deep networks as reported previously [10]. Table 1 shows the sequence training
results of the plain DNN and HDNN systems, from which,
we see that sequence training improves the recognition accuracy comparably for both DNN and HDNN systems, and the
improvements are consistent for both eval and dev sets as
shown by Table 1. Again, the HDNN model with around 5
million model parameters is on par with the plain DNN system with 30 million model parameters in terms of the recognition accuracy. In what follows, we only present results of
the eval set.
In the previous experiments, we updated all the model parameters in the HDNNs during sequence training. To look
into the effect of a specific parameter set, we performed a set
of ablation experiments, in which we switched off the update
of some model parameters. These results are given in Table 2,
which show that only updating the parameters in the gates θg
can retain most of the improvement given by sequence train-

ing, while updating θg and θc is close to optimum. Note that,
θg only accounts for a small fraction of the total number of parameters, e.g., ∼ 10% for the HDNN-H512 L10 system and ∼
7% for the HDNN-H256 L10 system. However, these results
demonstrate that the gate functions can largely manipulate the
behaviour of the neural network feature extractor.
We then investigated the effect of the regularisation term
in Eq. (7) for sequence training. We performed the experiments with and without the CE regularisation for two system
settings, i.e., i) update all the model parameters; ii) update
only the gate functions. Our motivation is to validate if only
updating the gate parameters is more resistant to overfitting.
The results are given in Table 3, from which we see that by
switching off the CE regularisation term, we can achieve even
slightly lower WER when updating the gate functions only.
However, when updating all the model parameters, the regularisation term turned to be an important stabiliser for convergence. Figure 1 shows the convergence curves for the
two system settings. Overall, while the gate functions can
largely control the behaviour of the highway networks, they
are not prone to overfitting when other model parameters are
switched off for update.
3.3. Unsupervised Adaptation
The observations in the sequence training experiments inspired us to study the speaker adaptation of the gate functions, because they can control the behaviour the feature extractor with relatively small number of model parameters. In
this paper, we use the term of speaker adaptation as con-

30

30
HDNN-H512L10-CE
HDNN-H256L10-CE
HDNN-H256L10-sMBR
HDNN-H512L10-sMBR

HDNN-H512L10-CE
HDNN-H256L10-CE
HDNN-H256L10-sMBR
HDNN-H512L10-sMBR

28

28

Word Error Rate (%)

Word Error Rate (%)

29

27
26
25

26

24

22

20

24

18

23
0

1

2

3

4

5

0

1

2

3

4

5

6

7

8

9

10

number of iterations

number of iterations

Fig. 2. Unsupervised adaptation results with different number of adaptation iterations. The speaker-independent models
were trained by CE or sMBR, and we used CE criterion for
all adaptation experiments.

Fig. 3. Supervised adaptation results with oracle labels.

vention, though the speaker can be defined as a cluster of
acoustic frames at any granularity. We firstly performed the
experiments in the unsupervised speaker adaptation setting,
in which we decoded the evaluation set using the speakerindependent models, and then used the pseudo labels to fine
tune the parameters in θg in the second pass. The evaluation
set has around 8.6 hours of audio, and the number of speakers
is 63. On average, each speaker has around 8 minutes speech,
which corresponds to about 50 thousand frames. Compared to
the size of θg in HDNNs, the amount of the adaptation data is
still small, e.g., the size of θg in the HDNN-H512 L10 system
is around 0.5 million. We set the learning rate to be 2 × 10−4
per sample, and we updated θg for 5 iterations.
Table 4 shows the adaptation results, from which we
observe small but consistent WER reduction with different
model configurations on top of the speaker adapted features
using fMLLR. Notably, the improvements are consistent for
both seed models, where the speaker-independent models are
trained using either the CE or the sMBR criterion. Updating all the model parameters yields smaller improvements as
shown by the table. With speaker adaptation and sequence
training, the HDNN system with 5 million model parameters
(HDNN-H512 L10 ) works slightly better than the DNN baseline with 30 million parameters (24.1% vs. 24.6%), while the
HDNN model with 2 million parameters (HDNN-H256 L10 )
achieves only slightly higher WER compared to the baseline
(25% vs. 24.6%). In Figure 2 we show the adaptation results with different number of iterations. We observe that the
best results can be achieved by only 2 or 3 adaptation iterations, thought updating the gate functions θg further does
not yield overfitting. To further validate this, we also did experiments with 10 adaptation iterations, but we still did not
observe overfitting. This observation is in line with the that
in the sequence training experiments, demonstrating that the

Table 4. Results of unsupervised speaker adaptation. Here,
we only updated θg using the CE criterion, while the speakerindependent (SI) models were trained by either CE or sMBR.
SD denotes speaker-dependent models.
Model
HDNN-H512 L10
HDNN-H256 L10
HDNN-H512 L15
HDNN-H256 L15
HDNN-H512 L10
HDNN-H256 L10
HDNN-H512 L15
HDNN-H256 L15
HDNN-H512 L10
HDNN-H256 L10

Seed

Update

CE
θg

sMBR
{θh , θg , θc }

WER (eval)
SI
SD
27.2
26.5
28.6
27.9
27.1
26.4
28.4
27.6
24.9
24.1
26.0
25.0
24.7
24.0
25.9
24.9
24.9
24.5
26.0
25.4

gate functions are relatively resistant to overfitting.
3.4. Adaptation with Oracle Labels
In the previous experiments, we studied the unsupervised
adaptation condition, in which we obtained the labels for
adaptation from the first-pass decoding. In order to evaluate the impact of the accuracy of the labels to this adaptation method, we performed a set of diagnostic experiments,
in which we used the oracle labels for adaptation. We obtained the oracle labels from the force alignment using the
DNN model trained with the CE criterion and word level transcriptions. We have also fixed the alignment for all the adaptation experiments in order to compare the results from different seed models. Figure 3 shows the adaptation results with
oracle labels, which demonstrates that significant WER reduction can be achieved when the supervision labels are more
accurate. Therefore, the gate functions may have large capacity for adaptation with high quality pseudo labels. To further
study this aspect, in the future, we shall investigate supervised
adaptation of highway networks.

4. CONCLUSIONS
Highway deep neural networks are structured, depth-gated
feedforward neural networks. In this paper, we studied sequence training and adaptation of these networks for acoustic modelling. In particular, we investigated the roles of the
parameters in the hidden layers, gate functions and classification layer in the case of sequence training. We show that the
gate functions, which only accounts for a small fraction of
the whole parameter set, are able to control the information
flow and adjust the behaviour of the neural network feature
extractors. We demonstrated this in both sequence training
and adaptation experiments, in which, considerable improvements were achieved by only updating the gate functions.
By this two techniques, we obtained comparable or slightly
lower WERs with much smaller acoustic models compared
to a strong baseline set by the conventional DNN acoustic
model with sequence training. Since the number of model parameters are still relative large compared to the speaker-level
adaptation data, this adaptation technique may be more applicable in the domain adaptation scenarios, where the amount
of adaptation data is relatively large. In the future, we shall
also investigate the model compression techniques to further
improve the results of our small-footprint acoustic models.
5. REFERENCES
[1] S. Tan, K. C. Sim, and M. Gales, “Improving the interpretability of deep neural networks with stimulated learning,” in Proc.
ASRU. IEEE, 2015, pp. 617–623.
[2] J. Xue, J. Li, and Y. Gong, “Restructuring of deep neural network acoustic models with singular value decomposition.” in
Proc. INTERSPEECH, 2013, pp. 2365–2369.
[3] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and
B. Ramabhadran, “Low-rank matrix factorization for deep neural network training with high-dimensional output targets,” in
Proc. ICASSP. IEEE, 2013, pp. 6655–6659.
[4] J. Li, R. Zhao, J.-T. Huang, and Y. Gong, “Learning small-size
dnn with output-distribution-based criteria,” in Proc. INTERSPEECH, 2014.
[5] J. Ba and R. Caruana, “Do deep nets really need to be deep?”
in Proc. NIPS, 2014, pp. 2654–2662.
[6] R. Adriana, B. Nicolas, K. Samira Ebrahimi, C. Antoine,
G. Carlo, and B. Yoshua, “Fitnets: Hints for thin deep nets,”
in Proc. ICLR, 2015.
[7] Q. Le, T. Sarlós, and A. Smola, “Fastfood-approximating kernel expansions in loglinear time,” in Proc. ICML, 2013.
[8] V. Sindhwani, T. N. Sainath, and S. Kumar, “Structured transforms for small-footprint deep learning,” in Proc. NIPS, 2015.
[9] M. Moczulski, M. Denil, J. Appleyard, and N. de Freitas,
“ACDC: A Structured Efficient Linear Layer,” in Proc. ICLR,
2016.
[10] L. Lu and S. Renals, “Small-footprint deep neural networks with highway connections for speech

recognition,” in Proc. INTERSPEECH, 2016. [Online]
http://arxiv.org/pdf/1512.04280v3.pdf.
[11] R. K. Srivastava, K. Greff, and J. Schmidhuber, “Training very
deep networks,” in Proc. NIPS, 2015.
[12] B. Kingsbury, “Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling,” in Proc.
ICASSP. IEEE, 2009, pp. 3761–3764.
[13] B. Kingsbury, T. N. Sainath, and H. Soltau, “Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization,” in Proc. INTERSPEECH, 2012.
[14] K. Veselý, A. Ghoshal, L. Burget, and D. Povey, “Sequencediscriminative training of deep neural networks,” in Proc. INTERSPEECH, 2013.
[15] H. Su, G. Li, D. Yu, and F. Seide, “Error back propagation
for sequence training of context-dependent deep networks for
conversational speech transcription,” in Proc. ICASSP. IEEE,
2013, pp. 6664–6668.
[16] M. Gibson and T. Hain, “Hypothesis spaces for minimum
bayes risk training in large vocabulary speech recognition.” in
Proc. INTERSPEECH. Citeseer, 2006.
[17] Y. Zhang, G. Chen, D. Yu, K. Yao, S. Khudanpur, and
J. Glass, “Highway Long Short-Term Memory RNNs for Distant Speech Recognition,” Proc. ICASSP, 2015.
[18] O. Abdel-Hamid and H. Jiang, “Fast speaker adaptation of hybrid nn/hmm model for speech recognition based on discriminative learning of speaker code,” in Proc. ICASSP. IEEE,
2013, pp. 7942–7946.
[19] P. Swietojanski and S. Renals, “Learning hidden unit contributions for unsupervised speaker adaptation of neural network
acoustic models,” in Proc. SLT. IEEE, 2014, pp. 171–176.
[20] C. Wu and M. J. Gales, “Multi-basis adaptive neural network
for rapid adaptation in speech recognition,” in Proc. ICASSP.
IEEE, 2015, pp. 4315–4319.
[21] G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, “Speaker
adaptation of neural network acoustic models using i-vectors.”
in Proc. ASRU, 2013, pp. 55–59.
[22] S. Renals, T. Hain, and H. Bourlard, “Recognition and understanding of meetings the AMI and AMIDA projects,” in Proc.
ASRU. IEEE, 2007, pp. 238–247.
[23] D. Yu, A. Eversole, M. Seltzer, K. Yao, Z. Huang, B. Guenter,
O. Kuchaiev, Y. Zhang, F. Seide, H. Wang et al., “An introduction to computational networks and the computational network
toolkit,” Tech. Rep. MSR, Microsoft Research, Tech. Rep.,
2014.
[24] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek,
N. Goel, M. Hannemann, P. Motlıcek, Y. Qian, P. Schwarz,
J. Silovský, G. Semmer, and K. Veselý, “The Kaldi speech
recognition toolkit,” in Proc. ASRU, 2011.

