Phonemic and Graphemic Multilingual CTC Based Speech Recognition
Markus Müller1 , Sebastian Stüker1 , Alex Waibel1,2
1

Interactive Systems Lab, Institute for Anthropomatics and Robotics
Karlsruhe Institute of Technology, Karlsruhe, Germany
2
Carnegie Mellon University, Pittsburgh PA, USA

arXiv:1711.04564v1 [eess.AS] 13 Nov 2017

m.mueller@kit.edu

Abstract
Training automatic speech recognition (ASR) systems requires large amounts of data in the target language in order
to achieve good performance. Whereas large training corpora
are readily available for languages like English, there exists
a long tail of languages which do suffer from a lack of resources. One method to handle data sparsity is to use data
from additional source languages and build a multilingual
system. Recently, ASR systems based on recurrent neural
networks (RNNs) trained with connectionist temporal classification (CTC) have gained substantial research interest. In
this work, we extended our previous approach towards training CTC-based systems multilingually. Our systems feature
a global phone set, based on the joint phone sets of each
source language. We evaluated the use of different language
combinations as well as the addition of Language Feature
Vectors (LFVs). As contrastive experiment, we built systems based on graphemes as well. Systems having a multilingual phone set are known to suffer in performance compared to their monolingual counterparts. With our proposed
approach, we could reduce the gap between these mono- and
multilingual setups, using either graphemes or phonemes.

1. Introduction
Automatic speech recognition systems have matured dramatically in recent years, lately with reported recognition accuracies similar to those of humans on certain tasks [1, 2]. A
large amount of carefully prepared training data is required to
achieve this level of performance. While such data is available for well-researched and -resourced languages like English, there exists a long tail of languages for which such
training material does not exist. Various methods have been
proposed to handle data sparsity. In this work, we focus on
multilingual systems: A common approach is to incorporate
data from supplementary source languages in addition to data
from the target language.
Lately, systems based on RNNs trained with connectionist temporal classification (CTC) [3] have become popular.
In this work we focus on building multilingual RNN/CTC
systems, instead of systems based on either GMM/HMM
or DNN/HMM, with the goal of applying them in a multilingual manner and are planning crosslingual experiments in

the future. For this future crosslingual case, the multilingual
RNN acts as a network that can be adapted to multiple languages for which only very little adaptation data is available.
In the multilingual scenario of this paper, we have one multilingual model that is able to recognize speech from multiple
languages simultaneously, while for all languages a comparatively large amount of training data is available. This is particular useful in environments with fast language changes.
Recently, we demonstrated the use of a second language
in addition to the target language when building a phoneme
based CTC system [4]. We now extend this approach by using data from up to 4 languages (English, French, German
and Turkish). Building systems using phones as acoustic
modeling unit requires a pronunciation dictionary. But, creating these dictionaries is a time-consuming, resource intense
process and often a bottle-neck when building speech recognition systems for new languages. While automatic methods to create pronunciations for new words given an existing
dictionary exist [5, 6], such approaches are based on an existing seed dictionary. Using graphemes as acoustic modeling units, instead has the advantage of loosing the need
for a pronunciation dictionary at the cost that graphemes
might not always be a good modeling unit, depending on
the grapheme-to-phoneme relation of the target language.
[7, 8, 9] This is particularly challenging in a multilingual setting, because different languages, although they might share
the same writing system, do feature different pronunciation
rules [10, 11, 12].
This paper is organized as follows: Next, in Section (2),
we provide an overview of related work in the field. In Section 3, we describe our proposed approach, followed by the
experimental setup in Section 4. The results are presented in
Section 5. This paper concludes with Section 6, where we
also provide an outlook to future work.

2. Related Work
2.1. Multi- and Crosslingual Speech Recognition Systems
Using GMM/HMM based systems was considered state of
the art prior to the emergence of systems with neural networks. Data sparsity has been addressed in the past, by

training systems multi- and crosslingually [13, 14]. Methods for crosslingual adaptation exist [15], but also methods
for adapting the cluster tree were proposed [16]. Traditional
systems typically use context-dependent phones. When
trained multi- or crosslingually, the clustering of phones into
context-dependent phones needs to be adapted [17].
But when using an RNN, the system is trained on
context-independent targets, so that in the multilingual case
this kind of adaptation is unnecessary, as the network learns
the context-dependency during training.
2.2. Multilingual Bottleneck Features
Deep Neural Networks (DNNs) are a data-driven method
with many parameters to be trained, failing to generalize
if trained on only a limited data set. Different methods
have been proposed to train networks on data from multiple source languages. Training DNNs typically involves a
pre-training and a fine-tuning step. It has been shown, that
the pre-training is language independent [18]. Several approaches exist to fine-tune a network using data from multiple languages. One method is to share hidden layers between languages, but to use language specific output layers
[19, 20, 21, 22]. Combining language specific output layers into one layer is also possible [23]. By dividing the output layer into language specific blocks, the setup uses language dependent phone sets. Training DNNs simultaneously
on data from multiple languages on the other hand can then
be considered a form of multi-task learning [24, 25].

Given enough training data, even training on whole words is
possible [34]. Multi-task learning has also been proposed
[35, 36, 37]. CTC based systems are able to outperform
HMM based setups on certain tasks [38].

3. Language Adaptive Multilingual CTC
Based Systems
Traditional speech recognition systems typically rely on a
pronunciation dictionary which maps words to phone sequences. It is also possible to train systems on graphemes
as acoustic units, but this affects the performance depending on the language. While there are languages with a close
mapping between letters and sounds, e.g., Spanish, this does
not hold for every language. Pronunciation rules are quite
complex, with groups of characters being mapped to different sounds based on their context. An example of such complex mappings would be English. The string “ough” has 8
different acoustic realizations, depending on the context, as
in, e.g., “rough”, “ought” or “through”.
3.1. Multilingual Systems

By supplying additional input features, neural networks can
be adapted to various conditions. One of the most common
methods is to adapt neural nets to different speakers by providing a low dimensional code representing speaker characteristics. These so called i-Vectors [26] allow to train speaker
adaptive neural networks [27]. An alternative method for
adaptation are Bottleneck Speaker Vectors (BSVs) [28].
Similar to BSVs, we proposed an adaptation method for
adapting neural networks to different languages when trained
on multiple languages. We first proposed using the language
identity information via one-hot encoding [29]. One of the
shortcomings of this approach is that it does not supply language characteristics to the network. To address this issue, we proposed Language Feature Vectors (LFVs) [30, 31]
which have shown to encode language properties, even if the
LFV net was not trained on the target language.

Speech recognition systems are typically built to recognize
speech of a single language. Training traditional systems
multilingually involves a hybrid DNN/HMM setup where the
hidden layers of the DNN are shared between languages and
the output layers are kept language dependent. Such systems can be seen as individual, language dependent systems,
trained jointly. Training language universal systems using
a global phones set is possible, however HMM based systems do not generalize well when being trained on multiple languages. In this work, we propose an approach using
RNN based systems trained using CTC on data from multiple languages, with a global set of units modeling the acoustics (graphemes or phones). The main advantage of such a
system is the ability to recognize speech from multiple languages simultaneously, without knowledge of the input language’s identity.
In the past, we proposed a setup for training CTC-based
systems multilingually using a universal phone set [4]. In
this work, we extended our previous work in three ways: 1)
we increased the number of languages used 2) we used multilingually trained bottleneck features (BNFs) 3) in addition
to phones, we evaluated the use of graphemes. In the past,
we demonstrated the use of LFVs using DNN/HMM-based
systems for multilingual speech recognition. We now apply
this technique to CTC-based speech recognition.

2.4. CTC Based ASR Systems

3.2. Language Feature Vectors

Recently, RNN-based systems trained using the CTC loss
function [3] have become popular. Similar to traditional ASR
systems, CTC based ones are trained using either phones,
graphemes, or both [32]. Training on units larger than characters is also possible [33]. This method, called Byte Pair
Encoding (BPE), derives larger units based on the transcripts.

LFVs are a low dimensional representation of language properties, extracted using a neural network. The setup consisted
of two networks, Figure 1 shows the network architecture.
The first network was used to extract BNFs from acoustic
input features. It was trained using a combination of lMel
and tonal features as input and phone states as targets. The

2.3. Neural Network Adaptation

second network was trained for language identification using
BNFs as input features. In contrast to networks trained for
speech recognition, we used a much larger input context because of the language information being long-term in nature.
This network was trained to detect languages and featured a
bottleneck layer, which was used to extract the LFVs after
training.
3.3. Input Features
Using BNFs as input features is common for traditional
speech recognition systems. By forcing the information
to pass through a bottleneck, the network creates a lowdimensional representation of features relevant to discriminate between phones. DNN/HMM or GMM/HMM based
systems benefit from using such features over plain features
like, e.g., MFCCs. We evaluated training our CTC systems
on multilingual BNFs.

plying all filtering steps, approximately 50h of data per language was available. We split the available data on a speaker
basis into a 45h training and 5h test set.
4.2. Acoustic Units
We conducted experiments using both phones and graphemes
as acoustic units. As graphemes we used the provided transcripts, while we used MaryTTS [45] to generate a pronunciation dictionary automatically to map words to phones. In
addition, we included a marker to indicate word boundaries.
4.3. Input Features

We built our systems using a framework based on PyTorch
[40], as well as warp-ctc [41] for computing the CTC loss
during network training. To extract acoustic features from
the data, we used the Janus Recognition Toolkit (JRTk) [42],
which features the IBIS single-pass decoder [43].

As input features, we used log Mel and tonal features (FFV
[46] and pitch [47]), extracted using a 32ms window with
a 10ms frame-shift. We included tonal features as part of
our standard pre-processing pipeline because previous experiments showed a reduction in the word error rate (WER) of
speech recognition systems, even if the language is not tonal
[48].
Based on these features, we trained a network for extracting multilingual bottleneck features (BNFs). The network featured 5 feed-forward layers, with 1,000 neurons per
layer, with the second last layer being the bottleneck with
only 42 neurons. The acoustic features were fed with a context of +/− 6 frames into the network. While the hidden
layers were shared between languages, we used language dependent output layers. 6,000 context-dependent phone states
were used as targets, with data from 5 languages (French,
German, Italian, Russian, Turkish). To obtain phone state
labels, DNN/HMM systems for each language were trained.
After training, all layers after the bottleneck were discarded
and the output activations of this layer were taken as BNFs.

4.1. Dataset

4.4. LFV Network Training

We conducted our experiments using data from the Euronews
Corpus [44], a dataset containing recordings of TV broadcast
news from 10 different languages (Arabic, English, French,
German, Italian, Polish, Portuguese, Russian, Spanish, Turkish), with orthographic transcripts at utterance level. The advantage of this dataset is that the channel conditions do not
differ between languages, ensuring that we are adapting our
systems to different languages instead of different channel
conditions, like, e.g., different environmental noises present
in different languages. We filtered the available data, retaining only utterances with a length of at least 1s and a transcript
length of at most 639 symbols, because of an internal limitation within CUDA1 .
Noises were annotated in a very basic way, consisting
of only one generic noise marker covering both human and
non-human noises. With noises accounting for a quite large
amount of utterances, we only selected a small subset of them
to account for a more balanced set of training data. After ap-

Training the network for the extraction of LFVs is a two step
process. First, BNFs are being trained (see Section 4.3), and
then based on these BNFs, a second network is trained to
recognize the language. This network features 6 layers with
1,600 neurons per layer, except for the bottleneck layer with
only 42 neurons. In contrast to networks trained for speech
recognition, this network featured a large context spanning
+/− 33 frames. To reduce the dimensionality of the input,
only every third frame was taken. For training this network,
we used data from 9 languages( all available languages in the
corpus except English).

3.4. Network Architecture
The network architecture chosen was based on Baidu’s Deepspeech2 [39]. As shown in Figure 2, the network consists of
two TDNN / CNN layers. We add LFVs to the output of
the second TDNN / CNN layer as input to the bi-directional
LSTM layers. We use a feed-forward output layer to map the
output of the last LSTM layer to the targets.

4. Experimental Setup

1 see:

https://github.com/baidu-research/warp-ctc, accessed 2017-10-09

4.5. CTC RNN Network Training
The RNN network was trained using either log Mel / tonal
features or BNFs. As targets, we used both graphemes
and phonemes as acoustic units, with an additional symbol
added for separating words. The networks were trained using
stochastic gradient descent (SGD) with Nesterov momentum
[49] of 0.9 and a learning rate of 0.0003. Mini-batch updates

AF stack

LFV

BNF stack

BNF

Language Feature Network
Figure 1: Overview of the network architecture used to extract language feature vectors (LFV). The acoustic features (AF) are
being pre-processed in a DBNF in order to extract BNFs. These BNFs are being stacked and fed into the second network to
extract LFVs.
obtained with it should indicate whether the improvements in
TER of the pure CTC model measured on English also lead
to a better word level speech recognition system.

LFV

5. Results
We first evaluated using multilingual BNFs over plain log
Mel / tone features. Next, we used multilingual BNFs to
train systems using a combination of 4 languages (English,
French, German, Turkish).
2D Convolution
Layers

Bi-directional LSTM Layers

Output
Layer

Figure 2: Network layout, based on Baidu’s Deepspeech2
[39]. LFVs are being added after the final convolution layer.
with a batch size of 20 and batch normalization were used.
During the first epoch, the network was trained with utterances sorted ascending by length to stabilize the training, as
shorter utterances are easier to align.

5.1. Multilingual BNFs
First, we evaluated the use of multilingually trained BNFs as
input features. To assess the performance, we trained systems for English and German monolingually on all available
data. The results are shown in Table 1. The gain by the addition of BNFs is larger for German which can be explained by
German being among the languages the BNF net was trained
on (see Section 4.3). But the BNFs also show an improvement for English, although they did not see this language
during training.

4.6. Evaluation

Condition

To evaluate our setup, we used the same decoding procedure
as in [3] and greedily search the best path without an external
language model and evaluated our systems by computing the
token error rate (TER) as primary measure. In addition, we
trained a character based neural network language model for
English on the training utterances, as described in [50], so
that for the recognition of English we could also measure
a word error rate (WER) by decoding the network outputs
with this language model. As the language model is only
trained on only a small amount of data, the word error rate

log Mel + Tone
ML BNF

English TER

German TER

13.0%
10.2%

10.8%
7.8%

Table 1: Comparison of using ML-BNFs over log Mel + tone
features

5.2. Multilingual Phoneme Based Systems
Next, we evaluated the performance using 4 languages (English, French, German, Turkish). We evaluated adding the

LFVs after the TDNN / CNN layers. As baseline, we did
not apply our language adaptation technique and used only
multilingual BNFs. As shown in Table 2, adding LFVs after
the TDNN / CNN layer shows improvements over the baseline. The relative improvements vary and while the language
adapted systems are not en par with the monolingual ones,
the adaptation does decrease the gap between the multi- and
monolingual setup.
Condition

DE

EN

FR

TR

Monolingual

7.8%

10.2%

8.3%

7.1%

ML
ML + LFV

9.9%
8.9%

14.1%
12.9%

12.8%
10.7%

8.4%
7.6%

Table 2: Term Error Rate (TER) of multilingual (ML)
phoneme CTC based systems, trained on 4 languages.

5.3. Multilingual Grapheme Based Systems
In addition to using phones, we also evaluated the performance using only the transcripts, without a pronunciation
dictionary. As shown in Table 3, using LFVs improves
the performance in this condition as well. For English and
French, the TER is higher compared to their phoneme counterpart, whereas lower TERs could be observed for both German and Turkish. One explanation could be that English
and French feature more complex pronunciation rules that
are better reflected by MaryTTS’ language definitions. The
generated pronunciations for German and Turkish appear to
worsen the performance. The RNN seems to capture the letter to sound rules for these languages better.
Condition

DE

EN

FR

TR

Monolingual

7.5%

12.9%

11.5%

6.6%

ML
ML + LFV

9.1%
7.9%

15.6%
14.3%

13.4%
12.5%

7.9%
7.3%

Table 3: Term Error Rate (TER) of multilingual (ML)
grapheme CTC based systems, trained on 4 languages.
For English, we also trained a basic character based language model to decode the network output and compute the
WER. As shown in Table 4, similar improvements can be
observed by adding LFVs.
Condition

Mono

ML

ML + LFV

English

25.2%

30.8%

28.1%

Table 4: Word Error Rate (WER) of English phoneme CTC
based systems. Adding LFVs improves the multilingual performance.

6. Conclusion
We have presented an approach to adapt recurrent neural networks to multiple languages. Using multilingual BNFs improved the performance, as well as providing LFVs for language adaptation. These language adaptive networks are able
to capture language specific peculiarities in a multilingual
setup which results in an increased performance. Such multilingual systems are able to recognize speech from multiple
languages simultaneously.
Future work includes the use of different language
combinations and working towards cross-lingual knowledge
transfer. We aim at further closing the gap between monoand multilingual systems using additional adaptation techniques.

7. References
[1] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer,
A. Stolcke, D. Yu, and G. Zweig, “Achieving Human
Parity in Conversational Speech Recognition,” arXiv
preprint arXiv:1610.05256, 2016.
[2] ——, “The microsoft 2016 conversational speech
recognition system,” in Acoustics, Speech and Signal
Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 2017, pp. 5255–5259.
[3] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling
unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning. ACM, 2006, pp. 369–
376.
[4] M. Müller, S. Stüker, and A. Waibel, “Multilingual ctc
speech recognition,” in SPECOM, 2017.
[5] M. Bisani and H. Ney, “Joint-sequence models for
grapheme-to-phoneme conversion,” Speech communication, vol. 50, no. 5, pp. 434–451, 2008.
[6] J. R. Novak, D. Yang, N. Minematsu, and K. Hirose,
“Phonetisaurus: A wfst-driven phoneticizer,” The University of Tokyo, Tokyo Institute of Technology, pp. 221–
222, 2011.
[7] C. Schillo, G. A. Fink, and F. Kummert, “Grapheme
based speech recognition for large vocabularies,” in
Proceedings of the Sixth International Conference on
Spoken Language Processing (ICSLP 2000). Beijing,
China: ISCA, October 2000, pp. 584–587.
[8] S. Kanthak and H. Ney, “Context-dependent acoustic
modeling using graphemes for large vocabulary speech
recognition,” in Proceedings the 2002 IEEE International Conference on Acoustics, Speech, and Signal
Processing (ICASSP’02), vol. 1.
Orlando, Florida,
USA: IEEE, 2002, pp. 845–848.

[9] M. Killer, S. Stüker, and T. Schultz, “Grapheme based
speech recognition,” in Proceedings of the 8th European Conference on Speech Communication and Technology EUROSPEECH’03.
Geneva, Switzerland:
ISCA, September 2003, pp. 3141–3144.
[10] S. Kanthak and H. Ney, “Multilingual acoustic modeling using graphems,” in Proceedings of the 8th European Conference on Speech Communication and Technology EUROSPEECH’03.
Geneva, Switzerland:
ISCA, September 2003, pp. 1145–1148.
[11] S. Stüker, “Modified polyphone decision tree specialization for porting multilingual grapheme based asr systems to new languages,” in Proceedings of the 2008
IEEE International Conference on Acoustics, Speech,
and Signal Processing. Las Vegas, NV, USA: IEEE,
April 2008, pp. 4249–4252.
[12] ——, “Integrating thai grapheme based acoustic models into the ml-mix framework - for language independent and cross-language asr,” in Proceedings of
the First International Workshop on Spoken Languages
Technologies for Under-resourced languages (SLTU),
Hanoi, Vietnam, May 2008.
[13] B. Wheatley, K. Kondo, W. Anderson, and
Y. Muthusamy, “An evaluation of cross-language
adaptation for rapid hmm development in a new
language,” in Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., 1994 IEEE International
Conference on, vol. 1. IEEE, 1994, pp. I–237.
[14] T. Schultz and A. Waibel, “Fast bootstrapping of
lvcsr systems with multilingual phoneme sets.” in Eurospeech, 1997.
[15] S. Stüker, “Acoustic modelling for under-resourced languages,” Ph.D. dissertation, Karlsruhe, Univ., Diss.,
2009, 2009.
[16] T. Schultz and A. Waibel, “Language-independent and
language-adaptive acoustic modeling for speech recognition,” Speech Communication, vol. 35, no. 1, pp. 31–
51, 2001.
[17] ——, “Polyphone decision tree specialization for language adaptation,” in Acoustics, Speech, and Signal Processing, 2000. ICASSP’00. Proceedings. 2000
IEEE International Conference on, vol. 3. IEEE, 2000,
pp. 1707–1710.

[20] S. Scanzio, P. Laface, L. Fissore, R. Gemello, and
F. Mana, “On the use of a multilingual neural network
front-end,” in Proceedings of the Interspeech, 2008, pp.
2711–2714.
[21] G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen,
M. Ranzato, M. Devin, and J. Dean, “Multilingual
Acoustic Models Using Distributed Deep Neural Networks,” in Proceedings of the ICASSP, Vancouver,
Canada, May 2013.
[22] K. Vesely, M. Karafiat, F. Grezl, M. Janda, and
E. Egorova, “The language-independent bottleneck features,” in Proceedings of the Spoken Language Technology Workshop (SLT), 2012 IEEE. IEEE, 2012, pp.
336–341.
[23] F. Grézl, M. Karafiát, and K. Vesely, “Adaptation of
multilingual stacked bottle-neck neural network structure for new language,” in Acoustics, Speech and Signal
Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 7654–7658.
[24] R. Caruana, “Multitask learning,” Machine learning,
vol. 28, no. 1, pp. 41–75, 1997.
[25] A. Mohan and R. Rose, “Multi-lingual speech recognition with low-rank multi-task deep neural networks,”
in Acoustics, Speech and Signal Processing (ICASSP),
2015 IEEE International Conference on. IEEE, 2015,
pp. 4994–4998.
[26] G. Saon, H. Soltau, D. Nahamoo, and M. Picheny,
“Speaker Adaptation of Neural Network Acoustic
Models Using i-Vectors,” in ASRU. IEEE, 2013, pp.
55–59.
[27] Y. Miao, H. Zhang, and F. Metze, “Towards Speaker
Adaptive Training of Deep Neural Network Acoustic
Models,” 2014.
[28] H. Huang and K. C. Sim, “An Investigation of Augmenting Speaker Representations to Improve Speaker
Normalisation for DNN-based Speech Recognition,” in
ICASSP. IEEE, 2015, pp. 4610–4613.
[29] M. Müller and A. Waibel, “Using Language Adaptive Deep Neural Networks for Improved Multilingual
Speech Recognition,” IWSLT, 2015.

[18] P. Swietojanski, A. Ghoshal, and S. Renals, “Unsupervised cross-lingual knowledge transfer in DNN-based
LVCSR,” in SLT, IEEE. IEEE, 2012, pp. 246–251.

[30] M. Müller, S. Stüker, and A. Waibel, “Language Adaptive DNNs for Improved Low Resource Speech Recognition,” in Interspeech, 2016.

[19] A. Ghoshal, P. Swietojanski, and S. Renals, “Multilingual training of Deep-Neural networks,” in Proceedings
of the ICASSP, Vancouver, Canada, 2013.

[31] ——, “Language Feature Vectors for Resource Constraint Speech Recognition,” in Speech Communication; 12. ITG Symposium; Proceedings of. VDE, 2016.

[32] D. Chen, B. Mak, C.-C. Leung, and S. Sivadas,
“Joint acoustic modeling of triphones and trigraphemes
by multi-task learning deep neural networks for lowresource speech recognition,” in Acoustics, Speech and
Signal Processing (ICASSP), 2014 IEEE International
Conference on. IEEE, 2014, pp. 5592–5596.
[33] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,”
arXiv preprint arXiv:1508.07909, 2015.
[34] H. Soltau, H. Liao, and H. Sak, “Neural speech
recognizer: Acoustic-to-word lstm model for large
vocabulary speech recognition,” arXiv preprint
arXiv:1610.09975, 2016.
[35] S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention
based end-to-end speech recognition using multi-task
learning,” arXiv preprint arXiv:1609.06773, 2016.
[36] L. Lu, L. Kong, C. Dyer, and N. A. Smith, “Multi-task
learning with ctc and segmental crf for speech recognition,” arXiv preprint arXiv:1702.06378, 2017.
[37] H. Sak and K. Rao, “Multi-accent speech recognition
with hierarchical grapheme based models,” 2017.
[38] Y. Miao, M. Gowayyed, X. Na, T. Ko, F. Metze, and
A. Waibel, “An empirical exploration of ctc acoustic models,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on.
IEEE, 2016, pp. 2623–2627.
[39] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai,
E. Battenberg, C. Case, J. Casper, B. Catanzaro,
Q. Cheng, G. Chen, et al., “Deep speech 2: End-to-end
speech recognition in english and mandarin,” in International Conference on Machine Learning, 2016, pp.
173–182.
[40] “PyTorch,” http://pytorch.org, accessed: 2017-04-13.
[41] “warp-ctc,” https://github.com/baidu-research/warpctc, accessed: 2017-04-13.
[42] M. W. et al., “JANUS 93: Towards Spontaneous Speech
Translation,” in International Conference on Acoustics,
Speech, and Signal Processing 1994, Adelaide, Australia, 1994.
[43] H. Soltau, F. Metze, C. Fugen, and A. Waibel, “A
One-Pass Decoder Based on Polymorphic Linguistic
Context Assignment,” in Automatic Speech Recognition and Understanding, 2001. ASRU’01. IEEE Workshop on. IEEE, 2001, pp. 214–217.
[44] R. Gretter, “Euronews: A Multilingual Benchmark
for ASR and LID,” in Fifteenth Annual Conference of
the International Speech Communication Association,
2014.

[45] M. Schröder and J. Trouvain, “The German text-tospeech synthesis system MARY: A tool for research,
development and teaching,” International Journal of
Speech Technology, vol. 6, no. 4, pp. 365–377, 2003.
[46] K. Laskowski, M. Heldner, and J. Edlund, “The Fundamental Frequency Variation Spectrum,” in Proceedings of the 21st Swedish Phonetics Conference (Fonetik
2008), Gothenburg, Sweden, June 2008, pp. 29–32.
[47] K. Schubert, “Grundfrequenzverfolgung und deren Anwendung in der Spracherkennung,” Master’s thesis,
Universität Karlsruhe (TH), Germany, 1999, in German.
[48] F. Metze, Z. Sheikh, A. Waibel, J. Gehring, K. Kilgour, Q. B. Nguyen, V. H. Nguyen, et al., “Models
of Tone for Tonal and Non-tonal Languages,” in Automatic Speech Recognition and Understanding (ASRU),
2013 IEEE Workshop on. IEEE, 2013, pp. 261–266.
[49] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On
the importance of initialization and momentum in deep
learning,” in Proceedings of the 30th International
Conference on Machine Learning (ICML-13), 2013,
pp. 1139–1147.
[50] T. Zenkel, R. Sanabria, F. Metze, J. Niehues, M. Sperber, S. Stüker, and A. Waibel, “Comparison of decoding strategies for ctc acoustic models,” arXiv preprint
arXiv:1708.04469, 2017.

