Leveraging Native Language Speech for Accent Identification using Deep Siamese Networks
Aditya Siddhant Preethi Jyothi Sriram Ganapathy
cs.CL cs.LG cs.SD eess.AS



Android Malware Detection using Deep Learning on API Method Sequences
    ElMouatez Billah Karbab, Mourad Debbabi, Abdelouahid Derhab, Djedjiga Mouheb
    December 30, 2023
================================================================================



The problem of automatic accent identification is important for several applications like speaker profiling and recognition as well as for improving speech recognition systems. The accented nature of speech can be primarily attributed to the influence of the speaker's native language on the given speech recording. In this paper, we propose a novel accent identification system whose training exploits speech in native languages along with the accented speech. Specifically, we develop a deep Siamese network based model which learns the association between accented speech recordings and the native language speech recordings. The Siamese networks are trained with i-vector features extracted from the speech recordings using either an unsupervised Gaussian mixture model (GMM) or a supervised deep neural network (DNN) model. We perform several accent identification experiments using the CSLU Foreign Accented English (FAE) corpus. In these experiments, our proposed approach using deep Siamese networks yield significant relative performance improvements of 15.4% on a 10-class accent identification task, over a baseline DNN-based classification system that uses GMM i-vectors. Furthermore, we present a detailed error analysis of the proposed accent identification system. 



Accent identification, i-vectors, Deep Siamese networks, Multi-lingual modeling.





§ INTRODUCTION



Over the recent years, many of voice-driven technologies have achieved significant robustness needed for mass deployment.
 This is largely due to significant advances in automatic speech
recognition (ASR) technologies and deep learning algorithms. However, the variability in speech accents pose
a significant challenge to state-of-the-art speech systems.  In
particular, large sections of the English-speaking population in the world
face difficulties interacting with voice-driven agents in English due to
the mis-match in speech accents seen in the training data. The accented nature of speech can be primarily
attributed to the influence of the speaker's native language. In this work
we focus on the problem of accent identification, where the user's
native language is automatically determined from their non-native speech. This
can be viewed as a first step towards building accent-aware voice-driven
systems. 

Accent identification from non-native speech bears resemblance to the task
of language identification <cit.>. However, accent
identification is a harder task as many cues about the speaker's native
language are lost or suppressed in the non-native speech. Nevertheless,
one may expect that the speaker's native language is reflected in the
acoustics of the individual phones used in non-native language speech, along with
pronunciations of words and grammar. In this work, we focus on the
acoustic characteristics of an accent induced by a speaker's native
language. 



Our main contributions:


  * We develop a novel deep Siamese network based model which
learns the association between accented speech and native
language speech.

  * We explore i-vector features extracted using both an unsupervised
Gaussian mixture model (GMM) and a supervised deep neural network (DNN)
model.

  * We present a detailed error analysis of the proposed system which
reveals that the confusions among accent predictions are contained within the language family of the corresponding native language. 


Section <ref> outlines the i-vector feature extraction
process. Section <ref> describes our Siamese network-based
model for accent identification. Our experimental results are detailed in
Section <ref> and Section <ref> provides an error
analysis of our proposed approach.



§ RELATED WORK



The prior work on foreign accent identification has drawn inspiration from
techniques used in language identification <cit.>. The 
phonotactic model based approaches <cit.> and acoustic model based
approaches <cit.> have been explored for accent identification
in the past. More recently, i-vector based representations,  which is part of the
state-of-the-art speaker recognition <cit.> and
language recognition <cit.> systems, have been applied to the task of
accent recognition. The i-vector systems that used GMM-based background models were
found to outperform other competitive baseline
systems <cit.>.

In the recent years, the language recognition systems and speaker recognition systems have shown promising results with the use of deep neural network (DNN) model based i-vector extraction <cit.>. 
Also, none of the previous approaches have exploited speech
in native languages while training accent identification systems to the best of our knowledge. This work attempts to
develop accent recognition systems using both these components.



§ FACTOR ANALYSIS FRAMEWORK FOR I-VECTOR EXTRACTION


The techniques outlined here are derived from previous work on joint
factor analysis (JFA) and i-vectors <cit.>. We
follow the notations used in <cit.>. The training data from all the
speakers is used to train a GMM with model parameters λ = {π_c ,
μ_c, Σ_c} where π_c,
μ_c and Σ_c denote the mixture
component weights, mean vectors and covariance matrices respectively for
c=1,..,C mixture components. Here, μ_c  is a vector of
dimension F and Σ_c is of assumed to be a diagonal
matrix of dimension F× F.


 §.§ GMM-based i-vector Representations

Let ℳ_0 denote the universal background model (UBM) supervector which is the
concatenation of μ_c for c=1,..,C and is of dimensionality 
D × 1 (where D = C · F). Let Σ denote the block diagonal
matrix of size D × D whose diagonal blocks are
Σ_c. Let
𝒳(s)={x_i^s,i=1,...,H(s)} denote the
low-level feature sequence for input recording s where i denotes the
frame index. Here H(s) denotes the number of frames in the recording. Each
x_i^s is of dimension F × 1.

Let ℳ(s) denote the recording supervector which is
the concatenation of speaker adapted GMM means μ_c(s) for
c=1,..,C for the speaker s. Then, the i-vector model is,

    ℳ(s) = ℳ_0 + Vy(s)

where V denotes the total variability matrix of dimension D
× M and y(s) denotes the i-vector of dimension M. The
i-vector is assumed to be distributed as
𝒩(0,I).

In order to estimate the i-vectors, the iterative EM algorithm is used. We
begin with a random initialization of the total variability matrix
V. Let p_λ(c | x_i^s) denote the
alignment probability of assigning the feature vector x_i^s
to mixture component c. The sufficient statistics are then computed as,

    N_c(s)     = ∑ _i=1^H(s) p_λ(c | x_i^s) 
    S_X,c(s)     = ∑ _i=1^H(s) p_λ(c | x_i^s) (x_i^s - μ _c)

Let N(s) denote the D× D block diagonal matrix with
diagonal blocks N_1(s)I,
N_2(s)I,..,N_C(s)I where I is
the F× F identity matrix. Let S_X(s) denote the
D× 1 vector obtained by splicing
S_X,1(s),..,S_X,C(s).

It can be easily shown <cit.> that the posterior distribution of the
i-vector p_λ(y(s) | 𝒳(s)) is Gaussian with
covariance l^-1(s) and mean
l^-1(s)V^*Σ^-1S_X(s),
where

    l(s) = I + V^*Σ^-1N(s)V

The optimal estimate for the i-vector y(s) obtained as
_y  [ p_λ(y(s) | 𝒳(s))  ] is
given by the mean of the posterior distribution.

For re-estimating the V matrix, the maximization of the
expected value of the log-likelihood function (EM algorithm), gives the
following relation <cit.>,

    ∑ _s=1^SN(s) V E[y(s) y^*(s)  ] = ∑ _s=1^SS_X(s)  E [y^*(s)  ]

where E[.] denotes the posterior expectation operator. The
solution for Eq. (<ref>) can be computed for each row of
V. Thus, the i-vector estimation is performed by iterating
between the estimation of posterior distribution  and the update of the
total variability matrix (Eq. (<ref>)).



 §.§ DNN i-vectors
 

Instead of using a GMM-UBM based computation of i-vectors, we can also use
DNN-based context dependent state (senone) posteriors to generate the sufficient
statistics used in the i-vector computation <cit.>.  The GMM
mixture components will be replaced with the senone classes present at the
output of the DNN. Specifically, p_λ(c | x_i^s)
used in Eq. (<ref>) is replaced with the DNN posterior probability
estimate of the senone c given the input acoustic feature vector
x_i^s and the total number of senones is the parameter C. The
other parameters of the UBM model  λ = {π_c , μ_c,
Σ_c}  are computed as


    π_c     = ∑ _s=1^S∑_i=1^H(s) p(c | x_i^s)/∑_c=1^C∑ _s=1^S∑_i=1^H(s)p(c | x_i^s) 
    μ_c     = ∑ _s=1^S∑_i=1^H(s) p(c | x_i^s) x_i^s /∑_c=1^C∑ _s=1^S∑_i=1^H(s)p(c | x_i^s) 
    Σ_c     = ∑ _s=1^S∑_i=1^H(s) p(c | x_i^s) (x_i^s - μ_c) (x_i^s-μ_c)^*/∑_c=1^C∑ _s=1^S∑_i=1^H(s)p(c | x_i^s)




§ PROPOSED APPROACH



Siamese networks <cit.> are neural network models
that are designed to learn a similarity function between pairs of input representations. This architecture consists of two identical neural networks with shared weights where the input is a pair of samples. The objective is to find a function that minimizes or maximizes the similarity between the pair of inputs depending on whether they belong to the same category or not. This is achieved by optimizing a contrastive loss function containing dual terms - a component that reduces the contribution from the positive training examples (i.e. pairs of inputs belonging to the same category) and a component that increases the contribution from negative training examples (i.e. pairs of inputs from different categories).




Figure <ref> shows an illustration of the Siamese network we
used for accent identification. Each training example comprises a pair of
input i-vectors, {𝐢_1, 𝐢_2} corresponding to
an accented speech sample and a language speech sample, and a binary
label y ∈{0,1} indicating whether or not the native language
underlying the accented speech sample exactly matches that of the language
speech sample. The positive training examples correspond to accented speech 
i-vectors that are paired with native language i-vectors from the same
speaker. For the negative examples, we paired up accented speech i-vectors
with i-vectors from languages different from the one underlying the accented
speech sample. These training instances are fed to twin networks with shared
parameters which produce two outputs
{ℱ(𝐢_1),ℱ(𝐢_2)} corresponding to
the input i-vectors 𝐢_1 and 𝐢_2. The whole network is
then trained to minimize the following contrastive loss function:

    ℒ(𝐢_1,𝐢_2,y)    = (1-y) ·(d(ℱ(𝐢_1),ℱ(𝐢_2)))^2 
       + y
    ·(max(0, 1-d(ℱ(𝐢_1),ℱ(𝐢_2)))^2


where d(·, ·) is a distance function between the output 
representations. 

We use a large number of positive and negative training samples to learn a distance metric between
the accented speech i-vectors and the language i-vectors. During test time,
we compare the accented speech test i-vector with a representative language
i-vector and choose the language whose i-vector representations are the nearest 
from the accented speech i-vector, according to the distance metric
learned by the Siamese network. We experiment with different strategies to
determine how the language i-vectors should be constructed during test time.
Section <ref> discusses more details of these test strategies.



§ EXPERIMENTAL RESULTS






 §.§ Data




For our experiments, we used the CSLU Foreign Accented English Release 1.2
database <cit.> that consists of telephone-quality spontaneous English
speech by native speakers of 22 different languages. We set up a 10-class
accent identification task using accented English from speakers of 10
different languages which had the most amount of data: Brazilian Portuguese (BP),Hindi
(HI), Farsi (FA), German (GE), Hungarian (HU), Italian (IT), Mandarin
Chinese (MA), Russian (RU), Spanish (SP) and Tamil (TA). For native language
speech, we used the CSLU 22 Languages Corpus <cit.> which contains
telephone-quality continuous speech in all the above-mentioned 10 languages.
Many of the speakers in the CSLU 22 Languages corpus also recorded speech
samples for the CSLU Foreign Accented English corpus. The samples from these speakers were used to construct positive examples for training our Siamese network. Table <ref> gives detailed statistics about the data used in our experiments, along with the training, development and test set splits. These splits were created using a stratified random sampler so that the proportion of different accents in each set remains almost same.

For the GMM based i-vectors, a 2048-component UBM was trained and
400 dimensional i-vectors were extracted using the formulation given in Sec. <ref>. These features will be referred to as GMM i-vectors in the rest of the paper. The UBM was trained with 39 dimensional MFCC features which were mean and variance normalized at the utterance level. The training data used in the UBM was obtained from the multilingual corpora from NIST SRE 2008 (consisting of telephone-quality speech) and the Switchboard English database <cit.>. For training the DNN i-vectors, an acoustic model was developed for Switchboard using the Kaldi toolkit <cit.>. The DNN model generates 4677-dimensional senone posterior features which are used in the i-vector extraction (Sec. <ref>). The i-vector training based on the DNN-UBM uses data from NIST SRE08 and Switchboard. We use 300 dimensional i-vectors from the DNN model (henceforth referred to as DNN i-vectors).    Both i-vectors were length normalized before the classifier training.  



Performance evaluation: We used accent identification
accuracy as the primary metric to evaluate our proposed approach. This is computed
as the percentage of utterances which are correctly identified as having one
of the 10 above-mentioned accents. (A classifier based on chance would give an accuracy of 10% on this task.)



 §.§ Comparing GMM i-vectors with DNN i-vectors




Table <ref> shows the performance of various baseline accent identification systems using
both GMM i-vectors and DNN i-vectors as input features. We used an LDA-based  classifier which reduces the dimensionality of the input vectors by linear projection onto a lower dimensional space that maximizes the separation between classes. We also built an SVM classifier using a radial basis function (RBF) kernel. Finally, NNET is a feed-forward neural network (with a single 128-node hidden layer) that is trained to optimize categorical cross-entropy using the Adam optimization algorithm <cit.>. LDA and SVM were implemented using the scikit-learn library <cit.> and NNET was implemented using the Keras toolkit <cit.>. The hyper-parameters of all these models were tuned using the validation set. From Table <ref>, we observe that the classifiers using DNN i-vectors clearly outperform the classifiers using the GMM i-vectors. This is intuitive because the DNN i-vectors carry more information about the underlying phones. Both the SVM and NNET classifiers perform comparably and outperformed the linear (LDA) classifier due to the inherent non-linearity in the data. 

In all subsequent experiments, we use the 300-dimensional DNN i-vectors (unless mentioned otherwise).



 §.§ Evaluating the Siamese network





We tried various configurations of the Siamese networks, along with varying ratios of positive and negative training examples. After tuning on the validation set, the Siamese architecture which yielded the best result consisted of 2 hidden layers with 128 nodes each and a dropout rate of 0.2 in the first hidden layer <cit.>. We used the RMSprop optimizer and the Glorot initialization for the network <cit.>. The network was trained on 100,000 positive and 900,000 negative training pairs. The 10 accents were equally distributed across the positive and negative pairs.

Table <ref> lists the accuracies of the  Siamese network-based classifiers using different strategies to choose language i-vectors during test time. We first extracted a random sample of 30 language i-vectors and computed the mean of the lowest five output scores. (Here, 30 and 5 were tuned on the validation set.) The language that was least distant from the accented test sample was chosen as the predicted output. This is the system referred to as “Siamese-1". “Siamese-2" refers to a system where we computed a mean language i-vector across all the i-vectors for a particular language. For “Siamese-3", we first clustered the language i-vectors into 4 clusters using the k-means algorithm, following which we computed cluster means and chose the language whose cluster mean was minimally distant from the accented test sample. Finally, “Siamese-4" augments “Siamese-3" with a neural network classifier (consisting of two 8-node hidden layers with 40 input nodes and 10 output nodes). This second DNN is trained on the distance measures computed from Siamese-3 model for the 10 accent classes (4 scores for each file obtained from the four cluster mean vectors) and it predicts the target accent class. This network is trained on the validation data. In our experiments, “Siamese-4" provided the best performance. 

Table <ref> compares the performance of our best-performing Siamese network to the two best-performing baseline systems from
Table <ref>.  We see consistent improvements from using the
Siamese network classifier over the best baseline system on both the validation set and the test set. These improvements hold when both GMM i-vectors and DNN i-vectors are used as input features.





 §.§ Comparison with other systems using native language i-vectors


All the baseline systems used so far only made use of the accented English samples during training and did not make use of the native language speech samples. We compare our Siamese network-based approach with other techniques that exploit speech data from native languages during training. First, analogously to
“NNET”, we train a 2-layer feed-forward neural network but with input
features consisting of language i-vectors concatenated with accent i-vectors. This system is referred to as “NNET-append”. We build a second system, which we call “NNET-nonid-twin", that is identical to the twin network Siamese architecture shown in Figure <ref>, except the weights of the twin networks are not shared. Finally, we also investigate a transfer learning based
approach, referred to as “NNET-transfer”. For this system, we train a 2-layer
feed-forward neural network using only language i-vectors to predict the
underlying language. Then, we use the resulting weights from the hidden
layers as an initialization for a neural network that uses accent i-vectors
as inputs to predict the underlying accent. Table <ref>
compares these three systems with “Siamese-4" introduced in Table <ref>. We
observe that our proposed Siamese-network system outperforms all
the other systems which also have access to native language i-vectors.






 §.§ Accuracies on accented speech utterances with varying accent strengths


The CSLU Foreign Accented Speech corpus contains perceptual judgments about the accents in the utterances. Each accented speech sample was independently annotated for accent strength on a scale of 1-4 (where 1 denotes a very mild accent and 4 denotes a very strong accent) judged by three native American English speakers. Table <ref> shows how the Siamese network-based classifier (“Siamese-4") performs on utterances when grouped according to accent strength. Intuitively, as the accent strength increases, our classifier accuracy increases. Despite the fact that our test set predominantly contains utterances of accent strength 3, it is interesting to see that the average accuracy on these test samples is much higher than the samples rated 1 and 2 on accent strength. 






 §.§ System Combination




Table <ref> shows the accuracies from our two top baseline systems and our best Siamese network based system, Siamese-4, when the correct accent is among the top 2 and top 3 accent predictions. We observe that the accuracies dramatically improve across all three individual systems when moving from 1-best to 2-best and 3-best accuracies. This is indicates that a significant part of the confusions seen across the classes are confined to the top 3 predictions. 

We also combine the outputs from the three individual systems. We adopt a simple majority voting strategy: Choose the prediction that 2 or more systems agree upon. If all three systems predict different accent classes, then choose the accent predicted by Siamese-4. We also use a weighted voting strategy. The outputs from Siamese-4 were converted into posterior probabilities using a softmax layer. A weighted combination of the probabilities from SVM, NNET and Siamese-4 was used to determine the accent with the highest probability. (Weights of 0.3, 0.3 and 0.4 were used for SVM, NNET and Siamese-4, respectively.) The N-best accuracies for both these combined systems are shown in Table <ref>. As expected, we observe performance gains from combining the outputs of all three individual systems; the gains are larger in the 2-best and 3-best cases.



§ DISCUSSION



It is illustrative to visualize the accent predictions made by our proposed Siamese network-based classifier in order to learn more about the main confusions that are prevalent in the system. We visualize the confusion matrix of the Siamese-4 classifier on the test samples using a heat map as shown in Figure <ref>. Darker/bigger bubbles against a column for a given row indicate a larger number of samples were predicted as being of the accent labeled against the column. For each accent class, Figure <ref> shows there are a sizable number of test examples that are correctly predicted as evidenced by the dark bubbles along the diagonal. We also see more than one darkly-colored bubble along the non-diagonal indicating strong evidence for confusion. For example, Hindi-accented English samples are often confused as being Tamil-accented and conversely, Tamil-accents in some test samples are mistaken for Hindi-accents. This is very intuitive given that both these accents are very closely related and correspond to the same geographical region. Indeed, if we group the languages according to the language families that they belong to, i.e. {BP, RU and IT}, {SP, GE and HU}, {MA}, {HI and TA} and {FA}, the corresponding confusion matrix exhibits far less confusion as shown in Figure <ref>.










§ CONCLUSIONS

In this paper, we explore the problem of accent identification from non-native English speech. We propose a novel approach based on deep Siamese neural networks that uses i-vectors extracted from both accented speech and native language speech samples and learns a semantic distance between these feature representations. On a 10-class accent identification task, our proposed approach outperforms a neural network-based classifier using both GMM-based i-vectors and DNN-based i-vectors with relative accuracy improvements of 15.4% and 7.0%, respectively.

In this work, we focused on the acoustic characteristics of an accent induced by a speaker's native language. Accents are also correlated with specific lexical realizations of words in terms of pronunciations and variations in word usage and grammar. As future work, we plan to explore how to incorporate the pronunciation model and language model based features to automatic identification of speech accents.



§ ACKNOWLEDGEMENTS

The authors thank the organizers of the MSR-ASI workshop, Monojit Choudhury, Kalika Bali and Sunayana Sitaram for  the opportunity and all their help, as well as the other project team members Abhinav Jain, Gayatri Bhat and Sakshi Kalra for their support. We also gratefully acknowledge the logistical support from Microsoft Research India (MSRI) for this project as well as access to Microsoft Azure cloud computing resources. 

IEEEbib


