Joint Bayesian Gaussian discriminant analysis for speaker verification

State-of-the-art i-vector based speaker verification relies on variants of Probabilistic Linear Discriminant Analysis (PLDA) for discriminant analysis. We are mainly motivated by the recent work of the joint Bayesian (JB) method, which is originally proposed for discriminant analysis in face verification. We apply JB to speaker verification and make three contributions beyond the original JB. 1) In contrast to the EM iterations with approximated statistics in the original JB, the EM iterations with exact statistics are employed and give better performance. 2) We propose to do simultaneous diagonalization (SD) of the within-class and between-class covariance matrices to achieve efficient testing, which has broader application scope than the SVD-based efficient testing method in the original JB. 3) We scrutinize similarities and differences between various Gaussian PLDAs and JB, complementing the previous analysis of comparing JB only with Prince-Elder PLDA. Extensive experiments are conducted on NIST SRE10 core condition 5, empirically validating the superiority of JB with faster convergence rate and 9-13% EER reduction compared with state-of-the-art PLDA.


INTRODUCTION
The state-of-the-art in speaker recognition is currently dominated by the i-vector approach, that models both speaker and channel variabilities in a single low-dimensional space termed the total variability subspace [1]. An i-vector based speaker verification system mainly consists of three components, which are i-vector extractor based on Gaussian Mixture Models (GMMs) or Deep Neural Networks (DNNs) [2], i-vector post-processing (e.g. length normalization) and discriminant analysis. Note that this approach defers the decomposition of speaker and intersession variabilities to the stage of discriminant analysis, which is particularly important for this approach.
An attractive discriminant analysis technique is to construct likelihood ratio score based on probabilistic generative models such as the widely used Probabilistic Linear Discriminant Analysis (PLDA) [3] with many variants. Except the heavy-tailed PLDA [4], most variants are Gaussian, such as Prince-Elder PLDA [3], Simplified PLDA (SPLDA) [5], Kaldi PLDA [6], two-covariance model [7], and Ioffe PLDA [8]. Basically, Gaussian PLDA assumes that the j-th i-vector from speaker i obeys the following decomposition 1 : This work is supported by NSFC grant 61473168. 1 Throughout the paper, we assume the data have zero-mean after a standard centering preprocessing step and omit the global mean in the model.
where the latent speaker factor zi and the intersession residual ǫij are both Guassians and independently distributed. F is the loading matrix spanning the speaker subspace. Denote by HI the intra-personal hypothesis that one set of i-vectors x1 and another set of i-vectors x2 belong to the same speaker, and HE the extra-personal hypothesis that they are from different speakers. The speaker verification problem can then be solved by thresholding the likelihood ratio score p(x1, x2|HI )/p(x1, x2|HE).
The performance of PLDA largely depends on how it can be trained effectively to learn the within-class variability, which characterizes intersession residuals, and the between-class variability, which characterizes differences among speakers. Some improvements include using data domain adaptation of PLDA parameters [5] and discriminative training [9]. In this paper, we are primarily concerned with addressing the two basic challenging issues for the current Gaussian PLDA family. First, for PLDAs with subspace modeling, it is difficult to determine the subspace dimension which is crucial for performance. Low subspace dimension often leads to under-fitting, while high subspace dimension results in over-fitting. Second, whether using subspace modeling or not, current PLDAs suffer from slow convergence of their implemented EM iterations. As analyzed in [10], different parameterizations and selections of hidden variables in designing EM updates have significant effect on the convergence performance. These basic issues hinder improving the performance of current PLDAs.
We are mainly motivated by recent the work of Joint Bayesian (JB) method [10], which was originally proposed for face verification. In JB, there is no need to determine the subspace dimension, and it achieved faster convergence and more accuracy in [10] for face verification. We apply JB to speaker verification and make three main contributions. 1) We find that the EM updates with approximated statistics suggested in [10] does not work in speaker verification problem. Instead, the EM iterations with exact statistics are employed and give better performance. 2) Inspired by Fisher LDA, we propose to do simultaneous diagonalization (SD) of the within-class and between-class covariance matrices to achieve efficient testing. Compared to the SVD-based efficient testing method in [10], the new SD method can still be applied to reduce the testing complexity even in the case that the number of training samples per subject are different. 3) We scrutinize similarities and differences between various Gaussian PLDAs and JB, complementing the analysis of comparing JB only with Prince-Elder PLDA in [10]. Moreover, extensive experiments are conducted on NIST SRE10 core condition 5, empirically validating the superiority of JB in term of EM convergence rate and EER performance.

JOINT BAYESIAN GAUSSIAN DISCRIMINANT ANALYSIS
Joint Bayesian (JB) Gaussian discriminant analysis was first proposed in [10] for face verification. Its model formulation is similar to the two-covariance model [11] but with different parameterizations and selections of hidden variables in EM training. For speaker verification, we use two independent Gaussians to represent speaker identity and intersession residuals respectively. The j-th i-vector of speaker i, denoted by xij ∈ R d , is decomposed as: The parameters Θ are estimated by the EM algorithm through iteratively optimizing the expected complete log-likelihood function as follows: where Θ t = {S t µ , S t ε } are the parameters from the t-th EM update, and Θ t+1 the parameters to be updated in the (t + 1)-th iteration. Under this auxiliary objective function, the terms related to Sµ and Sǫ are effectively decoupled, resulting in very elegant update equations for Sµ and Sǫ [10].
In speaker verification testing, we calculate the log-likelihood ratio score to determine whether one set of i-vectors x1 (including m1 i-vectors) and another set of i-vectors x2 (including m2 ivectors) are from the same speaker 2 : Note that all the data likelihoods p(x1, x2), p(x1) and p(x2) can be calculated through Eq. 1, which involves matrix inversions.
To accelerate testing, [10] employed SVD to obtain low rank approximations of the matrices appearing in the three log-likelihood terms in Eq. 3, which depend on m1 + m2, m1 and m2 respectively. Therefore, this speedup is more useful under the condition that the number of i-vectors is the same across all subjects, i.e. mi = m. This is often satisfied in the task of face verification and face search.
Here we propose first to do simultaneous diagonalization (SD) of Sµ and Sε, Φ T SµΦ = K and Φ T SǫΦ = I. Similar to Fisher LDA, we keep the first s < d largest eigenvalues of S −1 µ Sε, giving the low-rank diagonal matrix K. Denote by Φ the corresponding 2 By abuse of notation, here x 1 is not the data corresponding to speaker 1. low-rank eigenvector matrix. By defining Ψ = Φ −T , we have Sµ = ΨKΨ T , Sǫ = ΨΨ T , and moreover, where Ω = diag(Ψ; . . . ; Ψ). Based on this decomposition of Σx i , the calculation of data likelihood p(xi) could be accelerated, if we take Ω to transform the i-vectors via pre-computation. The likelihood calculation then only involves inversion of diagonal matrices, reducing the complexity from O(d 3 ) to O(d). Moreover, it can be seen that this speedup does not depend on mi and thus has broader applicability. In this paper, we also conduct experiments to compare these two speedup methods over speaker verification accuracy.

CONNECTION WITH PLDA
In this section, we investigate the connections between joint Bayesian (JB) [10], Simplified PLDA (SPLDA) [5] and Kaldi-PLDA [12]. We mainly show that different parameterization and selection of hidden variables lead to different behavior of the EM algorithm, and JB is superior to PLDAs in terms of EM convergence. For the advantages of JB in allowing the data to implicitly determine the subspace dimensionality for maximal discrimination and favoring low-rank esimates of Sµ and Sε, the reader could refer to [10]. Table 1 summarizes the similarities and differences between JB, SPLDA, Kaldi PLDA and the two-covariance model.

Simplified PLDA (SPLDA)
Basically, SPLDA [5] assumes that j-th i-vector from speaker i obeys the following decomposition : where the latent speaker factor zi ∼ N (0, I) and the intersession residual ǫij ∼ N (0, Λ) are both Guassians and independently distributed. F is the loading matrix spanning the speaker subspace. The speaker subspace could be full rank, which is also known as the two-covariance model [11]. The parameters Θ = {F, Λ} [5] are estimated by the EM algorithm through iteratively optimizing the complete log-likelihood logp(xi, zi; Θt+1) averaged over p(zi|xi; Θt) where Θt = {Λt, Ft} Different from Eq. 2 in JB, the hidden variables in SPLDA are only zi's, excluding ǫij 's. 3 Now we analyze the convergence property of the EM updates for SPLDA, analogous to [10]. Note that maximizing Eq. 5 over Ft+1 is equivalent to minimizing over Ft+1 It can be seen that  N (0, Sµ) is the identity variable for speaker i, modeled by the between-class covariance Sµ. εij ∼ N (0, Sǫ) is the intersession residual, modeled by the within-class covariance Sǫ. For SPLDA, zi ∼ N (0, I) stands for the identity variable. where When Λt is small, by setting Ft+1 as Ft, we find that Hence updating Ft+1 as Ft approximately optimize the M-step and the EM-algorithm stalls upon a single iteration.
Note that theoretically the EM algorithm is only actually guaranteed to produce non-decreasing optimization of data likelihood through a series of parameter updates. Strict convergence to local minima (or stationary points) requires further strong assumptions. Combining this understanding of the EM algorithm and the above analysis of halt upon a single iteration, we could realize that the EM update for SPLDA could be easily stuck into a non-local minimum with small Λt. The EM update for JB does not have such problem, since JB has different parameterization and selection of hidden variables. The faster convergence of the EM iterations for JB is also empirically observed in our experiments.

Kalid PLDA
The Kaldi is a widely used open-source speech recognition toolkit [12]. Here we examine the PLDA implementation in Kaldi code repository [6]. The conceptual starting point for Kaldi PLDA is the SPLDA model as shown in Eq. 4 with full rank F . Next, Kaldi PLDA is only concerned with modeling the average i-vector for each speakerxi = m i j=1 xij/mi, which is distributed according to where mi is the numbers of extracted i-vectors for speaker i. Eq. 6 is then treated as the data likelihood function. All the extracted i-vectors for each speaker are collapsed as a single sample -the average i-vector, which is assumed to obey the decomposition : where µi ∼ N (0, Γ) models the between-class variability with the covariance Γ = F F T and the average residualεi = m i j=1 εij/mi ∼ N (0, 1 m i Λ) models the within-class variability. The expected complete log-likelihood function for the EM algorithm is optimized to iteratively estimate Γ and Λ, as follows: The parameterization of Kaldi PLDA is similar to JB, i.e. using two covariance matrices. Hence the EM iterations in Kaldi PLDA can also select the total hidden variables (µi, εi1), with good convergence. However, the additive decomposition only applies to the average i-vector in Kaldi PLDA. This is helpful for estimating betweenclass covariance but is detrimental for estimating within-class covariance. At the testing phase, Kaldi PLDA also performs simultaneous diagonalization of Λ and Γ. However, the significance of computational saving is less than the SD applied in JB, because JB calculates the joint likelihood of a number of i-vectors while Kaldi only calculates the likelihood of a single average i-vector.

Dataset
We conduct speaker verification experiments with different discriminant analysis techniques on the NIST SRE10 core condition 5, which includes 11982 speakers, 7169 target and 408950 nontarget trials [13]. The DNN used in the experiments is trained on part of the Fisher data including about 600 hours of speech cuts. The i-vector extractor training data comprises 57517 speech cuts of 5767 speakers, which are from Switchboard, Fisher and NIST SRE 04, 06, and 08. Both JB and SPLDA are trained on SRE data, consisting of 36612 speech cuts and 3805 speakers from NIST SRE 04, 06, and 08.  Table 2. The data used to train the DNN-UBM, i-vector extractor, SPLDA and JB for speaker verification.

System Configuration
The features used in the experiments are 40-dimensional Mel Frequency Cepstral Coefficients (MFCCs), including 20-dimensional static features and first-order derivatives. The speaker verification system uses a DNN-UBM with 5 hidden layers, 5335 senones, and a 600-dimensional i-vector extractor. The input of the DNN-UBM is the MFCCs extracted using 21 frames (11 frames before and 9 frames after). For discriminant analysis techniques, we implement three references namely LDA+COS, SPLDA and Kaldi PLDA. We apply the LDA to the i-vectors to obtain 200-dimension features and use cosine distance metric for testing. For SPLDA, we set the dimension of the subspace to 300. For Kaldi PLDA, we use the default configuration of the Kaldi toolkit [6]. The system performances are reported by equal error rate (EER) and minimum decision cost function (DCF) defined in NIST SRE08 and SRE10 [13].

Speaker Verification Performance
We evaluate LDA+COS, Kaldi PLDA, SPLDA and JB on the NIST SRE10 core condition 5. These four models share the same training and test data. Fig. 1 illustrates the detection error trade-off (DET) curves of LDA+COS, SPLDA, Kaldi PLDA and JB for discriminant analysis with configurations described in Section 4.2. From the results show in Tab. 3, we can conclude that : • Compared to distance-based discriminant analysis LDA+COS, probabilistic generative model based methods such SPLDA, Kaldi PLDA and JB achieve better performance on EER.
• In terms of EER, JB improves 13.0% and 45.3% compared to SPLDA and Kaldi PLDA respectively on SRE10 male tests, 9.2% and 30.9% on female tests. This verifies that JB with careful selection of hidden variables achieves better parameter estimation due to efficient EM updates.
• SPLDA achieves better results on EER than Kaldi PLDA, because SPLDA utilizes the joint likelihood of i-vectors rather than the single average i-vector as used in Kaldi PLDA to estimate the parameters.

Subspace Dimensionality
Here we investigate the impact of the sub-space dimensionality of JB and SPLDA for discriminant analysis, which is shown in Fig. 2. Two methods (SVD, SD) are used to reduce the subspace dimensionality of JB in speaker verification testing. It can be seen that: 1) The   Table 3. Performance comparison of four different discriminant analysis back-ends on NIST SRE10 core condition 5.
dimension of the subspace plays an important role in SPLDA that lower may cause under-fitting, while higher may cause over-fitting.
2) The JB performance fluctuates slightly with the change of subspace dimension in testing, but the time complexity reduces from O(d 3 ) to O(s 3 ). Loose parameterization for JB makes it more robust since the dimension of the subspace is automatically fitted via data rather than manual defined. 3) SVD and SD have close performances but SD has wider applicability.

Convergence Rate
As discussed before, EM iterations for SPLDA are easier to stall in a single iteration. Fig. 3 shows the neg-loglikelihood curve of SPLDA with the optimal subspace dimensionality and that of JB trained by EM with exact and approximated statistics. From Fig. 3, first we find that JB converges faster than SPLDA with better parameter estimation. Second, we find that JB trained by EM with approximated statistics proposed by [10] will diverge in Fig 3b, Table 4. The effect of dimensionality reduction for JB and PLDA. The dimension of subspace for JB-SVD is determined by dim = rank(A) = 403 (A is defined in [10]) and the dimension of the subspace for JB-SD is determined by dim = rank(Sµ) = 407.
Tab. 4 shows the differences on EER between SPLDA and JB with or without dimension reduction. It is observed that even with the same dimension of the subspace learned by JB, SPLDA is still worse than JB. This justifies our analysis that model formulation and hidden variable selection of JB leads to better parameter estimation than SPLDA.

CONCLUSIONS
In this paper, we propose to apply JB to model i-vectors with careful parameterization and hidden variable selection that benefits EM iterations. Both theoretical derivation and experiments conducted on the NIST SRE10 core condition demonstrate that: 1) the parameterization of JB enables it to learn the intrinsic dimensionality of the identify subspace, which can reduce the system complexity without performance degradation; 2) Hidden variables selection of JB makes EM iterations converge faster with better parameter estimation; 3) The EM with exact statistics performs better than with approximated statistics. For future work, it is interesting to apply data domain adaption [5] and feature compensation [14,15] and nearestneighbor discriminant analysis (NDA) [16][17] that have been successfully applied to PLDA to JB to further improve performance.