Gaussian meta-embeddings for efficient scoring of a heavy-tailed PLDA model
Niko Brummer Anna Silnova Lukas Burget Themos Stafylakis

Time Varying Channel Tracking with Spatial and Temporal BEM for Massive MIMO Systems
    Jianwei Zhao, Hongxiang Xie,
        Feifei Gao,
         Weimin Jia, Shi Jin, and Hai Lin 
This work was presented in part at the 2017 IEEE ICC <cit.>.
J. Zhao, H. Xie and F. Gao are with  Tsinghua National Laboratory for Information Science and Technology (TNList) Beijing 100084, P. R. China, (e-mail:zhaojw15@mails.tsinghua.edu.cn, xiehx14@mails.tsinghua.edu.cn, feifeigao@ieee.org). J. Zhao is also with
High-Tech Institute of Xian, Xian, Shaanxi 710025, China.
  W. Jia is with High-Tech Institute of Xian, Xian, Shaanxi 710025, China (e-mail: jwm602@163.com).
  S. Jin is with the National Communications Research Laboratory, Southeast University, Nanjing 210096, P. R. China (email: jinshi@seu.edu.cn). 
 H. Lin is with the Department of Electrical and Information Systems,
Graduate School of Engineering, Osaka Prefecture University, Sakai,
Osaka, Japan (e-mail: hai.lin@ieee.org).


    December 30, 2023
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





Embeddings in machine learning are low-dimensional representations of complex input patterns, with the property that simple geometric operations like Euclidean distances and dot products can be used for classification and comparison tasks. We introduce meta-embeddings, which live in more general inner product spaces and which are designed to better propagate uncertainty through the embedding bottleneck. Traditional embeddings are trained to maximize between-class and minimize within-class distances. Meta-embeddings are trained to maximize relevant information throughput. As a proof of concept in speaker recognition, we derive an extractor from the familiar generative Gaussian PLDA model (GPLDA). We show that GPLDA likelihood ratio scores are given by Hilbert space inner products between Gaussian likelihood functions, which we term Gaussian meta-embeddings (GMEs). Meta-embedding extractors can be generatively or discriminatively trained. GMEs extracted by GPLDA have fixed precisions and do not propagate uncertainty. We show that a generalization to heavy-tailed PLDA gives GMEs with variable precisions, which do propagate uncertainty. Experiments on NIST SRE 2010 and 2016 show that the proposed method applied to i-vectors without length normalization is up to 20% more accurate than GPLDA applied to length-normalized i-vectors.  

  




§ INTRODUCTION

Embeddings are familiar in modern machine learning. Neural nets to extract word embeddings[<en.wikipedia.org/wiki/Word_embedding>] were already proposed in 2000 by Bengio <cit.>. Now embeddings are used more generally, for example in state-of-the-art face recognition, e.g. Facenet <cit.>.

Embeddings are becoming popular also in speech and speaker recognition. At Interspeech 2017, eighteen papers had the word `embedding' in the title.[<www.interspeech2017.org/program/technical-program/>.] In speaker recognition and spoken language recognition, we have been using i-vectors—embeddings extracted by a generative model—for almost a decade <cit.>. More general embeddings extracted by discriminatively trained DNNs are now appearing in speaker recognition, see for example the OK Google system <cit.>, the Voxceleb paper <cit.> and JHU's x-vectors <cit.>. Similar embeddings are also being used for spoken language recognition <cit.>. 
 
Input patterns (sequences of acoustic feature vectors, images, text, …) live in large, complex spaces, where probability distributions and geometric concepts such as distance are difficult to formulate. The idea with embeddings is that they are representations of complex input patterns that live in simpler spaces, e.g. ^d (multidimensional Euclidean space), where distance is naturally defined and can be put to work to compare patterns. 

At the Johns Hopkins HLTCOE SCALE 2017 Workshop[<http://hltcoe.jhu.edu/research/scale/scale-2017>] the ongoing research on embeddings for speaker recognition <cit.> inspired the generalization to meta-embeddings. The bulk of the work on meta-embeddings remains unpublished, but a current draft of that work can be followed on GitHub <cit.>.

Traditional embeddings can be interpreted as point estimates for hidden variables of interest and they typically live in low-dimensional Euclidean spaces, where comparisons between them are based on ordinary dot products. Meta-embeddings are likelihood functions for those hidden variables. They (meta-embeddings) typically live in infinite-dimensional Hilbert function spaces and comparisons between them are based on more generally defined inner products. This is a considerable generalization, which provides many new opportunities, but also complex challenges, both theoretical and computational. In this work, we restrict attention to multivariate Gaussian likelihood functions, for which the required inner products can be evaluated in closed form. 

In future we hope to apply meta-embeddings in speaker recognition in a similar way to i-vectors or x-vectors, in the sense that they will be extracted from the acoustic feature vectors (MFCCs). We regard the work in this paper as a warm-up exercise and a proof on concept, in which we use i-vectors, rather than MFCCs as input. With i-vector inputs, we can profit from simple generative models (Gaussian or heavy-tailed PLDA) that provide elegant closed-form formulas for extracting meta-embeddings.





§ MOTIVATION

In speaker recognition, we already have generative (i-vector) and discriminative (x-vector) embeddings that represent the state of the art and beyond. What additional advantages could we expect from meta-embeddings? The main motivation is to re-design from the ground up a vehicle for the propagation of uncertainty. 

The generative i-vector extractor model <cit.> provides a natural measure of uncertainty in the form of the i-vector posterior. In standard i-vector scoring recipes (PLDA <cit.> and cosine scoring <cit.>), only the expected value of that posterior is retained, while the precision (inverse covariance) is discarded. Work to propagate this source of uncertainty through PLDA has had limited success <cit.>. According to Patrick Kenny <cit.>, i-vector uncertainty propagation does not do what it is supposed to do, but instead gives some benefit as a channel compensator. We speculate that simplifying modelling assumptions and the mean-field variational Bayes approximation (which is required to make the i-vector extractor tractable) may be factors contributing to this problem.

To our knowledge, publications on discriminatively extracted embeddings for speaker recognition have not yet addressed the issue of uncertainty propagation. [Although the discriminative x-vector extractor does make use of standard deviations in its temporal pooling stage <cit.>, the uncertainty thus captured is not propagated through to the subsequent PLDA scoring backend.]

As we show below, meta-embeddings, whether extracted discriminatively or generatively, are designed to propagate uncertainty. Let us motivate this in a more general pattern recognition context. Quantifying the uncertainty is very important if a pattern recognizer is to be applicable to variable and sometimes challenging conditions. In speaker recognition, a short, noisy, narrow-band recording should leave much more uncertainty about the speaker than a long, clean, wideband recording. In face recognition, compare a well-lit, high resolution, full-frontal face image to a grainy, low resolution, partly occluded face. In fingerprint recognition, compare a clean, high-resolution ten-print, to a single, distorted, smudged fingermark retrieved from a crime scene.  
 


§ META-EMBEDDINGS

For an elaborate tutorial introduction to meta-embeddings, the interested reader is encouraged to read the first four chapters of <cit.>. Our summary here is limited to a few paragraphs.

In very general terms, we can describe speaker recognition as the problem of partitioning sets of recordings according to speaker <cit.>. Set sizes can vary from entire databases to binary trials that contain just a pair of recordings. For simplicity we assume: each recording has a single speaker; recordings from different speakers are independent; and the recordings of a given speaker are exchangeable. By De Finetti's theorem <cit.>, exchangeability is equivalent to the concept of the hidden speaker identity variable, which is familiar in speaker recognition thanks to the work of Patrick Kenny in JFA <cit.> and PLDA <cit.>. Meta-embeddings are likelihood functions for the speaker identity variable, of the form: 

    f()∝ P(r|)

where  is the identity variable and r denotes some representation of a recording, e.g. raw speech, MFCCs, i-vector, etc.

For traditional embeddings, the intuitive idea is to retain in the output representation as much as possible of the relevant information that was present in the input, r. For meta-embeddings, that idea is formalized: by definition all of the relevant information about the speaker of r must be present in the likelihood function of that recording. (Keep in mind that information content has meaning only relative to some probability model <cit.> and that for a poor likelihood model, asserting we have all of the relevant information won't do us any good in practice! Just as in any other probabilistic machine learning task, we need to choose the likelihood models wisely and find ways of training the parameters of these models.)

Let ={r_j}_j=1^n denote a set of n recordings. [In this paper, the r_j are i-vectors, but in future work they will be sequences of MFCCs.] In this paper, we let ∈^d denote a d-dimensional hidden speaker identity variable, for which we assign the standard normal distribution as prior: π()=(|,). The meta-embedding, f_j, extracted from input r_j is the likelihood function:

    f_j()    = k_j P(r_j|)

where k_j>0 is an arbitrary constant that may in general depend on r_j. Take careful note: the meta-embedding is the whole function, f_j, rather than some point estimate that lives in ^d. 

Let A,B represent two different hypotheses of how  might be partitioned w.r.t. speaker. Partition A has m hypothesized speakers, indexed by the subsets, {_i}_i=1^m, where _i⊆{1,…,n}. Likewise, B has m' speakers, indexed by {'_i}_i=1^m'. Using within-speaker exchangeability and between-speaker independence, the likelihood ratio (LR) comparing A to B can be expressed in terms of the meta-embeddings, f_j as:[This formula is a variation of the principle of Q-scoring, which we have introduced in <cit.>.]

    P(| A)/P(| B)   =
    ∏_i=1^m ∏_j∈_i P(r_j|)π/∏_i=1^m'∏_j∈'_i P(r_j|)π
       = ∏_i=1^m ∏_j∈_i f_j()π/∏_i=1^m'∏_j∈'_i f_j()π

where the triangle brackets denote expectation w.r.t. π(). The arbitrary scaling constants, {k_j}_j=1^n, are the same in the numerator and denominator and cancel. This equation is very general: any speaker recognition problem that can be formulated in terms of partitions can be expressed purely in terms of the meta-embeddings. This illustrates the principle that the likelihoods represent all the relevant information in the inputs. 

If we look only at the first line of (<ref>), we might conclude that to recognize speakers with a principled probabilistic model, we would always need a generative model, which requires possibly complex probability distributions for the observed data, of the form P(r_j|). Keeping in mind that f_j() is essentially an un-normalized posterior for :

    P(| r_j)∝π() f_j()

the last line of (<ref>) shows however that we require only much simpler distributions for ∈^d. 

Although we can extract meta-embeddings from generative models (as we do below), the generative models are by no means required. The RHS of (<ref>) shows that we can score—and therefore also train—meta-embedding systems in purely discriminative ways, without requiring complex generative models for the input data. 

This paves the way for discriminatively trained, DNN-based speaker recognizers, with principled uncertainty propagation.





 §.§ LR, inner product and pooling

For a binary trial, when ={r_1,r_2}, there are only two possible hypotheses, H_1: there is one speaker; and H_2: there are two speakers. Let f_1 and f_2 denote the meta-embeddings extracted from r_1,r_2, as defined by (<ref>). The likelihood ratio (<ref>) simplifies to:

    P(| H_1)/P(| H_1)   = f_1()f_2()π/f_1()πf_2()π
       =ḟ_̇1̇f_2/ḟ_̇1̇ḟ_̇2̇

where we have defined the constant function, ()=1 as well as the inner product between two meta-embeddings as:

    ḟ_̇1̇f_2   = ∫_^d f_1() f_2() π() d

We further need the concept of pooling. Let {r_j}_j=1^k, be a number of recordings all assumed to be from the same speaker and let {f_j}_j=1^k, be the associated meta-embeddings. Then the pooled meta-embedding is the product 

    f̅()   =∏_i=1^k f_i()

This is the likelihood function conditioned on those k recordings, which is the un-normalized form of the pooled posterior:

    P(|{r_j}_j=1^k)    ∝π() f̅()

As shown in <cit.>, all likelihood ratios of the form (<ref>) can be expressed in terms of these two primitive operations: pooling and inner products. Since our inner products are expectations of products, we can alternatively let our primitive operations be pooling and expectation.

Given some regularity conditions on the likelihood functions, our meta-embeddings live in a Hilbert space, which is a vector space equipped with an inner product. Although this Hilbert space is typically infinite-dimensional, it has a geometry just like Euclidean space. In meta-embedding space, norms, distances and angles are well-defined and have meaningful interpretations and practical applications in scoring and training of speaker recognizers. We lack space here, but the details are in <cit.>.



 §.§ Gaussian meta-embeddings

In practice, we need our primitive operations (pooling and expectation) to be computationally tractable. In this paper we restrict attention to multivariate Gaussian likelihood functions. The Gaussian meta-embedding (GME), extracted from a recording r_j is defined as:

    f_j()    = exp['_j-1/2'_j]
 
where f_j is represented by its natural parameters: _j∈^d and the d-by-d positive semi-definite precision matrix, _j. In future work, we envisage a discriminatively trained meta-embedding extractor where a DNN would process r_j (a sequence of MFCCs) and then output _j and some sensible representation of _j. In this paper, as warm-up exercise, we let the r_j be i-vectors and we use the PLDA model to derive relatively simple functions to extract _j,_j.

Since Gaussians are closed w.r.t. products and our representation is essentially logarithmic, pooling is easy: we simply need to add the natural parameters. The toolbox of primitive operations is completed by a closed-form expression for Gaussian expectations. For a (raw or pooled) meta-embedding, f, represented by ,, the expectation w.r.t. the prior is <cit.>:

    log E(,)    = logfπ = log∫_^d'-1/2'(+) d
       = 1/2'(+)^-1 - 1/2+

In the general case, Cholesky factorization would be the standard tool for this computation, but the GMEs that we extract in this paper have diagonalizable precisions that allow for much faster computations. 

We can combine the pooling and expectation formulas to compute arbitrary likelihood ratios of the form (<ref>). As an example, for binary trials, the LR becomes:

    P(| H_1)/P(| H_2)   = E(_1+_2,_1+_2)/E(_1,_2)E(_2,_2)





§ PLDA AS META-EMBEDDING EXTRACTOR



𝐆
In what follows, we let our recording representations be D-dimensional i-vectors: _j∈^D, and we derive a meta-embedding extractor via a heavy-tailed PLDA model. For more details see the chapter on Generative meta-embeddings in <cit.>.

The hidden variable is ∈^d and we require: d<D. The standard normal prior, π()=(|,), forms part of our PLDA model. (This is a simplification of Kenny's original model, which had a heavy-tailed prior <cit.>.) The PLDA model says that for every speaker, the identity variable  is sampled independently from π(); and every i-vector, _j, for a speaker having identity variable , is generated as:

    _j    =  + _j,    _j    ∼(,,ν)

where  is a D-by-d factor loading matrix and where the `channel' noise, _j, is drawn from a multivariate t-distribution <cit.>, having zero mean,[We can let the model have non-zero mean, but in practice it is simpler just to zero the mean of the training data.] precision  and degrees of freedom, ν>0. The PLDA model parameters are ,,ν. The meta-embedding that this model extracts from _j is simply:

    f_j()    ∝(_j |, ,ν)

This is a t-distribution for _j, but what does it look like as a function of ? Provided that D>d and ' is invertible, it is shown in <cit.> that it is another t-distribution, with increased degrees of freedom, ν'=ν+D-d

    f_j()    ∝(|_j^-1_j,_j,ν')
 
where

    _j    = b_j ,    _j    = b_j '_j,    
    b_j    = ν+D-d/ν+'_j_j

and

    = ',       = -^-1'

In a typical PLDA model, we have d∈ [100,200] and D∈ [400,600], so that ν'=ν+D-d is large, making the meta-embedding practically Gaussian. We therefore approximate:

    f_j()    ≈(|_j^-1_j,_j^-1) 
    ∝exp['_j-1/2'_j]

Several comments are in order:

	
  * For the heavy-tailed case, with small ν, the meta-embedding precisions, _j=b_j, vary as a function of the data. This is the uncertainty propagation.
	
  * Since precisions differ only by a scalar, we can simplify pooling and expectation. For pooling, addition of precisions simplifies to scalar addition. For the expectations (<ref>), we do not have to keep Cholesky factorizing every time. By precomputing an eigenanalysis of , we can diagonalize all +_j, which allows fast scoring and training <cit.>.
	
  * Notice that =, so that 
	
    '_j_j='_j_j
 
	is an ancillary statistic, independent of the speaker variable , but nevertheless important for complete inferences about the speaker. The heavy-tailed noise, _j is essentially Gaussian noise with random variance. For a `bad' i-vector, with a large noise vector,  '_j_j will also be large and the precision scaling factor, b_j will be small. For a `good' i-vector, with small noise, the opposite happens.
	
  * For the more familiar case of Gaussian PLDA, when ν→∞, we have b_j→ 1 and the meta-embedding precisions remain constant. In this case we cannot exploit the informtation in _j'_j.




 §.§ Length normalization

In <cit.> heavy-tailed PLDA was shown to be a better model of i-vectors than Gaussian PLDA, but the computational cost was considerable. Subsequently, <cit.> showed that the i-vectors could instead be Gaussianized via a simple length normalization procedure. This matched the accuracy of heavy-tailed PLDA, with negligible extra computational cost. The heavy-tailed nature of i-vectors was also addressed in <cit.>, where an iterative scaling procedure was proposed.

In this paper, our approximation (<ref>) has the advantage of a fast, closed-form estimate of the scale factor: _j'_j. Of course, length normalization would interfere with that estimate. We therefore apply the proposed Gaussian meta-embeddings (GMEs) to i-vectors without length normalization.  In our experiments below we compare against Gaussian PLDA applied to i-vectors both with and without length normalization.  




 §.§ GME extractor and scoring

In summary, the PLDA model (whether Gaussian or heavy-tailed), provides the functional form (<ref>), (<ref>) and (<ref>), for extracting Gaussian meta-embeddings from i-vectors. We shall explore both generative and discriminative methods for training the parameters of this GME extractor. 

It is also worth emphasizing the following difference between i-vectors and x-vectors on the one hand, versus meta-embeddings on the other:

	
  * The extractors for i-vectors and x-vectors are typically trained separately from the PLDA scoring backend. (In practice, this has many advantages.)
	
  * Meta-embeddings, once extracted, do not need an additional backend for scoring. The meta-embeddings contain within themselves everything that is needed to produce scores and this is done in general by (<ref>), or more specifically for GMEs applied to binary trials by (<ref>).

Of course, i-vectors and x-vectors can be scored in a parameterless backend via cosine scoring, but this is usually less accurate.

In machine learning, e.g. in Facenet <cit.>, the parameterless scoring strategy is preferred. The philosophy there can be summarized as: Train the embedding extractor such that embeddings of the same face are close (in Euclidean distance), while embeddings of different faces are far apart. That brute-force, geometric strategy has been very successful and should be appreciated as such. We do however hope that additional benefits can be eventually reaped as a result of the probabilistic strategy of the meta-embedding design.

Finally, it is worth repeating that comparisons between meta-embeddings can be interpreted in terms of a slightly more complex geometry, in terms of angles and norms. The interested reader is referred to the chapter entitled The structure of
meta-embedding space in <cit.>.   









§ TRAINING

𝐙
In what follows, let  denote the recordings in the training database; , the `labels', or true partition of the database w.r.t. speaker;  all of the hidden speaker identity variables (one per speaker); θ the generative model parameters (in this paper θ=(,,ν); and ϕ a set of variational parameters which would be needed for variational Bayes (VB) training strategies. 

Training strategies for complex probabilistic models are perhaps best understood via comparison with the celebrated variational autoencoder (VAE) <cit.> as described next. 



 §.§ Generative training

VB training effects approximate maximization of the marginal likelihood—here P(|,θ). Classical VB maximizes a lower bound to the marginal likelihood and is applicable to conjugate-exponential models with intractable posteriors and marginal likelihoods <cit.>. The VAE generalizes this to a wider class of (deep generative) models by using a stochastic approximation of the VB lower bound <cit.>. The VAE has two parts, decoder and encoder. The decoder is the generative model likelihood, P(,|,,θ). The encoder is a tractable variational posterior, Q(|,,ϕ), which approximates the true intractable posterior, P(|,,θ). VAE training is accomplished by maximization of the stochastic VB lower bound w.r.t. both decoder and encoder parameters, θ and ϕ. 

In this paper, we did not use VAE. Instead we used a crude shortcut. We trained a Gaussian PLDA model with the usual EM-algorithm and then simply plugged in a hand-selected value for ν to make the model heavy-tailed. 

Future work may however investigate VAE as a more powerful generative training strategy. The heavy-tailed PLDA model of section <ref> has intractable posteriors, P(| r_j), formed by the product of the Gaussian prior and the t-distribution likelihoods. As we pointed out however, the t-distribution likelihood is almost Gaussian, so that Gaussian variational posteriors can be expected to work well. Unfortunately, the VB lower bound (Gaussian expectations of logarithms of t-distributions) does not have a closed form, so that VAE rather than classical VB would be necessary. For this model, the variational parameters could be tied to the generative parameters, using (<ref>) and (<ref>). However, VB allows unconstrained optimization of the variational parameters, which could give advantages in both accuracy and computational complexity.

Alternatively, another solution for training is the classical mean-field VB solution for heavy-tailed PLDA of <cit.>, where the channel noise scaling factors are treated as hidden variables.




 §.§ Discriminative training


VAE training might be a good idea for simpler models such as heavy-tailed PLDA applied to i-vectors. However, for more complex models applied to acoustic features, discriminative training starts to look more attractive. 

	
  * For VAE training, we have a complexity doubling effect—we have to build a complex encoder, a complex decoder and also manage the non-trivial interface between them—see for example <cit.>. Once training is complete, the decoder is no longer needed for runtime scoring. 
	
  * In discriminative training, no decoder is needed and we only need to train the equivalent of the encoder.

In future work, for more complex models that extract meta-embeddings from acoustic features, we envisage that purely discriminative training methods could provide an easier route to success. 

Binary cross-entropy (BXE), applied to pairs of recordings, is a popular discriminative training criterion in speaker recognition <cit.>. BXE can indeed also be used to train meta-embedding extractors and that is what we do in this paper. The scoring formula needed during BXE training of the GME extractor is (<ref>).[Although we did not do that experiment here, notice that discriminative training of the GME extractor with ν→∞ is equivalent to discriminatively trained Gaussian PLDA <cit.>.] 




For future work, we note that BXE is by no means the only option—see <cit.> for a variety of other proposed discriminative training criteria. In particular, we would like to highlight the computationally attractive pseudolikelihood criterion, which does not rely on a quadratic expansion into binary trials. In addition, pseudolikelihood is a proper scoring rule for this training problem in a stricter sense than BXE and may give advantages as a calibration-sensitive training and evaluation criterion.



§ EXPERIMENTS




 §.§ I-vector extraction

In this paper our recordings are represented by i-vectors. For all of the experiments we use a single database of i-vectors, extracted as described below. 

We used 60-dimensional spectral features: 20 MFCCs, including C_0, augmented with Δ and ΔΔ. The features were short-term mean and variance normalized over a 3 second sliding window. With those features, we train a GMM UBM with 2048 diagonal components in gender independent fashion. Then, we collect sufficient statistics and train the i-vector extractor, with 600-dimensional i-vectors. These i-vectors serve us as input to either the PLDA baseline or to the new meta-embedding extractor. In both cases, i-vectors are transformed with global mean normalization. Then, for one of the baseline PLDA systems (baseline 1) we also apply length normalization. The second PLDA system (baseline 2) is applied to i-vectors without length normalization.




 §.§ Datasets and evaluation metrics

UBM, i-vector extractor and PLDA models are trained on the PRISM dataset <cit.>, containing Fisher parts 1 and 2, Switchboard 2, 3 and Switchboard cellphone phases. Also, NIST SRE 2004–2008 (from the MIXER collections) are added to the training. In total, the set contains  approximately 100K utterances coming from 16241 speakers. We used 8000 randomly selected files for UBM training and full set to train i-vector extractor and PLDA.

For training the GME extractor we use the same training list. However, for discriminative training via stochastic gradient descent (SGD), we split the set into training and cross-validation subsets. Cross validation was done on a randomly selected subset of 10% of the speakers, leaving the other 90% for training. This gave 8740 utterances for cross validation and 90309 for training. 

We evaluate performance on the female part of NIST SRE 2010, condition 5, which consists of English telephone data <cit.>. Additionally, we report the results on the NIST SRE 2016 evaluation set (both males and females). We report the results on the whole SRE'16 and also separately for each of the two language subsets, Cantonese and Tagalog.
As evaluation metrics, we use the equal error rate (EER, in %) as well as the average minimum detection cost function for two operating points (C^ Prm_ min). The two operating points are the ones of interest in the NIST SRE 2016 <cit.>, namely the probability of target trials  0.01 and 0.005.



 §.§ GME extractor initialized from PLDA


As mentioned in section <ref>, the GME extractor defined by (<ref>) with ν→∞ is equivalent to Gaussian PLDA. In this case, we can set the parameters of the GME extractor in such a way that log-likelihood scores computed with meta-embeddings will be equal to the scores provided by Gaussian PLDA. To see this, recall that in Gaussian PLDA the log-likelihood-ratio score for two i-vectors, _1 and _2, can be expressed as

    S_PLDA=2_1'_2+_1'_1+_2'_2 +(_1+_2)'+k,

where the parameters , ,  and k are calculated from the parameters of the PLDA model (see eq. (8) of <cit.>). Since we subtracted the global mean from the i-vectors, we have =, so that the linear term can be omitted.

Substituting for  and  in (<ref>) with expressions given by (<ref>) and setting b_j=1, the GME log-likelihood ratio score is:

    S_GME   =1/2_1''((+2)^-1-(+)^-1)'_1
       +1/2_2''((+2)^-1-(+)^-1)'_2 
       +_1''((+2)^-1)'_2 
       -1/2log|+2|+log|+|

Now, comparing (<ref>) and (<ref>), we see that S_GME=S_PLDA when

    =1/2'((+2)^-1-(+)^-1)',
       =1/2'((+2)^-1)',
    
    k   =-1/2log|+2|+log|+|.

Our GME extractor was initialized by solving for  and , while various values for ν were plugged in by hand. In the heavy-tailed regime (small ν), our results are not very sensitive to the exact value—we report results for ν=2. As a sanity check, we also tried ν→∞ to verify the equivalence with Gaussian PLDA. This is achieved by setting the b_j=1.

In our experiments below, we try the generatively initialized GME extractor as-is, as well as a discriminatively trained extractor as described next.   








 §.§ Discriminative GME extractor training

As mentioned in section <ref> we used binary cross-entropy (BXE) scored on pairs of i-vectors to discriminatively train the parameters of the GME extractor. We used the initialization from Gaussian PLDA as described above, with ν plugged in, followed by minibatch stochastic gradient descent (SGD) on the BXE objective. 

The (rather large) minibatches are formed as follows. We randomly select[with replacement] two sets of 5000 i-vectors each from the training data to serve as enrollment and test sets. Then, all i-vectors in the first set are scored against all in the second. We do not filter out any trials except for the cases where i-vectors are scored against themselves. As expected, the amount of non-target trials in a batch is much higher than target trials and the ratio can vary between the batches. To compensate for this, we separately compute BXE for target and non-target examples in the current batch and normalize each individual term by correct number of targets and non-targets respectively. 

The extractor parameters,  and , are updated by backpropagating gradients through the BXE objective, through the scoring formula (<ref>) and the extractor formula (<ref>) and (<ref>). The value of ν remains fixed at the plugged in value throughout training. Training continues until the BXE objective stops improving on the held out cross-validation set. 



 §.§ Multi-enroll trials

The SRE'16 evaluation set includes some trials with multiple enrollment recordings. For the results reported here, we took the shortcut of simply averaging enrollment i-vectors. Of course, both PLDA and meta-embeddings provide for more principled enrollment pooling and that will be explored in future. 




 §.§ Results

Table <ref> compares the results for the two Gaussian PLDA baselines against three variants of Gaussian meta-embeddings.

The first part of the table shows the results of our two baselines. Baseline 1 is Gaussian PLDA with D=600 and d=200, applied to length-normalized i-vectors. Baseline 2 is the same, but without length normalization. As usual, length normalization helps a lot—it makes the data more Gaussian to better fit the Gaussian PLDA model. 

















The second part of the table shows results for three GME configurations, all of them applied to i-vectors without length normalization. The first GME result is the sanity check that shows the equivalence between Baseline 2 and the GME extractor initialized from it, with ν→∞. The second GME result is the same as the previous one (PLDA initialization, no further training) but now with ν=2. Notice that this already does better in all cases than Gaussian PLDA without length normalization. Finally the third GME result shows that after further discriminative training, GME without length normalization can do better than Gaussian PLDA with length normalization.

By changing the degrees of freedom parameter, ν, we have effectively relaxed the Gaussian modelling assumptions. In our experiments, we have tried several different values for ν. We found that parameter ν can vary in a wide range of values where all of them provide similar performance. Here, we picked ν=2. As results indicate, the training not only mitigates the degradation brought by the lack of length normalization but even brings further improvements compared to both baselines in most of the cases. 





 §.§ Computational complexity

In our experiments, starting from i-vectors, Gaussian PLDA scoring of the whole SRE'10 and SRE'16 evaluation sets required respectively 0.4s and 1.5s in wall clock time. The GME solution required roughly double the time for the same tasks. We did not have an implementation of Kenny's heavy-tailed PLDA <cit.> to hand for direct comparison, but we do know that its computational complexity has thus far prevented it from being widely adopted. 




§ DISCUSSION

This paper introduces meta-embeddings, which are intended as a future alternative to i-vectors and x-vectors in speaker recognition, and indeed in other areas of machine learning as an alternative to traditional embeddings. The chief motivation for meta-embeddings is to build discriminatively trainable recognizers that allow the principled propagation of uncertainty, all the way from the input to the final output. We expect these advantages to be most noticeable in applications with varying and sometimes challenging quality of the inputs.

We do not yet have a full meta-embedding replacement for i-vectors or x-vectors, but were able to demonstrate the utility of our new design principles by creating a new i-vector scoring backend that is more accurate than the long-standing state of the art represented by length normalization and Gaussian PLDA. For SRE'10, we showed a 20% relative improvement in EER and for SRE'16 a 1% absolute improvement. This improvement was achieved purely by replacing the backend—without resorting to data augmentation, fusion, domain adaptation, or score normalization.

Our ongoing work on meta-embeddings can be followed on GitHub at <cit.>.




 §.§ The shrinkage problem

Although our results show that heavy-tailed PLDA performs better than Gaussian PLDA on i-vectors without length normalization, there remains a problem with i-vectors extracted from recordings with short durations. In the usual i-vector extractor, the effect of the standard normal prior is to shrink short-duration i-vectors towards the origin. The heavy-tailed PLDA model breaks down in such cases, because  when _j'_j decreases, it extracts meta-embeddings with higher precision. This is the opposite of what we want—shorter durations should give more uncertainty, not less. This inconsistency can be explained as follows. In an ideal world, the whole PLDA model should form the prior for the i-vector extractor. This is however not practical and we are forced to compromise by using the simpler standard normal prior instead. 

We conjecture that the π-vectors of <cit.> might help to mitigate this problem. The π-vectors are extracted similarly to i-vectors, but without the regularizing prior and do not shrink towards zero for short durations. Another way to see it is that π-vectors are point-estimates extracted from the i-vector likelihood function, rather than from the i-vector posterior. The likelihood function is uncontaminated by the inaccurate prior. 

This is however not a completely satisfactory solution, because we are still going via a point-estimate, which does not properly convey the uncertainty inherent in short-duration recordings. As mentioned, our next goal is to construct meta-embedding extractors that work directly on the acoustic features, rather than via i-vector-like point-estimates.




§ FAQ FOR FUTURE RESEARCH

𝐱
𝐲
Q: I have an existing DNN that extracts x-vectors from MFCCs, followed by a Gaussian PLDA backend. How do I generalize this to a meta-embedding extractor?

A: One solution is to replace your PLDA with our new GME backend. Replace our _j with your x-vectors. Initialize the backend from PLDA (with D ≫ d) as we did. It is probably not worthwhile learning ν, just fix it to say 1 or 2. Discriminative training of the GME backend (with fixed extractor) may already improve accuracy, as it did for us. The next step is to jointly optimize the extractor and the backend. This should encourage the uncertainty to be propagated from the extractor to the backend, through the (D-d)-dimensional complement of the speaker subspace.

A: Variants of the above recipe, where the interface between the original extractor and the back-end is simplified may ultimately give better solutions, but they may be more difficult to initialize. For example, one could replace (<ref>) and (<ref>) as follows. Split your D-dimensional x-vector into two parts: say _j∈^d and _j∈^D-d. Then do b_j=ν+D-d/ν+', _j=b_j_j and _j=b_j, where  is a trainable diagonal matrix with positive entries.





Q: Which discriminative training criterion should I use? 

A: In our experience, the multiclass cross-entropy that is currently used to train the Kaldi x-vector extractors of <cit.> and <cit.> will not work for training the GME backend, and by implication also not for the joint training stage. Do use your existing method to pre-train your extractor, but then for the further training, change the criterion. The first option to try is BXE as used in this paper. We were not successful in using BXE to train from random initialization, but it did work after PLDA initialization. A look at pseudolikelihood, or some of the other criteria proposed in <cit.> may be worthwhile. 


Q: Are there any important tricks that are not mentioned in the paper?

A: Yes. For example, to simplify the backend, set = and learn a linear transform of the _j instead.  Constrain (or coerce with a suitable L2 regularization penalty) = ' to be diagonal. We would be happy to assist with further details.

A: The generative heavy-tailed PLDA model can be used to generate synthetic data, with known properties. We found this invaluable in experiments to explore various discriminative training criteria. 




§ ACKNOWLEDGEMENTS

This work was started at the Johns Hopkins University HLTCOE SCALE 2017 Workshop. The authors thank the workshop organizers for inviting us to attend and (in the case of Niko Brümmer) for generous travel funding. Themos Stafylakis has been funded by the European Commission program Horizon 2020, under grant agreement no.706668 (Talking Heads). The work was also supported by Czech Ministry of Interior project No. VI20152020025 “DRAPAK” Google Faculty Research Award program,  Technology Agency of the Czech Republic project No. TJ01000208 “NOSICI”, and by Czech Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPU II) project “IT4Innovations excellence in science - LQ1602”. 



IEEEtran






