Infinite Mixture of Inverted Dirichlet Distributions
Zhanyu Ma Yuping Lai

figures/












Shell et al.: Bare Advanced Demo of IEEEtran.cls for Journals



In this work, we develop a novel Bayesian estimation method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the VI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichelt mixture model (InIDMM) that allows the automatic determination of the number of mixture components from data. Therefore, the problem of pre-determining the optimal number of mixing components has been overcome. Moreover, the problems of over-fitting and under-fitting are avoided by the Bayesian estimation approach. Comparing with several recently proposed DP-related methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations.



Dirichlet process, inverted Dirichlet distribution, Bayesian estimation, extended variational inference, lower bound approximation


Infinite Mixture of Inverted Dirichlet Distributions
    Zhanyu Ma and Yuping Lai

Preliminary work of ongoing work.

    December 30, 2023 (Original: July 27, 2018)
================================================================











§ INTRODUCTION

Finite mixture modeling <cit.> is a flexible and powerful probabilistic modeling tool for univariate and multivariate data that are assumed to be generated from heterogeneous populations. It has been widely applied to many areas, such as pattern recognition, machine learning, data mining, computer vision <cit.>. Among all finite mixture models, the finite Gaussian mixture model (GMM) has been the most popular method for modeling continuous data. Much of its popularity is due to the fact that any continuous distribution can be arbitrarily well approximated by a GMM with unlimited number of mixture components. Moreover, the parameters in a GMM can be estimated efficiently via maximum likelihood (ML) estimation with the expectation maximum (EM) algorithm <cit.>. By assigning prior distributions to the parameters in a GMM, Bayesian estimation of GMM can be carried out with conjugate prior-posterior pair matching <cit.>. Both the ML and the Bayesian estimation algorithms can be represented by an analytically tractable form <cit.>.

Recent studies have shown that non-Gaussian statistical models, e.g., the beta mixture model (BMM), the Dirichlet mixture model (DMM), the Gamma mixture model (GaMM) <cit.>, the von Mises-Fisher mixture model (vMM) <cit.>, can model the non-Gaussian distributed data more efficiently, compared to the conventional GMM. For example, BMM has been widely applied in modeling grey image pixel values and DNA methylation data <cit.>. In order to efficiently model proportional data <cit.>, DMM can be utilized to describe the underlying distribution. In generalized-K (K_G) fading channels, GaMM has been used to analyze the capacity and error probability <cit.>. The vMM has been widely used in modeling directional data, such as yeast gene expression <cit.> and topic detection <cit.>. The finite inverted Dirichlet mixture model (IDMM), among others, has been demonstrated to be an efficient tool for modeling data vector with positive elements <cit.>. For instance, IDMM has been widely used for software module classification <cit.>, visual scene analysis and classification <cit.>.

An essential problem in finite mixture modeling, no matter with Gaussian or non-Gaussian mixture component, is how to automatically decide the appropriate number of mixture components based on the data. The component number has a strong effect on the modeling accuracy <cit.>. If the number of mixture components is not properly chosen, the mixture model may over-fit or under-fit the observed data. To deal with this problem, many methods have been proposed. These can be categorized into two groups: deterministic approaches <cit.> and Bayesian methods <cit.>. Deterministic approaches are generally implemented by ML estimation under an EM-based and require the integration of entropy measures or some information theoretic criteria, such as the minimum message length (MML) <cit.>, the Bayesian information criterion (BIC) <cit.>, and the Akaike information criterion (AIC) <cit.>, to determine the number of components in the mixture model. It is worth noting that, in general, the EM algorithm converges to a local maximum or a saddle point and its solution is highly dependent on its initialization. On the other hand, the Bayesian methods, which are not sensitive to initialization by introducing proper prior distributions to the parameters in the model, have been widely used to find a suitable number of components in a finite mixture model. In this case, the parameters of a finite mixture model (including the parameters in a component and the weighting coefficients) are treated as random variables under the Bayesian framework. The posterior distributions of the parameters, rather than simple point estimates, are computed <cit.>. The model truncation in Bayesian estimation of finite mixture model is carried out by setting the corresponding weights of the unimportant mixture components to zero (or a small value close to zero) <cit.>. However, the number of mixture components should be properly initialized, as it can only decrease during the training process.

The increasing interest in mixture modeling has led to the development of the model selection methodHere, model selection means selecting the best of a set of models of different orders. Recent work has shown that the non-parametric Bayesian approach <cit.> can provide an elegant solution for automatically determining the complexity of model. The basic idea behind this approach is that it provides methods to adaptively select the optimal number of mixing components, while also allows the number of mixture components to remain unbounded. In other words, this approach allows the number of components to increase as new data arrives, which is the key difference from finite mixture modeling. The most widely used Bayesian nonparametric <cit.> model selection method is based on the Dirichlet process (DP) mixture model <cit.>. The DP mixture model extends distributions over measures, which has the appealing property that it does not need to set a prior on the number of components. In essence, the DP mixture model can also be viewed as an infinite mixture model with its complexity increasing as the size of dataset grows. Recently, the DP mixture model has been applied in many important applications. For instance, the DP mixture model has been adopted to a mixture of different types of non-Gaussian distributions, such as the DP mixture of beta-Liouville distributions <cit.>, the DP mixture of student's-t distributions <cit.>, the DP mixture of generalized Dirichlet distributions <cit.>, the DP mixture of student's-t factors <cit.>, and the DP mixture of hidden Markov random field models <cit.>.

Generally speaking, most parameter estimation algorithms for both the deterministic and the Bayesian methods are time consuming, because they have to numerically evaluate a given model selection criterion <cit.>. This is especially true for the fully Bayesian Markov chain Monte Carlo (MCMC) <cit.>, which is one of the widely applied Bayesian approaches with numerical simulations. The MCMC approach has its own limitations, when high-dimensional data are involved in the training stage <cit.>. This is due to the fact that its sampling-based characteristics yield a heavy computational burden and it is difficult to monitor the convergence in the high-dimensional space. To overcome the aforementioned problems, variational inference (VI), which can provide an analytically tractable solution and good generalization performance, has been proposed as an efficient alternative to the MCMC approach <cit.>. With an analytically tractable solution, the numerical sampling during each iteration in the optimization stage can be avoided. Hence, the VI-based solutions can lead to more efficient estimation. They have been successfully applied in a variety of applications including the estimation of mixture models <cit.>.

Motivated by the ability of the Bayesian non-parametric approaches to solve the model selection problem and the good performance recently obtained by the VI framework, we focus on the variational learning of the DP mixture of inverted Dirichlet distributions (a.k.a. the infinite inverted Dirichlet mixture model (InIDMM)). Since InIDMM is a typical non-Gaussian statistical model, it is not feasible to apply the traditional VI framework to obtain an analytically tractable solution for the Bayesian estimation. To derive an analytically tractable solution for the variational learning of InIDMM, the recently proposed extended variational inference (EVI), which is particularly suitable for non-Gaussian statistical models, has been adopted to provide an appropriate single lower bound approximation to the original object function. With the auxiliary functiond, an analytically tractable solution for Bayesian estimation of InIDMM is derived. The key contributions of our work are three-fold: 1) The finite inverted Dirichlet mixture model (IDMM) has been extended to the infinite inverted Dirichlet mixture model (InIDMM) under the stick-breaking framework <cit.>. Thus, the difficulty in automatically determining the number of mixture components can be overcome. 2) An analytically solution is derived with the EVI framework for InIDMM. Moreover, comparing with the recently proposed algorithm for InIDMM <cit.>, which is based on multiple lower bound approximation, our algorithm can not only theoretically guarantee convergence but also provide better approximations. 3) The proposed method has been applied in several important applications, such as image categorization and object detection. The good performance has been illustrated with both synthesized and real data evaluations.

The remaining part of this paper is organized as follow: Section <ref> provides a brief overview of the finite inverted Dirichlet mixture and the DP mixture. The infinite inverted Dirichlet mixture model is also proposed. In Section <ref>, a Bayesian learning algorithm with EVI is derived. The proposed algorithm has an analytically tractable form. The experimental results with both synthesized and real data evaluations are reported in Section <ref>. Finally, we draw conclusions and future research directions in Section <ref>.



§ THE STATISTICAL MODEL


In this section, we first present a brief overview of the finite inverted Dirichlet mixture model (IDMM). Then, the DP mixture model with stick-breaking representation is introduced. Finally, we extend the IDMM to InIDMM.


 §.§ Finite inverted Dirichlet mixture model

Given a D-dimensional vector x⃗ = {x_1, ⋯ ,x_D} generated from a IDMM with M components, the  probability density function (PDF) of x⃗ is denoted as <cit.>

    IDMM(x⃗|π⃗,Λ)=∑_m=1^Mπ_miDir(x⃗|α⃗_m),

where Λ={α⃗_m}_m=1^M and π⃗= {π_m}_m=1^M is the mixing coefficient vector subject to the constraints 0≤π_m≤ 1 and ∑_m=1^Mπ_m=1. Moreover, iDir(x⃗|α⃗) is an inverted Dirichlet distribution with its (D+1)-dimensional positive parameter vector α⃗={α_1,⋯ ,α_D+1} defined as

    iDir(x⃗|α⃗)=Γ (∑_d=1^D+1α_d)/∏_d=1^D+1Γ (α_d)∏_d=1^Dx_d^α_d-1( 1+∑_d=1^Dx_d)^-∑_d=1^D+1α_d,

where x_d>0 for d=1,⋯,D and Γ (·) is the Gamma function defined as Γ (a)=∫_0^∞t^a-1e^-tdt.




 §.§ Dirichlet Process with Stick-Breaking


The Dirichlet process (DP) <cit.> is a stochastic process used for Bayesian nonparametric data analysis, particularly in a DP mixture model (infinite mixture model). It is a distribution over distributions rather than parameters, i.e., each draw from a DP is a probability distribution itself, rather than a parameter vector <cit.>. We adopt the DP to extend the IDMM to the infinite case, such that the difficulty of the automatic determination of the model complexity (i.e., the number of mixture components) can be overcome. To this end, the DP is constructed by the following stick-breaking formulation <cit.>, which is an intuitive and simple constructive definition of the DP.

Assume that H is a random distribution and φ is a positive real scalar. We consider two countably infinite collections of independently generated stochastic variables Ω_m∼ H and λ_m∼Beta(λ_m;1,φ )To avoid confusion, we use f(x;a) to denote the PDF of x parameterized by parameter a. f(x|a) is used to denote the conditional PDF of x given a, where both x and a are random variables. Both f(x;a) and f(x|a) have exactly the same mathematical expressions. for m={1,⋯,∞}, where Beta(x;a,b) is the beta distribution defined as Beta(x;a,b)=Γ(a+b)/Γ(a)Γ(b)x^a-1(1-x)^b-1. A distribution G is said to be DP distributed with a concentration parameter φ and a base measure or base distribution H (denoted as G∼DP(φ ,H)), if the following conditions are satisfied:

    G=∑_m=1^∞π_mδ_Ω_m, π_m=λ_m∏_l=1^m-1(1-λ_l),

where {π_m} is a set of stick-breaking weights with constraints ∑_m=1^∞π_m=1, δ_Ω_m is a delta function whose value is 1 at location Ω_m and 0 otherwise. The generation of the mixing coefficients {π_m} can be considered as process of breaking a unit length stick into an infinite number of pieces. The length of each piece, λ_m, which is proportional to the rest of the “stick" before the current breaking, is considered as an independent random variable generated from Beta(λ_m;1,φ ). Because of its simplicity and natural generalization ability, the stick-breaking construction has been a widely applied scheme for the inference of DPs <cit.>.



 §.§ Infinite Inverted Dirichlet Mixture Model

Now we consider the problem of modeling x⃗ by an Infinite Inverted Dirichlet Mixture Model (InIDMM), which is actually an extended IDMM with an infinite number of components. Therefore, (<ref>) can be reformulated as

    InIDMM(x⃗|π⃗,Λ )=∑_m=1^∞π_miDir(x⃗|α⃗_m),

where π⃗={π_m}_m=1^∞ and Λ={α⃗_m}_m=1^∞
Then, the likelihood function of the InIDMM given the observed dataset 𝒳={x⃗_n}_n=1^N is given by

    InIDMM(𝒳|π⃗,Λ )=∏_n=1^N{∑_m=1^∞π_miDir(x⃗_n|α⃗_m) }.

In order to clearly illustrate the generation process of each observation x⃗_n in the mixture model, we introduce a latent indication vector variable
z⃗_n={z_n1,z_n2, ⋯}. z⃗ has only one element equal to 1 and the other elements in z⃗ are 0. For example, z_nm=1 indicates the sample x⃗_n comes from the mixture component m. Therefore, the conditional distribution of 𝒳 given the parameters Λ and the latent variables 𝒵={z_nm} is

    InIDMM(𝒳|𝒵,Λ )=∏_n=1^N∏_m=1^∞iDir(x⃗_n|α⃗_m)^z_nm.

Moreover, to exploit the advantages of the Bayesian framework, conjugate prior distributions are introduced for all the unknown parameters according to their distribution properties. In this work, we place the conjugate priors over the unknown stochastic variables 𝒵, Λ, and λ⃗=(λ_1, λ_2,⋯ ) such that a full Bayesian estimation model can be obtained.

In the aforementioned full Bayesian model, the prior distribution of 𝒵 given π⃗ is given by

    p(𝒵|π⃗)=∏_n=1^N∏_m=1^∞π _m^z_nm.

As π⃗ is a function of λ⃗ according to the stick-breaking construction of the DP as shown in (<ref>), we rewrite (<ref>) as

    p(𝒵|λ⃗)=∏_n=1^N∏_m=1^∞[ λ_m∏_l=1^m-1(1-λ_l)]^z_nm.


As previously mentioned in Section <ref>, the prior distribution of λ⃗ is

    p(λ⃗|φ⃗)=∏_m=1^∞Beta(λ_m;1,φ_m)=∏_m=1^∞φ_m(1-λ_m)^φ_m-1,

where φ⃗=(φ_1,φ_2,⋯ ). Based on (<ref>), we can obtain the expected value of π_m. In order to do this, the expected value of λ_m will first be calculated as

    ⟨λ_m⟩ =1/(1+φ_m).

Then, the expected value of π_m is denoted as

    ⟨π_m⟩ =⟨λ_m⟩∏_l=1^m-1(1-⟨λ_l⟩).

It is worth to note that, when the value of φ_m is small, ⟨λ_m⟩ will become large. Therefore, the expected of the mixing coefficients π_m are controlled by the parameters φ_m, i.e., small value of φ_m will yield small π_m such that the distribution of π_m will be sparse.

As φ_m is positive, we assume φ⃗ follows a product of gamma prior distributions as

    p(φ⃗;s⃗,t⃗)=∏_m=1^∞Gam(φ_m;s_m,t_m)=∏_m=1^∞t_m^s_m/Γ (s_m)φ _m^s_m-1e^-t_mφ_m,

where Gam(· ) is the gamma distribution. s⃗=(s_1,s_2,⋯ ) and t⃗=(t_1,t_2,⋯ ) are the hyperparamters and subject to the constraints s_m>0 and t_m>0.

Next, we introduce an approximating conjugate prior distribution to parameter Λ in InIDMM. The inverted Dirichlet distribution belongs to the exponential family and its formal conjugate prior can be derived with the Bayesian rule <cit.> as

    p(α⃗|μ⃗_0,v_0)=C(μ⃗_0,v_0)[ Γ (∑_d=1^D+1α_d)/∏_d=1^D+1α_d]^ν_0e^-μ⃗_0(α⃗^T-I⃗_D+1),

where μ⃗_0=[μ_1_0,⋯μ_D+1_0] and ν_0 are the hyperparameters in the prior distribution, C(μ⃗_0,v_0) is a normalization coefficient such that ∫p(α⃗|μ⃗_0,v_0)dα⃗=1. I⃗_d is a D-dimensional vector with all elements equal to one. Then, we can write the posterior distribution of α⃗ as (with N i.i.d. observations  𝒳)

    f(α⃗|𝒳)=iDir(𝒳|α⃗)f(α⃗|μ⃗_0,ν_0)/∫iDir(𝒳|α⃗)f(α⃗|μ⃗_0,ν_0)dα⃗ 
                   =C(μ⃗_N,ν_N)[ Γ (∑_d=1^D+1α_d)/∏_d=1^D+1Γ (α_d)]^ν_Ne^-μ⃗_N(α⃗^T-I⃗_D+1)

where the hyperparameters ν_N and μ⃗_N in the posterior distribution are




    ν_N=ν_0+N, μ⃗_N=μ⃗_0-[ln𝒳^+-I⃗_D+1ln (1+I⃗_D+1^T𝒳^+)]I⃗_N.

In (<ref>), 𝒳^+ is a (D+1)× N matrix by connecting I⃗_D+1^T to the bottom of 𝒳.
However, it is not applicable in our VI framework due to the analytically intractable normalization factor in (<ref>). Because Λ is positive, we adopt gamma prior distributions to approximate conjugate prior for Λ as well. By assuming the parameters of inverted Dirichlet distribution are mutually independent, we have

    p(Λ )=Gam(Λ ;U,V)=∏_m=1^∞∏_d=1^D+1v_md^u_md/Γ (u_md)α _md^u_md-1e^-v_mdα_md,

where all the hyperparameters U ={u_md} and V ={v_md} are positive.


With the Bayesian rules and by combining (<ref>) and (<ref>)-(<ref>) together, we can represent the joint density of the observation 𝒳 with all the i.i.d. latent variables Θ = (𝒵,Λ,λ⃗,φ⃗) as

    p(𝒳,Θ )=   p(𝒳|𝒵,Λ)p(𝒵|λ⃗)p(λ⃗|φ⃗)p(φ⃗)p(Λ) 
    
     =   ∏_n=1^N∏_m=1^∞[ λ_m∏_j=1^m-1(1-λ_j)Γ( ∑_d=1^D+1α_md)/∏_d=1^D+1Γ (α_md).
       .  ×∏_d=1^Dx_nd^α_md-1( 1+∑_d=1^Dx_nd)^-∑_d=1^D+1α_md]^z_nm
       ×∏_m=1^∞[ φ_m(1-λ_m)^φ_m-1t_m^s_m/Γ (s_m)φ _m^s_m-1e^-t_mφ_m] 
       ×∏_m=1^∞∏_d=1^D+1v_md^u_md/Γ (u_md)α _md^u_md-1e^-v _mdα_md.

The structure of the InIDMM can be represented in terms of a directed probabilistic graphical model, as shown in Fig. <ref>, which illustrates the relations among the variables and the observations.




§ VARIATIONAL LEARNING FOR INIDMM


In this section, we develop a variational Bayesian inference framework for learning the InIDMM. With the assistance of recently proposed EVI, an analytically tractable algorithm, which prevents numerical sampling during each iteration and facilitates a training procedure, is obtained. The proposed solution is also able to overcome the problem of overfitting and automatically decide the number of mixture components.


 §.§ Variational Inference

The purpose of Bayesian analysis is to estimate the values of the hyperparameters as well as the posterior probability distribution of the latent variables. Although we can formulate the posterior distribution p(Θ|𝒳) by using the Bayesian rule as

    p(Θ |𝒳)=p(𝒳,Θ )/p(𝒳).

The calculation of p(𝒳) from the joint distribution p(𝒳, Θ), which involves the summation and the integration over the latent variables, is analytically intractable for most of non-Gaussian statistical models. Therefore, we apply the VI framework to approximate the actual posterior p(𝒳|Θ) with an approximating distribution q(Θ). In principle, q(Θ) can be of arbitrary form. To make this approximation as close as possible to the actual posterior distribution, we can find the optimal approximation by minimizing the Kullback-Leibler (KL) divergence of p(Θ|𝒳) from q( Θ) as

    KL(q||p)=-∫q(Θ )ln{p( Θ |𝒳)/q(Θ )} dΘ.

By some mathematical manipulations, we obtain the following expression:

    ln p(𝒳)=∫q( Θ )ln{p(𝒳, Θ)/q( Θ )} dΘ_ℒ(q)-∫q( Θ )ln{p(Θ |𝒳)/q( Θ )}dΘ_KL(q||p),

where the KL divergence KL(q||p) is nonnegative and can be equal to zero if and only if q(Θ)=p(Θ|𝒳) <cit.>. However, it is infeasible to solve q(Θ) by minimizing KL(q( Θ )||p(Θ |𝒳)), as p(Θ|𝒳) is unknown. As the logarithm of the marginal evidence ln p(𝒳) is fixed by a given 𝒳, minimizing the KL divergence is equivalent to maximizing the lower bound ℒ(q) (which is known as the variational objective function). Hence, it is usual to find the optimal approximation q( Θ ) by maximizing ℒ(q) <cit.> in the VI framework. Within the variational inference framework, the variational objective function that needs to be maximized is

    ℒ(q)=E_q(Θ)[ln p(𝒳,Θ)]-E_q(Θ)[ln q(Θ)].



 §.§ Extended Variational Inference

For most of the non-Gaussian mixture models (e.g., the beta mixture model, the Dirichlet mixture model, the beta-Liouville mixture model <cit.>, the inverted Dirichlet mixture model <cit.>), the term E_q( Θ)[ln p(𝒳, Θ)] is analytically intractable such that the lower bound ℒ(q) cannot be maximized directly by a closed-form solution. Therefore, the EVI method <cit.> was proposed to overcome the aforementioned problem. With an auxiliary function p̃(𝒳, Θ ) that satisfies

    E_q( Θ )[ln p(𝒳, Θ )]≥E_q( Θ )[lnp̃(𝒳, Θ )]

and substituting (<ref>) into (<ref>), we can still reach the maximum value of ℒ(q) at some given points by maximizing a lower bound of ℒ̃(q)

    ℒ(q)≥ℒ̃(q)=E_q(Θ )[lnp̃(𝒳, Θ )]-E_q( Θ )[ln q( Θ )].

If p̃(𝒳, Θ ) is properly selected, an analytically tractable solution can be obtained. The strategy for selecting a proper p̃(𝒳, Θ ) can be found.

In order to properly formulate the variational posterior q(Θ), we truncate the stick-breaking representation for the InIDMM at a value M as

    λ _M = 1,    π _m = 0     when  m > M,     and∑_m = 1^M π _m  = 1.

Note that the model is still a full DP mixture. The truncation level M is not a part of our prior infinite mixture model, it is only a variational parameter for pursuing an approximation to the posterior, which can be freely initialized and automatically optimized without yielding overfitting during the learning process. Additionally, we make use of the following factorized variational distribution to approximate p( Θ |𝒳) as

    q(Θ ) =   [ ∏_m=1^Mq(λ_m)q(φ_m)][ ∏_n=1^N∏_m=1^Mq(z_nm)] 
       ×[ ∏_m=1^M∏_d=1^D+1q(α_md)],

where the variables in the posterior distribution are assumed to be mutually independent (as illustrated by the graphical model in Fig. <ref>). This is the only assumption we introduced to the posterior distribution. No other restrictions are imposed over the mathematical forms of the individual factor distributions <cit.>.

Applying the full factorization formulation and the truncated stick-breaking representation for the proposed model, we can solve the variational learning by maximizing the lower bound  ℒ̃(q) shown in (<ref>). The optimal solution in this case is given by

    lnq_s(Θ_s)=⟨lnp̃(𝒳, Θ ) ⟩_j s+Con.,

where ⟨·⟩_j s refers to the expectation with respect to all the distributions q_j(Θ _j) except for variable s. In addition, any term that does not include Θ_s are absorbed into the additive constant “Con.” <cit.>. In the variational inference, all factors q_s(Θ _s) need to be suitably initiated, then each factor is updated in turn with a revised value obtained by (<ref>) using the current values of all the other factors. Convergence is theoretically guaranteed since the lower bound is a convex with respect to each factor q_s(Θ_s) <cit.>. It is is worth noting that, although convergence is promised, the algorithm may also fall in local maxima or saddle points.



 §.§ EVI for the Optimal Posterior Distributions

According to the principles of EVI, the expectation of the logarithm of the joint distribution, given the joint posterior distributions of the parameters, can be expressed as

    ⟨lnp(𝒳,Θ )⟩
    
     =   ∑_n=1^N∑_m=1^M⟨z_nm⟩[ ℛ_m+∑_d=1^D(⟨α_md⟩-1)lnx_nd. 
       . -∑_d=1^D+1⟨α_md⟩(1+∑_d=1^Dx_nd)+⟨lnλ_m⟩ +∑_j=1^m-1⟨ln (1-λ_j)⟩] 
       +∑_m=1^M[ ⟨lnφ_m⟩ +(⟨φ_m⟩ -1)⟨ln (1-λ_m)⟩] 
       +∑_m=1^M∑_d=1^D+1[ (u_md-1)⟨lnα_md⟩ -v_md⟨α_md⟩] 
         +∑_m=1^M[ (s_m-1)⟨lnφ_m⟩ -t_m⟨φ_m⟩]+Con.,

where ℛ_m=⟨lnΓ (∑_d=1^D+1α_md)/∏_d=1^D+1Γ (α_md)⟩.




With the mathematical expression in (<ref>), an analytically tractable solution is not feasible, which is due to the fact that ℛ_m cannot be explicitly calculated (although it can be simulated by some numerical sampling methods). In order to apply (<ref>) to explicitly calculate the optimal posterior distributions and with the principles of the EVI framework, it is required to introduce an auxiliary function ℛ̃_̃m̃ such that ℛ_m≥ℛ̃_m.



According to, we can select ℛ̃_m as

    ℛ̃_m
    
    =   lnΓ (∑_d=1^D+1⟨α_md⟩)/∏_d=1^D+1Γ (⟨α_md⟩)+∑_d=1^D+1[ Ψ (∑_k=1^D+1⟨α_md⟩)-Ψ (⟨α_md⟩) ]
       ×[ ⟨lnα_md⟩ -ln⟨α_md⟩]⟨α_md⟩,

where Ψ (· ) is the digamma function defined as Ψ (a)=∂lnΓ (a)/∂ a.

Substituting (<ref>) into (<ref>), a lower bound to ⟨lnp(𝒳,Θ )⟩ can be obtained as

    ⟨lnp̃(𝒳,Θ )⟩
    
     =   ∑_n=1^N∑_m=1^M⟨z_nm⟩[ ℛ̃_m+∑_d=1^D(⟨α_md⟩-1)lnx_nd. 
       . -∑_d=1^D+1⟨α_md⟩(1+∑_d=1^Dx_nd)+⟨lnλ_m⟩ +∑_j=1^m-1⟨ln (1-λ_j)⟩] 
       +∑_m=1^M[ ⟨lnφ_m⟩ +(⟨φ_m⟩ -1)⟨ln (1-λ_m)⟩] 
       +∑_m=1^M∑_d=1^D+1[ (u_md-1)⟨lnα_md⟩ -v_md⟨α_md⟩] 
         +∑_m=1^M[ (s_m-1)⟨lnφ_m⟩ -t_m⟨φ_m⟩]+Con..

With (<ref>), we can get analytically tractable solutions for optimally estimating the posterior distributions of 𝒵, λ⃗, φ⃗, and Λ. We now consider each of these in more detail:
1) The posterior distribution of q(𝒵)

As any term that is independent of z_nm can be absorbed into the additive constant, we have

    lnq^*(z_nm)=Con.+z_nm[ ℛ_m+⟨lnλ_m⟩ +∑_j=1^m-1⟨ln (1-λ_j)⟩. 
               +∑_d=1^D(⟨α_md⟩-1)lnx_nd+. ∑_d=1^D+1⟨α_md⟩ln(1+∑_d=1^Dx_nd) ],

which has same logarithmic form of the prior distribution (i.e., the categorial distribution). Therefore, we can write ln q^*(𝒵) as

    ln q^*(𝒵)=∑_n=1^N∑_m=1^Mz_nmlnρ_nm+Con.

with the definition that

    lnρ_nm=⟨lnλ_m⟩ +∑_j=1^m-1⟨ln (1-λ_j)⟩+ℛ̃_m
              +∑_d=1^D(⟨α_md⟩-1)lnx_nd-∑_d=1^D+1⟨α_md⟩(1+∑_d=1^Dx_nd).

Recalling that z_nm∈ (0,1) and ∑_m=1^Mz_nm=1, we define

    r_nm=ρ_nm/∑_m=1^Mρ_nm.

Taking the exponential of both sides of (<ref>), we have

    q^*(𝒵)=∏_n=1^N∏_m=1^Mr_nm^z_nm,

which is the optimal posterior distribution of 𝒵.

The posterior mean ⟨z_nm⟩ can be calculated as ⟨z_nm⟩ =r_nm. Actually, the quantities {r_nm} are playing a similar role as the  responsibilities in the conventional EM <cit.> algorithm.



2) The posterior distribution of q(λ⃗)

The optimal solution to the posterior distribution of λ_m is given by

    lnq(λ_m)=Con.+lnλ_m∑_n=1^N⟨z_nm⟩ 
                  +ln (1-λ_m)[ ∑_n=1^N∑_j=m+1^M⟨z_nj⟩ +⟨φ_m⟩ -1],

which has the logarithmic form of the beta prior distribution. Hence, the optimal posterior distribution is

    q(λ⃗)=∏_m=1^MBeta(λ_m;g_m^*,h_m^*),

where the hyperparameters s_m^* and q_m^* are

    g_m^*=1+∑_n=1^N⟨z_nm⟩,     h_m^*=⟨φ_m⟩+∑_n=1^N∑_j=m+1^M⟨z_nj⟩.


3) The posterior distribution of q(φ⃗)

For variable φ_m, we have

    ln q^*(φ_m)=Con.+s_mlnφ_m+[ ⟨ln (1-λ_m)⟩ -t_m]φ_m.

It can be observed that (<ref>) has the logarithmic form of the gamma prior distribution. By taking the exponential of the both sides of (<ref>), we have

    q^*(φ⃗)=∏_m=1^MGam(φ_m;s_m^*,t_m^*),

where the optimal solutions to the hyperparamters s_m^* and t_m^* are

    s_m^*=1+s^0_m,      t_m^*=t^0_m-⟨ln (1-λ_m)⟩,

where s^0_m and t^0_m denote the hyperparameters initialized in the prior distribution, respectively.

4) The posterior distribution of q(Λ)

Similar to the above derivations, for the variable α_md, 1≤ d≤ D+1, the optimal approximation to the posterior distribution is

    ln q^*(α_md)
    
    =   {⟨z_nm⟩[ Ψ (∑_k=1^D⟨α_mk⟩)-Ψ (⟨α_md⟩) ]+u^0_md-1 }×lnα_md
        -{v^0_md-∑_n=1^N⟨z_nm⟩[ lnx_nd-ln(1+∑_d=1^Dx_nd) ]}×α_md
        +Con.

Since the posterior distribution of α_md has the logarithmic form of gamma distribution, we have

    q^*(Λ)=∏_m=1^M∏_d=1^D+1Gam(α_md;u_md^*,v _md^*),

where the optimal solutions to the hyperparameters u_md^* and v_md^* are given by

    u_md^*=u^0_md+∑_n=1^N⟨z_nm⟩[ Ψ (∑_k=1^K+1⟨α_mk⟩)-Ψ (⟨α_md⟩) ]⟨α_md⟩

and

    v_md^*=v^0_md-∑_n=1^N⟨z_nm⟩[ lnx_nd-ln (1+∑_d=1^Dx_nd) ].

In the above equations, u^0_md and v^0_md are the hyperparameters in the prior distribution and we set x_n,D+1=1.
The following expectations are needed to calculate the aforementioned update equations:

    ⟨ln (1-λ_m)⟩ =   Ψ (h_m^*)-Ψ (g_m^*+h_m^*), 
    ⟨lnλ_m⟩ =   Ψ (g_m^*)-Ψ (g_m^*+t_m^*), 
    ⟨lnα_md⟩ =   Ψ (u_md^*)-ln v_md^*,
    ⟨φ_m⟩ =s_m^*/t_m^*,    ⟨α_md⟩=u_md^*/v_md^*.





 §.§ Full Variational Learning Algorithm

As can be observed from the above updating process, the optimal solutions for the posterior distributions are dependent on the moments evaluated with respect to the posterior distributions of the other variables. Thus, the variational update equations are mutually coupled. In order to obtain optimal posterior distributions for all the variables, iterative updates are required until convergence. With the obtained posterior distributions, it is straightforward to calculate the lower bound ℒ̃(q)

    ℒ̃(q)=   ∫q(Θ )lnp̃(Θ ,𝒳)/q(Θ )dΘ
    
      =   ⟨lnp̃(𝒳,Θ )⟩ -⟨ln q(Θ )⟩
    
      =   ⟨lnp̃(𝒳,Θ )⟩ -⟨ln q(𝒵)⟩ -⟨ln q(λ⃗)⟩
         -⟨ln q(φ⃗)⟩ -⟨ln q(Λ)⟩,

which is helpful in monitoring the convergence. In (<ref>), each term with expectation (i.e., ⟨·⟩) is evaluated with respect to all the variables in its argument as

    ⟨ln q(𝒵)⟩ =r_nmlnr_nm,


    ⟨ln q(λ⃗)⟩ =   ∑_m=1^M[ lnΓ (g_m^*+h_m^*)-lnΓ (g_m^*)-lnΓ (h_m^*) .
       . +(g_m^*-1)⟨lnλ_m⟩ +(h_m^*-1)⟨ln (1-λ_m)⟩],


    ⟨ln q(φ⃗)⟩ =   ∑_m=1^M[ s_m^*ln t_m^*-lnΓ (s_m^*) .
       . +(s_m^*-1)⟨lnφ_m⟩ -t_m^*φ̅_m],

and

    ⟨ln q(α⃗)⟩ =   ∑_m=1^M∑_d=1^D+1[ u_md^*ln v_md^*-lnΓ (u_m^*) .
       . +(u_m^*-1)⟨lnα_md⟩ -v_md^*α̅_md].

Additionally, ⟨lnp̃(𝒳,Θ )⟩ is given in (<ref>) .

The algorithm of the proposed EVI-based Bayesian estimation of InIDMM is summarized in Algorithm <ref>.





§ EXPERIMENTAL RESULTS


In this section, both synthesized data and real data are utilized to demonstrate the performance of the proposed algorithm for InIDMM. In the initialization stage of all the experiments, the truncation level M is set to 15 and the hyperparameters of the gamma prior distributions are chosen as u_0=s_0=1 and v_0=t_0=0.005, which provide non-informative prior distributions. Note that these specific choices were based on our experiments and were found convenient and effective in our case. We take the posterior means as point estimates to the parameters in an InIDMM.


 §.§ Synthesized Data Evaluation


As shown in the previous studies for EVI-based Bayesian estimation, the single lower bound (SLB) approximation can guarantee the convergence while the multiple lower bound (MLB) approximation cannot. We use the synthesized data evaluation to compare the Bayesian InIDMM using the SLB approximation (proposed in this paper and denoted as InIDMM_SLB) with the Bayesian InIDMM using the MLB approximation (proposed in <cit.> and denoted as InIDMM_MLB). Three models (see Tab. <ref> for details) were selected to generate the synthesized datasets.








  §.§.§ Observations of Oscillations

We ran the InIDMM_MLB algorithm and monitored the value of the variational objective function during each iteration. It can be observed that the variational objective function was not always increasing in Bayesian estimation with the InIDMM_MLB. Figure <ref> illustrates the decreasing values during iterations. On the other hand, the variational objective function obtained with the InIDMM_SLB algorithm was always increasing until convergence, as the SLB approximation insures the convergency theoretically. The observations of oscillations demonstrate that the convergence with MLB approximation cannot be guaranteed. The original variational object function was numerically calculated by employing sampling method. In order to monitor the parameter estimation process of InIDMM_SLB, we show the value of the variational objective function during iterations in Fig. <ref>. It can be observe that the variational objective function obtained by InIDMM_SLB increases during iterations and in most cases it increases very fast.














































  §.§.§ Quantitative Comparisons

Next, we compare the InIDMM_SLB with the InIDMM_MLB quantitatively. With a known IDMM, 2000 samples were generated. The InIDMM_SLB and the InIDMM_MLB were applied to estimate the posterior distributions of the model, respectively. In Tab. <ref>, we list the estimated parameters  by taking the posterior means. It can be observed that, both the InIDMM_SLB and the InIDMM_MLB can carry out the estimation properly. However, with 20 repeats of the aforementioned “data generation-model estimation” procedure and calculating the variational objective function with sampling method, superior performance of the InIDMM_SLB over the InIDMM_MLB can be observed from Tab. <ref>. The mean values of the objective function obtained by InIDMM_SLB are larger than those obtained by the InIDMM_SLB while the computational cost (measured in seconds) required by the InIDMM_SLB are smaller than those required by the InIDMM_MLB. Moreover, smaller KL divergencesHere, the KL divergence is calculated as KL(p(𝒳|Θ)p(𝒳|Θ)) by sampling method. Θ denotes the point estimate of the parameters from the posterior distribution. of the estimated models from the corresponding true models also verify that the InIDMM_SLB yields better estimates than the InIDMM_MLB. In order to examine if the differences between the InIDMM_SLB and the InIDMM_MLB are statistically significant, we conducted the student's t-test with the null-hypothesis that the results obtained by these two methods have equal means and equal but unknown variances. All the p-values of in Tab. <ref> are smaller than the significant level 0.1, which indicates that the superiority of the InIDMM_SLB over the InIDMM_MLB is statistically significant. The distributions of the objective function values are shown by the boxplots in Fig. <ref>.


























































































































































































§ CONCLUSIONS


The inverted Dirichlet distribution has been widely applied in modeling the positive vector (vector that contains only positive elements). The Dirichlet processing mixture of the inverted Dirichlet mixture model (InIDMM) can provide good modeling performance to the positive vectors. Compared to the conventional finite inverted Dirichlet mixture model (IDMM), the InIDMM has more flexible model complexity as the number of mixture components can be automatically determined. Moreover, the over-fitting and under-fitting problem is avoided by the Bayesian estimation of InIDMM. To obtain an analytically tractable solution for Bayesian estimation of InIDMM, we utilized the recently proposed extended variational inference (EVI) framework. With single lower bound (SLB) approximation, the convergence of the proposed analytically tractable solution is guaranteed, while the solution obtained via multiple lower bound (MLB) approximations may result in oscillations of the variational objective function. Extensive synthesized data evaluations and real data evaluations demonstrated the superior performance of the proposed method.



IEEEtran
10

url@samestyle








01
B. Everitt and D. Hand, Finite Mixture Distributions.1em plus
  0.5em minus 0.4emChapman and Hall, London, UK, 1981.

02
G. McLachlan and D. Peel, Finite Mixture Models.1em plus 0.5em
  minus 0.4emNew York, NY, USA: Wiley, 2000.

28
C. M. Bishop, Pattern Recognition and Machine Learning (Information
  Science and Statistics).1em plus 0.5em minus 0.4emSpringer-Verlag New York, Inc., 2006.

03
N. Bouguila, D. Ziou, and J. Vaillancourt, “Unsupervised learning of a finite
  mixture model based on the Dirichlet distribution and its application,”
  IEEE Transactions on Image Processing, vol. 13, no. 11, pp.
  1533–1543, 2004.

Reynolds1995
D. A. Reynolds and R. C. Rose, “Robust text-independent speaker identification
  using Gaussian mixture speaker models,” IEEE Transactions on Speech
  and Audio Processing, vol. 3, no. 1, pp. 72–83, 1995.

Nasios2006
N. Nasios and A. G. Bors, “Variational learning for Gaussian mixture
  models,” IEEE Transactions on Systems Man and Cybernetics Part B
  (Cybernetics), vol. 36, no. 4, pp. 849–862, July 2006.


Jung2014
J. Jung, S. R. Lee, H. Park, S. Lee, and I. Lee, “Capacity and error
  probability analysis of diversity reception schemes over generalized-K
  fading channels using a mixture gamma distribution,” IEEE Transactions
  on Wireless Communications, vol. 13, no. 9, pp. 4721–4730, Sept 2014.

Taghia2014
J. Taghia, Z. Ma, and A. Leijon, “Bayesian estimation of the von-Mises
  Fisher mixture model with variational inference,” IEEE Transactions
  on Pattern Analysis and Machine Intelligence, vol. 36, no. 9, pp.
  1701–1715, Sept 2014.

Houseman2008
E. A. Houseman, B. C. Christensen, R. F. Yeh, C. J. Marsit, M. R. Karagas,
  M. Wrensch, H. H. Nelson, J. Wiemels, S. Zheng, J. K. Wiencke, and K. T.
  Kelsey, “Model-based clustering of DNA methylation array data: a
  recursive-partitioning algorithm for high-dimensional data arising as a
  mixture of beta distributions,” Bioinformatics, vol. 9, p. 365, 2008.

Nascimento2012
J. M. P. Nascimento and J. M. Bioucas-Dias, “Hyperspectral unmixing based on
  mixtures of Dirichlet components,” IEEE Transactions on Geoscience
  and Remote Sensing, vol. 50, no. 3, pp. 863–878, March 2012.

Ma2013
Z. Ma, A. Leijon, and W. B. Kleijn, “Vector quantization of LSF parameters
  with a mixture of Dirichlet distributions,” IEEE Transactions on
  Audio, Speech, and Language Processing, vol. 21, no. 9, pp. 1777–1790, Sept
  2013.

Xu2018
P. Xu, Q. Yin, Y. Huang, Y.-Z. Song, Z. Ma, L. Wang, T. Xiang, W. B. Kleijn,
  and J. Guo, “Cross-modal subspace learning for fine-grained sketch-based
  image retrieval,” NEUROCOMPUTING, vol. 278, pp. 75–86, Feb. 2018.

Yu2018
H. Yu, Z. H. Tan, Z. Ma, R. Martin, and J. Guo, “Spoofing detection in
  automatic speaker verification systems using DNN classifiers and dynamic
  acoustic features,” IEEE Transactions on Neural Networks and Learning
  Systems, vol. PP, no. 99, pp. 1–12, 2018.

Ma2018
Z. Ma, J. H. Xue, A. Leijon, Z. H. Tan, Z. Yang, and J. Guo, “Decorrelation of
  neutral vector variables: Theory and applications,” IEEE Transactions
  on Neural Networks and Learning Systems, vol. 29, no. 1, pp. 129–143, Jan
  2018.




He2010
Q. He, K. Chang, E. P. Lim, and A. Banerjee, “Keep it simple with time: A
  reexamination of probabilistic topic detection models,” IEEE
  Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 10,
  pp. 1795–1808, Oct 2010.

10
T. Bdiri and N. Bouguila, “Positive vectors clustering using inverted
  Dirichlet finite mixture models,” Expert Systems with Applications,
  vol. 39, no. 2, pp. 1869–1882, 2012.

Bdiri2013
——, “Bayesian learning of inverted Dirichlet mixtures for SVM kernels
  generation,” Neural Computing and Applications, vol. 23, no. 5, pp.
  1443–1458, 2013.

Bdiri2013a
T. Bdiri, N. Bouguila, and D. Ziou, “Visual scenes categorization using a
  flexible hierarchical mixture model supporting users ontology,” in
  IEEE International Conference on TOOLS with Artificial Intelligence,
  2013, pp. 262–267.

Markley2010
S. C. Markley and D. J. Miller, “Joint parsimonious modeling and model order
  selection for multivariate Gaussian mixtures,” IEEE Journal of
  Selected Topics in Signal Processing, vol. 4, no. 3, pp. 548–559, June
  2010.

11
Z. Liang and S. Wang, “An EM approach to MAP solution of segmenting tissue
  mixtures: a numerical analysis.” IEEE Transactions on Medical
  Imaging, vol. 28, no. 2, pp. 297–310, 2009.

12
N. Bouguila and D. Ziou, “Unsupervised selection of a finite Dirichlet
  mixture model: an MML-based approach,” IEEE Transactions on
  Knowledge and Data Engineering, vol. 18, no. 8, pp. 993–1009, June 2006.

Richardson1996
S. Richardson and P. J. Green, “Corrigendum: On bayesian analysis of mixtures
  with an unknown number of components,” Journal of the Royal
  Statistical Society, vol. 60, no. 3, p. 661, 1996.

Huang2016
L. Huang, Y. Xiao, K. Liu, H. C. So, and J. K. Zhang, “Bayesian information
  criterion for source enumeration in large-scale adaptive antenna array,”
  IEEE Transactions on Vehicular Technology, vol. 65, no. 5, pp.
  3018–3032, May 2016.

Chen2013
X. Chen, “Using Akaike information criterion for selecting the field
  distribution in a reverberation chamber,” IEEE Transactions on
  Electromagnetic Compatibility, vol. 55, no. 4, pp. 664–670, Aug 2013.

Bousmalis2015
K. Bousmalis, S. Zafeiriou, L. P. Morency, M. Pantic, and Z. Ghahramani,
  “Variational infinite hidden conditional random fields,” IEEE
  Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9,
  pp. 1917–1929, Sept 2015.

Meila2016
M. Meilǎ and H. Chen, “Bayesian non-parametric clustering of ranking
  data,” IEEE Transactions on Pattern Analysis and Machine
  Intelligence, vol. 38, no. 11, pp. 2156–2169, Nov 2016.

Xu2016
Y. Xu, M. Megjhani, K. Trett, W. Shain, B. Roysam, and Z. Han, “Unsupervised
  profiling of microglial arbor morphologies and distribution using a
  nonparametric Bayesian approach,” IEEE Journal of Selected Topics in
  Signal Processing, vol. 10, no. 1, pp. 115–129, Feb 2016.

18
T. S. Ferguson, “A Bayesian analysis of some nonparametric problems,”
  Annals of Statistics, vol. 1, no. 2, pp. 209–230, 1973.

19
C. E. Antoniak, “Mixtures of Dirichlet processes with applications to
  Bayesian nonparametric problems,” Annals of Statistics, vol. 2,
  no. 6, pp. 1152–1174, 1974.



Hjort2010
N. L. Hjort, C. Holmes, P. Müller, and S. G. Walker, Eds., Bayesian
  Nonparametrics.1em plus 0.5em minus 0.4emCambridge University
  Press, 2010.

20
Y. W. Teh and D. M. Blei, “Hierarchical Dirichlet processes,” Journal
  of the American Statistical Association, vol. 101, no. 476, pp. 1566–1581,
  2006.

Foti2015
N. J. Foti and S. A. Williamson, “A survey of non-exchangeable priors for
  Bayesian nonparametric models,” IEEE Transactions on Pattern
  Analysis and Machine Intelligence, vol. 37, no. 2, pp. 359–371, Feb 2015.

17
W. Fan and N. Bouguila, “Online learning of a Dirichlet process mixture of
  beta-Liouville distributions via variational inference,” IEEE
  Transactions on Neural Networks and Learning Systems, vol. 24, no. 11, pp.
  1850–1862, 2013.

21
X. Wei and C. Li, “The infinite student's t -mixture for robust modeling,”
  Signal Processing, vol. 92, no. 1, pp. 224–234, 2012.

22
N. Bouguila and D. Ziou, “A Dirichlet process mixture of generalized
  Dirichlet distributions for proportional data modeling,” IEEE
  Transactions on Neural Networks, vol. 21, no. 1, pp. 107–122, 2010.

23
X. Wei and Z. Yang, “The infinite student's t -factor mixture analyzer for
  robust clustering and classification ,” Pattern Recognition,
  vol. 45, no. 12, pp. 4346–4357, 2012.



24
S. P. Chatzis and G. Tsechpenakis, “The infinite hidden Markov random field
  model.” IEEE Transactions on Neural Networks, vol. 21, no. 6, pp.
  1004–14, 2010.


Bouguila2007
N. Bouguila and D. Ziou, “High-dimensional unsupervised selection and
  estimation of a finite generalized Dirichlet mixture model based on minimum
  message length.” IEEE Transactions on Pattern Analysis and Machine
  Intelligence, vol. 29, no. 10, pp. 1716–31, Aug. 2007.

Bouguila2012
N. Bouguila, “Hybrid generative/discriminative approaches for proportional
  data modeling and classification,” IEEE Transactions on Knowledge and
  Data Engineering, vol. 24, no. 12, pp. 2184–2202, July 2012.

14
M. Wedel and P. Lenk, Markov Chain Monte Carlo.1em plus 0.5em
  minus 0.4emBoston, MA: Springer US, 2013, pp. 925–930.

Robert2007
C. P. Robert, The Bayesian Choice: From Decision-Theoretic Foundations to
  Computational Implementation.1em plus 0.5em minus 0.4emSpringer-Verlag New York, 2007.

Pereyra2016
M. Pereyra, P. Schniter, E. Chouzenoux, J. C. Pesquet, J. Y. Tourneret, A. O.
  Hero, and S. McLaughlin, “A survey of stochastic simulation and optimization
  methods in signal processing,” IEEE Journal of Selected Topics in
  Signal Processing, vol. 10, no. 2, pp. 224–241, Mar. 2016.






15
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An introduction
  to variational methods for graphical models,” Machine Learning,
  vol. 37, no. 2, pp. 183–233, 1999.

Ma2010a
Z. Ma and A. Leijon, “Modeling speech line spectral frequencies with dirichlet
  mixture models,” in Proceedings of INTERSPEECH, 2010.

06
J. Taghia and A. Leijon, “Variational inference for Watson mixture model,”
  IEEE Transactions on Pattern Analysis and Machine Intelligence,
  vol. 38, no. 9, pp. 1886–1900, 2015.


Paisley2015
J. Paisley, C. Wang, D. M. Blei, and M. I. Jordan, “Nested hierarchical
  Dirichlet processes,” IEEE Transactions on Pattern Analysis and
  Machine Intelligence, vol. 37, no. 2, pp. 256–270, Feb. 2015.

Fan2015
W. Fan and N. Bouguila, “Topic novelty detection using infinite variational
  inverted Dirichlet mixture models,” in IEEE International Conference
  on Machine Learning and Applications (ICMLA), Dec 2015, pp. 70–75.




Frigyik2010
B. A. Frigyik, A. Kapila, and M. R. Gupta, “Introduction to the Dirichlet
  distribution and related processes,” Department of Electrical Engineering,
  University of Washington, Tech. Rep., 2010.



Paisley2009
J. Paisley and L. Carin, “Hidden Markov models with stick-breaking priors,”
  IEEE Transactions on Signal Processing, vol. 57, no. 10, pp.
  3905–3917, June 2009.

27
S. Kullback and R. A. Leibler, “On information and sufficiency,” Annals
  of Mathematical Statistics, vol. 22, no. 22, pp. 79–86, 1951.

16
H. Attias, “A variational bayesian framework for graphical models,”
  Advances in Neural Information Processing Systems, vol. 12, pp.
  209–215, 2000.

30
A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from
  incomplete data via EM algorithm,” Journal of the Royal Statistical
  Society, vol. 39, pp. 1–38, 1977.








































