Complex-Valued Restricted Boltzmann Machine for Direct Speech Parameterization from Complex Spectra
Toru Nakashika Shinji Takaki Junichi Yamagishi










































Journal of  Class Files, Vol. 11, No. 4, December 2012
Shell et al.: Bare Demo of IEEEtran.cls for Journals


























Complex-Valued Restricted Boltzmann Machine for Direct Speech Parameterization from Complex Spectra
    Toru Nakashika, Member, IEEE,, Shinji Takaki, Member, IEEE
        and Junichi Yamagishi, Senior Member, IEEE
T. Nakashika is with the Graduate School
of Informatics and Engineering, the University of Electro-Communications, Tokyo,
Japan e-mail: nakashika@uec.ac.jp
S. Takaki and J. Yamagishi are with National Institute of Informatics,
Tokyo, Japan
e-mail: takaki@nii.ac.jp, jyamagis@nii.ac.jp
Manuscript received April 19, 2005; revised December 27, 2012.
    December 30, 2023
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================






This paper describes a novel energy-based probabilistic distribution that represents complex-valued data and explains how to apply it to direct feature extraction from complex-valued spectra.
The proposed model, the complex-valued restricted Boltzmann machine (CRBM), is designed to deal with complex-valued visible units as an extension of the well-known restricted Boltzmann machine (RBM).
Like the RBM, the CRBM learns the relationships between visible and hidden units without having connections between units in the same layer, which dramatically improves training efficiency by using Gibbs sampling or contrastive divergence (CD).
Another important characteristic is that the CRBM also has connections between real and imaginary parts of each of the complex-valued visible units that help represent the data distribution in the complex domain.
In speech signal processing, classification and generation features are often based on amplitude spectra (e.g., MFCC, cepstra, and mel-cepstra) even if they are calculated from complex spectra, and they ignore  phase information.
In contrast, the proposed feature extractor using the CRBM directly encodes the complex spectra (or another complex-valued representation of the complex spectra) into binary-valued latent features (hidden units).
Since the visible-hidden connections are undirected, we can also recover (decode) the complex spectra from the latent features directly.
Our speech coding experiments demonstrated that the CRBM outperformed other speech coding methods, such as methods using the conventional RBM, the mel-log spectrum approximate (MLSA) decoder, etc.




Restricted Boltzmann machine, deep learning, complex-valued representation, feature extraction, speech synthesis.
















§ INTRODUCTION

















Deep learning is one of the recent hottest topics in a wide range of research fields, such as artificial intelligence, machine learning, and signal processing that includes image classification, speech recognition, etc<cit.>.
Many models have been proposed as tools of deep learning; one of the most widely-used and famous models is a deep belief-net (DBN) <cit.> that stacks multiple restricted Boltzmann machines (RBMs) layer-by-layer.
The RBM is a probabilistic model that consists of visible and hidden units and has often been used alone as a feature extractor, a generator, and as a classifier and pre-training scheme of deep neural networks.
Many extensions of the RBM have been proposed for task specification <cit.>.
Although the RBM has been used in many tasks, the RBM traditionally identified visible units as either binary-valued or real-valued <cit.>.

Representations based on the amplitude spectra of speech (such as MFCC, cepstra, and mel-cepstra) are traditionally used in speech signal processing as input features of speech recognition or output features of speech synthesis because the amplitude spectra are more effective and relevant to our auditory field for such tasks than phase spectra.
Raw amplitude spectral representation can be also used <cit.>.
However, these features that include the amplitude spectra theoretically lack phase information, and single use of the amplitude-based features cannot completely recover the original complex spectra with reasonable computational resources easily, even when using the well-known Griffin-Lim algorithm <cit.>.
In <cit.>, it is reported that the generated speech signals from direct waveform modification or synthesis are much more natural than those from methods that are based on phase reconstruction from amplitude spectra.
Furthermore, there are many cases in other kinds of signal processing in which we have to deal with complex-valued actual data such as fMRI images, wireless signals, acoustic intensity, etc.
Other machine learning models—that is, neural networks, Boltzmann machines, and non-negative matrix factorization (NMF) <cit.>—have their extensions proposed to represent complex-valued data <cit.>.

In our previous work <cit.>, we proposed an extended model of the RBM, namely “complex-valued RBM (CRBM), ” to tackle representing complex-valued data in the RBM-based approach in particular.
The CRBM includes three important characteristics.
Firstly, the CRBM has no connections across dimensions in the same layers but has connections between visible and hidden units like the RBMs.
These restrictions make it exceedingly easy to estimate the parameters using Gibbs sampling or CD <cit.>, which cannot be seen in an extension of a Boltzmann machine (directional-unit Boltzmann machine (DUBM) <cit.>) that feeds complex-valued data that has connections across dimensions and has difficulties in parameter estimation.
Secondly, unlike the conventional RBM, the CRBM restricts the connections between different visible units but still has connections between real and imaginary parts of each visible unit.
Therefore, the CRBM represents the complex-valued data distribution more accurately than the RBMs, especially when there are correlations between the real and imaginary parts.
Thirdly, the CRBM represents the complex-valued visible units in a rectangular form that consists of real and imaginary components, while traditional representation methods of complex-valued data that include a DUBM are based on a polar form of phase and amplitude components.
We can generate samples from the distribution straightforwardly in the CRBM.
The conditional probability of visible units given hidden units form a complex-normal distribution, which makes the real and imaginary components Gaussian-distributed.
In <cit.>, we showed that the CRBM sufficiently recovered the amplitude and phase components as well as the real and imaginary components in the speech coding experiments.

We also propose some improvements and learning techniques for the CRBM-based speech parameterization.
First, we reduce the number of dimensions by feeding complex-valued visible features obtained by the complex principal component analysis (CPCA) <cit.> into the CRBM instead of the raw complex spectra.
Next, we employ the maximum likelihood parameter generation (MLPG) <cit.> to generate the trajectories of the CPCA features for better representation of speech sequences.
Finally, we extend the Adam algorithm to deal with the complex-valued parameters (referred to as “complex Adam” or “CAdam”), which makes convergence of the model training faster than the steepest descent/ascent.
In the experiments, we compare the performance of the improved CRBM method with other speech coding methods, such as the conventional RBM, the mel-log spectrum approximate (MLSA), etc.

This paper is organized as follows.
In Section 2, we briefly review the conventional RBM and we present our proposed model, CRBM, in Section 3.
In Section 4, we present improvement methods for the CRBM using the CPCA.
In Section 5, we propose a complex-valued sequence generation method based on MLPG.
In Section 6, we show our experimental results, and we conclude our findings in Section 7. 





§ PRELIMINARY


A restricted Boltzmann machine (RBM) is one of the most widely used energy-based models and is convenient for representing latent features that cannot be observed but surely exist in the background.
The Bernoulli-Bernoulli RBM (BB-RBM) RBM was originally introduced by Freund et. al <cit.>.
It defines the distribution of binary-valued visible variables v∈𝔹^I and binary-valued hidden (latent) variables h∈ℝ^J with their undirected real-valued connection weights 𝐖∈ℝ^I × J, as shown in Fig. <ref> (a) where I and J are the numbers of dimensions in their respective visible and hidden units and 𝔹≜{0, 1} indicates the binary set.
The RBM was later extended to deal with real-valued data known as a Gaussian-Bernoulli RBM (GB-RBM) <cit.>, as shown in Fig. <ref> (b).
However, it has been reported that there were some difficulties with the original GB-RBM because of the unstable training of the parameters.
Later, Cho et al. <cit.> proposed an improved learning method for a GB-RBM to overcome the difficulties.
In the remainder of this paper, we refer to this improved GB-RBM just as an RBM unless otherwise stated.
In the modeling using an RBM, the joint probability p(v,h) of real-valued visible units v and binary-valued hidden units h is defined as follows:

    p(v,h;θ)    = 1/U(θ) e^-E(v,h;θ)
    
    E(v,h;θ)    = 1/2v^⊤Σ^-1v - b^⊤Σ^-1v - c^⊤h - v^⊤Σ^-1𝐖h
    
    U(θ)    = ∫∑_h e^-E(v,h;θ) d v

where θ = {b,c,𝐖,σ} indicates a set of parameters that contains bias parameters of the visible units b∈ℝ^I, bias parameters of the hidden units c∈ℝ^J, the connection weight parameters between visible-hidden units 𝐖∈ℝ^I × J, and the standard deviation parameters associated with the dimension independent Gaussian visible units σ∈ℝ^I that define Σ≜ diag(σ^2) (the function diag(·) returns a diagonal matrix whose diagonal vector is the argument, and ·^2 indicates the element-wise square operation).
The parameters θ are often estimated using the maximum likelihood (ML) and the gradient descent/ascent given the training set D ∋v.
The partial gradients of the parameters to the expected log likelihood:

    L(θ) = 𝔼_D [log p(v; θ)] = 𝔼_D [log∑_h p(v, h; θ)]

can be calculated as:

    ∂ L/∂θ = ⟨ -∂ E/∂θ⟩_ data - ⟨ - ∂ E/∂θ⟩_ model,

where ⟨·⟩_ data and ⟨·⟩_ model indicate expectations of the training data and the inner model, respectively.
Although exact calculation of the inner model has an order of 2^I + J, the expectation value can be approximated using the Gibbs sampling, or more efficiently, the contrastive divergence (CD) <cit.>.
From the definition of RBM in Eqs. (<ref>), (<ref>), and (<ref>), the conditional probabilities p(v|h) and p(h|v) form quite simple distributions as:

    p(v|h)    = 𝒩(v; b + 𝐖h, Σ) 
    
    p(h|v)    = ℬ(h; f(c + 𝐖^⊤Σ^-1v))

where 𝒩(·; μ, Σ), ℬ(·; π), and f(·) indicate the multivariate Gaussian distribution with the mean μ and the variance matrix Σ, the multi-dimensional Bernoulli distribution with the success probabilities π, and an element-wise sigmoid function, respectively.
As Eqs. (<ref>) and (<ref>) indicates, we can easily compute the iteration of drawing samples h given v, and v given h, which is used in Gibbs sampling or CD.
The same is true for BB-RBM.
In the case of BB-RBM, the conditional probabilities p(v|h) and p(h|v) turn into the following:

    p(v|h)    = ℬ(v; f(b + 𝐖h) 
    
    p(h|v)    = ℬ(h; f(c + 𝐖^⊤v))

under the energy function:

    E(v,h;θ) = - b^⊤v - c^⊤h - v^⊤𝐖h.







§ COMPLEX-VALUED RBM


Conventional RBMs assume that data is either binary-valued or real-valued.
Therefore, complex-valued data should not be fed into the conventional RBMs directly because the conditional probability of visible units specifies binary- or real-valued variables, as Eqs. (<ref>) and (<ref>) indicate.
In other words, the conditional probability of visible units should specify complex-valued variables in order to feed complex-valued variables into the model.
In our approach, we define an extension of the RBM that feeds complex-valued data and forms the conditional probability of visible units as complex normal distribution <cit.>.
A real-valued cost function (the likelihood) is still used in parameter estimation for an extended RBM—namely a complex-valued RBM (CRBM—because the probability distribution is real-valued.
Like conventional RBMs, the CRBM consists of two layers: complex-valued visible units z and binary-valued hidden units h with undirected connection weights 𝐖, as shown in Fig. <ref> (a).
Furthermore, in the CRBM, wegive a “restriction” where there are no connections between visible units or hidden units, which enables easy estimation of parameters just as an RBM does.
However, we allow the model to have connections between the real and imaginary parts in order to capture the relationships between the real and imaginary parts of each complex-valued visible unit.



 §.§ Definition


Based on the above discussion, we formulated a CRBM that has I-dimensional complex-valued visible units z∈ℂ^I and J-dimensional binary-valued hidden units h∈𝔹^J as follows:

    p(z,h;θ) =   1/U(θ) e^-E(z,h;θ)
     E(z,h;θ) =   1/2[ [  z; z̅ ]]^H Φ^-1[ [  z; z̅ ]] 
       - [ [  b; b̅ ]]^H Φ^-1[ [  z; z̅ ]] - 2 c^⊤h
       - [ [  z; z̅ ]]^H Φ^-1[ [  𝐖; 𝐖̅ ]] h
    
    U(θ) =   ∫∑_h e^-E(z,h;θ) d z,

where ·̅ denotes complex-conjugate and ·^H denotes Hermitian-transpose.
b∈ℂ^I, c∈ℝ^J, and 𝐖∈ℂ^I × J are bias parameters of the visible units and the hidden units, and the biased connection weights between visible and hidden units, respectively.
In order to make the restrictions, the extended covariance matrix Φ consists of a covariance matrix Γ and a pseudo-covariance matrix 𝐂—both of which are diagonal matrices—as

    Φ≜[ [   Γ   𝐂; 𝐂^H Γ^H ]]

and

    [ Γ≜ diag(γ),      γ∈ℝ+^I; 𝐂≜ diag(δ),       δ∈ℂ^I ]

where γ and δ are variance and pseudo-variance parameters of the complex-valued visible units, respectively.
To summarize, the set of parameters of the CRBM is θ = {b,c,𝐖,γ,δ}.

Introducing auxiliary precision vectors p and q defined as

    p   ≜γ/γ^2 - |δ|^2∈ℝ^I 
    q   ≜ - δ/γ^2 - |δ|^2∈ℂ^I

where the fraction bar denotes element-wise division, we can rewrite the energy function in Eq. (<ref>) as follows:

    E(z,h;θ) = 
       z^H  diag(p) z + (z^H  diag(q)z̅) - 2(z^H  diag(p) b) 
       -2(z^H  diag(q) b̅) - 2 c^⊤h -2(z^H  diag(p)𝐖) h
       -2(z^H  diag(q)𝐖̅) h,

which confirms that 1) the above energy function E and the probability distribution are real-valued and that 2) there are connections between the complex-valued visible units and their conjugates for each dimension but there are no connections between different dimensions.

Furthermore, when we use unbiased parameters:

    b'    ≜ diag(p) b +  diag(q) b̅
    𝐖'    ≜ diag(p) 𝐖 +  diag(q) 𝐖̅,

the energy function E becomes

    E(z,h;θ) = 
       1/2z^H  diag(p) z + 1/2z̅^H  diag(p) z̅ + z^H  diag(q)z̅
       + z̅^H  diag(q̅) z - z^H b' - z̅^H b̅' - 2 c^⊤h
       - z^H 𝐖' h - z̅^H 𝐖̅' h,

which indicates that z and z̅ are symmetric to each other, as shown in Figure <ref> (b).

From the above definition, the conditional probabilities p(z|h) and p(h|z) can be derived as follows:

    p(z|h)    = 𝒞𝒩(z; b + 𝐖h, Γ, 𝐂) 
    
    p(h|z)    = ℬ(h; f(2 c + 2  (𝐖'^H z ) ))

where 𝒞𝒩(·; μ, Γ, 𝐂) is a multivariate complex normal distribution <cit.> a mean vector μ, a covariance matrix Γ, and a pseudo-covariance matrix 𝐂:

    p(z) = 1/π^D √( det(Γ) det(𝐐))
       ·exp{ -1/2[ [     z-μ; z̅ - μ̅ ]]^H [ [   Γ   𝐂; 𝐂^H Γ^H ]]^-1[ [     z-μ; z̅ - μ̅ ]] }
       𝐐 = Γ̅ - 𝐂^H Γ^-1𝐂.





 §.§ Parameter estimation


To estimate the model parameters θ of the CRBM, we employed the complex-valued gradient method.
In this approach, the parameters θ are estimated so as to maximize the expected log-likelihood L of the complex-valued training data set D ∋z:

    L(θ)    = 𝔼_D [log p(z;θ) ]
       = 𝔼_D [ log∑_h p(z,h;θ) ]
       = 𝔼_D [log∑_h e^-E(z,h;θ)] - log∫∑_h̃ e^-E(z̃,h̃;θ) dz̃.

The complex-valued gradient ascend iteratively updates each parameter as:

    θ^(l+1)θ^(l) + g^(l) (∂ L/∂θ)

where θ^(l) indicates the parameters and g^(l) indicates the complex-valued gradient at the l-th iteration.
One of the simplest gradient functions is the complex-valued steepest ascent (CSA) <cit.>, which is:

    g^(l) (∂ L/∂θ) = 2 α∂ L/∂θ̅

where α∈ℂ, (α)>0 is a complex-valued learning rate.
A simple CSA is not be suitable for a large amount of training data of speech due to the slow convergence speed.
Therefore, we propose another, more efficient learning method, the complex-valued adaptive momentum (CAdam), which is motivated by the real-valued Adam algorithm <cit.>.
In the CAdam, we introduce auxiliary parameters m^(n) and v^(n) and update the parameters as:

    m^(l)   = β_1 m^(n-1) + (1-β_1) ∇_θ̅ L 
    v^(l)   = β_2 v^(n-1) + (1-β_2) | ∇_θ̅ L |^2 
    Δθ^(l)   = 2 α1-β_2^l/1-β_1^lm^(l)/v^(l),

where β_1, β_2 ∈ℂ, 0 < |β_1|, |β_2| < 1, and α∈ℂ, (α)>0.

Calculating partial gradients to the parameters θ, we obtain:

    ∂ L/∂θ = ⟨ - ∂ E/∂θ⟩_ data - ⟨ - ∂ E/∂θ⟩_ model,

where the complex-valued partial gradients here indicate the Wirtinger derivatives:

    ∂ L/∂θ   = 1/2( ∂ L/∂(θ) - i ∂ L/∂(θ)) 
    ∂ L/∂θ̅   = 1/2( ∂ L/∂(θ) + i ∂ L/∂(θ)).

The negative partial gradients of the energy function with respect to each parameter - ∂ E/∂θ can be further derived as:

    - ∂ E/∂b   =  diag(p) z̅ +  diag(q̅) z
    
    - ∂ E/∂c   = h
    
    - ∂ E/∂𝐖   = ( diag(p) z̅ +  diag(q̅) z) h^⊤
    
    - ∂ E/∂γ   = (p^2 + |q|^2) ∘∂ E/∂p + 2 (p∘q∘∂ E/∂q) 
    
    - ∂ E/∂δ   = p^2 ∘∂ E/∂q + q̅^2 ∘∂ E/∂q̅ + 2 p∘q̅∘∂ E/∂p,

where ∘ denotes an element-wise product and |·| denotes the absolute, and

    ∂ E/∂p   = 1/2 |z |^2 - (z∘ (b̅ + 𝐖̅h)) 
    ∂ E/∂q   = 1/2z̅^2 - z̅∘ (b̅ + 𝐖̅h).

The gradients of variance and pseudo variance tend to be larger than those of the other parameters.
For stable training, we replace the parameters as γ≜ e^r and δ≜ e^s and update using the gradients of r and s in a manner similar to the improved GB-RBM <cit.>.

The second term on the right-hand side in Eq. (<ref>) usually requires high computational cost.
However, because of the restrictions of the CRBM, the second term can be efficiently approximated using  Gibbs sampling or CD <cit.> in a way similar to conventional RBMs.



 §.§ Relationships with complex representation using GB-RBM




We can also represent a complex-valued vector z = x + i y (where x, y∈ℝ^I) in the real space using a conventional GB-RBM that feeds the double-sized concatenated vector z' = [ x^⊤ y^⊤ ]^⊤∈ℝ^2I as:

    p(x,y)    = ∑	_h p(x,y,h) 
    
    p(x,y,h)    = 1/U e^-E(x,y,h)
     E(x,y,h)    = 1/2x^⊤Σ^-1_x x + 1/2y^⊤Σ^-1_y y
          - b_x^⊤Σ^-1_x x - b_y^⊤Σ^-1_y y - c^⊤h
          - x^⊤Σ^-1_x 𝐖_x h - y^⊤Σ^-1_y 𝐖_y h
    
    U    = ∫∫∑_h e^-E(x,y,h) dx dy,

where Σ_x =  diag (σ_x^2), Σ_y =  diag (σ_y^2) and we decompose the GB-RBM parameters as b = [ b_x^⊤ b_y^⊤ ]^⊤, σ = [ σ_x^⊤ σ_y^⊤ ]^⊤, 𝐖 = [ 𝐖_x^⊤ 𝐖_y^⊤ ]^⊤ in Eqs. (<ref>), (<ref>), and (<ref>).
Fig. <ref> (a) depicts this concatenating representation of the complex-valued data using GB-RBM.
For example, the negative partial differentials of the real and imaginary parts of the bias parameters to the energy function in this representation can be derived as:

    - ∂ E/∂b_x   = x/σ_x^2
     - ∂ E/∂b_y   = y/σ_y^2.


On the other hand, when we put z = x + i y, b = b^R + i b^I, 𝐖 = 𝐖^R + i 𝐖^I, q = q^R + i q^I where b^R, b^I ∈ℝ^I, 𝐖^R, 𝐖^I ∈ℝ^I × J, q^R, q^I ∈ℝ^I, we can rewrite the energy function of CRBM in Eq. (<ref>) as:

    E =    1/2x^⊤Σ^-1_x x + x^⊤𝐑_xyy + 1/2y^⊤Σ^-1_y y
       - b_x^⊤Σ^-1_x x - b_y^⊤Σ^-1_y y - c^⊤h
       - x^⊤Σ^-1_x 𝐖_x h - y^⊤Σ^-1_y 𝐖_y h,

where we introduce

    Σ_x    =  diag(1/2(p + q^R)) 
    Σ_y    =  diag(1/2(p - q^R)) 
    𝐑_xy   =  diag(2q^I) 
    b_x    = b^R + q^I/p+q^R∘b^I 
    b_y    = b^I + q^I/p-q^R∘b^R 
    𝐖_x    = 𝐖^R +  diag(q^I/p+q^R) 𝐖^I 
    𝐖_y    = 𝐖^I +  diag(q^I/p-q^R) 𝐖^R.


Comparing the energy functions in Eqs. (<ref>) and (<ref>), the latter energy function includes a cross term (x^⊤𝐑_xyy) between x and y while the former energy function does not.
Therefore, we can claim that the CRBM representation extends the conventional GB-RBM, which has connections between the real and imaginary parts for each dimension with the weights r_xy = 2 q^I as in Fig. <ref> (b), where r_xy is the diagonal vector of 𝐑_xy.

Furthermore, in the GB-RBM representation, the gradients regarding the real and imaginary parts of the bias of visible units, for example, are calculated independently of each other as Eqs. (<ref>) and (<ref>) indicate, while the gradients of the bias of visible units in the CRBM representation are calculated using both the real and imaginary terms as Eq. (<ref>) indicates.
This will make the model convergence better than the GB-RBM.




§ COMPLEX SPECTRA COMPRESSION USING CPCA



The aim of this paper is to represent trajectories of complex-valued speech spectra.
In general, the number of dimensions of the raw complex spectra tends to be large (e.g., when analyzing speech with the window length of 1,024, the complex spectra has the dimensions of 513), which makes it difficult to use dynamic features or segment features as input for the model due to the sizable number of parameters.
Therefore, we reduced the dimensions using the complex principal component analysis (CPCA) <cit.> in this paper.

Letting o_t be the complex spectra at the frame t, the complex-valued features z_t whose dimensions are reduced to P using CPCA calculated as:

    z_t = Λ_1:P^-1/2𝐔_:,1:P^H o_t,

where Λ_1:P^-1/2 and 𝐔_:,1:P indicate a diagonal matrix where the diagonal elements are the inverse of the top P eigenvalues of the empirical covariance matrix and a complex matrix whose columns are the complex eigenvectors corresponding to the eigenvalues.
Conversely, when we recover the complex spectra o_t from z_t, we just calculate the inversion as:

    o_t = 𝐔_:,1:PΛ_1:P^1/2z_t.


In our speech modeling experiments that will be discussed later, we used the concatenated features Z_t ≜ [z_t^H  Δz_t^H]^H as visible units in  CRBM, where z_t^H was calculated using the CPCA with the degree of P=40 from the complex spectra analyzed with the window length of 256, and their dynamics Δz_t^H were calculated as 0.5z_t+1 - 0.5z_t-1.
The total dimensions of visible units in CRBM were I=80.
In these experiments, the CRBM was trained so as to maximize the expected likelihood L(θ) = 𝔼[log p(Z;θ)] of the concatenated feature set.




§ COMPLEX-VALUED SEQUENCE GENERATION BASED ON MLPG


When we apply the CRBM to represent speech spectra, we need further improvements to compare it to other feature extraction methods of speech.
In this section, improved methods of dealing with trajectory modeling will be presented.

In our first work on the CRBM <cit.>, complex-valued visible units z_t were probabilistically encoded into binary values by calculating the expectations of hidden units as ĥ_t ≜𝔼[p(h_t|z_t)] and inversely decoded (recovered) from ĥ_t by calculating the expectations of visible units as ẑ_t ≜𝔼[p(z_t|ĥ_t)] = b + 𝐖ĥ_t frame-by-frame.
However, speech signals are sequences; there are correlations between adjacent frames of speech.
In this paper, we employ trajectory modeling and sequence generation instead of frame-wise modeling.
The proposed, efficient method recovers complex-valued visible units involving correlations among the neighbor frames based on the maximum likelihood parameter generation (MLPG) <cit.>.
The MLPG is the algorithm to estimate the optimum sequence of features from static and dynamic features based on a maximum likelihood estimation.
This cannot be directly applied to complex-valued features; therefore, we present the following formulation.

After training the CRBM, we estimated the optimum sequence of CPCA features ẑ_1:T≜ [ẑ_1^H  ẑ_2^H  ⋯ ẑ_T^H]^H, where T is the number of frames of the test speech, from the encoded features (the expectations of hidden units) ĥ_t ≜𝔼[p(h_t|z_t)] that were calculated from the original concatenated features Z_1:T≜ [Z_1^H  Z_2^H  ⋯ Z_T^H]^H of the test speech.
ẑ_1:T is the sequence that maximizes the conditional probability p(Z_1:T | ĥ_1:T), which is defined as:

    ẑ_1:T = _z_1:T p(Z_1:T | ĥ_1:T).

Now introducing the weight matrix 𝐒∈ℝ^IT × PT that is:

    𝐒   ≜ [𝐒_1  𝐒_2  ⋯𝐒_T]^⊤⊗I_P × P
    𝐒_t    ≜ [s_t^(1) s_t^(2)],

where s_t^(1)∈ℝ^T and s_t^(2)∈ℝ^T are the sparse vectors where only the t-th element is 1 otherwise 0, and where the (t-1)-th element has the value of -0.5 and the (t+1)-th element of 0.5 otherwise 0, respectively, the sequence can be rewritten as Z_1:T = 𝐒z_1:T.
Since the conditional probability in Eq.(<ref>) has a single mode, the objective 𝒬≜log p(Z_1:T | ĥ_1:T, θ) can be calculated as:

    Q =    - z_1:T^⊤S^⊤ diag(q̃̅̃) Sz_1:T - z_1:T^⊤S^⊤ diag(p̃) Sz̅_1:T
        + z_1:T^⊤S^⊤μ_1:T + K
    ,

where K is a constant that can be ignored in the estimation, x̃ indicates a vector that put x for T times in a column, and

    μ_1:T≜    [μ_1^H  μ_2^H  ⋯ μ_T^H]^H 
    μ_t ≜     diag(p) (b + 𝐖ĥ_t) +  diag(q) (b̅ + 𝐖̅ĥ_t).

In this paper, we estimate the optimum sequence ẑ_1:T using a complex-valued gradient method in a way similar to that discussed in the previous section.
Specifically, using the initial sequence as the frame-wise optima of the static features from:

    _Z_t p(Z_t|ĥ_t, θ) = b + 𝐖ĥ_t, ∀ t,

we iteratively update the sequence as:

    z_1:T^(l+1)z_1:T^(l) + g^(l) (∂ Q/∂z_1:T),

where ∂ Q/∂z_1:T indicates the Wirtinger derivative and can be calculated as:

    ∂ Q/∂z_1:T = - 2 S^⊤ diag(q̃̅̃) Sz_1:T - S^⊤ diag(p̃) Sz̅_1:T + S^⊤μ̃̅̃.

In our experiments, we employ the CSA for the gradient function g^(l) in the sequence generation.




§ EXPERIMENTAL EVALUATION







 §.§ Evaluation using artificial data


In order to confirm the effectiveness of the proposed CRBM, we first conducted a simple experiment using one-dimensional complex-valued artificial data (the number of training data N=2000).
The artificially created data is illustrated in Fig. <ref> as black dots, which has correlations between the real and imaginary parts.
In this experiment, we compared the CRBM to a conventional RBM that has two real-valued visible units; one is for the real part, another is for the imaginary part.
We trained both models with two hidden units using the steepest gradient ascent with a learning rate of 0.01, a momentum of 0.1, a batch size of 20, and a number of epochs as 200.
After the training, we randomly generated samples from the models; the samples from the CRBM are shown as red dots on the top of Fig. <ref> and the RBM are shown as red dots on the bottom of Fig. <ref>.
As shown in Figure <ref>, we can see that the proposed CRBM could represent the distribution of the complex-valued artificial data more accurately than the RBM.
This is because the CRBM captures the relationships between the real and imaginary parts while the conventional RBM does not. 



 §.§ Evaluation using speech data


Secondly, we conducted speech encoding experiments using speech signals of 50 sentences (approx. 4.2 min) for training and another 53 for tests pronounced by a female announcer (“FTK”) from the set “A” of the ATR speech corpora.
The speech signals were downsampled from the original 20kHz to 16kHz, and processed into 129-dimensional complex spectra using the short-time Fourier transform (STFT) with a window length of 256 and a hop size of 64, followed by the CPCA to obtain complex-valued features.
The total number of the training data was 64,438.
In order to decide how much to reduce the number of dimensions by the CPCA, we examined the perceptual evaluation of speech quality (PESQ) of the recovered signals using the inverse short-time Fourier transform (ISTFT) and the overlap-add method from the CPCA features by changing the number of dimensions P to 20, 40, 60, 80, and 100, as shown in Table <ref>.
Table <ref> shows that the PESQ with P=40 is similar to those with higher P; therefore, we used P=40 in the rest of our experiments in terms of sufficient quality and dimensional reduction.













  §.§.§ Methods compared


We compared the proposed method (“CRBM+T”) to our previous model <cit.> (“CRBM”), the RBM that feeds concatenated real-valued vectors of real and imaginary parts of the CPCA features (“RBM”), and its trajectory version that feeds static and dynamic features (“RBM+T”).
Another RBM-based method (“RBM+GL”) we compared was trained using 40-dimensional real-valued features obtained by amplitude spectra followed by PCA as visible units and recovered speech signals using the Griffin-Lim algorithm <cit.>.
These models were evaluated by changing the number of hidden units H to 1,000, 2,000, and 4,000.
The CRBMs were trained using the stochastic gradient method of 100-size mini-batches and 200 epochs with a learning rate α = 0.01 for the CSA and α = 0.001, β_1 = 0.9, β_2 = 0.999 for the complex Adam (CAdam).
We set the same parameters for the RBMs except for using the real-valued steepest ascent (SA) and Adam.
For the gradient method to estimate the sequence in Eq. (<ref>), we used the CSA (SA for the RBM method) of 100 epochs with a learning rate of 0.01.
We also compared the proposed method to the traditional speech coding of cepstral (“CEP”) and mel-cepstral (“MCEP”) analysis.
The cepstral coefficients were 40 and recovered speech using the log magnitude approximation (LMA) filter <cit.>.
From 20-dimensional mel-cepstral coefficients, we restored speech using the mel-log spectrum approximate (MLSA) filter <cit.>.
Finally, we compared it to the WORLD <cit.> as a high-quality speech analysis-by-synthesis system.






  §.§.§ Objective evaluation


Fig. <ref> shows the mean-squared error (MSE) calculated during the training, comparing the CRBM with CAdam (“CRBM+CAdam”) to the counterparts.
As shown in Fig. <ref>, the CRBMs converged more quickly than the counterparts of RBMs, and the CAdam algorithm was considerably effective for the CSA.
Table <ref> illustrates the performance of the CRBM and RBM methods in the test set, showing that the proposed method outperformed the rest regardless of the number of hidden units.
While the performance of the CRBMs with H=1,000 was comparable to that of the RBMs with H=1,000, the CRBMs with more hidden units highly improved the performance compared to the RBMs.
This is because each hidden unit in the CRBMs represents complex-valued patterns more independently than that in RBMs; i.e., CRBMs have a higher ability of representing complex-valued data than RBMs.

Table <ref> summarizes the performance of each method under their best conditions.
All methods based on CRBM and RBM were trained using the CAdam and Adam algorithms.
Interestingly, the CRBM with frame-wise modeling (“CRBM”) outperformed the RBM with trajectory modeling (“RBM+T”) because the CRBM implicitly represents the phase information of complex-valued data, and the frame-wise features from the CRBM recovered speech sufficiently.
Furthermore, the proposed trajectory modeling (“CRBM+T”) improved accuracy by extracting the correlations of complex-valued features between adjacent frames and performed the best out of all the training-based methods.
The performance of the proposed method is even comparable to that of the WORLD, which is one of the most high-quality synthesis methods.
Unlike traditional speech coding (mel-cepstrum, cepstum, and WORLD), the CRBMs directly encode arbitrary complex-valued features into binary-valued features, which indicates that the CRBMs have a high compression ability and high versatility to complex-valued data and can be applied to speech and to other signals.

Table <ref> compares the peak signal-to-noise ratio (PSNR) of the CRBM and RBM in order to analyze which magnitude or phase in the reconstruction of the CRBM is actually effective.
According to Table <ref>, the CRBM got 2.58% relative improvement points to the RBM in terms of magnitude and 4.76% relative improvement points to the RBM in terms of phase.
This demonstrates that the CRBM can effectively represent complex-valued data in particular with respect to phase.
Therefore, although the magnitude spectra of the CRBM reconstruction are very similar to those of the RBM (as shown in Fig. <ref>), the generated speech from the CRBM outperformed that from the RBM in terms of the PESQ, as shown in Table <ref>.





  §.§.§ Subjective evaluation


Finally, we conducted subjective experiments based on the mean opinion score (MOS) of 95 participants gathered through crowdsourcing.
Each participant was asked to rank the synthesized and natural speech (NAT) on a 5-point scale (1: poor, 2: fair, 3: good, 4: very good, and 5: excellent) in terms of speech quality (naturalness).
Because the MCEP, the CEP, and the WORLD are based on frame-wise synthesis, we used frame-wise estimation for the CRBM and the RBM rather than trajectory estimation in these experiments.
Fig. <ref> shows the results of the subjective evaluation.
As shown in Fig. <ref>, the CRBM performed the best out of all the methods except the WORLD.
We also conducted pairwise t-tests for each combination and observed significant differences with a 95% confidence for all pairs except for the difference between the CEP and MCEP.




§ CONCLUSION



We proposed a “complex-valued RBM” (CRBM), a novel probabilistic model that extends an RBM in order to feed complex-valued data.
This paper also includes its improved learning methods in modeling speech: the dimensionality-reduction of complex-valued data using CPCA, the CAdam learning algorithm to estimate complex-valued parameters more effectively, and the trajectory modeling and the generation method of complex-valued data based on MLPG.
Experimental results showed the effectiveness of the proposed method with objective and subjective criteria compared to the other speech coding methods except the WORLD.
Although the CRBM fell just one step short of the WORLD in terms of quality as a specialized coding method for speech, the CRBM can be also used for coding other signals such as music, images, array signals, etc.
We will further investigate the high ability of the CRBM in such applications in the future.

We presented the CRBM in this paper as a very basic model and believe that the model can be easily extended.
For example, we could define extensions by stacking two or more hidden layers like the deep Boltzmann machine <cit.> by adding connections from the previous to the current hidden/visible units like the recurrent temporal Boltzmann machine <cit.>, or by changing the energy function to form the conditional probability of hidden units such as Gaussian distribution or complex normal distribution, etc.
The deep extension can be also used as a pre-training method for complex neural networks <cit.>.
Future work includes such extensions.






























§ ACKNOWLEDGMENT

This work was partially supported by JST ACT-I, by MEXT KAKENHI Grant Numbers (15H01686, 16K16096, 16H06302,17H04687), and by the Telecommunications Advancement Foundation Grant.









































IEEEtran














Toru Nakashika
received his B.E. and M.E. degrees in computer science from Kobe University in 2009 and 2011, respectively. In the summer in 2010, he was a student researcher at IBM Research, Tokyo Research Laboratory. From September 2011 to August 2012, he was a visiting researcher in the image group at INSA Lyon in France. In that year, he continued his research as a doctoral student at Kobe University and received his Dr.Eng. degree in computer science in 2014. He was an Assistant Professor at Kobe University until April 2015. He is currently an Assistant Professor at the University of Electro-Communications in Chofu, Japan. He received the IEICE ISS Young Researcher's Award in Speech Field in 2013 and the IPSJ Ongaku Symposium Excellent Paper Award in 2016. He is a member of IEEE, IEICE, and ASJ.


Shinji Takaki
 received a B.E. degree in computer science and received an M.E. and a Ph.D. degree in scientific and engineering simulation from the Nagoya Institute of Technology, Nagoya, Japan in 2009, 2011, and 2014, respectively. From September 2013 to January 2014, he was a visiting researcher at the University of Edinburgh University. Since April 2014, he has been a project researcher at the National Institute of Informatics in Japan. His research interests include statistical machine learning and speech synthesis. He is a member of the Acoustical Society of Japan and the Information Processing Society of Japan.



Junichi Yamagishi
(SM'13) is an associate professor of the National Institute of Informatics in Japan. He is also a senior research fellow at the Centre for Speech Technology Research (CSTR) at the University of Edinburgh, UK. He was awarded a Ph.D. by the Tokyo Institute of Technology in 2006 for a thesis that pioneered speaker-adaptive speech synthesis and was awarded the Tejima Prize for the best Ph.D. thesis of the Tokyo Institute of Technology in 2007. Since 2006, he has authored and co-authored over 100 refereed papers in international journals and conferences. He was awarded the Itakura Prize from the Acoustic Society of Japan, the Kiyasu Special Industrial Achievement Award from the Information Processing Society of Japan, the Young Scientists’ Prize from the Minister of Education, Science, and Technology, and the JSPS prize in 2010, 2013, 2014, and 2016, respectively. 

He  was one of organizers for special sessions on “Spoofing and Countermeasures for Automatic Speaker  Verification” at  Interspeech  2013,  “ASVspoof evaluation”  at  Interspeech  2015, “Voice conversion challenge 2016” at Interspeech 2016, and “2nd ASVspoof evaluation” at Interspeech 2017.  He  has been  a member of  the  Speech  & Language  Technical  Committee  (SLTC). He served as an Associate  Editor  of  the IEEE/ACM Transactions on Audio, Speech, and Language Processing and as a Lead Guest Editor for the IEEE Journal of Selected Topics in Signal Processing (JSTSP) special issue on Spoofing and Countermeasures for Automatic Speaker Verification.

















