A Multi-Discriminator CycleGAN for Unsupervised Non-Parallel Speech Domain Adaptation
Ehsan Hosseini-Asl Yingbo Zhou Caiming Xiong Richard Socher


Non-LTE Calculations of the Fe1 6173 Å Line in a Flaring Atmosphere
    Mats Carlsson
    December 30, 2023
===================================================================




Domain adaptation plays an important role for speech recognition models, in particular, for domains that have low resources.
We propose a novel generative model based on cyclic-consistent generative adversarial network (CycleGAN) for unsupervised non-parallel speech domain adaptation.
The proposed model employs multiple independent discriminators on the power spectrogram, each in charge of different frequency bands.
As a result we have 1) better discriminators that focus on fine-grained details of the frequency features, and 2) a generator that is capable of generating more realistic domain-adapted spectrogram.
We demonstrate the effectiveness of our method on speech recognition with gender adaptation, where the model only has access to supervised data from one gender during training, but is evaluated on the other at test time. 
Our model is able to achieve an average of 7.41% on phoneme error rate, and 11.10% word error rate relative performance improvement as compared to the baseline, on TIMIT and WSJ dataset, respectively. 
Qualitatively, our model also generates more natural sounding speech, when conditioned on data from the other domain.
  

Index Terms: generative models, speech domain adaptation, non-parallel data, unsupervised learning

Sound demos can be found at https://einstein.ai/research/a-multi-discriminator-cyclegan-for-unsupervised-non-parallel-speech-domain-adaptationhttps://einstein.ai/research/a-multi-discriminator-cyclegan-for-unsupervised-non-parallel-speech-domain-adaptation
footnote0



§ INTRODUCTION


Neural-based acoustic models have shown promising improvements in building automatic speech recognition (ASR) systems <cit.>. 
However, it tends to perform poorly when evaluated on out-of-domain data, because of mismatch between the training and testing distribution (Table <ref>). 

Domain mismatch is mainly due to variation in non-linguistic features, such as different speaker identity, unseen environmental noise, large accent variations, etc. 
Therefore, training a robust ASR system is highly dependent on factorizing linguistic features (text) from non-related variations, or adapting the inter-domain variations of source and target. 

Voice conversion (VC) has been widely used to adapt the non-linguistic variations, such as statistical methods <cit.>, and Neural-based models <cit.>. 
However, traditional VC methods require parallel data of source and target that is difficult to obtain in practice.
In addition, the requirement of parallel data prevent these methods from using more abundant unsupervised data. Therefore, an unsupervised domain adaptation is desirable for building a robust ASR system. 





In this paper, we propose a new generative model based on CycleGAN <cit.> for unsupervised non-parallel domain adaptation. 
Since differences in magnitude of frequency is the main variation across domains for spectrogram representations, it is imperative that CycleGAN correctly catch the spectro-temporal variations between different frequency bands across domains during training. 
This will allow the generator to learn the mapping function which can convert spectrogram from source to target domain. 
In this paper, we show that the original CycleGAN model is failing to learn the correct mapping function between domains, and the generator collapses into learning an identity mapping function, which results in generating a noisy and unnatural-sounding audio.














To accommodate generative adversarial network for training on non-parallel spectrogram domains, the generator should be back-propagated with multiple gradient signals (from different discriminators), that each represents the variations between source and target domains at a specific frequency band. 

To achieve this goal, we propose to use multiple and independent discriminators for each domain, similar to generative multi adversarial network (GMAN) <cit.>. 
We show that the proposed Multi-Discriminator CycleGAN, without pretraining the discriminators, outperforms CycleGAN <cit.> with pretrained discriminator, for spectrogram adaptation. 

Furthermore, we show that the multi discriminator architecture can overcome the checkerboard artifacts problem caused by deconvolution layer in generator <cit.>, and generates natural clean audio. To evaluate the performance of the proposed model, gender-based domains are selected as domain adaptations. 








 §.§ Related Work



Generative Adversarial Network (GAN) <cit.> is a family of non-parametric density estimation models which learn to model the data generating distribution using adversarial training. Conditional GANs (CGAN) <cit.> was first proposed for supervised (parallel) domain adaptation, where the goal is to convert source distribution to match the target. 
CGAN has been used in various data domains, especially image domains, both for parallel <cit.> and non-parallel domain adaptation <cit.>. 

Recently, CGAN is used for speech enhancement on parallel datasets <cit.>. Speech denoising is achieved by conditioning the generator on noisy speech to learn the de-noised version <cit.>. Donahue et al. <cit.> proposed a GAN model on audio (WaveGAN) and spectrogram (SpecGAN), which is actually trained CGAN on parallel domains. 
Kaneko et al. <cit.> proposed a cycle-consistent adversarial network (CycleGAN) <cit.> with gated convolutional neural network (CNN) as the generator part, where the model is trained on Mel-cepstral coefficients (MCEPs) features. Hsu et al. <cit.> proposed a combination of variational inference network, using variational autoencoder (VAE) <cit.>, and adversarial network, using Wasserstein GAN (WGAN) <cit.>. In <cit.>, the goal is to disentangle the linguistic from nuisance latent variables via VAE using spectra (SP for short), aperiodicity (AP), and pitch contours (F0) features, followed by adversarial training to learn the target distribution from the inferred linguistic latent distribution. A recurrent VAE is also proposed <cit.> to capture the temporal relationships in the disentangled representation of sequence data, using Mel-scale filter bank (FBank).  






Contributions of the proposed generative model are, (1) It is a robust GAN model developed for non-parallel unsupervised domains, compared to parallel-based SpecGAN and WaveGAN <cit.>, (2) The choice of multiple discriminator is adjustable to the spectro-temporal structure of the intended domains, compared to domain-specific model design of <cit.>, (3) Proposed GAN model training is robust and invariant to the choice of adversarial objective, i. e. binary cross-entropy or least square (LS-GAN <cit.>), while the CycleGAN in <cit.> is only stable using least square 

loss, with additional using of identity mapping loss in generator, (4) Source and target domains in <cit.> are sampled from same speakers, both including male and female, only uttering different sentences, while our approach is more natural as source and targets distribution is strongly diverged due to different speaker, gender, and uttered sentences. (5) Compared to FHVAE <cit.>, our models improves ASR performance on TIMIT female set by 2.067% PER (Table <ref>), when only trained on male. 
















§ PROPOSED MODEL




In this section, the proposed model is explained. We first describe the generative model based on adversarial network. Generative Model based on adversarial training (GAN) has been proposed by Goodfellow et al. <cit.> to model the data generating distribution. Training GAN is based on minimizing Jensen-Shannon divergence between data generating distribution p_data(x) and model data distribution p_z(z). Learning is through minimization of the adversarial loss between generator network G(z), which learns a mapping function G: Z→ X, and discriminator network D(x). The generator is learning to model the data distribution p_data(x) by generating indistinguishable samples x̂=G(z) from x, using a source noise signal z to minimize (<ref>),  whereas discriminator is learning to discriminate between real data x and generated x̂ by maximizing the adversarial loss,

    ℒ_GAN(G, D) =    𝔼_x∼ p_data(x)[log D(x)] +  
       𝔼_z∼ p_z(z)[log(1-D(G(z)))]





 §.§ Domain Adaptation via GAN




For domain adaptation between parallel domains X and Y, Conditional GAN (CGAN)<cit.> is proposed, using a generator that directly learns the mapping function G: X→ Y, by minimizing parallel conditional adversarial loss ℒ_P-CGAN, 



    ℒ_P-CGAN(G, D) = 𝔼_(x,y)∼ p_data(x,y)[log D(x,y)] + 
       𝔼_x∼ p_data(x), z∼ p_z(z)[log(1-D(x,G(z,x)))]

where D is discriminating between pair of real parallel data (x,y) and generated pair (x,G(z,x)).
To apply CGAN for adaptation between non-parallel domains X and Y, a conditional GAN using cycle consistent adversarial loss (CycleGAN) has been proposed <cit.>. In CycleGAN <cit.>, there are two conditional generators, i. e., G_X:X→ Y and G_Y:Y→ X, each trained in adversarial setting with D_Y and D_X, respectively. In other words, there are two pairs of Non-parallel conditional adversarial loss ℒ_NP-CGAN(G_X, D_Y) and ℒ_NP-CGAN(G_Y, D_X), where, 

    ℒ_NP-CGAN(G_X, D_Y) = 𝔼_(y)∼ p_Y(y)[log D_Y(y)] + 
       𝔼_x∼ p_X(x), z∼ p_z(z)[log(1-D_Y(G_X(z,x)))]

In non-parallel situation, the goal is to find the correct pseudo pair (x,y) across X and Y domains in an unsupervised way. To ensure that G_X and G_Y will learn such mapping function, CycleGAN<cit.> proposed to minimize a cycle consistency loss using ℓ_1 norm,

    ℒ_cycle =    𝔼_x∼ p_X(x)[∥ G_Y(G_X(x))-x∥_1]+
       𝔼_y∼ p_Y(y)[∥ G_X(G_Y(y))-y∥_1]

Therefore, CycleGAN<cit.> learns unsupervised mapping functions between X and Y domains by combining (<ref>) and (<ref>), to maximize the adversarial loss ℒ_CycleGAN, where, 

    ℒ_CycleGAN =    ℒ_NP-CGAN(G_X, D_Y) + 
       ℒ_NP-CGAN(G_Y, D_X) + 
        -λℒ_cycle(G_X, G_Y)






 §.§ Multi-Discriminator CycleGAN (MD-CycleGAN)




In this section, we propose a multiple discriminator generative model based on cycle consistency  loss (<ref>). The model is based on generative multi adversarial network (GMAN) <cit.>. In this paper, X and Y represents spectrogram feature datasets of different speech domains. Spectrogram feature represents the frequency variation of audio data through time dimension. In order to allow CycleGAN to learn the mapping function of spectrogram between different speech domains, the generators { G_X,G_Y} should be able to learn the variations in each frequency band for each aligned time window, across domains.


In order to learn the frequency-dependent mapping functions { G_X, G_Y} that catch the variation per each frequency bands, we define multiple frequency-dependent discriminators { D_X^f_j∈ n, D_Y^f_i∈ m}, where f_j∈ n represents the i^th frequency band of domain X with n frequency bands, and f_i∈ m represents j-th frequency band od domain Y, respectively. The frequency band definition in each domain can share a portion of frequency spectrum, or be exclusive, based on the domain spectrogram distribution. We are also using the non-saturating version of GAN<cit.>, NS-GAN, where the generator G is learned through maximizing the probability of predicting generated samples x̂ as drawn from data generating distribution p_data(x). Accordingly, the adversarial loss for each pair of generator and discriminator {(G_X, D^f_i∈ m_Y), (G_Y, D^f_j∈ n_X)} in (<ref>) and (<ref>) is

    ℒ_MD-CGAN(G_X, D^f_i∈ m_Y) = 𝔼_(y)∼ p_Y(y)[∑_i=1^mlog D^f_i_Y(y)] 
        + 𝔼_x∼ p_X(x), z∼ p_z(z)[∑_i=0^mlog(D^f_i_Y(G_X(z,x)))]


The Multi-Discriminator CycleGAN (MD-CycleGAN) is training by maximizing ℒ_MD-CycleGAN, where,

    ℒ_MD-CycleGAN =    ℒ_MD-CGAN(G_X, D^f_i∈ m_Y) + 
       ℒ_MD-CGAN(G_Y, D^f_j∈ n_X) + 
        -λℒ_cycle(G_X, G_Y)

A natural extension to the proposed MD-CycleGAN is to use multiple frequency-dependent generators <cit.> jointly with discriminators as well. This can follow in two configurations. In one-one setting, each generator is trained on a specific frequency band with the corresponding discriminator, i. e., set of {(G_X^f_i, D_Y^f_i): i∈ m}. Additionally, in one-many setting, each frequency-dependent generator is trained with all frequency-dependent discriminators, i. e., set of {(G_X^f_j, D_Y^f_i∈ m): j∈ n} trained in adversarial setting. 





§ EXPERIMENT



We used TIMIT  <cit.> and Wall Street Journal (WSJ) corporas to evaluate the performance of proposed model on domain adaptation. TIMIT dataset contains broadband
16kHz recordings of phonetically-balanced read speech of 6300 utterances (5.4 hours). Male/Female ratio of speakers across train/validation/test sets are approximately 70% to 30%. WSJ contains ≈80 hours of standard si284/dev93/eval92 for train/validation/test sets, with equally distributed genders.





The spectrogram representation of audio is used for training the CycleGAN and ASR models, which is computed with a 20ms window and 10ms step size. Each spectrogram is normalized to have zero mean and unit variance. 

To implement MD-CycleGAN, three non-overlapping frequency bands are defined, i. e. m=n=3 with [53, 53, 55] bandwidth, for male and female domains. 

We denote the size of the convolution layer by the tuple (C, F, T, SF, ST), where C, F, T, SF, and ST denote number of channels, filter size in frequency dimension, filter size in time dimension, stride in frequency dimension and stride in time dimension respectively. CycleGAN model architecture is based on <cit.> with some modifications. The generator is based on U-net <cit.> architecture with 4 convolutional layers of sizes (8,3,3,1,1), (16,3,3,1,1), (32,3,3,2,2), (64,3,3,2,2) with corresponding deconvolution layers. We noticed that the discriminator in <cit.> outputs a vector with dimension equal to the channel size of final convolution layer, instead of outputting a scalar <cit.>. It was observed that this causes instability in a balanced training between generator and discriminator. We modified this by adding a fully connected layer as final layer, to match the discriminator in <cit.>.

Discriminator has 4 convolution layers of sizes (8,4,4,2,2), (16,4,4,2,2), (32,4,4,2,2), (64,4,4,2,2), as default kernel and stride sizes in <cit.>. We used Griffin-lim algorithm <cit.> for audio reconstruction, to assess its quality. ASR model is based on <cit.>, trained with maximum likelihood, and no policy gradient. The model has one convolutional layer of size (32,41,11,2,2), and five residual convolution blocks of size (32,7,3,1,1), (32,5,3,1,1), (32,3,3,1,1), (64,3,3,2,1), (64,3,3,1,1) respectively. Following the convolutional layers are 4 layers of bidirectional GRU RNNs with 1024 hidden units per direction per layer, one fully-connected hidden layer of size 1024 and final output layer. 





 §.§ Quantitative Evaluation



In this section, ASR model is employed to evaluate the performance of proposed model, where domains are different genders. 

First, gender generators { G_M→ F, G_F→ M} [G_M→ F: Male→ Female, G_F→ M: Female→ Male] are trained on gender-separated train set. These generators are then evaluated for train→test and test→train adaptation using ASR model. In former, ASR model is retrained on the adapted train set, while in latter, a more applicable case, ASR model is fixed and evaluated on the new adapted test sets.


















  §.§.§ Train→Test Adaptation




Results on adapting TIMIT train set are shown in Table <ref> and <ref>. As ablation study to CycleGAN-VC <cit.>, performance is significantly improved with three discriminator compared to single one. Compared to FHVAE <cit.>, phoneme error rate is improved by 2.067% in Table <ref>.  
To evaluate the generalization of the generators, we used them on WSJ dataset without retraining. As shown in Table <ref>, ASR performance is significantly improved by reducing the gap to the corresponding male and female baselines. For a fair comparison, ASR performance trained on WSJ train set is 5.55% WER. It is worth mentioning that relatively lower performance on TIMIT is due to smaller size of dataset.








  §.§.§ Test→Train Adaptation




Test set adaptation of TIMIT and WSJ are shown in Table <ref> and <ref>. It is clear that using the proposed model, ASR performance is significantly improved by adapting test→train 

, compared to original CycleGAN. Qualitative assessment of ASR predictions are shown in Tables <ref> and Appendix <ref>.















 §.§ Qualitative Evaluation



In this section, the quality of generated spectrogram for male↔female adaptation is assessed. The characteristic difference between male and female spectrograms is the variation rate of frequency for a fixed time window. As shown in Figure <ref>, top row depicts the original spectrograms, where male is characterized by smooth frequency variation, opposed to peaky and high-rate variations of female. Well-trained generators should catch these inter-domain variations. As ablation study, we are also showing the generated spectrogram by CycleGAN <cit.> (One-D CycleGAN), in middle row, comparing with Three-D CycleGAN in bottom row. One-D CycleGAN learns to convert the spectrogram only using a pretrained discriminator. It is noticeable that the converted spectrogram in One-D CycleGAN fails to match the target domain characteristics, at some frequency bands, and simply copied the source spectrogram. However, with no pretraining of Three-D CycleGAN, it learns a better mapping function, by either suitably smoothing the spectrogram (female→male), or generating peaky variations (male→female). The checkerboard artifacts <cit.> is a common problem in deconvolution-based generators. This problem is visible in One-D CycleGAN, with discontinuous artifacts through time and frequency dimensions, which results in a noisy and unnatural-sounding audio. This problem is mitigated in Three-D CycleGAN, by learning the target domain characteristics using multiple independent discriminators. 









§ CONCLUSION AND FUTURE DIRECTIONS




In this paper, a new cyclic consistent generative adversarial network based on multiple discriminators is proposed (MD-CycleGAN) for unsupervised non-parallel speech domain adaptation. Based on the frequency variation of spectrogram between domains, the multiple discriminators enabled MD-CycleGAN to learn an appropriate mapping functions that catch the frequency variations between domains. The performance of MD-CycleGAN is measured by ASR prediction, when train and test set are sampled from different genders. 


It was shown that MD-CycleGAN can improve the ASR performance on unseen domains. As future extension, this model will be evaluated on datasets adaptation, e. g. TIMIT↔WSJ, and accent, e. g. American↔Indian adaptations. 

IEEEtran










§ ASR PREDICTION ASSESSMENT



In this section, qualitative assessment of ASR predictions are shown, when test set is sampled from the opposite gender, and when MD-CycleGAN model is used to adapt the test set distribution to the train set. Tables <ref>, <ref> show the selected ASR predictions when adapting female→male, whereas Tables <ref>, <ref> show adaptation for male→female. Note that MD-CycleGAN is trained on TIMIT, and used for adaptation on WSJ dataset.



















  






