title: Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks
authors: Hirokazu Kameoka Takuhiro Kaneko Kou Tanaka Nobukatsu Hojo
text: 
  
  
  
  
  
  
  
  
  
  
  Nonparallel Voice Conversion with 
  Augmented Classifier 
  Star Generative Adversarial Networks
  
      Hirokazu Kameoka, 
  
  Takuhiro Kaneko,
  Kou Tanaka,
  and Nobukatsu Hojo
  H. Kameoka, T. Kaneko, K. Tanaka, 
  and N. Hojo are with NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Atsugi, Kanagawa, 243-0198 Japan (e-mail: hirokazu.kameoka.uh@hco.ntt.co.jp).
  
  
  This work was supported by 
  JSPS KAKENHI 17H01763 and JST CREST Grant Number JPMJCR19A3, Japan. 
  
      December 30, 2023
  =====================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
  
  We previously proposed a method that allows for nonparallel voice conversion (VC) by using a variant of generative adversarial networks (GANs) called StarGAN. The main features of our method, called StarGAN-VC, are as follows: First, it requires no parallel utterances, transcriptions, or time alignment procedures for speech generator training. Second, it can simultaneously learn mappings across multiple domains using a single generator network and thus fully exploit available training data collected from multiple domains to capture latent features that are common to all the domains. Third, it can generate converted speech signals quickly enough to allow real-time implementations and requires only several minutes of training examples to generate reasonably realistic-sounding speech. In this paper, we describe three formulations of StarGAN, including a newly introduced novel StarGAN variant called “Augmented classifier StarGAN (A-StarGAN)”, and compare them in a nonparallel VC task. We also compare them with several baseline methods. 
  
  
  
  Voice conversion (VC), nonparallel VC, multi-domain VC, 
  generative adversarial networks (GANs), 
  CycleGAN, StarGAN, A-StarGAN.
  
  
  
  
  
  
  § INTRODUCTION
  
  
  
  Voice conversion (VC) is a task of converting the voice of
  a source speaker without changing the uttered sentence. 
  Examples of the applications of VC techniques include 
  speaker-identity modification <cit.>, speaking assistance <cit.>, 
  speech enhancement <cit.>, 
  bandwidth extension <cit.>,
  and accent conversion <cit.>.
  
  One successful VC framework involves 
  approaches that utilize acoustic models represented by Gaussian mixture models (GMMs) for feature mapping <cit.>.
  Recently, frameworks based on neural networks (NNs)  
  <cit.> and an exemplar-based framework based on nonnegative matrix factorization (NMF) <cit.> 
  have also proved successful. 
  Many conventional VC methods, including those mentioned above, require accurately aligned parallel 
  source and target speech data. However, in many scenarios, 
  it is not always possible to collect parallel utterances. 
  Even if we could collect such data, we typically need to perform 
  time alignment procedures, which becomes relatively
  difficult when there is a large acoustic gap between
  the source and target speech. Since many frameworks are weak as regards the misalignment
  found with parallel data, careful pre-screening and
  manual correction may be required to make these frameworks work reliably.
  To bypass these restrictions, this paper is concerned with developing a nonparallel  
  VC method, which requires no parallel utterances, transcriptions, or time alignment procedures.
  
  In general, the quality and conversion effect obtained with nonparallel methods
  are usually limited compared with methods using parallel data
  due to the disadvantage related to the training condition. 
  Thus, developing nonparallel methods whose speech quality and a conversion effect are as high as those of parallel methods can be very challenging. 
  Recently, some attempts have been made to develop nonparallel methods <cit.>.
  One example is a method using automatic speech recognition (ASR) <cit.>.
  The idea is to convert input speech under the restriction that the posterior state probability of the acoustic model of an ASR system is preserved so that the transcription of the converted speech becomes consistent with that of the input speech. 
  Since the performance of this method depends heavily on the quality of the acoustic model of ASR,
  it can fail to work if ASR does not function reliably. 
  A method using i-vectors <cit.>, known as a feature for speaker verification, was proposed in <cit.>.
  Conceptually, the idea is to
  shift the acoustic features of input speech towards target speech in the i-vector space so that
  the converted speech is likely to be recognized as the target speaker by a speaker recognizer.
  While this method is also free from parallel data, one limitation is that 
  it is applicable only to speaker identity conversion tasks.
  
  Recently, a framework based on conditional variational autoencoders (CVAEs) <cit.> was proposed in <cit.>.
  As the name implies, variational autoencoders (VAEs), 
  consisting of encoder and decoder networks,
  are probabilistic counterparts of autoencoders (AEs).
  CVAEs <cit.> are an extended version of VAEs where the encoder and decoder networks can take a class indicator variable 
  as an additional input. 
  By using acoustic features as the training examples and 
  the associated domain class labels, 
  the networks learn how to convert source speech to a target domain 
  according to the domain class label fed into the decoder. 
  This CVAE-based VC approach is notable in that it is completely free from parallel data and works even with unaligned corpora. 
  However, one well-known problem as regards VAEs is that outputs from the decoder tend to be oversmoothed. 
  For VC applications, this can be problematic since it usually results in poor quality buzzy-sounding speech. 
  
  One powerful framework that can potentially overcome the weakness of VAEs involves generative adversarial networks (GANs) <cit.>.
  GANs offer a general framework 
  for training a generator network 
  so that it can generate fake data samples 
  that can deceive a real/fake discriminator network 
  in the form of a minimax game. 
  While they have been found to be effective for use with image generation, 
  in recent years they 
  have also been employed with notable success for various speech processing tasks <cit.>.
  We previously reported a nonparallel VC method using a GAN variant called cycle-consistent GAN (CycleGAN) <cit.>, which
  was originally proposed as a method for translating images using unpaired training examples <cit.>. 
  Although this method, which we call CycleGAN-VC, was shown to work reasonably well, 
  one major limitation is that it only learns mappings between a single pair of domains. 
  In many VC application scenarios, it is desirable to 
  be able to convert speech into multiple domains, not just one. 
  One naive way of applying CycleGAN to multi-domain VC tasks would be to prepare and train a 
  different mapping 
  pair for each domain pair. 
  However, this can be ineffective 
  since each mapping pair
  fails to use the training data of the other domains for learning,
  even though there must be a common set of latent features that can be shared 
  across different domains.
  
  To overcome the shortcomings and limitations of CVAE-VC <cit.> and CycleGAN-VC <cit.>, we previously proposed a nonparallel VC method 
  <cit.> using
  another GAN variant called StarGAN <cit.>, which 
  offers the advantages of CVAE-VC and CycleGAN-VC concurrently. 
  Unlike CycleGAN-VC and as with CVAE-VC, our method, which we call StarGAN-VC, is 
  capable of simultaneously learning multiple mappings using a single generator network 
  and can thus fully
  use available training data collected from multiple domains.
  Unlike CVAE-VC and as with CycleGAN-VC, StarGAN-VC uses an adversarial loss 
  for generator training 
  to encourage 
  the generator outputs to become indistinguishable from real speech.
  It is also noteworthy that unlike CVAE-VC and CycleGAN-VC, 
  StarGAN-VC does not require any information about the domain of the input speech at test time.
  
  In this paper, we describe three formulations of StarGAN, including a newly introduced novel StarGAN variant called “Augmented classifier StarGAN”, and compare them in a nonparallel VC task.
  The remainder of this paper is organized as follows.
  After reviewing other related work in relatedwork,
  we briefly describe the formulation of CycleGAN-VC in cyclegan-vc,
  present the three formulations of StarGAN-VC in stargan-vc,
  and show experimental results in experiments.
  
  
  
  
  § RELATED WORK
  
  
  
  Other natural ways of overcoming the weakness of VAEs includes
  the VAE-GAN framework <cit.>.
  A nonparallel VC method based on this framework has already been proposed in <cit.>.
  With this approach, an adversarial loss derived using a GAN discriminator is incorporated into the training loss to encourage the decoder outputs of a CVAE to be indistinguishable from real speech features. 
  Although the concept is similar to our StarGAN-VC approach, 
  we will show in experiments that our approach outperforms this method 
  in terms of both speech quality and the conversion effect.
  
  Another related technique worth noting is 
  the vector quantized VAE (VQ-VAE) approach <cit.>, 
  which has performed impressively in nonparallel VC tasks. 
  This approach 
  is particularly notable in that it offers a novel way of overcoming the weakness of VAEs by using
  the WaveNet model <cit.>, a sample-by-sample neural signal generator, to devise both the encoder and decoder of a discrete counterpart of CVAEs.
  The original WaveNet model is a recursive model that makes it possible to predict the distribution of a sample conditioned on the samples the generator has produced. 
  While a faster version <cit.> has recently been proposed,
  it typically requires huge computational cost to generate a stream of samples, which can cause difficulties when implementing real-time systems.
  The model is also known to require a huge number of training examples to generate natural-sounding speech.
  By contrast, our method is noteworthy in that it is able to generate signals quickly enough to allow real-time implementation and requires only several minutes of training examples to generate reasonably realistic-sounding speech.
  
  Meanwhile, 
  given the recent success of the sequence-to-sequence (S2S) learning framework
  in various tasks, 
  several VC methods based on S2S models have been
  proposed, including the ones we proposed previously <cit.>. 
  While S2S models usually require parallel corpora for training, 
  an attempt has also been made to train an S2S model using nonparallel utterances 
  <cit.>.
  However, 
  it requires phoneme transcriptions as auxiliary information
  for model training.
  
  
  
  § CYCLEGAN VOICE CONVERSION
  
  
  
  Since StarGAN-VC is an extension of CycleGAN-VC, which we proposed previously <cit.>,  
  we start by briefly reviewing its formulation (cyclegan).
  
  Let x∈ℝ^Q× N and y∈ℝ^Q× M
  be acoustic feature sequences of speech belonging to domains X and Y, respectively,
  where Q is the feature dimension and N and M are the lengths of the sequences.
  In the following, we will restrict our attention to speaker identity conversion tasks,
  so when we use the term domain, we will mean speaker.
  The aim of CycleGAN-VC is to learn 
  a mapping G that converts the domain of x into Y
  and a mapping F that does the opposite. 
  Now, we introduce discriminators D_X and D_Y,
  whose roles are to predict whether or not their inputs are the acoustic features of real speech belonging to X and Y,
  and define 
  
      ℒ_ adv^D_Y(D_Y) 
      =   - 
      𝔼_y∼ p_Y(y)[log D_Y(y)]
         -𝔼_x∼ p_X(x)[log (1- D_Y(G(x)))],
      
      ℒ_ adv^G(G) 
      =   𝔼_x∼ p_X(x)[log (1- D_Y(G(x)))],
      
      ℒ_ adv^D_X(D_X) 
      =   -
      𝔼_x∼ p_X(x)[log D_X(x)]
         -
      𝔼_y∼ p_Y(y)[log (1- D_X(F(y)))],
      
      ℒ_ adv^F(F) 
      =   𝔼_y∼ p_Y(y)[log (1- D_X(F(y)))],
  
  as the adversarial losses for D_Y, G, D_X and F, 
  respectively.
  ℒ_ adv^D_Y(D_Y) 
  and 
  ℒ_ adv^D_X(D_X)
  measure how indistinguishable 
  G(x) and F(y) are 
  from acoustic features of real speech belonging to Y and X. 
  Since the goal of D_X and D_Y is to correctly distinguish 
  the converted feature sequences obtained via G and F
  from real speech feature sequences, 
  D_X and D_Y attempt to minimize these losses to avoid being fooled by G and F.
  Conversely, since one of the goals of G and F is to generate natural-sounding speech that is indistinguishable from real speech, G and F attempt to maximize these losses or minimize 
  ℒ_ adv^G(G) and ℒ_ adv^F(F) to fool D_Y and D_X.
  It can be shown that the output distributions of G and F trained in this way will match the empirical distributions 
  p_Y(y) and p_X(x) 
  if G, F, D_X, and D_Y have enough capacity <cit.>. 
  Note that since 
  ℒ_ adv^G(G) and ℒ_ adv^F(F) 
  are minimized when D_Y(G(x))= 1 and D_X(F(y))= 1,
  we can also use 
  - 𝔼_x∼ p_X(x)[log D_Y(G(x))]
  and 
  - 𝔼_x∼ p_X(x)[log D_Y(G(x))] 
  as the adversarial losses for G and F.
  
  As mentioned above,
  training G and F using the adversarial losses
  enables mappings G and F to produce outputs identically distributed 
  as target domains Y and X, respectively.
  However,  
  using them alone does not guarantee that 
  G or F will preserve the linguistic contents of 
  input speech since there are infinitely many mappings 
  that will induce the same output distributions.
  One way to let G and F preserve the linguistic contents of input speech 
  would be to encourage them to make only minimal changes from the inputs.
  To incentivize this behaviour,
  
  we introduce a cycle consistency loss <cit.>
  
      ℒ_ cyc(G,F) 
         = 𝔼_x∼ p_X(x)[ F(G(x)) - x_ρ^ρ]
      
         +𝔼_y∼ p_Y(y)[ G(F(y)) - y_ρ^ρ],
  
  to enforce F(G(x))≃x and G(F(y))≃y. 
  In image-to-image translation tasks, 
  this regularization loss contributes to enabling G and F to change only the textures and colors of input images while preserving the domain-independent contents.
  However, the effect this loss would have on VC tasks was nontrivial.
  Our previous work <cit.> was among the first  
  to show that it enables G and F to change only the voice characteristics
  of input speech while preserving the linguistic content.
  This regularization technique has recently proved 
  effective also in the VAE-based VC methods <cit.>. 
  With the same motivation, we also consider an identity mapping loss
  
      ℒ_ id(G,F) 
         = 𝔼_x∼ p_X(x)[ F(x) - x_ρ^ρ]
      
         +𝔼_y∼ p_Y(y)[ G(y) - y_ρ^ρ],
  
  to ensure that inputs to G and F 
  are kept unchanged when the inputs already belong to Y and X.
  The full objectives of CycleGAN-VC to be minimized with respect to 
  G, F, D_X, and D_Y 
  are thus given as 
  
      ℐ_G,F(G,F) =   λ_ advℒ_ adv^G(G) +
      λ_ advℒ_ adv^F(F) 
      
         + 
      λ_ cycℒ_ cyc(G,F) + 
      λ_ idℒ_ id(G,F),
      ℐ_D(D_X,D_Y) =   ℒ_ adv^D_X(D_X)
      +
      ℒ_ adv^D_Y(D_Y),
  
  where 
  λ_ adv≥ 0,
  λ_ cyc≥ 0, and λ_ id≥ 0 are 
  regularization parameters, which weigh the importance of 
  the adversarial,
  cycle consistency, and identity mapping losses.
  In practice,
  we alternately update  
  G, F, D_X, and D_Y one at a time
  while keeping the others fixed.
  
  
  
  
  
  § STARGAN VOICE CONVERSION
  
  
  
  While CycleGAN-VC 
  can only learn mappings between a single pair of domains, 
  StarGAN-VC <cit.> 
  can learn mappings among multiple speech domains using a single generator network,
  thus allowing us to fully utilize available training data collected from
  multiple domains.
  In this section, we describe three formulations of StarGAN. 
  While the first and second formulations respectively correspond to the ones 
  presented in <cit.> and <cit.>,
  the third formulation is newly proposed in this paper 
  with the aim of further improving the former two.
  
  
  
  
   §.§ Cross-Entropy StarGAN formulation
  
  
  First, we describe the formulation we introduced in <cit.>.
  Let G be a generator that takes an acoustic feature sequence x∈ℝ^Q× N 
  belonging to an arbitrary domain and a target domain class index k∈{1,…,K} as the inputs and generates an acoustic feature sequence ŷ = G(x,k). 
  For example, if we consider speaker identities as the domain classes, 
  each k will be associated with a different speaker.
  One of the goals of StarGAN-VC is to make 
  ŷ = G(x,k) as realistic as real speech features
  and belong to domain k. 
  To achieve this, we introduce a real/fake discriminator D as with CycleGAN 
  and a domain classifier C, 
  whose role is to predict to which classes an input belongs. 
  D is designed to produce a probability D(y,k) that an input y is a real speech feature whereas 
  C is designed to produce class probabilities p_C(k|y) of y.
  
  Adversarial Loss:
  First, we define
  
      ℒ_ adv^D(D) =    
      - 𝔼_k ∼ p(k), y∼ p_ d(y|k)[log D(y,k)] 
      
         - 𝔼_k∼ p(k), x∼ p_ d(x)
      [log (1-D(G(x,k),k))],
      
      ℒ_ adv^G(G) =   
      - 𝔼_k∼ p(k), x∼ p_ d(x)
      [log D(G(x,k),k)],
  
  as adversarial losses for discriminator D and generator G, respectively, 
  where 
  p(k) is a uniform categorical distribution (p(k)=1/K),
  y∼ p_ d(y|k) denotes a training example of an acoustic feature sequence of real speech in domain k, and 
  x∼ p_ d(x) denotes that in an arbitrary domain. 
  ℒ_ adv^D(D) takes a small value when D correctly classifies 
  G(x,k) and y as fake and real speech features 
  whereas
  ℒ_ adv^G(G) takes a small value when G successfully deceives D so that 
  G(x,k) is misclassified as real speech features by D.
  Thus, we would like to minimize ℒ_ adv^D(D) with respect to D and
  minimize ℒ_ adv^G(G) with respect to G.
  Note that 
  𝔼_k∼ p(k)[·] is a simplified notation for
  1/K∑_k=1^K(·), and
  when k denotes a speaker index,
  𝔼_y∼ p_ d(y|k)[·]
  and
  𝔼_x∼ p_ d(x)[·]
  denote 
  the sample means over the training examples of speaker k
  and all speakers, respectively.
  Note also that 
  the order of the variables over which each expectation is taken
  corresponds to the order of the for-loop in our implementation.
  
  Domain Classification Loss: 
  Next, we define
  
      ℒ_ cls^C(C) =    
      - 𝔼_k∼ p(k), y∼ p_ d(y|k)
      [log p_C(k|y)],
      
      ℒ_ cls^G(G) =    
      - 𝔼_k∼ p(k), x∼ p(x)
      [log p_C(k|G(x,k))],
  
  as domain classification losses for classifier C and generator G.
  ℒ_ cls^C(C) and ℒ_ cls^G(G) take small values
  when C correctly classifies y∼ p_ d(y|k) and 
  G(x,k) as belonging to domain k. 
  Thus, we would like to minimize ℒ_ cls^C(C) with respect to C 
  and ℒ_ cls^G(G) with respect to G.
  
  Cycle Consistency Loss: 
  Training G, D, and C using only the losses presented above does not guarantee that 
  G will preserve the linguistic content of input speech.  
  As with CycleGAN-VC,
  we introduce a cycle consistency loss to be minimized
  
      ℒ_ cyc(G) 
      
      = 
      𝔼_k∼ p(k), k'∼ p(k), x∼ p_ d(x|k')
      [ G(G(x,k),k') - x_ρ^ρ],
  
  to encourage G(x,k) to
  preserve the linguistic content of x,
  where 
  x∼ p_ d(x|k') denotes a training example of real speech feature sequences 
  in domain k', and ρ is a positive constant.
  We also consider an identity mapping loss
  
      ℒ_ id(G) = 
      𝔼_k'∼ p(k), x∼ p_ d(x|k')
      [G(x,k') - x_ρ^ρ],
  
  to ensure that an input into G will remain unchanged when the input already belongs to domain k'.
  
  To summarize, 
  the full objectives  
  to be minimized with respect to G, D, and C
  are given as 
  
      ℐ_G(G) =
         λ_ advℒ_ adv^G(G)
      +
      λ_ clsℒ_ cls^G(G)
      
         +
      λ_ cycℒ_ cyc(G)
      +
      λ_ idℒ_ id(G),
      
      ℐ_D(D) =
         λ_ advℒ_ adv^D(D),
      
      ℐ_C(C) =
         λ_ clsℒ_ cls^C(C),
  
  respectively, 
  where λ_ adv≥ 0, 
  λ_ cls≥ 0, λ_ cyc≥ 0 and λ_ id≥ 0 are 
  regularization parameters, which weigh the importance of 
  the adversarial,
  domain classification,
  cycle consistency, and 
  identity mapping losses.
  Since the adversarial and domain classification losses in 
  (<ref>), (<ref>), (<ref>) and (<ref>)
  are defined using cross-entropy measures, 
  we refer to this version of StarGAN as “C-StarGAN” (cstargan).
  
  
  
  
  
   §.§ Wasserstein StarGAN formulation
  
  
  Next, we describe the original StarGAN formulation
  <cit.>.
  It is frequently reported that optimization in regular GAN training 
  can often get unstable.
  It has been shown
  that
  using a cross-entropy measure as the minimax objective
  corresponds to optimizing  
  the Jensen-Shannon (JS) divergence between the real data distribution
  and the generator's distribution <cit.>.
  As discussed in <cit.>, 
  why regular GAN training tends to easily get unstable 
  can be explained by the fact that the JS divergence will be maxed out 
  when the two distributions are distant from each other and thus have disjoint supports.
  It is probable that this can also happen in the StarGAN training when
  using a cross-entropy measure.
  With the aim of stabilizing training, instead of the cross-entropy measure,
  the original StarGAN adopts 
  the Wasserstein distance as the training objective, 
  which provides a meaningful distance metric
  between two distributions even for those with disjoint supports. 
  By using
  the Kantorovich-Rubinstein duality theorem <cit.>,
  a tractable form of 
  the Wasserstein distance
  between 
  the real speech feature distribution p(x) and
  the distribution of the fake samples generated by the generator G(x,k), 
  where x∼ p_ d(x) and k∼ p(k),
  is given by
  
      𝒲(G) = 
      max_D∈𝒟{𝔼_y∼ p_ d(y)[D(y)] 
      
      -
      𝔼_k∼ p(k), x∼ p_ d(x)[D(G(x,k))] 
      },
   
  where 
  D must lie within the space
  𝒟 of 1-Lipschitz functions.
  A 1-Lipschtiz function is a differentiable function that has gradients 
  with norm at most 1 everywhere. 
  This Lipschtiz constraint is derived as a result of 
  obtaining the above form of the Wasserstein distance <cit.>.
  As wasserstein shows, 
  the computation of the Wasserstein distance requires
  optimization with respect to a function D.
  Thus, if we describe D using a neural network,
  the problem of minimizing 𝒲(G) with respect to G 
  leads to a minimax game played by G and D, as with regular GAN training, 
  where D plays a similar role to the discriminator.
  Now, recall that
  the function D must be 1-Lipschitz. 
  Although there are several ways to 
  constrain D, such as the weight clipping technique 
  adopted in <cit.>,
  one successful and convenient way involves imposing
  a penalty on the sampled gradients of D
  
      ℛ(D) = 
      𝔼_x̂∼ p(x̂)
      [(∇
      D(x̂) _2 -1)^2],
  
  and including it
  in the training objective <cit.>,
  where 
  ∇ denotes the gradient operator and
  x̂
  is a sample uniformly drawn along a straight line between
  a pair of a real and a generated samples. 
  We must also consider incorporating the domain classification loss to 
  encourage G(x,k) to belong to class k and 
  the cycle-consistency loss to 
  encourage G(x,k) to preserve 
  the linguistic information in the input x. 
  Overall, the training objectives to be minimized with respect to G, D, and C
  become
  
      ℐ_G(G) =    
      -
      λ_ adv𝔼_k∼ p(k), x∼ p_ d(x)[D(G(x,k))]
      
         
      +
      λ_ clsℒ_ cls^G(G)
      +
      λ_ cycℒ_ cyc(G)
      +
      λ_ idℒ_ id(G),
      
      ℐ_D(D) =   λ_ adv𝔼_k∼ p(k), x∼ p_ d(x)[D(G(x,k))]
      
         - λ_ adv𝔼_y∼ p_ d(y)[D(y)]
      
         + λ_ gp𝔼_x̂∼ p(x̂)
      [(∇
      D(x̂) _2 -1)^2],
      
      ℐ_C(C) =   λ_ clsℒ_ cls^C(C),
  
  where λ_ gp≥ 0
  is for weighing the importance of the gradient penalty.
  We refer to this version of StarGAN as “Wasserstein StarGAN (W-StarGAN)”.
  It should be noted that 
  the authors of
  <cit.>
  choose to implement D and C as a single
  multi-task classifier network that simultaneously 
  produces the values D(x) and p_C(k|x) (k=1,…,K) (wstargan). 
  
  
  
   §.§ Proposed New StarGAN formulation
  
  
  With the two StarGAN formulations presented above, 
  the ability of G to appropriately convert its input into a target domain
  depends on how the decision boundary is formed by C during training.
  The domain classification loss can be easily made almost 0 
  by letting 
  the samples of G(x,k) resemble, for example, only 
  a few of the real speech samples in domain k
  near the decision boundary.
  In such situations, 
  G will have no incentive to attempt to make the generated samples
  get closer to the rest of the real speech samples distributed in domain k.
  As a result, the conversion effect of the trained G will be limited. 
  One reasonable way 
  to avoid such situations
  would be to 
  consider additional classes for out-of-distribution samples 
  that do not belong to any of the domains
  and encourage G to not generate samples belonging to those classes. 
  This idea can be formulated as follows.
  
  First, we unify the real/fake discriminator and the domain classifier
  into a single multiclass classifier A that  
  outputs 2K probabilities p_A(k|x) (k=1,…,2K)
  where k=1,…,K and k=K+1,…,2K
  correspond to the real domain classes and the fake classes, respectively.
  Note that this differs from the multi-task classifier network 
  mentioned above in that 
  p_A(k|x) must now satisfy 
  ∑_k=1^2K p_A(k|x)=1.
  Here, the K fake classes can be seen as the classes for out-of-distribution samples.
  Next, by using this multiclass classifier, 
  we define 
  
      ℒ^A_ adv(A) =    
      -𝔼_k∼ p(k), y∼ p_ d(y|k)[log p_A(k|y)]
      
         -𝔼_k∼ p(k), x∼ p_ d(x)[log p_A(K+k|G(x,k))],
      
      ℒ^G_ adv(G) =   
      - 𝔼_k∼ p(k), x∼ p_ d(x)[log p_A(k|G(x,k))]
      
         + 𝔼_k∼ p(k), x∼ p_ d(x)[log p_A(K+k|G(x,k))],
  
  as adversarial losses for classifier A and generator G.
  ℒ^A_ adv(A) becomes small when A correctly classifies 
  y∼ p_ d(y|k) as real speech samples in domain k
  and G(x,k) as fake samples in domain k, whereas
  ℒ^G_ adv(G) becomes 
  small when G fools A so that G(x,k) is misclassified by A as 
  real speech samples in domain k and is not classified as fake samples.
  
  We will show below that this minimax game reaches a global optimum when 
  p_ d(y|k) = p_G(y|k) for k=1,…,K 
  if both G and A have infinite capacity, 
  where p_G(y|k) denotes 
  the distribution of y = G(x,k) with x∼ p_ d(x).
  We first consider the optimal classifier A for any given generator G.
  
  For fixed G, 
  ℒ^A_ adv(A) is minimized when
  
  
      p_A^*(k|y)    = 
      
      p(k) p(y|k)
      /∑_k
      p(k) p_ d(y|k)
      +
      ∑_k
      p(k) p_G(y|k)
      ,
      
      
      p_A^*(K+k|y)
         =
      
      p(k) p_G(y|k)
      /∑_k
      p(k) p_ d(y|k)
      +
      ∑_k
      p(k) p_G(y|k)
      ,
  
  for k=1,…,K.
  
   
  By differentiating the Lagrangian
  
      L(A,γ) = 
      ℒ^A_ adv(A) 
      + 
      ∫γ(y)
      (
      ∑_k=1^2K p_A(k|y) - 1 
      ) 
      y
  
  with respect to p_A(k|y)
  
      ∂L(A,γ)/∂ p_A(k|y)
      =
      
      - 
      
      p(k) p_ d(y|k)
      /
      p_A(k|y)
       + γ(y)
         (1≤ k ≤ K)
      
      
      -
      
      p(k) p_G(y|k)
      /
      p_A(k|y)
       + γ(y)
         (K+1≤ k ≤ 2K)
  
  and setting the result at zero, 
  we obtain
  
      p_A(k|y) = 
      
      p(k)p_ d(y|k)/γ(y)
         (1≤ k≤ K)
      
      
      p(k)p_G(y|k)/γ(y)
         (K+1≤ k≤ 2K)
      .
  
  Since p_A(k|y) must sum to unity,
  the multiplier γ must be
  
      γ(y) = ∑_k=1^K p(k)p_ d(y|k) + ∑_k=1^K p(k)p_G(y|k).
  
  Substituting p_C_opt_2 into p_C_opt_1 concludes the proof.
  
  
  
  The global optimum of the minimax game is achieved when 
  p_ d(y|k) = p_G(y|k) for k=1,…,K.
  
  
  By substituting p_C_opt_1toKp_C_opt_K+1to2K into 
  ℒ^G_ adv(G), we can describe 
  it as a function of G only:
  
      ℒ^G_ adv(G) 
         =
      - 𝔼_k∼ p(k), y∼ p_G(y|k)[log
      p_A^*(k|y)
      /
      p_A^*(K+k|y)
      ]
      
         =
      𝔼_k∼ p(k), y∼ p_G(y|k)[logp_G(y|k)/p_ d(y|k)]
      
         =
      𝔼_k∼ p(k) KL
      [
      p_G(y|k)
      p_ d(y|k)
      ]
      ,
  
  where KL[··] denotes the
  Kullback-Leibler (KL) divergence. 
  Obviously, ℒ^G_ adv(G) becomes 0 if and only if 
  p_ d(y|k) = p_G(y|k) for k=1,…,K,
  thus concluding the proof.
  
  
  As with 
  the first two formulations,
  we must also consider incorporating 
  the cycle-consistency and identity mapping
  losses to 
  encourage G(x,k) to preserve 
  the linguistic information in the input x. 
  Overall, the training objectives to be minimized with respect to G and A
  become
  
      ℐ_G(G) =   λ_ advℒ^G_ adv(G) 
      +
      λ_ cycℒ_ cyc(G)
      +
      λ_ idℒ_ id(G),
      
      ℐ_A(A) =   λ_ advℒ^A_ adv(A).
  
  We refer to this formulation as the “augmented classifier StarGAN (A-StarGAN)” (astargan).
  
  A comparative look at the C-StarGAN <cit.>
  and A-StarGAN formulations  
  may provide intuitive insights into the behavior of the A-StarGAN training.
  Although not explicitly stated, 
  with C-StarGAN, 
  the minimax game played by G and D
  using advloss_dadvloss_g alone is shown to
  correspond to 
  minimizing the JS divergence between p_G(y|k) and p_ d(y|k).
  While this minimax game 
  only cares whether G(x,k) resembles real samples in domain k  
  and is not concerned with whether G(x,k) is likely to belong to a different domain k' ≠ k,
  A-StarGAN
  is designed to require G(x,k) to keep away from all the speaker domains except k 
  by explicitly penalizing 
  G(x,k) for resembling real samples in domain k' ≠ k.
  We expect that this particular mechanism can contribute to enhancing the conversion effect.  
  The domain classification loss given as clsloss_g
  in C-StarGAN is expected to play this role; 
  however, its effect can be limited for the reason already mentioned.
  With A-StarGAN, the classifier augmented with 
  the fake classes creates
  additional decision boundaries, each of which is expected to 
  partition the region of each domain 
  into in-distribution and out-of-distribution regions
  thanks to the adversarial learning and thus 
  encourage the generator to generate samples that resemble real in-distribution samples only.
  It should also be noted that in C-StarGAN,
  when the domain classification loss comes into play, 
  the training objective does not allow for 
  an interpretation of the optimization process as distribution fitting, unlike in A-StarGAN. 
  This is also true for the W-StarGAN formulation.
  
  From the above discussion,
  we can also think of another version of the A-StarGAN formulation,
  in which the K fake classes are merged into a single fake class 
  (so the classifier A now produces only K+1 probabilities) and 
  the adversarial losses for classifier A and generator G are defined as
  
      ℒ^A_ adv(A) =    
      -𝔼_k∼ p(k), y∼ p_ d(y|k)[log p_A(k|y)]
      
         -𝔼_k∼ p(k), x∼ p_ d(x)[log p_A(K+1|G(x,k))],
      
      ℒ^G_ adv(G) =   
      - 𝔼_k∼ p(k), x∼ p_ d(x)[log p_A(k|G(x,k))]
      
         + 𝔼_k∼ p(k), x∼ p_ d(x)[log p_A(K+1|G(x,k))].
  
  It should be noted that
  the minimax game using these losses no longer leads to
  the minimization of the KL divergence between p_G(y|k) and p_ d(y|k).
  However, we still believe it can work reasonably well
  if the augmented classifier really behaves in the way discussed above. 
  
  
  
   §.§ Acoustic feature
  
  
  In this paper, we choose to use mel-cepstral coefficients (MCCs) computed 
  from a spectral envelope obtained using WORLD <cit.>
  as the acoustic feature to be converted.
  Although it would also be interesting to consider 
  directly converting time-domain signals (for example, like in <cit.>), 
  given the recent significant advances in high-quality neural vocoder systems <cit.>, 
  we would expect to be able to generate high-quality signals 
  using a neural vocoder once we obtain a sufficient set of acoustic features.  
  Such systems can be advantageous in that
  the model size for the generator can be made small enough to 
  allow the system to run in real-time and 
  work well even when a limited amount of training data is available.
  
  At training time, 
  we normalize 
  each element x_q,n of 
  the MCC sequence x to
  x_q,n← (x_q,n - ψ_q)/ζ_q
  where q denotes the dimension index of the MCC sequence,
  n denotes the frame index, 
  and ψ_q and ζ_q denote
  the means and standard deviations of the q-th MCC sequence 
  within all the [
  We chose to compute the mean and standard deviation only from the voiced segments in the training samples since we wanted them to be less dependent on the lengths of the silent segments. 
  The voiced segments were detected using WORLD.
  ]voiced segments of the training samples of the same speaker.
  
  
  
  
   §.§ Conversion process
  
  
  
  After training G, we can convert the acoustic feature sequence x 
  of an input utterance with
  
      ŷ = G(x, k),
  
  where k denotes the target domain.
  Once ŷ
  has been obtained, 
  we adjust
  the mean and variance of the generated feature sequence 
  so that they match the pretrained 
  mean and variance of the feature vectors of the target speaker.
  We can then generate a time-domain signal using the WORLD vocoder or any 
  recently developed neural vocoder <cit.>.
  
  
  
  
  
   §.§ Network architectures
  
  
  
  The architectures of all the networks are 
  detailed in generator2dclassifier.
  As detailed below, 
  G is designed to take an acoustic feature sequence as an input
  and output an acoustic feature sequence of the same length  
  so as to learn conversion rules that capture time dependencies.
  Similarly, 
  D, C and A are designed to take acoustic feature sequences as inputs
  and generate sequences of probabilities.
  There are two ways to incorporate the class index k into G or D.
  One is to simply represent it as a one-hot vector 
  and append it to the input of each layer.
  The other is to retrieve a continuous vector 
  given k from a dictionary of embeddings and append it to each layer input,
  as in our previous work <cit.>.
  In this work, we adopted the former way though both performed almost the same.
  As detailed in generator2dclassifier, all the networks are designed
  using fully convolutional architectures with gated linear units
  (GLUs) <cit.>. The output of the GLU
  block used is defined as 
  𝖦𝖫𝖴(X) = X_1 ⊙𝗌𝗂𝗀𝗆𝗈𝗂𝖽(X_2), 
  where X is the layer input, 
  X_1 and X_2 are equally sized arrays into which X 
  is split along the channel dimension,
  and 𝗌𝗂𝗀𝗆𝗈𝗂𝖽 is a sigmoid gate function.
  Like long short-term memory units, 
  GLUs can reduce the vanishing gradient
  problem for deep architectures by providing a linear path for
  the gradients while retaining nonlinear capabilities.
  
  Generator:
  As described in generator2dgenerator1d,
  we use 
   a 2D CNN 
  or a 1D CNN
  that takes 
  an acoustic feature sequence x as an input
  to design G, where x is treated
  as an image of size Q× N with 1 channel in the 2D case 
  or as a signal sequence of length N with Q channels in the 1D case.
  
  Real/Fake Discriminator:
  We leverage the idea of PatchGANs <cit.> to
  design a real/fake discriminator or a Lipschitz continuous function D, 
  which assigns a probability or a score 
  to each local segment of an input feature sequence to
  indicate whether it is real or fake. 
  More specifically, D takes 
  an acoustic feature sequence y as an input and 
  produces a sequence of probabilities (with C-StarGAN)
  or scores (with W-StarGAN) that measures how likely each segment of y 
  is to be real speech features.
  With C-StarGAN, the final output of D is given by the product of all these probabilities,
  and with W-StarGAN, the final output of D is given by the sum of all these scores.
  
  Domain Classifier/Augmented Classifier:
  We also design the domain classifier C and the augmented classifier A 
  so that each of them takes 
  an acoustic feature sequence y as an input and 
  produces a sequence of class probability distributions that measure 
  how likely each segment of y is to belong to domain k.
  The final output of p_C(k|y) or p_A(k|y)
  is given by the product of all these distributions.
  
  
  
  
  
  § EXPERIMENTS
  
  
  
  
  
   §.§ Datasets
  
  
  
  To confirm the effects of the proposed StarGAN formulations, 
  we conducted objective and subjective evaluation experiments involving 
  a nonparallel speaker identity conversion task.
  For the experiments, we used two datasets,
  the CMU ARCTIC database <cit.> and 
  the Voice Conversion Challenge (VCC) 2018 dataset <cit.>.
  The former consists of recordings of 
  two female US English speakers (`clb' and `slt') and
  two male US English speakers (`bdl' and `rms')
  sampled at 16,000 Hz. 
  The latter consists of recordings 
  of six female and six male US English speakers
  sampled at 22,050 Hz.
  From the VCC2018 dataset, 
  we selected two female speakers (`SF1' and `SF2') 
  and two male speakers (`SM1' and `SM2').
  Thus, for each dataset,
  there were K=4 speakers, so 
  in total there were twelve different combinations of source and target speakers.
  
  
  
    §.§.§ The CMU ARCTIC Dataset
  
  
  The CMU ARCTIC dataset consisted of four speakers, 
  each reading the same 1,132 short sentences.
  For each speaker, 
  we used the first 1,000 and the latter 132 sentences
  for training and evaluation.
  To simulate a nonparallel training scenario,
  we divided the first 1,000 sentences equally
  into four groups and used only
  the first, second, third, and fourth groups 
  for speakers clb, bdl, slt, and rms,
  so as not to use 
  the same sentences between different speakers.
  The training utterances of speakers clb, bdl, slt, and rms
  were about 12, 11, 11, and 14 minutes long in total, respectively.
  For each utterance, we extracted
  a spectral envelope,
  a logarithmic fundamental frequency (log F_0), and
  aperiodicities (APs) every 8 ms using the
  WORLD analyzer <cit.>. 
  We then extracted
  Q=28 MCCs from 
  each spectral envelope using the Speech Processing Toolkit (SPTK) <cit.>. 
  
  
  
    §.§.§ The VCC2018 Dataset
  
  
  The subset of the VCC2018 dataset consisted of four speakers,
  each reading 
  the same 116 short sentences (about seven minutes long in total).
  For each speaker, 
  we used the first 81 and the latter 35 sentences 
  (about five and two minutes long in total) 
  for training and evaluation.
  Although we could actually construct a parallel corpus using this dataset,
  we took care not to take advantage of it, because our purpose was 
  to simulate a nonparallel training scenario.
  For each utterance, 
  we extracted
  a spectral envelope,
  a log F_0, APs, and Q=36 MCCs every 5 ms using the
  WORLD analyzer <cit.> and 
  the SPTK <cit.> in the same manner. 
  
  For both datasets,
  the F_0 contours were converted using the logarithm Gaussian normalized
  transformation described in <cit.>. 
  The APs were used directly without modification. 
  The signals of the converted speech were obtained using the methods described in conversion.
  
  
  
   §.§ Baseline Methods
  
  
  We chose the VAE-based <cit.> and VAEGAN-based <cit.> 
  nonparallel VC methods
  and our previously proposed CycleGAN-VC <cit.>
  for comparison. 
  In CycleGAN-VC,
  we used the same network architectures shown in generator2ddiscriminator
  to design the generator and discriminator.
  To clarify how close the proposed method can get to the performance achieved 
  by one of the best-performing parallel VC methods,
  we also chose a GMM-based open-source method 
  called “sprocket” <cit.> for comparison.
  This method was used as a baseline in the VCC2018 <cit.>.
  Note that
  since sprocket is a parallel VC method, 
  we tested it only on the VCC2018 dataset.
  To run these methods,
  we used the source codes provided by the authors <cit.>.
  
  
  
  
   §.§ Hyperparameter Settings
  
  
  In the following, we use the abbreviations A-StarGAN1 and A-StarGAN2
  to indicate the A-StarGAN formulations using advloss3_cadvloss3_g 
  and using advloss4_cadvloss4_g 
  as the adversarial losses. 
  Hence, four different versions of the StarGAN formulations 
  (namely C-StarGAN, W-StarGAN, A-StarGAN1 and A-StarGAN2) were considered for comparison.
  
  All the networks were trained simultaneously with random initialization.
  Adam optimization <cit.> was used for model training, where 
  the mini-batch size was 16. 
  The settings of
  the regularization parameters 
  λ_ adv,
  λ_ cls, λ_ cyc, λ_ id,
  and λ_ gp,
  the learning rates α_G and α_D/C for the generator and the discriminator/classifier, and the iteration number I
  are listed in hyperparameter_settings.
  For CycleGAN and
  all the StarGAN versions,
  the exponential decay rate
  for the first moment was set at 0.9 for the generator and 
  0.5 for the discriminator and classifier.
  loss_curves shows the learning curves of 
  C-StarGAN, W-StarGAN, and A-StarGAN1 under the above settings.
  We also performed batch normalization with
  the training mode at test time.
  Note that all these hyperparameters were tuned on the VCC2018 database.
  
  
  
  
  
  
  
   §.§ Objective Performance Measure
  
  
  In each dataset,
  the test set consists 
  of speech samples of each speaker reading the same sentences.
  Thus, 
  the quality of a converted feature sequence can be assessed by
  comparing it with the feature sequence of the target speaker reading the 
  same sentence.
  Here, we used the average of the mel-cepstral distortions (MCDs) 
  taken along the 
  dynamic time warping (DTW)
  path between converted and target feature sequences
  as the objective performance measure for each test utterance.
  
  
  
   §.§ Objective Evaluations
  
  
  
  
  
  
  
  
  
  
  
  
  First,
  we evaluated the performance of each StarGAN version with
  different network configurations of G. The detailed settings for
  these configurations are shown in generator2dgenerator1d.
  The network architectures of the conditional discriminator and domain classifier
  in C-StarGAN, the multi-task classifier in W-StarGAN, and the
  augmented classifier in A-StarGAN are shown in 
  discriminatorclassifier.
  mcd_config_comp shows the average MCDs 
  along with standard errors
  
  obtained with these network configurations. 
  The results show 
  that 
  the CycleGAN, C-StarGAN, W-StarGAN, A-StarGAN1, and A-StarGAN2 methods
  performed better 
  with G designed using 
  1D-, 2D-, 2D-, 1D-, and 1D-CNNs, respectively.
  In the following,
  we only present the results obtained 
  with these configurations.
  
  mcd_baseline_comp_arcticmcd_baseline_comp_vcc2018
  show the MCDs obtained with the proposed and baseline methods.
  As the results show, 
  W-StarGAN 
  and A-StarGAN1
  performed best and next best on the CMU ARCTIC dataset,
  and A-StarGAN2 and
  A-StarGAN1 performed best and next best of all the nonparallel methods 
  on the VCC 2018 dataset.
  All the StarGAN versions
  performed consistently better than CycleGAN.
  Since both 
  CycleGAN and C-StarGAN use the cross-entropy measure to define the adversarial losses, 
  the superiority of C-StarGAN over 
  CycleGAN reflects 
  the effect of the many-to-many extension. 
  Now, let us turn to the comparisons of the four StarGAN versions.
  From the results, we can see that W-StarGAN performed better than C-StarGAN on both datasets, revealing the advantage of the training objective defined using the Wasserstein distance with the gradient penalty.
  We also confirmed that A-StarGAN1&2 
  performed even better than W-StarGAN on the VCC2018 dataset,
  though it performed
  slightly worse on the CMU ARCTIC dataset. 
  We also confirmed that all the StarGAN versions 
  could not yield higher performance than sprocket.
  Given the fact that sprocket had the
  advantage of using parallel data for the model training, 
  we consider the current result to be promising, since
  the proposed methods are already advantageous in that they can
  be applied in nonparallel training scenarios.
  
  
  
  Balancing the learning of the players in a minimax game is essential in the GAN framework. 
  The probabilities of the feature sequence converted from each test sample  
  being real and produced by the target speaker 
  may provide an indication of how successfully the generator, discriminator, and classifier
  have been trained in a balanced manner.
  dis_output_arcticcls_output_arctic show
  the mean outputs of the discriminator and classifier
  of C-StarGAN, W-StarGAN, and A-StarGAN1&2 at test time.
  Note that since 
  the discriminator in W-StarGAN produces scores (instead of probabilities), which are not straightforward to interpret, 
  we 
  have omitted them in dis_output_arctic.
  As for the augmented classifier in A-StarGAN1&2,
  if we use p_k,n to denote an element of the classifier output
  corresponding to the probability of the classifier input 
  belonging to class k at [
  Since the classifier is designed to contain 
  three downsampling layers, the output sequence becomes 2^3=8 times shorter than the input sequence. 
  Hence, n corresponds to a segment consisting of eight consecutive frames.
  This is why here we have used “segment” rather than “frame” to signify n. 
  ]segment n, 
  the values 
  ∑_k=1^K p_k,n 
  and
  p_k,n/∑_k'=1^K p_k',n
  correspond to the probabilities 
  of the classifier input being real and produced by speaker k, 
  respectively,
  at that segment[
  In A-StarGAN1, k ∈{1,…,K} indicates a class corresponding to real speaker k,
  whereas K+k indicates a class corresponding to fake speaker k.
  Thus,
  the marginal probability of an input being real at segment n
  can be expressed as ∑_k=1^K p_k,n.
  The conditional probability of an input being produced by speaker k at segment n, 
  given that the input is real,
  can be expressed as p_k,n/∑_k'=1^K p_k',n.
  The same applies to A-StarGAN2.
  ]. 
  The means of these values over n of all the utterances
  along with standard errors
  are shown in
  dis_output_arcticcls_output_arctic.
  As dis_output_arcticcls_output_arctic indicate, 
  the generators in all the StarGAN versions
  were successful in 
  confusing the discriminator and
  making the classifier believe
  that the feature sequence converted from each test sample
  was produced by the target speaker.
  
  The modulation spectra of MCC sequences are known to be  
  quantities that are closely related to the perceived quality and naturalness of speech <cit.>. 
  modspec_all shows an example of the average 
  modulation spectra of the converted MCC sequences obtained with the proposed and baseline methods
  along with those of the real speech of the target speaker.
  As this example shows, the modulation spectra obtained with 
  the CycleGAN-based method and 
  all the StarGAN-based methods were relatively closer to those of real speech 
  than the spectra obtained with the VAE-based and VAEGAN-based methods over the entire frequency range, 
  thanks to the adversarial training strategy. 
  
  
  
  
  
   §.§ Subjective Listening Tests
  
  
  
  
  
  
  We conducted 
  subjective listening tests to
  compare the speech quality and speaker similarity
  of the converted speech samples
  obtained with the proposed and baseline methods.
  For these tests, we used the CMU ARCTIC dataset.
  Twenty-four listeners (including 21 native Japanese speakers) 
  participated in both tests.
  The tests were conducted online, where each participant was asked 
  to use a headphone in a quiet environment.
  
  With the speech quality test, we evaluated the mean opinion score (MOS) for each speech sample.
  In this test, we included the speech samples 
  synthesized in the same way as the proposed and baseline methods (namely with the WORLD synthesizer) using
  the acoustic features directly extracted from real speech samples. 
  Hence, the scores of these samples are expected to show the upper limit of the performance. 
  Speech samples were presented in random orders to eliminate bias as regards the order of the stimuli. 
  Each listener was  
  asked to evaluate the naturalness
  by selecting 5: Excellent, 4: Good, 3: Fair, 2: Poor, or 1: Bad for each utterance.
  The obtained scores
  with 95% confidence intervals are shown in MOS_qlt.
  As the results show, 
  A-StarGAN1 performed slightly better than 
  W-StarGAN and A-StarGAN2 (although the differences were not significant)
  and significantly better than C-StarGAN and the VAE and VAEGAN methods.
  However, it also became clear that the speech quality obtained with all the 
  methods tested here
  was still perceptually distinguishable from real speech samples.
  
  With the speaker similarity test, 
  the subjective score for each sample was rated
  on a five-point scale, as with the speech quality test. 
  Each listener was given a converted speech sample and 
  a real speech sample of the corresponding target speaker 
  and asked to evaluate how likely they were to have been produced by the same speaker by selecting 
  5: Definitely, 4: Likely, 3: Fairly likely, 2: Not very likely, or 1: Unlikely. 
  The obtained scores
  with 95% confidence intervals
  are shown in MOS_sim.
  A scatter plot of the speech quality and speaker similarity scores 
  obtained with the tested methods is shown in scatter.
  As can be seen from the results,
  the W-StarGAN and A-StarGAN formulations
  performed comparably to each other and
  showed significantly
  better conversion ability 
  than the remaining four methods.
  
  Voice conversion examples 
  are provided at <cit.>.
  
  
  
  § CONCLUSION
  
  
  In this paper, we proposed a method that allows nonparallel multi-domain VC 
  based on StarGAN. We described three formulations of StarGAN and
  compared them and several baseline methods 
  in a nonparallel speaker identity conversion task.
  Through objective evaluations, we confirmed that
  our method was able to convert speaker identities reasonably well using
  only several minutes of training examples.
  Interested readers are referred to <cit.>
  for our investigations of other
  network architecture designs and improved techniques for CycleGAN-VC and StarGAN-VC.
  
  One limitation of the proposed method is that 
  it can only convert input speech to the voice of a speaker 
  seen in a given training set.
  This is due to the fact that  
  one-hot encoding (or a simple embedding) used for speaker conditioning is 
  nongeneralizable to unseen speakers.
  An interesting topic for future work includes 
  developing a zero-shot VC system that can convert input speech to 
  the voice of an unseen speaker by looking at only a few of his/her utterances.
  As in the recent work <cit.>,
  one possible way to achieve this involves using 
  a speaker embedding 
  pretrained based on a metric learning framework
  for speaker conditioning.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  IEEEtran
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Hirokazu Kameoka
  Hirokazu Kameoka received the B.E., M.S., and Ph.D. degrees, all from the University of Tokyo, Japan, in 2002, 2004  and  2007, respectively. He is currently a Senior Distinguished Researcher in NTT Communication  Science  Laboratories,  Nippon  Telegraph and Telephone Corporation and an Adjunct Associate Professor at the National Institute of Informatics. From 2011 to 2016, he was an Adjunct Associate Professor at the University of Tokyo. His research interests include audio, speech, and music signal processing and machine learning. He has been  an  associate  editor  of  the  IEEE/ACM  Transactions on Audio, Speech, and Language Processing since 2015, a Member of IEEE Audio and Acoustic Signal Processing Technical Committee since 2017, and a Member of IEEE Machine Learning for Signal Processing Technical Committee  since  2019.  He has received 17 awards, including the IEEE Signal Processing Society 2008 SPS Young Author Best Paper Award. He is the author or co-author of about 150 journal articles and peer-reviewed conference proceedings.
  
  
  Takuhiro Kaneko
  Takuhiro Kaneko received the B.E., M.S., and Ph. D degrees from the University of Tokyo, Japan, in 2012, 2014, and 2020, respectively. He is currently a Distinguished Researcher in NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation. His research interests include computer vision, signal processing, and machine learning. In particular, he is currently working on image synthesis, speech synthesis, and voice conversion using deep generative models. He has received the Hatakeyama Aawrd from the Japan Society of Mechanical Engineers in 2012, the ICPR2012 Best Student Paper Award at the 21st International Conference on Pattern Recognition in 2012, and Dean's Award for Best Doctoral Thesis from the University of Tokyo in 2020.
  
  
  Kou Tanaka
  Kou Tanaka received the B.E. degree from Kobe University, Japan, in 2012 and the M.E. and D.E. degrees from the Nara Institute of Science and Technology (NAIST), Japan, in 2014 and 2017, respectively. 
  He joined NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation as a research associate in 2017.
  He is currently a research scientist at NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation.
  He was a Research Fellow of the Japan Society for the Promotion of Science from 2015 to 2017. 
  His research interests include speech signal processing with a strong focus on deep generative models.
  
  
  
  Nobukatsu Hojo
  Nobukatsu Hojo recieved the B.E and M.E. degrees from the University
  of Tokyo, Japan, in 2012 and 2014, respectively. He joined NTT Media
  Intelligence Laboratories, Nippon Telegraph
  and Telephone Corporation in 2014, where he was engaged in the research
  and development of speech synthesis. 
  He is currently a research scientist at 
  NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation.
  He is a member of Acoustical
  Society of Japan (ASJ), the International Speech Communication
  Association (ISCA), and the Institute of Electronics, Information and
  Communication Engineers (IEICE) of Japan.
  
  
  
  
  
  
  
  
  
  
  
  
  