Learning to Fuse Music Genres with Generative Adversarial Dual Learning

FusionGAN is a novel genre fusion framework for music generation that integrates the strengths of generative adversarial networks and dual learning. In particular, the proposed method offers a dual learning extension that can effectively integrate the styles of the given domains. To efficiently quantify the difference among diverse domains and avoid the vanishing gradient issue, FusionGAN provides a Wasserstein based metric to approximate the distance between the target domain and the existing domains. Adopting the Wasserstein distance, a new domain is created by combining the patterns of the existing domains using adversarial learning. Experimental results on public music datasets demonstrated that our approach could effectively merge two genres.


I. INTRODUCTION
Computational creativity is a lively research area that focuses on understanding and facilitating human creativity through the implementation of software programs [2]. With the rapid advances in data-driven algorithms such as deep learning [7], the exploration into computational creativity via machine learning approaches has become increasingly popular. Examples showing the potential of deep learning for creative approaches are the artistic style transfer on images [4] and videos [12]. Music, with its complex hierarchical and sequential structure and its inherent emotional and aesthetic subjectivity, is an intriguing research subject at the core of human creativity. While there has been some work on generative models for music, research that investigates the capabilities of deep learning for creative applications such as style transfer is limited. This paper aims to fill this space by exploring the idea of style fusion in music with generative adversarial dual learning.
In the field of unsupervised generative learning, generative adversarial networks (GAN) [6] have recently gained considerable attention. It is important to note, however, that GANs are designed to learn from a single domain and cannot discover cross-domain knowledge. Inspired by dual learning in machine translation, several recent publications propose methods to pair unlabeled data points in different domains by optimizing bi-directional reconstruction errors [14], [8], [16]. Although these methods are designed to address the challenge of multimodal learning, they do not address the problem of domain fusion.
The goal of this paper is to provide a solution for fusing two or more groups of sequential patterns using unsupervised methods. Specifically, we focus on the problem of generating music with a fused music genre. To combine genres using generative adversarial learning, we propose a framework for integrating multiple GANs. Unlike previous work, our study targets the creation of an unknown domain by mixing two given domains; a multi-way GAN-based model is proposed to absorb existing patterns and yield a new mixture. With the utilization of the Wasserstein measure [1], we pose a distance constraint on the new domain that automatically balances its relation to the given domains. As a result, our model is capable of generating a mixed pattern after convergence. The main contributions of this paper are: • A novel framework for unsupervised music fusion: The dual learning scheme is extended to involve multiple domains by leveraging mutual regularization. In this way, the proposed framework enables the new domain to keep equal similarities with the given domains and produce a mixture of existing sequential patterns. • A sequence fusion method using GANs: To apply unsupervised fusion learning, we extend the generative adversarial model so that multiple generators and discriminators can be updated by one another. • Formulation of an objective function indicating fusion progress: An objective function based on Wasserstein distance is designed to integrate the information from multiple domains and represent learning progress.
II. RELATED WORK Music Genre Fusion: Fusion is mostly known as the sub-genre of jazz that emerged in the late '60s, and that combines several musical styles such as funk, rock, and blues with the jazz harmony and improvisation [5]. The term can be generalized, however, to any combinations of music genres. The creative combination of two or more genres obviously requires not only intimate knowledge of the stylistic characteristics of the involved genres but also extensive compositional experience and skill to combine genres in a meaningful and satisfying way. Recently, Engel et al. [3] proposed a new approach of generating new musical sounds through interpolating the latent space learned by WaveNet [11] autoencoders. Additionally, Gatys et al. demonstrated the Deep Neural Networks' (DNNs) capabilities of learning the artistic styles by blending the style of the training materials to the testing images using ConvNet [4]. Both examples show the great potential of using DNNs for merging two abstract concepts without the need for domain knowledge and explicitly defined rules.
Cross-domain GAN: GAN [6], which has quickly risen to one of the most popular generative approaches, learns patterns without requiring adversarial examples or labels. In CGAN [10], the generator G has a conditional parameter c and will learn the conditional distribution of the data.
However, those conditions need to be provided manually, somewhat similar to supervised learning. CoGAN [9] is proposed to learn a joint distribution with only samples drawn from the marginal distributions. DiscoGAN [8] solves the cross-domain pairing by minimizing bi-directional reconstruction loss, while DualGAN [14] takes advantage of the dual learning paradigm in machine translation. CycleGAN [16] presents a method whose mapping functions are cycleconsistent and proposes a cycle consistency loss function to further reduce the space of possible mapping functions.
Different from the previous studies, our work raises a new problem, i.e., merging two sequential patterns into one. We propose a GAN-based framework that merges two domains without manually tuning the distance between the given domains and the target domain.

III. LEARNING TO BLEND MUSIC WITH FUSIONGAN A. Problem Definition
This paper aims to characterize music fusion across different genres in an unsupervised manner. Based on the data X A and X B from existing music domains D A , D B respectively, our goal is to create a new domain D F which resembles both D A and D B . In the first phase, the new domain D F is expected to learn from D A , D B and keep an equal distance from both. Specifically, the generator G F of D F obtains feedback from the discriminators D A , D B of existing domains D A , D B , while the discriminator D F collects data from G F and D A , D B for iterative updating. The G F from the new domain minimizes the distance from the domains D A , D B , respectively, and keeps the same distance from D A and D B .

B. FusionGAN
First, the framework initializes individual GAN models for existing domains. Maximum likelihood estimation using Long Short Term Memory (LSTM) is applied to initialize the sequence generators G A (G B ). Then D A (D B ) are trained given the domain data and sequence sampled from G A (G B ). Then GAN is employed to iteratively enhance the G A (G B ) and D A (D B ). After pre-training, we build a GAN model for the new domain using the feedback from the models constructed in the pre-training procedure. First, a new domain D F is initialized randomly and trained by both D A and D B . Following the dual learning strategy, D A is enhanced by D B and D F in the second phase. Similarly, D B is improved by D A and D F . Such learning proceeds until convergence. The framework overview is shown in Figure 1. In FusionGAN, we employ the Wasserstein-1 distance for all discriminators because it is sensible when the learning distributions are supported by low dimensional manifolds [1]. To avoid the intractable infimum in Wasserstein-1 distance, we resort to its Kantorovich-Rubinstein duality [13]: For the vanilla GAN, the goal is to find the optimal configuration of the parameters φ of discriminator (f = D). When the discriminator is optimized, the maximized Wasserstein distance can be used as reward in the policy gradient process of the generator. Similarly, we define a three-way Wasserstein distance as the objective function: W (P XA , P XB , P θ ) = L to measure the integrated distance between P θ and P XA , P XB . This measure should consider the relationship between every pair of domains. The framework consists of three pairs of generators and discriminators, i.e., and two input data, i.e., X A , X B . For each discriminator, there are five possible inputs, i.e., X A , X B , G A , G B , G F . Collecting all possible inputs w.r.t. the three discriminators, the objective function to maximize is defined as: where the first five terms are for D A , the second five terms are for D B , and the last five terms are for D F . This loss is used to update all the generators and discriminators. The following two subsections, III-C and III-D, will present the update rules using Eq. 1.

C. D F Update of FusionGAN
Removing the terms that are unrelated to D F in Eq. 1 (zero derivative w.r.t. D F and G F ), we have the objective function for updating D F and G F : First, L F is calculated w.r.t. G F so as to update G F . Considering differentiating ∇ θ L F , the optimal G F is approximated as its convergence. The proof is provided below to show that this derivative is principled under the optimality assumption. Theorem III.1 (FusionGAN Optimality Theorem). Let P XA and P XB be any distribution, Let P θ be the distribution of G Fθ (Z) with Z, a random variable with density p and G Fθ a function satisfying E z∼p [L(θ, z)] < +∞, where L is Lipschitz constants. All functions are well-defined. Then there is a solution D F : X → R to the problem: and we have: Proof of Theorem III.1: Let us define By the Kantorovich-Rubinstein duality, there is an f ∈ F that satisfies: is continuous and differentiable for any D F ∈ X * (θ) when both terms are well-defined. Let D F ∈ X * (θ), which we knows exists since X * (θ)is non-empty for all θ. Then: since D F is 1-Lipschitz. Furthermore, G Fθ (z) is locally Lipschitz as a function of (θ, z). Therefore, G Fθ (z) is locally Lipschitz on (θ, z) with constants L(θ, z). By Radamachers Theorem, D F (G Fθ (z)) has to be differentiable almost everywhere for (θ, z) jointly. Rewriting this, the set By Fubinis Theorem, this implies that for almost every θ the section A θ = {z : (θ, z) ∈ A} has measure 0. Let's fix a θ 0 such that the measure of A θ0 is null. For this θ 0 we have ∇ θ D F (G Fθ (z))|θ 0 is well-defined for almost any z, and since p(z) has a density, it is defined p(z) almost everywhere. Given the condition E z∼p [L(θ, z)] < +∞, we have so E z∼p(z) [∇ θ f (g θ (z))] is well-defined for almost every θ 0 .
To keep the notation simple, we leave it implicit in the following equation that the E subjects to z ∼ p(z). : where i ∈ {A, B, F }. By differentiability, the term inside the integral converges p(z) to 0 as θ → θ 0 . Furthermore, and since E z∼p(z) [6L(θ 0 , z)] < +∞, we get by dominated convergence that Eq. 5 converges to 0 as θ → θ 0 . So the result of Eq. 5 equals to: Therefore, the update rules for the parameters θ of G F is Eq. 4. The physical meaning is that all three discriminators can offer feedback to generator G F as shown in Fig. 2. The left cyan rectangle indicates D A , the middle pink one is D F , and the right yellow one D B . Similarly, the gradient w.r.t. the parameters φ of D F is: As shown in Fig. 3, the update for D F actually implicitly balances itself between D A and D B . However, this update does not cover the degree of blending which may lead to an unbalanced mixture. Specifically, a good learning process for the discriminator D F should consider two factors: (1) minimizing the distance with D A , D B and D F simultaneously and (2) maintaining a equal distance from D A and D B so as to satisfy both domains. To satisfy (2) and push the discriminator D F to have a well-proportioned blending, an additional constraint is employed to further balance the ratio of existing domains, i.e., D A , D B :

D. D A and D B Update of FusionGAN
After updating D F , FusionGAN will improve D A and D B . Since D A and D B are symmetric in the framework, we only discuss D A as shown in Fig. 9. Keeping related terms in Eq. 1, we get the loss function for D A : Taking the derivative of L A w.r.t. D A , we have There are four negative terms and one positive term in Eq. 11 w.r.t. D A . The training implicitly increases the positive terms and decreases the value of the negative terms. However, the inequalities in the two groups are not controlled. To further control the distance among the three domains, we propose an inequality constraint for D A : Then combining Eq. 12 into loss function Eq. 11, it can be rewritten as:

E. Algorithm Description
The Algorithm 1 1 initializes the parameters of the discriminators and generators (line 1). TextCNN [15] and LSTM are employed to build the discriminators and generators respectively. Given X A , the generator is updated by maximum likelihood estimation (line 3). From lines 4 to 5, D A receives the sequences generated by G A as negative data, and real data as positive samples. Between lines 6 and 16, a GAN procedure is applied to optimize D A and G A . D A returns regression scores that are treated as rewards for the policy gradient of G A . Similarly, D A improves itself by the real world data X A and the data generated by G A . Our method proceeds to update D B and G B by using the same process from lines 3 to 16. D F repeats the same pre-training procedure from 3 to 16 with D F , G F , X F = X A + X B .

A. Quantitative Study
Due to the aesthetic nature of the task, a quantitative evaluation can only give a first impression of the power of the generative system. Here, we investigate the distance between pitch and note duration distributions, knowingly discarding all sequential information of the melodies.
The ideal output should keep equally close to the given domains and minimize this distance. Two distributions of properties are selected for the study.
1) Duration Distribution (DD): Notes are categorized by their durations. 2) Normalized Pitch Distribution (NPD): All tokens are categorized into C/C#/D/D#/E/F/F#/G/G#/A/A#/B. To quantify the similarity between distributions, two symmetric metrics, the Euclidean distance (EUD) and the Wasserstein-1 metric (Earth Mover distance (EM)), are employed to calculate the distance between the distributions of generated sequences and those of domains X A and X B . Intuitively, the distance between X A and X B , between X A , D F , and between X B , D F should satisfy the triangle inequality, and the optimal configuration should minimize the inequality, i.e., Figure 6 (top) shows the note duration distributions (DD) and Table I (top) gives the corresponding distance measures. The note duration distribution of jazz and folk have shapes that resemble the power law distribution. The shape of RM is very different from jazz or folk. Visually, RL's duration distribution does not match that of jazz and folk because the portion of notes at the length of 0 and 1 (the peak) were relatively higher than that of jazz or folk. The other baselines (GAN, MLE, MC) appear similar to the given domains because they conform to the power law distributions. Fusion is roughly at the same level of MC. Note that, given our evaluation metrics, MC has the best performance in this experiment since it directly samples notes from the true distribution. This disregard of sequential information, however, results in unnatural rhythmic qualities of the music clips generated by MC. Figure 6 (bottom) shows the pitch distributions (NPD) and Table I (bottom) gives the corresponding distance measures. RL's pitch distribution is -due to overfitting-different from that of jazz and folk, which shows its low capacity in modeling pitch distribution. The single peak at D# as generated by GAN does not match either the jazz or folk distributions. The distributions of the other baselines are difficult to categorize as a good or bad match. Since MC is directly sampled from real distributions, its configurations should be exactly the same as that of the optimal fusion. MC shows two valleys at C# and F#, and two peaks at C and G, which should be expected. RM satisfies one of them (F# is a valley), while RM meets only one standard (C is the peak). GAN fails in all the characteristic pitch classes, and both MLE and Fusion meet all of them. Quantitatively, the ratio of RM in NPD is the highest which means that RM is the worst in modeling pitch. GAN and RL improve RM by around 10%. As one of the two distributions that look most similar to the original distributions, MLE outperforms RM, RL, and GAN. Our method Fusion has the second lowest ratio after MC. However, as mentioned above, the MC does not consider the sequential consistence, resulting in bad evaluations in the listening test described in Sect. IV-B.

B. User Study via Listening Test
To evaluate the subjective quality of the generated sequences, a user study was conducted via Amazon Mechanical Turk. The listening test was divided into three steps: 1) Qualification test: "Listen to the music, and choose one genre below that most matches the music". This is to test the evaluators' qualification level. The provided choices are (A) jazz, (B) folk, (C) neither. 2) Fusion recognition: "Listen to the attached music, and then choose the choice that most matches the music" Possible answers are (A) pure jazz, (B) pure folk, (C) mixture of jazz and folk, (D) neither. Each of the pieces of music generated by our proposed method and the baselines are assigned to one such question. 3) Musicality: "Listen to the attached music in 2nd step, who do you think its composer", the allowed answers are (A) expert, (B) newbie, (C) robot. Similarly, each generated sample is associated with one such question. We randomly drew from the training data and prepared 800 sets for the first question. In the first step, 66.5% of candidates chose the correct answer, which is an acceptable rate given the inherent difficulties of genre identification for average listeners. The statistics resulting from the valid answers are shown in Table II. Each of the methods generated 500 samples, which were randomly selected for the second and third questions.
As shown in Table II, the best system based on the percentage of mixture is MLE. A closer look at MLE, however, reveals the unbalanced distribution between jazz and folk, which implies its bias towards the jazz genre. On the other hand, RM is the best system based on the balance between jazz and folk, but the high percentage of neither suggests the confusion of the listeners. It is clear that a single criterion is insufficient for determining the best system. Therefore, we design a metric to summarize the results and represent the fusion level (FL) of the evaluated systems with the following equation: where C i indicates the count of i. C jazz − C f olk means the unbalanced error, and C neither can be treated as another type of error. A higher FL value implies better fusion.
Overall, Fusion shows a balanced performance in both F usionrecognition and M usicality tests, which demonstrates the effectiveness of our proposed method. V. CONCLUSION In this paper, we proposed a three-way GAN-based learning framework to integrate multiple domains. A Wasserstein distance based metric is introduced to indicate the blending progress. The evaluation investigated objective metrics looking at the pitch and note duration distributions of the generated data. A user study explicitly illustrates the validity of the proposed method as compared to the baselines, as the melodies generated by our model were preferred by the majority of users. While the listening test results are encouraging, future work will benefit from removing key dependence of the training data, a careful look into perceptually meaningful evaluation metrics that take into account the sequential nature of music, and a careful listening test design, including human-composed melodies for comparison.