Mind2Mind : transfer learning for GANs
Yaël Frégier Jean-Baptiste Gouray



Simple source device-independent continuous-variable
 quantum random number generator
    A. J. Shields
    December 30, 2023
=====================================================================================




Training generative adversarial networks (GANs) on high quality (HQ) images involves important computing resources. This requirement represents a bottleneck for the development of applications of GANs. We propose a transfer learning technique for GANs that significantly reduces training time. Our approach consists of freezing the low-level layers of both the critic and generator of the original GAN. We assume an auto-encoder constraint in order to ensure the compatibility of the internal representations of the critic and the generator. This assumption explains the gain in training time as it enables us to bypass the low-level layers during the forward and backward passes. We compare our method to baselines and observe a significant acceleration of the training. It can reach two orders of magnitude on HQ datasets when compared with StyleGAN. We prove rigorously, within the framework of optimal transport, a theorem ensuring the convergence of the learning of the transferred GAN. We moreover provide a precise bound for the convergence of the training in terms of the distance between the source and target dataset.  






§ INTRODUCTION


The recent rise of deep learning as a leading paradigm in AI mostly relies on computing power (with generalized use of GPUs) and massive datasets. These requirements represent bottlenecks for most practitioners outside of big labs in industry or academia and are the main obstacles to the generalization of the use of deep learning.  Therefore, methods that can bypass such bottlenecks are in strong demand.  Transfer learning is one of them and various methods of transfer learning (for classification tasks) specific to deep neural networks have been developed <cit.>. 

 
 A generative problem is a situation in which one wants to be able to produce elements that could belong to a given data set 𝒟.
Generative Adversarial Networks (GANs) were introduced in 2014 <cit.> <cit.>  to tackle generative tasks with deep learning architectures and improved in <cit.> (Wasserstein GANs ). They immediately took a leading position in the world of generative models, comforted by progress with HQ images <cit.>, <cit.>.
The goal of our work (see section <ref>) is to develop in a generative setting, i.e., for GAN architectures, the analog of the cut-and-paste approach. 

The main idea of our method is to reuse, for the training of a GAN, the weights of an autoencoder already trained on a source dataset 𝒟.  The weights of the low level layers of the generator (resp. critic) will be given by those of the decoder (resp. encoder). We call MindGAN the high level layers of the generator. It is a subnetwork that is trained as a GAN on the encoded features of the target dataset 𝒟'.

We prove in section <ref> a theorem that controls the convergence of the transferred GAN in terms of the quality of the autoencoder, the domain shift between 𝒟 and 𝒟' and the quality of the MindGAN. As a consequence, our experimental results in section <ref> rely heavily on the choice of the autoencoder. ALAE autoencoders <cit.> are extremely good autoencoders that turned out to be crucial in our experiments with HQ images. Their use, in conjunction with our transfer technique, enables an acceleration of the training by a factor of 656, while keeping a good quality. 














§ PRELIMINARIES


 
A GAN consists of two networks trained adversarially. The generator g:Z→χ associates to a vector z sampled from a latent vector space Z a vector g(z) in another vector space χ while the discriminator c:χ→ℝ learns to associate a value close to 1 if the vector g(z) belongs to 𝒟 and zero otherwise. Their respective loss functions, L_g and L_c are recalled in section <ref>.

One can assume that elements of 𝒟 can be sampled from an underlying probability distribution ℙ_𝒟 on a space χ and try to approximate it by ℙ_θ, another distribution on χ that depends on some learnable parameters θ.  Generating then means sampling from the distribution ℙ_θ. The main idea behind a Wasserstein GAN is to use the Wasserstein distance (see appendix <ref>, and <cit.> definition 6.1) to define by W(ℙ_𝒟,ℙ_θ) the loss function for this optimisation problem.  More precisely, the Wasserstein distance is a distance on Borel probability  measures on χ (when compact metric space). In particular, the quantity W(ℙ_𝒟,ℙ_θ) gives a number which depends on the parameter θ since ℙ_θ depends itself on  θ. The main result of <cit.> asserts that
if ℙ_θ is of class 𝒞^k as a function of θ almost everywhere, W(ℙ_D, ℙ_θ) is also of class 𝒞^k with respect to θ. As a consequence, one can solve this optimization problem by doing gradient descent for the parameters θ (using a concrete gradient formula provided by the same theorem) until the two probability distributions coincide.

Among the distributions on χ, some can be obtained from a prior distribution ℙ_Z on an auxiliary latent space Z and a map g : Z →χ as follows. The push-forward of ℙ_Z under g <cit.> is defined so that a sample is given by g(z) the image through g of a sample z from the distribution ℙ_Z. We will denote this pushforward by g_♯ℙ_Z and when g depends on parameters θ use instead the notation ℙ_θ:=g_♯ℙ_Z. In practice, one can choose for ℙ_Z a uniform or Gaussian distribution, and for g a (de)convolution deep neural network. In our applications, we will consider for instance Z:=ℝ^128 equipped with a multivariate gaussian distribution and χ=[-1,1]^28× 28, the space of gray level images of resolution 28× 28. Hence, sampling from ℙ_θ will produce images.

In order to minimise the function W(ℙ_𝒟,ℙ_θ), one needs a good estimate of the Wasserstein distance. The Rubinstein-Kantorovich duality (See <cit.> theorem 5.9) states that W(ℙ, ℙ')=max_| c|_L≤ 1𝔼_x ∼ℙ c(x)-𝔼_x∼ℙ' c(x), where 𝔼_x ∼ℙf(x) denotes the expected value of the function f for the probability measure ℙ, while the max is taken on the unit ball for the Lipschitz semi-norm. Concretely, this max is obtained by gradient ascent on a function c_θ encoded by a deep convolution neural network.







In our case, when ℙ':=ℙ_θ, the term 𝔼_x∼ℙ' c(x) takes the form 𝔼_z∼ℙ_Z c_θ(g_θ(z)). One recovers the diagram 
    Zg_θ⟶χc_θ⟶ℝ
 familiar in the adversarial interpretation of GANs. With this observation, one understands that one of the drawbacks of GANs is that there are two networks to train. They involve many parameters during the training, and the error needs to backpropagate through all the layers of the two networks combined, i.e., through c_θ∘ g_θ. This process is computationally expensive and can trigger the vanishing of the gradient. Therefore, specific techniques need to be introduced to deal with very deep GANs, such as in <cit.>. The approach we present in section <ref> can circumvent these two problems.


 





 Transfer learning is a general approach in machine learning to overcome the constraints of the volume of data and computing power needed for training models. It leverages a priori knowledge from a learned task 𝒯 on a source data set 𝒟 in order to learn more efficiently a task 𝒯' on a target data set 𝒟'. It applies in deep learning in at least two ways: Cut and Paste and Fine tuning.
 
 
 
 

Cut and Paste takes advantage of the difference between high and low-level layers of the network c. It assumes that the network c is composed of two networks c_0 and c_1 stacked one on each other, i.e., mathematically that c=c_0∘ c_1 (one understands the networks as maps). While the low-level layers c_1 process low-level features of the data, usually common for similar datasets, the high-level layers c_0 are in charge of the high-level features which are very specific to each dataset. Hence, instead of retraining all the weights of an auxiliary network pre-trained on 𝒟, one can retrain only the parameters of the last layers of the network while keeping the other parameters untouched. This approach boils down to the following steps  :



  * identify a dataset 𝒟 similar to the data set 𝒟' we are interested in, both in the same space χ,

  * import a network c=c_0∘ c_1 , already trained on the dataset 𝒟, where c factors as 
[column sep=1.5em]
    M drc_0    

χurc_1 rrc        C  with C the space of classes.



  * pass the new dataset 𝒟' through c_1 and train c_0' on c_1(𝒟'), and

  * use c':=c_0'∘ c_1 as the new classifier on 𝒟'.


The main advantages of this approach are the following :


  a. much fewer parameters to train (only the parameters of c_0', which in practice correspond to a few dense layers),

  b. need to pass the data 𝒟' only once through c_1, and

  c. no need to backpropagate the error through c_1.
 
 




 Since the low-level features often represent the most time-consuming part of the training, eliminating the need to train their weights will accelerate the process. If the datasets are similar, only training the last layers generally leads to good results in a much shorter time and with fewer data.
 
  Fine tuning is based on the same first two steps than Cut and Paste, but instead of steps 3 and 4, uses the weights of c to initialise the training of the new network c' on 𝒟'. It is assumed that c and c' share the same architecture.

 






In both approaches, the network c must have been previously trained by a third party during a very long time on the source dataset 𝒟, which is potentially much more massive than 𝒟'. 
It turns out in practice that these approaches enable to train a network on a new task with much less computing power and data (see <cit.>). 




§ RELATED WORKS

In the following section, we survey the works to which this paper can be associated. We postpone the analysis of the main differences with these works in section <ref>.

 
Wasserstein autoencoders
A version of autoencoders in conjunction with GANs has been considered in <cit.> and later generalized with Wasserstein distance in <cit.> and subsequent works. In short, in this approach, one trains an adversarially an autoencoder. Moreover, one adds a regularizer term to get a generative model. More details in section <ref>.

Adversarial learned inference
A second stream of papers, <cit.>, <cit.>, <cit.>, <cit.> and <cit.> to cite a few, uses another blend of autoencoders with GANs. Their key idea is to learn adversarially an encoder g_x : χ→ M together with a decoder g_m : M →χ against a discriminator c : χ× M →ℝ in a way that the distributions given by the couples (g_m(m),m) and (x,g_x(x)) are indistinguishable from the discriminator point of view. Note that one does not explicitly train g_x and g_m to be inverses to each other. However theorem 2 of ( <cit.> ) shows that at optimality they are indeed.

 Adversarial Latent Autoencoders These autoencoders <cit.> have the property of learning the latent distribution to match the encoded distribution. This is very different from other traditional approaches that assume an a priori target latent distribution and learn the encoder to match the encoded distribution with this a priori target. From this perspective, this method is very close to the architecture we use in our work, though the objectives are very different : their point is to disentangle representations in order to be able to control the features, whereas our objective is to do transfer. The similarity in architecture probably explains why ALAE autoencoders are very suited to our method.

Fine-Tuning
On the side of papers addressing transfer for GANs, we are aware of <cit.>, <cit.>. Both apply to GANs fine-tuning, one of the techniques of transfer learning. It consists in initializing the training of a network on a target dataset 𝒟' with weights from another network with the same architecture, but already trained on a similar source dataset 𝒟. The two papers seem to have been written independently. While <cit.> is mainly targeting a specific application of de-noising in medical imagery, <cit.> is rather interested in understanding fine-tuning for GANs per se. Both report a faster convergence and a better quality, though <cit.> also observes that fine-tuning enables training with smaller datasets and that the distance between the source and target datasets influences the quality of the training.



§ MIND TO MIND ALGORITHM


We now adapt  to GANs the cut-and-paste procedure described in <ref>. The difference is that in addition to the classifier c (or critic in the language of WGAN), one also has a generator g. Let us consider a factorisation of the form  g=g_1∘ g_0 [column sep=1.5em]
    M' drg_1    

Zurg_0 rrg        χ.



 


Our algorithm assumes that g_1 comes from an autoencoder (c_1,g_1) that has been trained on a source dataset 𝒟. The algorithm passes the second data set 𝒟' through the encoder c_1 and trains a MindGAN (g_0',c'_0) on the encoded data c_1(𝒟'). One obtains the final generator as the composition g_1∘ g'_0 of g_1, the decoder of the autoencoder, with g'_0, the generator of the MindGAN. We denote by L_c and L_g the losses of the discriminator and the generator.


 In the remainder of the paper, we use for L_g and L_c  the losses of a WGAN with gradient penalty <cit.> :       L_g := 𝔼_z∼ℙ_Z  c'_0(g'_0(z)),  L_c := - 𝔼_m∼c_1_♯ℙ_𝒟' c'_0(m) +  L_g +λ𝔼_m∼c_1_♯ℙ_𝒟', z∼ℙ_Z, α∼(0,1){ (‖∇  c'_0(α m +(1-α)g'_0(z))‖_2-1)^2}.


 This algorithm can be applied (with minor modifications) to conditional GANs. We refer to Appendix <ref> for more details.


Motivation for the approach  The architectures of a generator and a critic of a GAN are symmetric to one another. The high-level features appear in g_0, the closest to the prior vector (resp. c_0, the closest to the prediction), while the low-level features are in g_1, the closest to the generated sample (resp c_1, the closest to the input image). Therefore, the analogy with cut and paste is to keep g_1 (the low level features of 𝒟) and only learn the high level features g_0' of the target data set 𝒟'. However, the only way a generator can access to information from 𝒟' is through the critic c via the value of c∘ g=c_0'∘ c_1 ∘ g_1 ∘ g_0'  <cit.>. Hence, the information needs to back-propagate through c_1∘ g_1 to reach the weights of g_0'. Our main idea is to bypass this computational burden and
train directly g_0' and c_0' on c_1(𝒟'). But this requires that the source of c_0' coincides with the target of g_0'. Therefore, we assume that M=M', a first hint that autoencoders are relevant for us. 

A second hint comes from an analogy with humans learning a task, like playing tennis, for instance. One can model a player as a function Zg→χ, where χ is the space of physical actions of the player. His/her coach can be understood as a function χc→ℝ, giving c(g(z)) as a feedback for an action g(z) of g. The objective of the player can be understood as to be able to generate instances of the distribution 𝒟' on χ corresponding to the “tennis moves". However, in practice, a coach rarely gives his/her feedback as a score. He instead describes what the player has done and should do instead. We can model this description as a vector c_1(g(z)) in M, the mind of c=c_0∘ c_1. In this analogy, c_1 corresponds to the coach analyzing the action, while c_0 corresponds to the coach giving a score based on this analysis. One can also decompose the player itself as g=g_1∘ g_0. Here g_0 corresponds to the player conceiving the set of movements he/she wants to perform and g_1 to the execution of these actions. Therefore, two conditions are needed for the coach to help efficiently his/her student : 


  * they must speak the same language in order to understand one each other,

  * the player must already have a good command of his/her motor system g_1.


 In particular, the first constraint implies that they must share the same feature space, i.e., M=M'. A way to ensure that both constraints are satisfied is to check whether the player can reproduce a task described by the coach, i.e., that 
    g_1(c_1(x))=x

 holds. One recognizes in (<ref>) the expression of an autoencoder. It is important to remark that usually, based on previous learning, a player already has a good motor control and he/she and his/her coach know how to communicate together. In other words g_1 and c_1 satisfy (<ref>) before the training starts. Then the training consists only in learning g_0' and c_0' on the high level feature interpretations of the possible tennis movements, i.e., on c_1(𝒟').
 
 




§ THEORETICAL GUARANTEE FOR CONVERGENCE

 
 

The following theorem (cf. Appendix <ref>) enables a very precise control of the convergence of the generated distribution towards the true target distribution. It gives an upper bound on the convergence error er_conv in terms of the domain shift er_shift, the autoencoder quality er_AE and the quality of the mindGAN er_mind.
 
There exist two positive constants a and b such that

    er_conv≤   a · er_shift+ er_AE +b · er_mind.




To be more precise, with the notations er_conv=W(ℙ_𝒟',ℙ_θ'), er_shift=W(ℙ_𝒟, ℙ_𝒟'), er_AE=W(AE(ℙ_𝒟),ℙ_𝒟) and  er_mind=W(c_1_♯ℙ_𝒟',g'_0_♯ℙ_Z). 

Very concretely, theorem <ref> tells us that in order to control the convergence of the transferred GAN towards the distribution of the target dataset 𝒟', we need the exact analogues of steps 1-3 of <ref> : 

  * choose two datasets 𝒟' and 𝒟 very similar, i.e., W(ℙ_𝒟, ℙ_𝒟') small,

  * choose a good autoencoder (c_1,g_1), i.e., W(ℙ_𝒟,AE(ℙ_𝒟)) small,

  * train well the MindGAN (g_0,c_0) on c_1(𝒟'), i.e., W(c_1_♯ℙ_𝒟',ℙ'^0_θ) small.


 The main theorem of <cit.>, theorem 3.1, guarantees the convergence of a Wasserstein autoencoder (WAE). We show in Appendix <ref> that it is a  direct consequence of our theorem.




§ EVALUATION
 

Datasets. We have tested our algorithm at resolution 28× 28 in grey levels (scaled in range [-1;1]) on  MNIST <cit.>, KMNIST <cit.>, FashionMNIST <cit.>
(60 000 images each) and at resolution 1024×  1024 in color on CelabaHQ <cit.> (30 000 images) from models trained on the 60 000 first images of FFHQ <cit.> (which consists of 70 000 images), using the library Pytorch. The hardware for our experiments with 28× 28 images consisted of a desktop with 16 Go of RAM and a GPU Nvidia GTX 1080 TI. Most of our experiments with HD color images used a node-gpu <cit.> with two CPU Intel Cascade Lake 6248 and a GPU Nvidia V100 SXM2 32 Go. We have also benchmarked the running time on entry level GPU GTX 1060. Despite its limitations <cit.>, we have used FID (Frechet Inception Distance) <cit.> as metric. It is the current standard for evaluations of GANs.   Our code is available at <cit.>.

 At resolution 28×28.  The encoder c_1 has three convolutional layers with instance normalisation (in) and relu : 32+in+relu, 64+in+relu, 128+in+relu, followed by a single dense layer 256+tanh. The decoder g_1 has a single dense layer 4*4*64 + relu and three deconvolutional layers with batch normalisation (bn) : 64+bn+relu, 64+bn+relu, 32+bn+relu, 1+tanh. The MindGAN is a MLP WGAN whose generator g_0 consists in three dense layers : 512+bn+relu, 512+bn+relu, 256+tanh and the critic c_0 consists also in three dense layers :  256+relu, 256+relu,1. Our hyper-parameters :  learning rate of 10^-3,  batch size of 128 for all the networks, 80 epochs for (g_1,c_1) and 100 for the other networks, gradient penalty with λ=10, beta parameters in Adam optimizer (.9,.9) for (g_1,c_1), (.1,.5) for the other networks.


 We report the results with 𝒟'=MNIST for c_1 trained on each dataset (see  Appendix <ref> for other 𝒟'). We compare our results to a Vanilla WGAN with architecture (g_1∘ g_0, c_0∘ c_1), so that the number of parameters agrees, for fair comparison. 
 

Baseline 1 Vanilla WGAN with gradient penalty trained on MNIST. We have used for the Vanilla WGAN a model similar to the one used in  <cit.>. However, this model would not converge properly, due to a problem of "magnitude race." We have therefore added an ϵ-term <cit.>, <cit.> to ensure its convergence. Our results appear in the first graph on the l.h.s of figure <ref>,  with time in seconds in abscissa. One can observe extremely fast convergence of the mindGAN to good scores, in comparison with the Vanilla WGAN. Note that we have not smoothed any of the curves. This observation suggests that our approach, beyond the gain in training time, provides a regularising mechanism. The stability of the training confirms this hypothesis. Indeed, statistics over 10 runs demonstrate a very small standard deviation, as shown in figure <ref> in the supplementary material.  In particular, this regularization enabled us to use a much bigger learning rate (10^-3 instead of 10^-4), adding to the speed of convergence. In terms of epochs, the MindGAN and the Vanilla WGAN learn similarly (cf Appendix <ref>).


Baseline 2 Fine tuning studied in <cit.>. We have trained a Vanilla WGAN with gradient penalty on 𝒟=KMNIST, the dataset the closest to 𝒟'=MNIST. We have then fine-tuned it on 𝒟', i.e., trained a new network on 𝒟', initialized with the weights of this previously trained Vanilla WGAN. 
We display it on the  r.h.s of figure <ref> 
  
  

  
    
     
   

  under the name Vanilla init Kmnist, together with our best result, namely a Mind2Mind transfer on 𝒟'=MNIST from 𝒟=KMNIST. One can observe that the Mind2Mind approach achieves significantly better performances in FID. 
  
   The bottom of figure <ref> displays samples of images produced by a MindGAN trained on MNIST images encoded using a KMNIST autoencoder.
 


At resolution 1024×1024. We have worked with the encoder and decoder of the ALAE model <cit.> pre-trained on FFHQ available at <cit.>. The generator of our MindGAN has an input dimension of 128,  three hidden dense layers with relu activation (128, 256, 512) followed by a dense output layer with 512 units (no activation). The critic has an input dimension of 512, three hidden dense layers with relu activation (512, 256, 128) followed by a dense output layer with one unit (no activation). Its hyperparameters were lr = 1e^-3, betas = [0., 0.5], gradient penalty = 10, epsilon penalty = 1e^-2, batch size = 256, critic iteration = 5, epochs = 300.


We have encoded the dataset CelebaHQ <cit.> and then trained a mindGAN on a V100  for 300 epochs at 16.95 s/epoch during 1h24m.  On a GTX 1060 the same training takes 30.67 s/epoch. We have reached (over 5 runs) an average FID of 15.18 with an average standard deviation of 0.8. This is a better result than the FID score (19.21) of an ALAE directly trained from scratch (see table 5  from <cit.>).  Samples are displayed on figure <ref>. Compared to the results reported on the ProGAN and StyleGAN official repositories <cit.>, <cit.>, our training (1.5 hour) on 1  GPU V100 is roughly 224 times faster than the training of a proGAN (2 weeks) and 656 times faster than the training of a StyleGAN (41 days). The training of a MindGAN on a GTX 1060 is about 112 times faster than the training  of a proGAN and 328 times faster than the training of a StyleGAN, both on a V100.  Note that a GTX 1060 costs around 200 $ while a V100 is around 8000 $. One has to mention however that, on CelebaHQ, the FID of a ProGAN is  8.03 (see table 5  from <cit.>), while the FID of a StyleGAN is 4.40 (see table 4 of <cit.>), so both are significantly better than ours. We see two factors that can explain the acceleration of the training. The first one is that there are much less parameters to train. Indeed, our mindGAN in HD has around 870K parameters, while the ALAE model (based on a StyleGAN architecture) has 51M parameters. So this already represents a difference of almost two orders of magnitude. One can suspect that the rest of the difference comes from the fact that we bypass 18 layers in the computation of the backpropagation. We believe that the validation of this hypothesis deserves a careful experimental study.




§ COMPARISON TO OTHER WORKS


Wasserstein autoencoders. WAEs do not provide a solution to the question we address here. Indeed, <cit.> do not consider at all transfer learning and work only with a single data set 𝒟 at a time. Its goal is rather to give a new approach to Variational Auto Encoders based on the use of the Wasserstein distance.

Adversarial learned inference. The works on adversarial leaned inference offer an alternative way to train autoencoders. They do not address transfer learning, however, it is possible to choose these types of auto-encoders as building block for our transfer method. We haven't yet conducted experiments with such auto-encoders. In particular, we do not know if theoretical results similar to theorem <ref> can be obtained in this setting.

 Adversarial Latent Autoencoders The architecture of this method is very close to the architecture we use in our work, though the objectives are very different : their point is to disentangle representations in order to be able to control the features, whereas our objective is to do transfer. The similarity in architecture probably explains why ALAE are very suited to our method. We did not have enough time to test wether the disentanglement properties of ALAE are preserved via transfer. We plan to investigate further this question.



Fine-Tuning. Our approach is different but can be, in theory, combined with fine-tuning. Indeed, one can initialise the training of our MindGAN (g_0,c_0) of algorithm <ref> on c_1(𝒟') with another MindGAN trained on c_1(𝒟). We have tried this on the MNISTs datasets; see figure <ref>, but no significant improvement has been observed, We have compared fine-tuning against Mind2Mind transfer for WGANs and report better results in terms of FID. Note that our theorem <ref>  gives a theoretical justification, in our setting, of the observation of <cit.> of the influence of the domain shift (distance between 𝒟 and 𝒟') on the convergence of the learning.



§ DISADVANTAGES


Our first limitation is that we do not learn the transferred layers during the transfer. On the one hand, it is a feature as it enables faster learning. On the other hand, it is possible that at the asymptote, i.e., after the training has converged, the transferred mindGAN offers a worse quality than a Vanilla WGAN. We have not observed this phenomenon with the ALAE architecture. To the contrary, we obtained a better  asymptote since we got a FID of 15.18 for the transferred model  compared to the FID of 19.21 of the ALAE model trained from scratch on CelebaHQ. Maybe this is due to the fact that we have focussed the training on a more significant part of the network (the MindGAN). 

The second limitation of our approach is that it does not provide improvement of the training on limited data. However, such a training was not the objective of our work as we believe that acceleration of the training is a useful goal on itself. 

Our results, compared to state of the art baselines, show a worse performance in terms of quality (FID of 15.18 compared to 8.03 for ProGAN and 4.40 for StyleGAN). However, this should not be an obstacle to the use of this algorithm. Indeed, the targeted users are practitioners without significant computing capacity. Their need is to reach a reasonable quality in a short time. With this regard, in terms of wall clock time, this algorithm learns roughly 224 times faster than a ProGAN and 656 times faster than a StyelGAN on CelebaHQ.  So merits of our algorithm will depend on the tradeoff between the needs  of the users in terms of quality and their constraints in terms of computing capacity.

 The bottleneck in our approach is the lack of a zoo of models of autoencoders trained on diverse datasets in high resolution. We hope that our method, in conjunction with the ALAE architecture, will lead the main players in the industry to train such models and make them accessible to the community.
 



§ CONCLUSION


We introduced a method that enables transfer learning for GANs. Given an autoencoder trained on a source dataset, one passes the target dataset through the encoder and uses the encoded features to train a GAN, called a MindGAN, in the latent space of the autoencoder. Composing the MindGAN with the decoder provides the transferred GAN, a generator for the target dataset. We provided theoretical results that guaranty that the transferred GAN converges to the target data. We have demonstrated that our method enables  to train GANs much faster (between 6 and 656 times faster, depending on the size of the network) than state of the art methods, in both 28× 28 gray scale and 1024× 1024 HQ color datasets.





§ BROADER IMPACT


The main impact of our work will be a democratization of the use of GANs. Indeed, without the barrier of computing time/costs, GANs for high quality images will become accessible to a pool of practitioners much broader than the researchers/engineers of big industry and academic labs. As a corrollary, one can expect a development of research in this field proportional to the increase of number of researchers who will gain access to these tools.  

Affordable computing time will offer the possibility to customize models for specific needs and datasets. This can lead to new applications in movie industry, virtual assistants or even telecommunications (it may be more efficient to learn a model of a person, send it and use it to reconstruct a video signal than to transmit the signal itself) to name a few. But this can also lead to malicious uses, in particular for the generation fake profiles and deep fakes that can be used for phishing or disinformation.









§ PROOF OF THINGS



We first recall notions that will be needed. We then give the proof of our main theoretical result.



 §.§ Wasserstein distance and Lipschitz functions


In the following, all the metric spaces considered will be subsets of normed vector spaces, with the metric on the subset induced by the norm.

First, one recalls some definitions (more details can be found in <cit.>).
[transference plan]
    Let (X, ℙ_X) and (Y,ℙ_Y) be two probability spaces. A transference plan γ is a measure on X × Y such that :
    
    ∫_A× Ydγ = ℙ_X(A),

    and,
    
    ∫_X× Bdγ = ℙ_Y(B).

    ℙ_X and ℙ_Y are called the marginals of γ. The set of transference plans with marginals ℙ_X and ℙ_Y is denoted by Π(ℙ_X,ℙ_Y).


[p-Wasserstein distance]
    Let (X, ‖ . ‖) be a metric space and p ∈ [1,+∞). For two probability measure ℙ_1, ℙ_2 on X, the p-Wasserstein distance between ℙ_1 and ℙ_2 is defined by the following
    
    W_p(ℙ_1,ℙ_2)  = ( inf_γ∈Π(ℙ_1,ℙ_2)𝔼_(x,y)∼γ‖ x -  y‖^p   )^1/p.



    
In this paper, we used the notation W(ℙ_1,ℙ_2) instead of W_1(ℙ_1,ℙ_2).



[Lipschitz function]
    Let ϕ : X → Y be a map between metric spaces X and Y. It is called a C-Lipschitz function if there exists a constant C such that :
    
    ∀x and y∈ X, ‖ϕ(x) - ϕ(y) ‖_Y ≤ C ‖ x - y ‖_X.




Let ϕ : X → Y be a locally Lipschitz map, with X compact, then there exists a constant C such that

    W_Y(ϕ_♯μ,ϕ_♯ν)≤  C W_X(μ,ν).




Let γ be a transference plan realising W_X(μ,ν). Define γ ':=(ϕ×ϕ)_♯γ. One can check that γ ' defines a transference plan between ϕ_♯μ and ϕ_♯ν.  Therefore, one has the following relation

    W_Y(ϕ_♯μ,ϕ_♯ν)    ≤   ∫‖  x-y ‖ dγ'(x,y)
        =    ∫‖ϕ(x)-ϕ(y) ‖ dγ(x,y)
       ≤   ∫ C ‖ x-y ‖ dγ(x,y)
        =     C W_X(μ,ν),

where the first inequality comes from the fact that  γ' is a transference plan, the first equality from the definition of the push forward of a measure by a map (recalled in section <ref>), the last inequality from lemma   <ref>, and the last equality from the choice of γ.






    Let ϕ : X → Y be a locally Lipschitz map, and X a compact metric space. Then there exists C such that ϕ is a C-Lipschitz function.


    By definition of a locally Lipschitz map, for all x in X, there exists U_x a neighbourhood of x and a constant C_x such that ϕ is C_x-Lipschitz on U_x. 

    So ⋃_x ∈ X U_x is a cover of X. Since X is compact, there exists a finite set I such that ⋃_i ∈ I U_i is a cover of X. 

    One can check that ϕ is C-Lipschitz on X, with C := max_i ∈ I(C_x_i).





 §.§ Proof of theorem <ref>


We can finally turn to the proof of theorem <ref> :

 

There exist two positive constants a and b such that

    W(ℙ_𝒟',ℙ_θ')  ≤    a     W(ℙ_𝒟, ℙ_𝒟')+ W(ℙ_𝒟,AE(ℙ_𝒟)) 
    
                                                        +     b     W(c_1_♯ℙ_𝒟',ℙ'^0_θ).







From the triangle inequality property of the Wasserstein metric and the definition of ℙ_θ', one has :

    W(ℙ_𝒟',ℙ_θ') ≤  W(ℙ_𝒟',AE(ℙ_𝒟'))+W(AE(ℙ_𝒟'),g_1_♯ℙ'^0_θ).

 One concludes with lemma <ref>  and lemma <ref> with ϕ=g_1.



There exist a positive constant a such that

    W(ℙ_𝒟',AE(ℙ_𝒟'))  ≤ a W(ℙ_𝒟, ℙ_𝒟')+ W(ℙ_𝒟,AE(ℙ_𝒟)).
 



Applying twice the triangle inequality, one has :

    W(ℙ_𝒟',AE(ℙ_𝒟'))    ≤   W(ℙ_𝒟',ℙ_𝒟) +W(ℙ_𝒟,AE(ℙ_𝒟))
        +     W(AE(ℙ_𝒟),AE(ℙ_𝒟')).

One concludes with lemma <ref> with ϕ=g_1∘ c_1.




It is important to remark that in order to be able to apply lemma <ref> in the proof of theorem <ref>,  one needs the assumption that ℙ'^0_θ and c_1_♯ℙ_𝒟' have compact support. But as χ is itself compact, this is not a problem for c_1_♯ℙ_𝒟' since the image of a compact χ by a continuous function c_1 is compact. However,  the compacity of the support of ℙ'^0_θ is not a priori granted. An easy fix is to choose a prior ℙ_Z with compact support. Therefore, we choose this setting in our applications. 





Our proof of theorem <ref> implicitly assumed that neural networks are locally Lipschitz maps (see lemmata  <ref> and <ref>). This assumption is justified by the following lemma.




    Let g : Z → X be a neural network and ℙ_Z a prior over Z such that 𝔼_z ∼ℙ_Z(‖ z ‖ )< ∞ (such as Gaussian) then g is locally Lipschitz and 𝔼_z ∼ℙ_Z(L_z)< ∞, where L_z are the local Lipschitz constants. 




 See Corollary 1. of <cit.>





 §.§ Application to Wasserstein autoencoders.
 The main theorem of <cit.>, theorem 3.1, guarantees the convergence of a Wasserstein autoencoder (WAE). We recall this theorem and show that it is a direct consequence of our theorem <ref>.

 

    W(ℙ_𝒟,g_1_♯ℙ_𝒵) ≤    W(ℙ_𝒟, AE(ℙ_𝒟)) 
    
                                                                +     bW(c_1_♯ℙ_𝒟,ℙ_𝒵).




Since WAEs do not involve transfer, one has 𝒟=𝒟', i.e. ℙ_𝒟=ℙ_𝒟'.
Then it suffices to replace ℙ'^0_θ by ℙ_𝒵 and ℙ_θ' becomes g_1_♯ℙ_𝒵.



Our proof of theorem <ref> is very similar to the proof of theorem <ref> given in <cit.>. Therefore, our contribution here consists rather in finding a versatile statement that applies to both problems (transfer and WAE) than in the originality of the tools used in the proofs.



 When one restricts our approach to the case when 𝒟=𝒟', it does not coincide with WAE. Indeed, with the notations of our paper, WAE work with a fixed prior ℙ_M on M that one tries to approximate by c_1_♯ℙ_𝒟, while constraining c_1 to be a right inverse (in measure) of g_1, and g_1_♯ℙ_M to approximate (in measure) ℙ_𝒟. On the other hand, our approach involves an extra auxiliary latent space Z. Therefore we can consider g_0_♯ℙ_Z as a replacement of  ℙ_M. Via the flexibility of the learnable weights of g_0, we use g_0_♯ℙ_Z to approximate c_1_♯ℙ_𝒟, instead of using c_1_♯ℙ_𝒟 to approximate ℙ_M as in <cit.>. This is fundamental, because in a setting where 𝒟≠𝒟', this decoupling permits to train c_1 and g_1 on 𝒟 and c_0 and g_0 on c_1(𝒟'), enabling us to do transfer.





  



§ MIND2MIND CONDITIONAL GANS
 

As suggested to us by L. Cetinsoy, the Mind2Mind approach also applies to conditional GANs. However, one needs to implement the following modifications : replace M by M× L and Z by Z× L in the diagram

 
    [baseline=(current  bounding  box.center)]
        M drg_1       M drc_0
    
    Zurg_0rrg      χurc_1rrc      ℝ,

where L stands for the space of conditions, in order to get

.45

 
    [column sep=1.5em]
        M× L drg_1×𝕀_L       M× L drc^c_0
    
    Z× Lurg^c_0rrg^c      χ× Lurc_1×𝕀_Lrrc^c      ℝ.



Here, (g^c_0,c^c_0) and (g^c,c^c) are conditional GANs, with the generators of the form g^c_0(z,l)=(m(z,l),l) and g^c(z,l)=(x(z,l),l). The autoencoder (c_1×𝕀_L,g_1×𝕀_L) can be trivially deduced from an autoencoder (c_1,g_1) via the formulas c_1×𝕀_L(x,l):=(c_1(x),l) and  g_1×𝕀_L(m,l):=(c_1(m),l).

In practice, the algorithm <ref> becomes a classical conditional GAN algorithm :





 







§ SUPPLEMENTARY EXPERIMENTS


 In figure <ref> we display additional samples from a MindGAN on CelebaHQ transferred from FFHQ.  We also display in figure  <ref> the mean and standard deviation over 10 runs of the training of a MindGAN in 28× 28.   The source dataset in figure <ref> is 𝒟'= FashionMNIST, while in figure <ref>, 𝒟'= MNIST. For comparison, we display in figure <ref> samples from a vanilla WGAN. 







 

 

 








