Adversarial Pixel-Level Generation of Semantic Images
Emanuele Ghelfi Paolo Galeone Michele De Simoni Federico Di Mattia


[
	Adversarial Pixel-Level Generation of Semantic Images



	equal*

	
		Emanuele Ghelfito
		Paolo Galeoneto
		Michele De Simonito
		Federico Di Mattiato
	

	toZuru Tech, Modena, Italy

	Emanuele Ghelfiemanuele@zuru.tech
	

	Machine Learning, ICML, Generative Adversarial Network, Semantic

	0.3in
]



  








	Generative Adversarial Networks (GANs) have obtained extraordinary success in the generation of realistic images, a domain where a lower pixel-level accuracy is acceptable. We study the problem, not yet tackled in the literature, of generating semantic images starting from a prior distribution. Intuitively this problem can be approached using standard methods and architectures. However, a better-suited approach is needed to avoid generating blurry, hallucinated and thus unusable images since tasks like semantic segmentation require pixel-level exactness. In this work, we present a novel architecture for learning to generate pixel-level accurate semantic images, namely Semantic Generative Adversarial Networks (SemGANs). The experimental evaluation shows that our architecture outperforms standard ones from both a quantitative and a qualitative point of view in many semantic image generation tasks.






§ INTRODUCTION


Generative Adversarial Networks <cit.> (GANs) are generative methods used for estimating the relationship between simple latent distributions to complex distributions. Recent advances over the original GANs formulation involve the architecture <cit.>, the training technique <cit.>, or the applications  <cit.>.

In this paper, we address the problem, not yet tackled, of generating semantic images starting from a latent representation. In this context, a semantic image is an image in which every pixel is assigned to a label indicating its class, thus assuming only values in a discrete and finite set. We propose a new architecture able to generate semantic images with higher precision than a standard architecture.

While the image-to-image approach <cit.> has proven to be effective in this domain, we argue that in this case, it is not possible to generate completely new images starting from an arbitrary latent representation, as an input image is always required. This limitation makes an image-to-image approach less flexible w.r.t. our approach since it cannot leverage the power of a learned latent, low-dimensional, representation. Moreover, the presented approach is also faster not requiring an encoder-like network embedded in the generator.

The contributions of this paper are the followings:

	
  * we present a new set of applications for the GAN framework: the generation of semantic images from latent vectors;
	
  * we propose, to the best of our knowledge, the first GAN architecture that generates semantic images directly, without the need of any post-process step;
    
  * the experimental evaluation of the new architecture on several semantic datasets; showing that our approach leads to improved image quality when compared to the standard GAN architecture.



This paper is organized as follows.
In <ref> we present two works related to our approach. In <ref> we first introduce the main concepts underlying the GAN framework, then we present the main contribution of this paper: the Semantic Generative Adversarial Network architecture. The evaluation metrics used in our tests are described in <ref>. <ref> contains the experimental evaluation of SemGAN. In <ref> we describe an interesting application of this approach. Finally, <ref> contains the conclusions and future research directions.



§ RELATED WORK









  
GAN-based Semantic Segmentation The generation of semantic masks can be related to <cit.>.  Semantic-segmentation using GAN is an adversarial approach to the problem of semantic-segmentation, that is detecting for each pixel the class it belongs to. In <cit.> the generator outputs a semantic map starting from a realistic image. The generator output map is then multiplied channel-wise by the input image before being fed into the discriminator to avoid the simple detection of distribution inconsistencies between the ground truth annotations (one-hot encoded) and the generator output. However, we highlight the fact that this approach does not allow to generate semantic images starting from a latent vector, like in unconditional generative models, therefore this model is not suitable for the generation of new artificial semantic environments. 



  
Image to Image translation
In <cit.> an adversarial approach to Image to Image translation is presented (pix2pix).
The <cit.> approach has been applied to translate semantic images to realistic images and vice versa. When used to produce semantic labels, the output must be post-processed since the generator is not able to generate labels (colors) that perfectly match the space of semantic labels. Like in <cit.>, the pix2pix model does not allow to generate entirely new semantic images, always requiring a source domain.



§ PIXEL-LEVEL GENERATIVE ADVERSARIAL NETWORKS




 §.§ Generative Adversarial Networks


Generative Adversarial Networks are based on a game-theoretic formulation in which there are two main components, namely the generator and  the discriminator, competing against each other in a minimax game. Given the data distribution p_data(·) over variables 𝐱, the goal of the generator G is to learn a mapping from a prior distribution p_g(·) over latent variables 𝐳 to p_data(·). On the other side, the discriminator D must learn to determine whether a sample comes from the data distribution or the model distribution. We denote with θ_G the generator parameters and with θ_D the discriminator parameters.
Both G and D are trained simultaneously considering the two player minimax game:

    min_G max_D V(D,G) =    𝔼_𝐱∼ p_data(·)[ log(D(𝐱) ] +
       𝔼_𝐳∼ p_g(·)[ log( 1 - D(G(𝐳)) ) ].

The goal of GAN training is to find a Nash equilibrium, that is a point (θ^*_G, θ^*_D) in the parameter space in which neither the generator nor the discriminator can improve their cost function unilaterally. Training GANs is well known for being unstable and prone to divergence <cit.>. Feature matching loss <cit.> is known to reduce the instability of GAN training. Instead of updating G parameters based on the output of the discriminator, it updates G based on the internal representation of D, e.g., the features of an intermediate layer. 





 §.§ Semantic Generative Adversarial Networks





We propose a new technique for generating semantic labels at a pixel-wise level based on Generative Adversarial Networks. The SemGAN generator outputs a tensor with shape (W,H,K), where K is the number of classes defined in the dataset; we denote with the name class channel the last dimension of the generator output.
The last layer of the generator is a softmax over the class channel applied in order to interpret every pixel vector as a probability distribution over the classes.
The SemGAN discriminator input has the same shape as the output of the generator, and it corresponds to the one hot notation of the semantic map in the case of real images and to the softmax distribution over classes in the case of generated images. The SemGAN architecture is illustrated in <ref> and in <ref>. Our architecture is a more principled way to generate semantic map rather than treating semantic maps as normal images.

The generator tends to generate distributions with very low variance since it tries to replicate the original images, that are one hot encoded. Using this approach, we are sure to generate only valid labels; moreover, we help the generator in its task because of the restricted subset to learn. Thus, the generator function G can be formalized as:

    G: 𝒳→Δ(L)^W × H,


where 𝒳 is the latent space and with Δ(L) we denote the probability distribution induced by the softmax operation over the set of labels L:

    L = {l_1, l_2, …, l_K}.


A prediction can be collapsed into a segmentation map by taking the  of each depth-wise pixel vector and after mapping the labels in L to their corresponding colors obtaining an RGB image naturally follows.

This architecture is very general, and there is the possibility to combine it with all defined losses as well as all training strategies available.
We highlight three important advantages of our architecture:

    
  * the generator always outputs a valid class label;
    
  * the gradient signal arriving at the generator is applied to the predicted class labels, not to the generated colors;
    
  * the limited generator co-domain speeds up the convergence: we empirically demonstrated to achieve better qualitative and quantitative results with fewer training steps w.r.t. the standard GAN architecture.





§ EVALUATION METRICS






Assessing image quality is always a challenging task in all GAN-based approaches. We evaluate our architecture using three metrics based on the assumption that the final goal of the generator is to generate images visually similar at every scale to the images contained in the training set. 


  
Sliced Wasserstein Distance The first metric we use in our evaluation is the Sliced Wasserstein Similarity (SWD), used in <cit.>. Following this approach we build a Laplacian pyramid downsampling the considered image, we extract patches of 7 × 7, with three color channels. A small SWD of a certain pyramid level indicates that the generated images are structurally similar to the training set at this level.


  
Fréchet Inception Distance The second metric we use is the Fréchet Inception Distance (FID) <cit.>. This metric is based on the Fréchet Distance between features extracted on a particular layer of the GoogLeNet <cit.>. Even if this net has been trained on real images, a visual inspection shows that the particular features we are extracting might be also interesting for semantic images. As for the SWD, a small FID indicates that features of the generated images are similar to the features of the training images. 


  
MultiScale Structural Similarity The third metric is the MultiScale Structural Similarity <cit.> (MS-SIM). This metric detects the similarity among generated images, and this is useful in order to detect mode collapse, a typical GAN problem. The downside of this metric is that it does not take into account the similarity between the generator output and the training set.



§ EXPERIMENTAL EVALUATION






In this section, we show the experimental evaluation of our architecture on a toy dataset of colored shapes (<ref>) and two standard semantic datasets, namely Cityscapes (<ref>) and Facades (<ref>). We compare our results with the results obtained with a traditional architecture working on RGB images. We highlight the fact that we use the same parameters, the same loss, and the same architecture both for the semantic GAN and for the standard GAN. The unique difference is the output layer, that in the case of the Semantic GAN is a volume with dimension W × H × K representing a probability distribution over labels for each pixel vector, while in the case of the GAN is a standard RGB image, thus with dimensions W × H × 3.



  §.§.§ Semantic Generation of Colored Shapes


The Colored Shapes dataset takes inspiration from <cit.>. It contains 2916 images 64×64×3 each. Each image contains a 10×10 px red square in a different location (54^2 available positions), a blue circle with radius 10 randomly placed in the image and a green rectangle in a random position. The three shapes can overlap, and the circle can completely cover the square. Thus, the classes in this dataset are 4: background, circle, rectangle, square.

We evaluate the image quality visually (<ref>) and using the SWD metric (<ref>). The semantic GAN architecture outperforms the GAN architecture, especially for the SWD 64 × 64 metric. When observing the generated images,   artifacts can be spotted in the GAN output, while the proposed Semantic GAN output is very precise.




  §.§.§ Semantic Generation of Street Scenes





The Cityscapes <cit.> dataset contains a diverse set of stereo video sequences recorded in street scenes from 50 different cities. We resized the images to 128×128, and we used the 22 main classes. The comparison between SemGAN and GAN for the SWD metric is reported in <ref>. In the lowest level of the pyramid the Sliced Wasserstein distance of the SemGAN is comparable to the one of the standard GAN, while in the highest, where the image resolution is greater, the gap increases notably. Considering the mean SWD, the SemGAN model outperforms the GAN model. <ref> depicts the generated images for each architecture. From a visual point of view, the generated images using the semantic architecture are precise, without artifacts of any sort. The generated images using the standard architecture have colors not related to any label. We also compare in <ref> the FID, the MS-SIM and the SWD of the image generated by our architecture with the images generated using the standard GAN architecture. Further details on the experiments are reported in <ref>.











  §.§.§ Semantic Generation of Facades


The Facades dataset <cit.> includes 606 rectified images of facades from different cities around the world and diverse architectural styles; it contains 12 classes. We resized the images to 128x128 in order to perform this experiment.
In <ref> we report the metrics used for evaluation. See <ref> for a visual comparison of the generated images for the best models for each architecture.
From a qualitative point of view, the generated images using SemGAN and GAN are not very impressive, this might relate to the fact that the Facades dataset has a smaller number of images with higher variance than the Cityscape dataset. We can see from the generated samples that our architecture is not able to generate precise geometric forms, unlike in the Colored Shapes experiment. Samples from the GAN are very blurry and the class distinction is not neat. Further details and experiments are reported in <ref>.









§ APPLICATION




 §.§ Dreaming realistic Street Scenes

The Semantic-GAN architecture can be combined with pix2pix in order to generate new, unseen street scenes; this is obtained by feeding the output of our generator into the pix2pix generator, after having applied the RGB mapping and scaling to match the input size. Results of the pix2pix model applied to the SemGAN and GAN outputs are depicted in <ref>.
The street scenes dreamed by the semantic GAN are much more realistic w.r.t. the street scenes dreamed by the standard GAN.
It is worth noting that the pix2pix model we used for generating images is trained on compressed RGB images, not on semantic maps. A pix2pix model trained on semantic maps can benefit from this representation.






§ CONCLUSIONS


In this paper, we presented Semantic-GAN, a new GAN architecture able to generate semantic images starting from a latent representation. The empirical evaluation suggests that using this type of architecture can be beneficial when working in semantic domains, where a pixel level accuracy is needed. Our approach outperforms the standard GAN architecture in the generation of images with pixels value in a finite domain since each pixel is modeled as a value sampled from a probability distribution induced over a finite set of discrete values.
While requiring to change only the generator output and the discriminator input, the Semantic-GAN approach can be applied to many different applications such as the learning of a semantic encoder able to map semantic maps to their latent representations. This idea paves the way for new applications like the generation of realistic worlds via latent space interpolation combining our method with a style transfer solution.



icml2019










§ FURTHER RESULTS




 §.§ Cityscapes



In this appendix we show further results for the cityscapes experiment, we provide a full comparison of SemGAN and GAN for each type of architecture we tested. For all experiments we use a batch size equal to 32.
The main parameters we focus on in our experiments are the learning rate (lr), the kernel size (ks), the latent space dimension (ld). Results are summarized in <ref>. Sample images are reported in <ref>.




 §.§ Facades



In this appendix we show further results for the facades experiment, we provide a full comparison of SemGAN and GAN for each type of architecture we tested. For all experiments we use a batch size equal to 32.
The main parameters we focused on in our experiments are the learning rate (lr), the kernel size (ks), the latent space dimension (ld). Results are summarized in <ref>. Sample images are reported in <ref>.










§ TRAINING DETAILS

For each experiment we train the generator with feature matching loss and the discriminator with the binary cross entropy loss. The SWD and the FID are measured over the whole set of images in the training set. We run each training for 5 × 10^4 steps.
Each component of the latent vector comes from the distribution 𝒩(0,1).
We use the same learning rate both for the generator and the discriminator using the Adam optimizer, with β_1 = 0.5, β_2=0.999 and ϵ=10^-8.


  
Colored shapes experiment
For the colored shapes experiment we use a batch size equal to 32, lr=10^-4, ks=11, ld=100.





