Controllable Person Image Synthesis with Attribute-Decomposed GAN
Yifang Men Yiming Mao Yuning Jiang Wei-Ying Ma Zhouhui Lian
cs.CV
















Controllable Person Image Synthesis with Attribute-Decomposed GAN
    Yifang Men^1, Yiming Mao^2, Yuning Jiang^2, Wei-Ying Ma^2, Zhouhui Lian^1Corresponding author. E-mail: lianzhouhui@pku.edu.cn

^1Wangxuan Institute of Computer Technology, Peking University, China

^2Bytedance AI Lab

    December 30, 2023. .../latex/6he-sdm-2/6he-sdm-2-3.tex
=============================================================================================================================================================================================================================

empty



This paper introduces the Attribute-Decomposed GAN, a novel generative model for controllable person image synthesis, which can produce realistic person images with desired human attributes (e.g., pose, head, upper clothes and pants) provided in various source inputs. The core idea of the proposed model is to embed human attributes into the latent space as independent codes and thus achieve flexible and continuous control of attributes via mixing and interpolation operations in explicit style representations. Specifically, a new architecture consisting of two encoding pathways with style block connections is proposed to decompose the original hard mapping into multiple more accessible subtasks. In source pathway, we further extract component layouts with an off-the-shelf human parser and feed them into a shared global texture encoder for decomposed latent codes. This strategy allows for the synthesis of more realistic output images and automatic separation of un-annotated attributes. Experimental results demonstrate the proposed method's superiority over the state of the art in pose transfer and its effectiveness in the brand-new task of component attribute transfer.







§ INTRODUCTION





Person image synthesis (PIS), a challenging problem in areas of Computer Vision and Computer Graphics, has huge potential applications for image editing, movie making, person re-identification (Re-ID), virtual clothes try-on and so on. An essential task of this topic is pose-guided image generation <cit.>, rendering the photo-realistic images of people in arbitrary poses, which has become a new hot topic in the community. Actually, not only poses but also many other valuable human attributes can be used to guide the synthesis process.

In this paper, we propose a brand-new task that aims at synthesizing person images with controllable human attributes, including pose and component attributes such as head, upper clothes and pants. As depicted in Figure <ref>, users are allowed to input multiple source person images to provide desired human attributes respectively. The proposed model embeds component attributes into the latent space to construct the style code and encodes the keypoints-based 2D skeleton extracted from the person image as the pose code, which enables intuitive component-specific (pose) control of the synthesis by freely editing the style (pose) code. Thus, our method can automatically synthesize high-quality person images in desired component attributes under arbitrary poses and can be widely applied in not only pose transfer and Re-ID, but also garment transfer and attribute-specific data augmentation (e.g., clothes commodity retrieval and recognition).

Due to the insufficiency of annotation for human attributes, the simplicity of keypoint representation and the diversity of person appearances, it is challenging to achieve the goal mentioned above using existing methods. Pose transfer methods firstly proposed by <cit.> and later extended by <cit.> mainly focus on pose-guided person image synthesis and they do not provide user control of human attributes such as head, pants and upper clothes. Moreover, because of the non-rigid nature of human body, it is difficult to directly transform the spatially misaligned body-parts via convolution neural networks and thus these methods are unable to produce satisfactory results. Appearance transfer methods <cit.> allow users to transfer clothes from one person to another by estimating a complicated 3D human mesh and warping the textures to fit for the body topology. Yet, these methods fail to model the intricate interplay of the inherent shape and appearance, and lead to unrealistic results with deformed textures. Another type of appearance transfer methods <cit.> try to model clothing textures by feeding the entire source person image into neural networks, but they cannot transfer human attributes from multiple source person images and lack the capability of component-level clothing editing.

The notion of attribute editing is commonly used in the field of facial attribute manipulation <cit.>, but to the best of our knowledge this work is the first to achieve attribute editing in the task of person image synthesis. Different from pervious facial attribute editing methods which require strict attribute annotation (e.g., smiling, beard and eyeglasses exist or not in the training dataset), the proposed method does not need any annotation of component attributes and enables automatic and unsupervised attribute separation via delicately-designed modules. In another aspect, our model is trained with only a partial observation of the person and needs to infer the unobserved body parts to synthesize images in different poses and views. It is more challenging than motion imitation methods <cit.>, which utilize all characters performing a series of same motions to disentangle the appearance and pose, or train one model for each character by learning a mapping from 2D pose to one specific domain.


To address the aforementioned challenges, we propose a novel controllable person image synthesis method via an Attribute-Decomposed GAN. In contrast to previous works <cit.> forcedly learn a mapping from concatenated conditions to the target image, we introduce a new architecture of generator with two independent pathways, one for pose encoding and the other for decomposed component encoding. For the latter, our model first separates component attributes automatically from the source person image via its semantic layouts which are extracted with a pretrained human parser. Component layouts are fed into a global texture encoder with multi-branch embeddings and their latent codes are recombined in a specific order to construct the style code. Then the cascaded style blocks, acting as a connection of two pathways, inject the component attributes represented by the style code into the pose code by controlling the affine transform parameters of AdaIN layer. Eventually, the desired image can be reconstructed from target features. In summary, our contributions are threefold:




  * We propose a brand-new task that synthesizes person images with controllable human attributes by directly providing different source person images, and solve it by modeling the intricate interplay of the inherent pose and component-level attributes.

  * We introduce the Attribute-Decomposed GAN, a neat and effective model achieving not only flexible and continuous user control of human attributes, but also a significant quality boost for the original PIS task.

  * We tackle the challenge of insufficient annotation for human attributes by utilizing an off-the-shelf human parser to extract component layouts, making an automatic separation of component attributes.









§ RELATED WORK





 §.§ Image Synthesis

Due to their remarkable results, Generative Adversarial Networks (GANs) <cit.> have become powerful generative models for image synthesis <cit.> in the last few years. The image-to-image translation task was solved with conditional GANs <cit.> in Pix2pix <cit.> and extended to high-resolution level in Pix2pixHD <cit.>. Zhu et al. <cit.> introduced an unsupervised method, CycleGAN, exploiting cycle consistency to generate the image from two domains with unlabeled images. Much of the work focused on improving the quality of GAN-synthesized images by stacked architectures <cit.>, more interpretable latent representations <cit.> or self-attention mechanism <cit.>. StyleGAN  <cit.> synthesized impressive images by proposing a brand-new generator architecture which controls generator via the adaptive instance normalization (AdaIN) <cit.>, the outcome of style transfer literature <cit.>. However, these techniques have limited scalability in handling attributed-guided person synthesis, due to complex appearances and simple poses with only several keypoints. Our method built on GANs overcomes these challenges by a novel generator architecture designed with attribute decomposition.




 §.§ Person Image Synthesis

Up to now, many techniques have been proposed to synthesize person images in arbitrary poses using adversarial learning. PG2 <cit.> firstly proposed a two-stage GAN architecture to generate person images, in which the person with the target pose is coarsely synthesized in the first stage, and then refined in the second stage. Esser et al. <cit.> leveraged a variational autoencoder combined with the conditional U-Net <cit.>to model the inherent shape and appearance. Siarohin et al. <cit.> used a U-Net based generator with deformable skip connections to alleviate the pixel-to-pixel misalignments caused by pose differences. A later work by Zhu et al. <cit.> introduced cascaded Pose-Attentional Transfer Blocks into generator to guide the deformable transfer process progressively. <cit.> utilized a bidirectional strategy for synthesizing person images in an unsupervised manner. However, these methods only focused on transferring the pose of target image to the reference person and our method achieved a controllable person image synthesis with not only pose guided, but also component attributes (e.g., head, upper clothes and pants) controlled. Moreover, more realistic person images with textural coherence and identical consistency can be produced.






§ METHOD DESCRIPTION


Our goal is to synthesize high-quality person images with user-controlled human attributes, such as pose, head, upper clothes and pants. Different from previous attribute editing methods <cit.> requiring labeled data with binary annotation for each attribute, our model achieves automatic and unsupervised separation of component attributes by introducing a well-designed generator. Thus, we only need the dataset that contains person images {I∈ R^3× H× W} with each person in several poses. The corresponding keypoint-based pose P∈ R^18× H× W of I, 18 channel heat map that encodes the locations of 18 joints of a human body, can be automatically extracted via an existing pose estimation method <cit.>. During training, a target pose P_t and a source person image I_s are fed into the generator and a synthesized image I_g following the appearance of I_s but under the pose P_t will be challenged for realness by the discriminators. In the following, we will give a detailed description for each part of our model.




 §.§ Generator

Figure <ref> shows the architecture of our generator, whose inputs are the target pose P_t and source person image I_s, and the output is the generated image I_g with source person I_s in the target pose P_t. Unlike the generator in  <cit.> which directly concatenates the source image and target condition together as input to a U-Net architecture and forcedly learns a result under the supervision of the target image I_t, our generator embeds the target pose P_t and source person I_s into two latent codes via two independent pathways, called pose encoding and decomposed component encoding, respectively. These two pathways are connected by a series of style blocks, which inject the texture style of source person into the pose feature. Finally, the desired person image I_g is reconstructed from target features by a decoder.





  §.§.§ Pose encoding



In the pose pathway, the target pose P_t is embedded into the latent space as the pose code C_pose by a pose encoder, which consists of N down-sampling convolutional layers (N=2 in our case), following the regular configuration of encoder.









  §.§.§ Decomposed component encoding

In the source pathway, the source person image I_s is embedded into the latent space as the style code C_sty via a module called decomposed component encoding (DCE). As depicted in Figure <ref>, this module first extracts the semantic map S of source person I_s with an existing human parser <cit.> and converts S into a K-channel heat map M ∈ R^K× H× W. For each channel i, there is a binary mask M_i ∈ R^H× W for the corresponding component (e.g., upper clothes). The decomposed person image with component i is computed by multiplying the source person image with the component mask M_i




    I_s^i = I_s ⊙ M_i,

where ⊙ denotes element-wise product. I_s^i is then fed into the texture encoder T_enc to acquire the corresponding style code C_sty^i in each branch by




    C_sty^i = T_enc(I_s^i),

where the texture encode T_enc is shared for all branches and its detailed architecture will be described below. Then all C_sty^i, i=1…K will be concatenated together in a top-down manner to get the full style code C_sty.


In contrast to the common solution that directly encodes the entire source person image, this intuitive DCE module decomposes the source person into multiple components and recombines their latent codes to construct the full style code. Such an intuitive strategy kills two birds with one stone: 1) It speeds up the convergence of model and achieves more realistic results in less time. Due to the complex structure of the manifold that is constituted of various person images with different clothes and poses, it is hard to encode the entire person with detailed textures, but much simpler to only learn the features of one component of the person. Also, different components can share the same network parameters for color encoding and thus DCE implicitly provides a data augmentation for texture learning. The loss curves for the effects of our DCE module in training are shown in Figure <ref> and the visualization effects are provided in Figure <ref> (d)(e). 2) It achieves an automatic and unsupervised attribute separation without any annotation in the training dataset, which utilizes an existing human parser for spatial decomposition. Specific attributes are learned in the fixed positions of the style code. Thus we can easily control component attributes by mixing desired component codes extracted from different source persons.







For the texture encoder, inspired by a style transfer method <cit.> which directly extracts the image code via a pretrained VGG network to improve the generalization ability of texture encoding, we introduce an architecture of global texture encoding by concatenating the VGG features in corresponding layers to our original encoder, as shown in Figure <ref>. The values of parameters in the original encoder are learnable while those in the VGG encoder are fixed. Since the fixed VGG network is pretrained on the COCO dataset <cit.> and it has seen many images with various textures, it has a global property and strong generalization ability for in-the-wild textures. But unlike the typical style transfer task <cit.> requiring only a roughly reasonable result without tight constraints, our model needs to output the explicitly specified result for a given source person in the target pose. It is difficult for the network with a fixed encoder to fit such a complex model and thus the learnable encoder is introduced, combined with the fixed one. The effects of the global texture encoding (GTE) are shown in Figure <ref> (c)(d).







  §.§.§ 	Texture style transfer 

Texture style transfer aims to inject the texture pattern of source person into the feature of target pose, acting as a connection of the pose code and style code in two pathways. This transfer network consists of several cascaded style blocks, each one of which is constructed by a fusion module and residual conv-blocks equipped with AdaIN. For the t^th style block, its inputs are deep features F_t-1 at the output of the previous block and the style code C_sty. The output of this block can be computed by






    F_t =φ_t(F_t-1, A) + F_t-1,

where F_t-1 firstly goes through conv-blocks φ_t, whose output is added back to F_t-1 to get the output F_t, F_0 = C_pose in the first block and 8 style blocks are adopted totally. A denotes learned affine transform parameters (scale μ and shift σ) required in the AdaIN layer and can be used to normalize the features into the desired style <cit.>. Those parameters are extracted from the style code C_sty via a fusion module (FM), which is an important auxiliary module for DCE. Because component codes are concatenated in a specified order to construct the style code, making a high correlation between the position and component features, this imposes much human ruled intervention and leads to a conflict with the learning tendency of the network itself. Thus we introduce FM consisting of 3 fully connected layers with the first two allowing the network to flexibly select the desired features via linear recombination and the last one providing parameters in the required dimensionality. FM can effectively disentangle features and avoid conflicts between forward operation and backward feedback. The effects of FM are shown in Figure <ref>. When DCE is applied to our model without FM, the result (see Figure <ref> (d)) is even worse than that without DCE (see Figure <ref> (c)). The fusion module makes our model more flexible and guarantees the proper performance of DCE.






  §.§.§ 	Person image reconstruction

With the final target features F_T-1 at the output of the last style block, the decoder generates the final image I_g from F_T-1 via N deconvolutional layers, following the regular decoder configuration.



 §.§ Discriminators

Following Zhu et al. <cit.>, we adapt two discriminators D_p and D_t, where D_p is used to guarantee the alignment of the pose of generated image I_g with the target pose P_t and D_t is used to ensure the similarity of the appearance texture of I_g with the source person I_s. For D_p, the target pose P_t concatenated with the generated image I_g (real target image I_t) is fed into D_p as a fake (real) pair. For D_t, the source person image I_s concatenated with I_g (I_t) is fed into D_t as a fake (real) pair. Both D_p and D_t are implemented as PatchGAN and more details can be found in  <cit.>.






 §.§ Training

Our full training loss is comprised of an adversarial term, a reconstruction term, a perceptual term and a contextual term




    ℒ_total = ℒ_adv+λ_recℒ_rec+λ_perℒ_per+λ_CXℒ_CX,

where λ_rec, λ_per and λ_CX denote the weights of corresponding losses, respectively.

 Adversarial loss.
We employ an adversarial loss ℒ_adv with discriminators D_p and D_t to help the generator G synthesize the target person image with visual textures similar to the reference one, as well as following the target pose. It penalizes for the distance between the distribution of real pairs (I_s(P_t), I_t) and the distribution of fake pairs (I_s(P_t), I_g) containing generated images


    L_adv=   𝔼_I_s,P_t,I_t[log(D_t(I_s,I_t) · D_p(P_t,I_t)) ]+ 
       𝔼_I_s,P_t[log((1-D_t(I_s,G(I_s,P_t)))
       ·(1-D_p(P_t,G(I_s,P_t))))].


 Reconstruction loss.
The reconstruction loss is used to directly guide the visual appearance of the generated image similar to that of the target image I_t, which can avoid obvious color distortions and accelerate the convergence process to acquire satisfactory results. ℒ_rec is formulated as the L1 distance between the generated image and target image I_t


    L_rec = || G(I_s,P_t)-I_t ||_1.


 Perceptual loss.
Except for low-level constraints in the RGB space, we also exploit deep features extracted from certain layers of the pretrained VGG network for texture matching, which has been proven to be effective in image synthesis <cit.> tasks. The preceputal loss is computed as <cit.>

    ℒ_per = 1/W_lH_lC_l∑_x=1^W_l∑_y=1^H_l∑_z=1^C_l∥ϕ_l(I_g)_x,y,z-ϕ_l(I_t)_x,y,z∥ _1,

where ϕ_l is the output feature from layer l of VGG19 network, and W_l, H_l, C_l are spatial width, height and depth of feature ϕ_l.






 Contextual loss.
The contextual loss proposed in  <cit.> is designed to measure the similarity between two non-aligned images for image transformation, which is also effective in our GAN-based person image synthesis task.
Compared with the pixel-level loss requiring pixel-to-pixel alignment, the contextual loss allows spatial deformations with respect to the target, getting less texture distortion and more reasonable outputs. We compute the contextual loss L_CX by


    ℒ_CX = -log(CX(ℱ^l(I_g), ℱ^l(I_t))),

where ℱ^l(I_g) and ℱ^l(I_t) denote the feature maps extracted from layer l=relu{3_2,4_2} of the pretrained VGG19 network for images I_g and I_t, respectively, and CX denotes the similarity metric between matched features, considering both the semantic meaning of pixels and the context of the entire image. More details can be found in  <cit.>. We show the effects of ℒ_CX in Figure <ref>, which enables the network to better preserve details with less distortion.

 Implementation details. 
Our method is implemented in PyTorch using two NVIDIA Tesla-V100 GPUs with 16GB memory. With the human parser  <cit.>, we acquire the semantic map of person image and merge original labels defined in  <cit.> into K (K=8) categories (i.e., background, hair, face, upper clothes, pants, skirt, arm and leg). The weights for the loss terms are set to λ_rec = 2, λ_per = 2, and λ_CX = 0.02. We adopt Adam optimizer  <cit.> with the momentum set to 0.5 to train our model for around 120k iterations. The initial learning rate is set to 0.001 and linearly decayed to 0 after 60k iterations. Following this configuration, we alternatively train the generator and two discriminators.






§ EXPERIMENTAL RESULTS

In this section, we verify the effectiveness of the proposed network for attributes-guided person image synthesis tasks (pose transfer and component attribute transfer), and illustrate its superiority over other state-of-the-art methods. Detailed results are shown in the following subsections and more are available in the supplemental materials (Supp).

 Dataset. 
We conduct experiments on the In-shop Clothes Retrieval Benchmark DeepFashion <cit.>, which contains a large number of person images with various appearances and poses. There are totally 52,712 images with the resolution of 256×256. Following the same data configuration in pose transfer <cit.>, we randomly picked 101,966 pairs of images for training and 8,750 pairs for testing.

 Evaluation Metrics. 
Inception Score (IS) <cit.> and Structural Similarity (SSIM) <cit.> are two most commonly-used evaluation metrics in the person image synthesis task, which were firstly used in PG2 <cit.>. Later, Siarohin et al. <cit.> introduced Detection Score (DS) to measure whether the person can be detected in the image. However, IS and DS only rely on an output image to judge the quality in itself and ignore its consistency with conditional images. Here, we introduce a new metric called contextual (CX) score, which is proposed for image transformation <cit.> and uses the cosine distance between deep features to measure the similarity of two non-aligned images, ignoring the spatial position of the features.
CX is able to explicitly assess the texture coherence between two images and it is suitable for our task to measure the appearance consistency between the generated image and source image (target image), recording as CX-GS (CX-GT). Except for these computed metrics, we also perform the user study to assess the realness of synthesized images by human.



 §.§ Pose transfer



  §.§.§ Person image synthesis in arbitrary poses


Pose is one of the most essential human attributes and our experiments verify the effectiveness of our model in pose-controlled person image synthesis. Given the same source person image and several poses extracted from person images in the test set, our model can generate natural and realistic results even when the target poses are drastically different from the source in scale, viewpoints, etc. We show some results of our method in Figure <ref> and more are available in Supp.






  §.§.§ Comparison with state-of-the-art methods

For pose transfer, we evaluate our proposed method with both qualitative and quantitative comparisons.

 Qualitative comparison. In Figure <ref>, we compare the synthesis results of our method with four state-of-the-art pose transfer methods: PG2 <cit.> , DPIG <cit.>, Def-GAN <cit.> and PATN <cit.>. All the results of these methods are obtained by directly using the source codes and trained models released by authors. As we can see, our method produced more realistic results in both global structures and detailed textures. The facial identity is better preserved and even detailed muscles and clothing wrinkles are successfully synthesized. More results can be found in Supp.

 Quantitative  comparison. In Table <ref>, we show the quantitative comparison with abundant metrics described before. Since the data split information in experiments of  <cit.> is not given, we download their pre-trained models and evaluate their performance on our test set. Although it is inevitable that testing images may be contained in their training samples, our method still outperforms them in most metrics. The results show that our method generates not only more realistic details with the highest IS value, but also more similar and natural textures with respect to the source image and target image, respectively (lowest CX-GS and CX-GT values). Furthermore, our method has the highest confidence for person detection with the best DS value. For SSIM, we observe that when the value of IS increases, this metric slightly decreases, meaning the sharper images may have lower SSIM, which also has been observed in other methods <cit.>.











 User study.
We conduct a user study to assess the realness and faithfulness of the generated images and compare the performance of our method with four pose transfer techniques. For the realness, participants are asked to judge whether a given image is real or fake within a second. Following the protocol of  <cit.>, we randomly selected 55 real images and 55 generated images, first 10 of which are used for warming up and the remaining 100 images are used for evaluation. For the faithfulness, participants are shown a source image and 5 transferred outputs, and they are asked to select the most natural and reasonable image with respect to the source person image. We show 30 comparisons to each participant and finally 40 responses are collected per experiment. The results in Table <ref> further validate that our generated images are more realistic, natural and faithful. It is worth noting that there is a significant quality boost of synthesis results obtained by our approach compared with other methods, where over 70% of our results are selected as the most realistic one.




 §.§ Component Attribute Transfer

Our method also achieves controllable person image synthesis with user-specific component attributes, which can be provided by multiple source person images. For example, given 3 source person images with different component attributes, we can automatically synthesize the target image with the basic appearance of person 1, the upper clothes of person 2 and the pants of person 3. This also provides a powerful tool for editing component-level human attributes, such as pants to dress, T-shirt to waistcoat, and head of man to woman.


By encoding the source person images to decomposed component codes and recombining their codes to construct the full style code, our method can synthesize the target image with desired attributes. In Figure <ref>, we edit the upper clothes or pants of target images by using additional source person images to provide desired attributes. Our method generates natural images with new attributes introduced harmoniously while preserving the textures of remaining components.











 Style Interpolation. 
Using our Attribute-Decomposed GAN, we can travel along the manifold of all component attributes of the person in a given image, thus synthesizing an animation from one attribute to another. Take for example the codes of upper clothes from person1 and person2 (C_uc1 and C_uc2), we define their mixing result as


    C_mix = β C_uc1+ (1-β) C_uc2,

where β∈ (0,1) and β decreases from 1 to 0 in specific steps. Results of style interpolation are available in Supp.






















 §.§ Failure cases

Although impressive results can be obtained by our method in most cases, it fails to synthesize images with pose and component attributes that extremely bias the manifold built upon the training data. The model constructs a complex manifold that is constituted of various pose and component attributes of person images, and we can travel along the manifold from one attribute to another. Thus, valid synthesis results are actually the mixtures of seen ones via the interpolation operation. As shown in Figure <ref>, the specific carton pattern in T-shirt of a woman fails to be interpolated with seen ones and the person in a rare pose cannot be synthesized seamlessly.




§ CONCLUSION

In this paper, we presented a novel Attribute-Decomposed GAN for controllable person image synthesis, which allows flexible and continuous control of human attributes. Our method introduces a new generator architecture which embeds the source person image into the latent space as a series of decomposed component codes and recombines these codes in a specific order to construct the full style code. Experimental results demonstrated that this decomposition strategy enables not only more realistic images for output but also flexible user control of component attributes. We also believed that our solution using the off-the-shelf human parser to automatically separate component attributes from the entire person image could inspire future researches with insufficient data annotation. Furthermore, our method is not only well suited to generate person images but also can be potentially adapted to other image synthesis tasks.



§ ACKNOWLEDGEMENTS


This work was supported by National Natural Science Foundation of China (Grant No.: 61672043 and 61672056), Beijing Nova Program of Science and Technology (Grant No.: Z191100001119077), Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology).



ieee_fullname



