title: Attribute-guided image generation from layout
authors: Ke Ma Bo Zhao Leonid Sigal
text: 
  Core-structure and lattice resistance of twinning dislocations in fcc metals
      Sri Sadgun R. Pulagam and Amlan Dutta 
      December 30, 2023
  ============================================================================
  
  
  
  
      Recent approaches have achieved great successes in image generation from structured inputs, e.g., semantic segmentation, scene graph or layout. Although 
      these methods allow specification of objects and their locations at image-level, they lack the fidelity and semantic control to specify visual appearance of these objects at an  instance-level.
      To address this limitation, 
      we propose a new image generation method that enables instance-level attribute control. 
      Specifically, the input to our attribute-guided generative model is a tuple that contains: (1) object bounding boxes, (2) object categories and (3) an (optional) set of attributes for each object. The output is a generated image where the requested objects are in the desired locations and have prescribed attributes.
      Several losses work collaboratively to encourage accurate, consistent and diverse image generation.
      Experiments on Visual Genome <cit.> dataset demonstrate our model's capacity to control object-level attributes in generated images, and validate plausibility of 
      disentangled object-attribute representation in the image generation from layout task. Also, the generated images from our model have higher resolution, object classification accuracy and consistency, as compared to the previous state-of-the-art. 
  
  
  
  
  
  § INTRODUCTION
  
  
  Controlled image generation methods have achieved great successes in recent years, driven by the advances in conditional Generative Adversarial Networks (GANs) <cit.> and disentangled representations <cit.>. The goal of these methods is to generate high-fidelity images from various user specified guidelines (conditions), such as textual descriptions <cit.>, attributes <cit.>,  scene graphs <cit.>, layout <cit.> and semantic segmentation <cit.>.  
  The high-level nature of most of these specifications is desirable from ease of use and control point of views, but severely impoverished in terms of pixel-level spatial and appearance information, leading to a challenging image generation problem. 
  
  In this paper, we specifically focus on image generation from layout, where the input is a course spatial layout of the scene (, bounding boxes and corresponding object class labels) and the output is an image consistent with the specified layout. Compared to text-to-image <cit.> and scene-graph-to-image <cit.> generation paradigms, layout-to-image provides an easy, spatially aware and interactive abstraction for specifying desired content. This makes this paradigm compelling and effective for users across the spectrum of potential artistic skill sets; from children and amateurs to designers. 
  Image generation from a layout is a relatively new problem, with earlier methods using layout only as an intermediate representation <cit.>, but not a core abstraction or specification exposed to the user.  
  
  
  
  
  Layout2Im <cit.> was the first model proposed for image generation from layout, followed by more recent LostGAN <cit.>, which improved on the performance in terms of overall image 
  quality. However, all current image generation from layout frameworks <cit.> are limited in a couple of fundamental ways. 
  First, they lack ability to semantically control individual object instances. While both Layout2Im and LostGAN model distributions over appearances of objects through appearance <cit.> or style <cit.> latent codes, neither is able to control these variations semantically.
  
  One can imagine using encoded sample patches depicting desired objects as an implicit control mechanism (i.e., generate an instance of a tree or sky that resembles an example in a given image patch),
  
  however, this is in the very least awkward and time consuming from the user perspective.
  Second, they generally lack consistency – are not spatially equivariant. Intuitively, shifting a location (bounding box) of an object in the layout specification, while keeping appearance/style latent code fixed, should result in the object simply shifting by the relatively same amount in the output image (property known as equivariance).
  
  However, current models fail to achieve this 
  intuitive consistency. 
  Finally, they are limited to low-resolution output images, typically of size 64×64. 
  
  In this paper, we address these challenges by proposing a new framework for attribute-guided image generation from layout, building on, and substantially extending, the backbone of <cit.>. In particular, we show that a series of simple and intuitive architectural changes: incorporating (optional) attribute information, adopting a global context encoder, and adding additional image generation path where object locations are shifted – leads to the instance-level fine-grained control over the generation of objects, while increasing the image quality and resolution. We call this model attribute-guided layout2im (see Figure <ref>). 
  
  
  
  Contributions: Our contributions are three fold: (1) our attribute-guided layout2im architecture allows (but does not require) direct instance-level granular semantic attribute control over individual objects; (2) is directly optimized to be consistent, or equivariant, with respect to spatial shifts of object bounding boxes in the input layout; and (3) allows for higher-resolution output images of size up to 128×128 by utilizing global context encoder and progressive upsampling. Both qualitatively and quantitatively we show state-of-the-art generative  performance on Visual Genome <cit.> benchmark dataset, while benefiting from the desirable control properties, unavailable in other models. The code and pretrained models will be made available [<https://github.com/UBC-Computer-Vision-Group/attribute-guided-image-generation-from-layout>].
  
  
  
  
  § RELATED WORK
  
  
  Image Generation from Scene Graph:
  Scene graph is a convenient directed graph structure designed to represent a scene,  where nodes correspond to objects and edges to the relationships between objects. Recently, scene graphs have been used in many image generation tasks due to their flexibility in explicitly encoding object-object relationships and interactions <cit.>. The typical generation process involves two phases: (1) a graph convolutional neural network <cit.> is applied to the scene graph to predict the scene layout (i.e., bounding boxes and segmentation masks); and (2) the layout is then decoded into an image. Different from methods that generate image as a whole, Li  <cit.> propose a semi-parametric approach and crop refining network to reconcile the isolated crops into an integrated image. 
  
  Unlike in our approach, the scene layout in these models is used as an  intermediate semantic representation to bridge abstract scene graph input and an image output. 
  
  
  
  
  
  
  Image Generation from Layout:
  Image layout, comprising bounding box locations, often serves as an intermediate step for image generation (see above). Zhao  <cit.> proposed image generation from layout as a task in its own right, where the image is generated from bounding boxes and corresponding object categories. To combine multiple objects,  <cit.> sequentially fuse object feature maps using a convolutional LSTM (cLSTM) network; the resulting fused feature map is then decoded to an image. Turkoglu  <cit.>, on the other hand, divide the generation into multiple stages where objects are added to the canvas one by one. To better bridge the gap between layouts and images, Li  <cit.> uses a shape generator to outline the object shape and provide the model fine-grained information from text using object-driven attention. Similarly,  <cit.> learns object-level fine-grained mask maps that outline the object shape. In addition,   <cit.> show that incorporating layout information into normalization layer yields better results: adopting instance normalization technique  <cit.> in generator realize multi-object style control  <cit.>, whereas spatially-adaptive normalization  <cit.> based on segmentation mask modulates activation in upsampling layers. Taking the inspiration from  <cit.>, we apply spatially-adaptive normalization in our generator to compute layout-aware affine transformations in normalization layers.
  
  
  
  
  
  Semantic Image Synthesis:
  Semantic image synthesis is an image-to-image translation task. While most methods use conditional adversarial training <cit.>, such as pix2pix <cit.>, pix2pixHD <cit.>, cVAE-GAN and cLR-GAN <cit.>, others such as Cascaded Refinement Networks <cit.> also yields plausible results. To preserve semantic information, normalization techniques like SPADE  <cit.> have recently been deployed to modulate the activations in normalization layers through a spatially adaptive and learned transformation. Semantic image synthesis can also serve as an intermediate step for image modeling <cit.>. In addition, some image-to-image translation tasks are achieved in unsupervised manner <cit.>, but most existing semantic image synthesis models require paired data. 
  
  
  
  
  Attribute-guided Image Generation:
  In image generation,
  various attempts have been made to specify the attributes of generated images and objects.
  For example, <cit.> aim to edit the attributes of the generated image using natural language descriptions. In <cit.> authors embed a visual attribute vector (, blonde hair) for attribute-guided face generation. 
  Li  <cit.> also incorporates object-level appearance information in the input, but it relies on external memory to sample objects. Another line of the literature concentrates on editing images by providing reference images (, <cit.>) to transfer style.
  
  
  Different from prior approaches, our model allows direct attribute control over  individual instances of objects in complex generative scenes.
  
  
  
  § APPROACH
  
  Let us start by formally defining the problem. 
  
  
  
  Let Λ be an image canvas (, of size 128×128) and let L = { (ℓ_i, 𝐚𝐭𝐭𝐫_i, 𝐛𝐛𝐨𝐱_i)^m_i=1} be a layout consisting of m labeled object instances with class labels ℓ_i ∈𝒞, attributes 𝐚𝐭𝐭𝐫_i = {a_ij}_j=1^n_i, 
  
  and bounding boxes defined by top-left and  bottom-right coordinates on the canvas, 𝐛𝐛𝐨𝐱_i  ⊆Λ, where |𝒞| is the total number of object categories and a_ij∈𝒜 is the j-th attribute of i-th object instance; 𝒜 is the attribute set. Note that each object might have more than one attribute. Let 𝐳_obj_i be the latent code for object instance (ℓ_i, 𝐚𝐭𝐭𝐫_i, 𝐛𝐛𝐨𝐱_i), modeling class- and attribute-unconstrained appearance.  We denote the set of all object instance latent codes in the layout L by Z_obj = {𝐳_obj_i}^m_i=1.
  
  Attribute-guided image generation from a layout can then be formulated as learning a generator function G which maps given input (L, Z_obj) to the output image I conforming to specifications:
  
  
  
      I = G(L, Z_obj; Θ_G),
  
  
  where Θ_G parameterizes the generator function G and corresponds to the set of  parameters that need to be learned. 
  
  Different from previous layout to image generation methods is explicit, but optional (𝐚𝐭𝐭_i can be ∅ or sampled from the prior), inclusion of the attributes. Further, we specifically aim to learn G which is, at least to some extent, equivariant with respect to location of objects 𝐛𝐛𝐨𝐱_i in the layout.
  
  
  Our attribute-guided layout2im formulation builds on and improves <cit.>, as such it shares some of the basic architecture design principles outlined in Zhao  <cit.>.  
  
  
  
  
  
  Training:
  The overall training pipeline of the proposed approach is illustrated  in  Figure <ref>. Given the input image I and its layout L = { (ℓ_i, 𝐚𝐭𝐭𝐫_i, 𝐛𝐛𝐨𝐱_i)^m_i=1}, our model first creates a word  embedding 𝐰_i for each object label ℓ_i and multi-hot attribute embedding 𝐞_i ∈{0,1}^|𝒜| for object attribute(s)[Note exactly n_i elements of 𝐞_i will be 1 and the rest are 0.] 𝐚𝐭𝐭𝐫_i, and form a  joint object-attribute embedding M(𝐰_i ⊕𝐞_i) where ⊕ is the vector concatenation and M is a MLP layer, composed of three fully connected layers that map the concatenated vector to a lower dimensional vector.  A set of object latent codes Z_obj = {𝐳_obj_i}^m_i=1 are sampled from the standard prior normal distribution 𝒩(0, 1), and a new L^shift = { (ℓ_i, 𝐚𝐭𝐭𝐫_i, 𝐛𝐛𝐨𝐱^shift_i)^m_i=1} is constructed, where 𝐛𝐛𝐨𝐱^shift_i represents bounding boxes that are randomly shifted in the canvas Λ. The shifts  are horizontal to maintain consistency. Then, our model estimates another set of latent codes Z^r_obj = {𝐳^r_obj_i}^m_i=1, where 𝐳^r_obj_i is sampled from the posterior Q(𝐳^r_obj_i | O_i) conditioned on CNN features of object O_i cropped from input image I. 
  We effectively end up with three datasets: 
  
     
    Set 1: (L, Z^r_obj) for reconstruction of the original image. 
     Mapping this input through generator G
     should result in an 
     
     image I^rec, which is a reconstruction of the training image I serving as the source of the layout L; 
     
  
     
    Set 2: (L, Z_obj) for generation of a new image sharing the original layout. The output of G here should be image 
     I^rand that shares the layout with above, but where appearance of each object instance is novel (sampled from the prior).
     
     
  
    
    Set 3: (L^shift, Z_obj) which is used to generate an image from reconfigured layout (i.e., reconfiguration path, see Suppl. Sec. 1.1 for details). The output should be a corresponding shifted image I^shift, which shares latent appearance codes with those from Set 2. 
  
   The same pipeline is applied to all three input sets simultaneously: multiple object feature maps F_i are constructed based on the layout L_i and (𝐳_obj_i⊕ M(𝐰_i ⊕𝐞_i)), and then fed into the object encoder and the objects fuser sequentially, generating a fused hidden feature map H containing information from all specified objects. Lastly, we incorporate a global context embedding 𝐠 onto H to form a context-aware feature canvas H^g, and decode it back to an image with a combination of deconvolution, upsampling and SPADE normalization layers <cit.>. For generated object O_i in I^rand and I^shift, we make the object estimator regress the sampled latent codes 𝐳_obj_i based on O_i to encourage O_i to be consistent with the latent code 𝐳_obj_i, and use an auxiliary object classifier and attribute classifier to ensure O_i has the desired category and attributes. To train the model adversarially,  we also introduce a pair of discriminators, D_img and D_obj, to classify results as being real/fake at image and object level. 
  
  
  
  Inference: At inference time, the model is able to synthesize a realistic image from a given (user specified) layout L and object latent codes Z_obj sampled from the prior normal object appearance distribution 𝒩(0, 1). The attributes can be specified by the user or sampled from prior class distributions of  object-attribute co-occurrence counts, which we also estimate from data during training. In this way, attribute can be treated as “optional" at individual instance level; i.e., one can specify sub-set of attributes for any sub-set of instances. 
  
  
  
   §.§ Attribute Encoder
  
  
  Visual Genome <cit.> dataset provides attribute descriptions for some objects. For example, a car object might have attributes red and parked, and a person object might have attributes smile and tall. There are over 40,000 attributes in the datasets. We only keep the most common 106 attributes for simplicity.
  In other words, |𝒜| = 106.
  Each object might have more than one attribute, hence we adopt multi-hot embedding for the attribute encoder. If no attributes are specified for the object, the attribute embedding would be a vector of zeros, i.e., 𝐞_i = 0. We concatenate the multi-hot embedding with object word embedding and pass it through an MLP layer to obtain the final object-attribute embedding M(𝐰_i ⊕𝐞_i), which is then concatenated with (prior sampled) latent code 𝐳_obj_i to construct the object feature map F_i.
  The feature map F_i is therefore constructed by filling 𝐛𝐛𝐨𝐱_i of the feature canvas with M(𝐰_i ⊕𝐞_i) ⊕𝐳_obj_i. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  Attribute Disentanglement:
  For two novel image generation paths, mainly (L, Z_obj) and (L^shift, Z_obj) we further entice the model to  use attribute embedding 𝐞_i to determine appearance of corresponding objects.
  
  To explicitly disentangle attribute information from 𝐳_obj_i and 𝐰_i during training, we randomly choose a subset of training objects and replace their GT attributes with new attributes sampled from the attributes frequency distribution for the object class. By doing this, we force the generator to use the attributes code 𝐞_i to generate objects with corresponding attributes, instead of encoding attribute information into 𝐳_obj_i and/or 𝐰_i.
  
  
  
  
   
   
  
  
  
  
   §.§ Global Context Encoder
  
  
  At the last stage of the generation process, the fused feature map H is decoded back to the output image by a stack of deconvolution layers. However, the generated image obtained from simply decoding H often contains objects that are not coherent within a scene. For example, it is observed that some generated objects and the background appear incoherent and exhibit sharp transitions (image patch pasting effect). Hence, it is desirable to explicitly incorporate global context information 𝐠 in each receptive field of the feature map H, so that, locally, object generation is more informed. Since H contains the information for all objects, it itself is a natural choice for encoding the global context 𝐠 (Figure <ref>). To encode 𝐠, we feed the 8x8 feature map H through two convolution layers to downsample it to a 2x2 feature map, which is average pooled to a global context vector. We then expand the vector to the size of the fused feature map H. 
  This concatenation, H^g = (𝐠⊕ H), is the final feature map used to decode the image. 
  
  
  
  
   §.§ SPADE normalization
  
  
  Spatially-adaptive (SPADE) normalization <cit.> is an improved normalization technique that prevents semantic information from being washed away by conventional normalization layers. In SPADE normalization, the learnable transformation (i.e., scale and shift) is learned directly from the semantic layouts. In our model, the feature map H resembles the semantic layout because H encodes both spatial and semantic information. Hence, we add SPADE normalization layers between the deconvolution layers in our image decoder where H is used as the semantic layout. As we show later, in the ablation study (Table <ref>), the object accuracy of generated results improves when we adopt SPADE. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Loss Function
  
  The structure of our discriminator D follows the discriminator proposed in layout2im <cit.> (see Appendices A.2 for details), but adds an additional term for attribute classifier cls^att, which predicts the attribute of cropped objects and is used to train the generator to synthesize objects with correct attributes. It is trained on real objects O and their attributes 𝒜.
  
  Our GAN model utilizes two 
  
  
  adversarial losses: Image Adversarial Loss ℒ_adv^img and Object Adversarial Loss ℒ_adv^obj. Five more losses, including KL Loss ℒ_KL, Image Reconstruction Loss ℒ_1^img, Object Latent code Reconstruction ℒ_1^latent, Auxiliar Object Classification Loss ℒ_AC^obj  and Auxiliar Attribute Classification Loss ℒ_AC^att, are added to facilitate the generation of realistic images. Due to lack of space we provide details of the loss terms in Appendices Material (Section A.3). 
  
  
  
  
  As the result, the generator G minimizes:
  
  
  
      ℒ_G= λ_1ℒ_adv^img+ λ_2ℒ_adv^obj+λ_3ℒ_AC^obj + λ_4ℒ_AC^att
      + λ_5ℒ_KL+λ_6ℒ_1^img+λ_7ℒ_1^latent
  
  
  and the discriminator D minimizes:
  
  
  
      ℒ_D= -λ_1ℒ_adv^img -λ_2ℒ_adv^obj+λ_3ℒ_AC^obj + λ_4ℒ_AC^att
  
  where λ_i are weights for different loss terms. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
    
    
  
  
  
  
    
  
  
    
  
  
    
  
  
    
  
  
    
    
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § EXPERIMENTS
  
  
  
  Datasets:
  We evaluate our proposed model on Visual Genome <cit.> datasets. We preprocess and split the dataset following the settings of  <cit.>. In total, we have 62,565 training, 5,506 validation and 5,088 testing images. Each image contains 3 to 30 objects from 178 categories, and each object has 0 to 5 attributes from 106 attribute set. We are unable to train on COCO <cit.> because it does not provide attribute annotations.
  
  
  
  
  
  
  Experimental setup:
  Our experiments use the pre-trained VGG-net <cit.> as the base model to compute the inception score for generated image. For object classification loss and the attribute classification loss,  our experiments adopted the ResNet-50 model <cit.> and replace its last fc layer with the corresponding dimensions. Both object and attribute classifier are trained using the objects cropped from real training images. For attribute accuracy, we estimate the attributes of generated objects using a separately trained attribute classifier which consists of five residual blocks, and compute the recall and precision against the GT attribute annotations. Lastly, we generate two sets of test images and use LPIPS metric <cit.> to compute the diversity score. More specifically, we take the activation of conv1 to conv5 from AlexNet <cit.>,  and normalize them in the  channel  dimension  and  take  the  L2  distance.   We  then  average across  spatial  dimension  and  across  all  layers  to  get  the  LPIPS  metric. 1-LPIPS metric is also used for consistency score, where we compute the foreground diversity between each object before and after it is shifted, and compute the background diversity for the rest of the image which did not undergo the shift. Higher consistency for both is better.  
  
  
  
  
  Baselines:
  We compare our model with Sg2Im <cit.>, Interactive Scene Graph <cit.>, Layout2im <cit.> and LostGAN <cit.>. 
  
  
  
  
  
   §.§ Quantitative Results
  
  
  Table <ref> and <ref> shows the image generation results when trained using different models. For 64 × 64 images, our attribute-guided image generation from layout outperforms all other models in terms of object accuracy, attribute score and consistency score. Notably, our attribute classification score (recall and precision) outperform other models with a substantial margin, demonstrating our model's capability to control the attributes of generated objects. For consistency in layout reconfiguration, our consistency score is the highest for both background and foreground in the generated images, reflecting the effectiveness of the layout reconfiguration path. Note, as expected, specifying attributes limits the diversity of output objects (0.10 ± 0.02). However, sampled from prior class distributions of object-attribute co-occurrence counts leads to much higher diversity of generated images (0.20 ± 0.01).
  
  
  
  
  We also conduct experiments at 128 × 128 resolution and compare with LostGAN <cit.>. 
  Our model obtains better results on the object accuracy, attribute score and consistency score.
  
  
  
   §.§ Qualitative Results
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Figure <ref> demonstrates our model's ability to control the attributes of generated objects. For each image, we pick an object and replace its current attribute with a different one, while keeping the rest of the layout unchanged. It can be seen from Figure <ref> that our model is able to change the attributes of the object of interest, and keep the rest of 
  the image unmodified.
  
  
  Figure <ref> compares the results before and after some object bounding boxes in the canvas are horizontally shifted. For each images pair, the image on the left is generated from the GT layout, and the image on the right from the reconfigured layout. Our model shows better layout reconfigurability than other methods. For example, in Figure <ref>(b') the boat is moved, in (d') two human are moved, and in (e') the horse is moved. In contrast, layout2im <cit.> and LostGan <cit.> either change the theme of the image (see <ref>(f')), or have missing objects (see <ref>(d')) for reconfigured layouts.  This is also reflected in their much lower consistency score. 
  
  Additional examples at 128 × 128 resolution are in Appendices, Figure 6 and 7. 
  Appendices Figure 9 shows generated images obtained using different SoTA models. Our method consistently outperforms baselines in quality and consistency of generated images. LostGan <cit.>  fails to generate plausible human faces, and layout2im <cit.> does not generate realistic objects such as food. 
  
  
  
  
  
  
  
  
   §.§ Ablation Study
  
  
  We demonstrate the necessity of our key components by comparing scores of several ablated models trained on Visual Genome <cit.> dataset.
  
  As shown in Table <ref>, removing any components is detrimental to the model's performance. Not surprisingly, attribute specification is the key to the successful attribute classification. The absence of layout reconfiguration path decreases the inception score by 0.2, slightly increases the classification accuracy and, more importantly, reduces the consistency for reconfigured layouts. SPADE <cit.> is 
  
  beneficial for object classification accuracy, and global context encoder improves inception score by 0.3. 
  
  
  
  
  
  
  
  
  
  § CONCLUSIONS
  
  
  
  
  This paper proposes attribute-guided image generation from layout, an effective approach to control the visual appearance of generated images in instance-level. We also showed that the model ensures visual consistence of generated images when bounding boxes in layout undergo translation. Qualitative and quantitative results on Visual Genome <cit.> datasets demonstrated our model's ability to synthesize images with instance-level attribute
  control and improved level of visual consistence. 
  
  
  
   §.§ Acknowledgments
  
  
  
  
  This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC and the NSERC DG and Discovery Accelerator Grants. Hardware support was provided by JELF CFI grant and Compute Canada under the RAC award.
  
  
  
  
  unsrt  
    
  
  
  
  
  
  
  § APPROACH DETAILS
  
  
  
  
  
   §.§ Layout Reconfiguration
  
  
   In addition to image reconstruction path and image generation path, described in the main paper, layout reconfiguration path is introduced in our model to increase the spatial equivariance of the generator. Here we describe the layout reconfiguration path a little more completely. Similar to image generation path, an object latent code 𝐳_obj_i is sampled from a normal prior distribution 𝒩(0, 1), and is concatenated to the object attribute embedding M(𝐰_i ⊕𝐞_i). When composing the feature map F_i^shift, however, the input bounding boxes are randomly shifted. We limit ourselves to horizontal shifts in order to preserve coherence of the scene and not introduce perspective inconsistencies. Hence, each F_i^shift is composed based on the a new L^shift. Then, the set of F_i^shift feature maps are downsampled and passed to a cLSTM network to form the fused map H^shift, which is then decoded back to an image I^shift. The same image discriminator is applied to the generated image I^shift, and the object discriminator, the object classifier and the attribute classifier are applied to each generated object O^shift cropped based on the shifted bounding boxes 𝐛𝐛𝐨𝐱^shift_i.
  
  
  
  
  
   §.§ Discriminator
  
  
  
  The structure of the discriminator D in our model follows the discriminator proposed in layout2im [40], but adds an additional term for the attributes (4):
  
  
  
      
    (1) Image discriminator classifies the input image I as real and the generated image I^rec, I^rand, I^shift as fake.
  
      
    (2) Object discriminator classifies the cropped objects O from I as real, and O^rec, O^rand and O^shift from I^rec, I^rand, I^shift, respectively, as fake.
  
      
    (3) Auxiliary object classifier cls^obj predicts the category of cropped objects and is used to train the generator to synthesize correct objects. It is trained on real objects O and their labels ℓ. 
  
      
    (4) Auxiliary attribute classifier cls^att predicts the attribute of cropped objects and is used to train the generator to synthesize objects with correct attributes. It is trained on real objects O and their attributes 𝒜. 
  
  
  
  
   §.§ Loss Function
  
  
  Our model follows the Generative Adversarial Networks framework [4]. Namely, one image generator and two discriminators are jointly trained in minimax game:
  
  
  
  
      min_Gmax_D
      𝔼_𝐱∼ p_𝐱 [ log D(𝐱)] + 𝔼_𝐳∼ p_𝐳 [ log (1 - D(G(𝐳))],
  
  where 𝐱 is the real image sampled from the data distribution p(𝐱) and 𝐳 is the latent codes that generator uses to produce fake image. Since we have two separate discriminators for images and objects, there are two adversarial losses:
  
    
    * Image Adversarial Loss.
    In each training iteration, our generator produces three images, which are: a generated image I^rand, a reconstructed image I^rec and a shifted image I^shift. Hence, the image adversarial loss  ℒ_adv^· is defined as in Eq. (<ref>) for all three types of generated images. By averaging the loss for I^rec, I^shift, I^rec, we obtain: 
    
  
    
      ℒ_adv^img = ℒ_adv^I^rand + ℒ_adv^I^rec + ℒ_adv^I^shift/3
  
    which generator G minimizes, and discriminator D maximizes.
  
    
    * Object Adversarial Loss.
    We crop and resize objects O^rand, O^rec and O^shift from I^rand, I^rec and I^shift, respectively. By treating cropped objects as images, the object adversarial loss ℒ_adv^· is also defined as in Eq. (<ref>):
    
  
    
      ℒ_adv^obj = ℒ_adv^O^rand + ℒ_adv^O^rec + ℒ_adv^O^shift/3
  
    
    
  
  In addition, we have another five losses to facilitate the generation of realistic images:
  
    
    * KL Loss. ℒ_KL = ∑_i=1^o𝔼[𝒟_KL(Q(𝐳^r_obj_i | O_i) 𝒩(𝐳^r_obj))] encourages the posterior distribution Q(𝐳^r_obj_i | O_i) for object i to be close to the prior 𝒩(𝐳^r_obj), for all of the o objects in the given image/layout.
    
    
    * Image Reconstruction Loss.
    ℒ_1^img = I-I^rec_1 is the L1 difference between ground-truth image I and reconstructed image I^rec produced by the generator.
    
    
    * Object Latent Code Reconstruction Loss.
    ℒ_1^latent  = ∑_i=1^o𝐳_obj_i - 𝐳^rand_obj_i_1 + 𝐳_obj_i - 𝐳^shift_obj_i_1  penalizes the ℒ_1 difference between the randomly sampled 𝐳_obj_i∼𝒩(𝐳_obj) and the re-estimated 𝐳^rand_obj_i and 𝐳^shift_obj_i from the generated objects O^rand and O^shift, respectively.
    
    
    * Auxiliar Object Classification Loss.
    ℒ_AC^obj is defined as the cross entropy loss from the object classifier. Cropped objects O^real with labels from real images are used to train the object classifier, and then the generator G is trained to generate realistic objects O^rand, O^rec and O^shift that minimize ℒ_AC^obj.
    
    
    * Auxiliar Attribute Classification Loss.
    ℒ_AC^att is defined as the weighted binary cross entropy loss from the attribute classifier. Similarly, real objects are used to train the classifier, and the generator G is trained to generate objects O^rand, O^rec and O^shift with correct attribute labels that minimize ℒ_AC^att.
    
    
  
  
  Therefore, the generator G minimizes:
  
  
      ℒ_G= λ_1ℒ_adv^img+ λ_2ℒ_adv^obj+λ_3ℒ_AC^obj + λ_4ℒ_AC^att
      + λ_5ℒ_KL+λ_6ℒ_1^img+λ_7ℒ_1^latent
  
  
  and the discriminator D minimizes:
  
  
      ℒ_D= -λ_1ℒ_adv^img -λ_2ℒ_adv^obj+λ_3ℒ_AC^obj + λ_4ℒ_AC^att
  
  where λ_i are weights for different loss terms.
  
  
  
  Implementation Details:
  
  We set image canvas size to 64 × 64 (128 × 128), and the object size to  32 × 32 (64 × 64).  The λ 1 ∼ λ 7 are 1.0, 1.0, 8.0, 2.0, 0.01, 5.0, 5.0. The dimension of the category embedding 𝐰 and the latent code 𝐳 are both 64. The model is trained using Adam with a learning rate of 0.0001 and a batch size of 6 for 300,000 iterations on 2 Geforce GTX 1080 Ti. In each training iteration, we first train the object classifier, the attribute classifier and the two discriminators, and then the generator. 
  
  
  
  
  § RESULTS
  
  Due to limited space in the main paper, we provide additional evaluations here. 
  
  
  
   §.§ Spatial Equivariance Experiments
  
  
  Figure <ref> and <ref> demonstrate the ability of our model to generate high quality images (at 128 × 128 resolution) and maintain consistency of objects when the boxes are shifted. We want to draw reader attention to 4-th row from the top in Figure <ref>. Note how our model can generate images where tree shifts from left to right based on the change in the layout (cyan), while largely maintaining the structure and appearance of the boat unchanged. In contrast, LostGAN [31], when presented with the same sifted layout, generates an image that is entirely incoherent with the original: boat is no longer discernible, sky changes color, etc. Similar behavior can also be observed in the last row, where our model is able to generate new version of the image with shifted placement of the elephants, while maintaining the tree line and overcast sky. The images produced with LostGAN [31] are highly incoherent with visible changes in both foreground and background objects, as well as their appearances (despite fixing appearance latent vectors). Similar behavior is also readily observed in Figure <ref>. For example, consider new shifted placement of the person in the third row from the top, or an almost mirror image produced by shifting trees and the house from right to left and vice versa in the 5-th row.  
  LostGAN [31], while generates plausible images, is consistently failing to maintain style, appearance, structure and placement of objects when the layout is modified to simply spatially  re-arrange the same objects. 
  
  Figure <ref> shows similar ability to maintain consistency with spatial shifts of objects in the layout at the lower, 64 × 64, resolution. Note that results of LostGAN are less blurry because, unlike all other methods in Figure <ref>, they are computed at 128 × 128 resolution (but illustrated at 64 × 64); authors of LostGAN do not provide a trained 64 × 64 model. As such, the comparison to LostGAN isn't exactly fair and is less favorable to us. Despite this, our model, is able to generate high-quality images that are consistent under spatial shifts in layout (see last row).  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Qualitative Generation Experiments
  
  
  Figure <ref> showcases our model's ability to generate plausible images for a wide variety of layout configurations (human, food, animal, furniture, house). Notably, results of sg2im [10] and layout2im [40] are of lower quality and blurry. LostGAN [31] does not perform well on human faces. Similar to Figure <ref>, results of LostGAN in Figure <ref> are less blurry because, unlike all other methods, they are at 128 × 128 resolution; LostGAN didn't provide trained 64 × 64 model, so we use a higher resolution model instead for visualization. 
  
  
  
  
  
  
  
  
   §.§ Attribute Modification Experiments
  
  
  Figure <ref> illustrates additional examples of our model's ability to modify attributes of various objects. The change of attributes does not affect the layout or other objects in the image.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  