Object-driven Text-to-Image Synthesis via Adversarial Training
Wenbo Li Pengchuan Zhang Lei Zhang Qiuyuan Huang Xiaodong He Siwei Lyu Jianfeng Gao







Object-driven Text-to-Image Synthesis via Adversarial Training
    
  Wenbo Li^†*1,2  Pengchuan Zhang^*2  Lei Zhang^3 
 Qiuyuan Huang^2  Xiaodong He^4  Siwei Lyu^1  Jianfeng Gao^2

  
  ^1University at Albany, SUNY^2Microsoft Research AI^3Microsoft^4JD AI Research

  {wli20,slyu}@albany.edu, {penzhan,leizhang,qihua,jfgao}@microsoft.com, xiaodong.he@jd.com

    
=======================================================================================================================================================================================================================================================================================================


† Work was performed when was an intern with Microsoft Research AI.
* indicates equal contributions.

  In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes. Following the two-step (layout-image) generation process, a novel object-driven attentive image generator is proposed to synthesize salient objects by paying attention to the most relevant words in the text description and the pre-generated semantic layout. In addition, a new Fast R-CNN based object-wise discriminator is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated layout. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale COCO benchmark, increasing the Inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the traditional grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.






§ INTRODUCTION


Synthesizing images from text descriptions (known as Text-to-Image synthesis) is an important machine learning task, which requires handling ambiguous and incomplete information in natural language descriptions and learning across vision and language modalities.
Approaches based on Generative Adversarial Networks (GANs) <cit.> have recently achieved promising results on this task <cit.>.

Most GAN based methods synthesize the image conditioned only on a global sentence vector, which may miss important fine-grained information at the word level, and prevents the generation of high-quality images. More recently, AttnGAN <cit.> is proposed which introduces the attention mechanism <cit.> into the GAN framework, thus allows attention-driven, multi-stage refinement for fine-grained text-to-image generation.



Although images with realistic texture have been synthesized on simple datasets, such as birds <cit.> and flowers <cit.>, most existing approaches do not specifically model objects and their relations in images and thus have difficulties in generating complex scenes such as those in the COCO dataset <cit.>. For example, generating images from a sentence “several people in their ski gear are in the snow” requires modeling of different objects (people, ski gear) and their interactions (people on top of ski gear), as well as filling the missing information (, the rocks in the background). In the top row of Fig. <ref>, the image generated by AttnGAN does contain scattered texture of people and snow, but the shape of people are distorted and the picture's layout is semantically not meaningful. <cit.> remedies this problem by first constructing a semantic layout from the text and then synthesizing the image by a deconvolutional image generator. However, the fine-grained word/object-level information is still not explicitly used for generation. Thus, the synthesized images do not contain enough details to make them look realistic (see the middle row of Fig. <ref>).

In this study, we aim to generate high-quality complex images with semantically meaningful layout and realistic objects.
To this end, we propose a novel Object-driven Attentive Generative Adversarial Networks (Obj-GAN) that effectively capture and utilize fine-grained word/object-level information for text-to-image synthesis. 
The Obj-GAN consists of a pair of object-driven attentive image generator and object-wise discriminator, and a new object-driven attention mechanism. The proposed image generator takes as input the text description and a pre-generated semantic layout and synthesize high-resolution images via multiple-stage coarse-to-fine process. At every stage, the generator synthesizes the image region within a bounding box by focusing on words that are most relevant to the object in that bounding box, as illustrated in the bottom row of Fig. <ref>. More specifically, using a new object-driven attention layer, it uses the class label to query words in the sentences to form a word context vector, as illustrated in Fig. <ref>, and then synthesizes the image region conditioned on the class label and word context vector. The object-wise discriminator checks every bounding box to make sure that the generated object indeed matches the pre-generated semantic layout. To compute the discrimination losses for all bounding boxes simultaneously and efficiently, our object-wise discriminator is based on a Fast R-CNN <cit.>, with a binary cross-entropy loss for each bounding box.

The contribution of this work is three-folded.
(i) An Object-driven Attentive  Generative Network (Obj-GAN) is proposed for synthesizing complex images from text descriptions. Specifically, two novel components are proposed, including the object-driven attentive generative network and the object-wise discriminator. 
(ii) Comprehensive evaluation on a large-scale COCO benchmark shows that our Obj-GAN significantly outperforms previous state-of-the-art text-to-image synthesis methods. Detailed ablation study is performed to empirically evaluate the effect of different components in Obj-GAN.
(iii) A thorough analysis is performed through visualizing the attention layers of the Obj-GAN, showing insights of how the proposed model generates complex scenes in high quality. Compared with the previous work, our object-driven attention is more robust and interpretable, and significantly improves the object generation quality in complex scenes.





§ RELATED WORK


Generating photo-realistic images from text descriptions, though challenging, is important to many real-world applications such as art generation and computer-aided design. There has been much research effort for this task through different approaches, such as variational inference <cit.>, approximate Langevin process <cit.>, conditional PixelCNN via maximal likelihood estimation <cit.>, and conditional generative adversarial networks <cit.>. Compared with other approaches, Generative Adversarial Networks (GANs) <cit.> have shown better performance in image generation <cit.>. However, existing GAN based text-to-image synthesis is usually conditioned only on the global sentence vector, which misses important fine-grained information at the word level, and thus lacks the ability to generate high-quality images. <cit.> uses the traditional grid visual attention mechanism in this task, which enables synthesizing fine-grained details at different image regions by paying attentions to the relevant words in the text description.

To explicitly encode the semantic layout into the generator, <cit.> proposes to decompose the generation process into two steps, in which it first constructs a semantic layout (bounding boxes and object shapes) from the text and then synthesizes an image conditioned on the layout and text description. <cit.> also proposes such a two-step process to generate images from scene graphs, and their process can be trained end-to-end. In this work, the proposed Obj-GAN follows the two-step generation process as <cit.>. However, <cit.> encodes the text into a single global sentence vector, which loses word-level fine-grained information. Moreover, it uses the image-level GAN loss for the discriminator, which is less effective at providing object-wise discrimination signal for generating salient objects. We propose a new object-driven attention mechanism to provide fine-grained information (words in the text description and objects in the layout) for different components, including an attentive seq2seq bounding box generator, an attentive image generator and an object-wise discriminator.

The attention mechanism has recently become a crucial part of vision-language multi-modal intelligence tasks. The traditional grid attention mechanism has been successfully used in modeling multi-level dependencies in image captioning <cit.>, image question answering <cit.>, text-to-image generation <cit.>, unconditional image synthesis <cit.> and image-to-image translation <cit.>, image/text retrieval <cit.>. In 2018, <cit.> proposes a bottom-up attention mechanism, which enables attention to be calculated over semantic meaningful regions/objects in the image, for image captioning and visual question-answering. Inspired by these works, we propose Obj-GAN which for the first time develops an object-driven attentive generator plus an object-wise discriminator, thus enables GANs to synthesize high-quality images of complicated scenes.



§ OBJECT-DRIVEN ATTENTIVE GAN


As illustrated in Fig. <ref>, the Obj-GAN performs text-to-image synthesis in two steps: generating a semantic layout (class labels, bounding boxes, shapes of salient objects), and then generating the image. In the image generation step, the object-driven attentive generator and object-wise discriminator are designed to enable image generation conditioned on the semantic layout generated in the first step.

The input of Obj-GAN is a sentence with T_s tokens. With a pre-trained bi-LSTM model, we encode its words as word vectors e ∈^D× T_s and the entire sentence as a global sentence vector e̅∈^D. We provide details of this pre-trained bi-LSTM model and the implementation details of other modules of Obj-GAN in  <ref>.






 §.§ Semantic layout generation


In the first step, the Obj-GAN takes the sentence as input and generates a semantic layout, a sequence of objects specified by their bounding boxes (with class labels) and shapes. As illustrated in Fig. <ref>, a box generator first generates a sequence of bounding boxes, and then a shape generator generates their shapes. This part resembles the bounding box generator and shape generator in <cit.>, and we put our implementation details in  <ref>.

Box generator. We train an attentive seq2seq model <cit.>, also referring to Fig. <ref>, as the box generator:

    B_1:T := [B_1, B_2, …, B_T] ∼ G_box(e).

Here, e are the pre-trained bi-LSTM word vectors, B_t = (l_t, b_t) are the class label of the t's object and its bounding box b = (x, y, w, h) ∈^4. In the rest of the paper, we will also call the label-box pair B_t as a bounding box when no confusion arises. Since most of the bounding boxes have corresponding words in the sentence, the attentive seq2seq model captures this correspondence better than the seq2seq model used in <cit.>.


Shape generator. Given the bounding boxes B_1:T, the shape generator predicts the shape of each object in its bounding box, ,

    M̂_1:T = G_shape(B_1:T, z_1:T).

where z_t∼𝒩(0,1) is a random noise vector. Since the generated shapes not only need to match the location and category information provided by B_1:T, but also should be aligned with its surrounding context, we build G_shape based on a bi-directional convolutional LSTM, as illustrated in Fig. <ref>.

Training of G_shape is based on the GAN framework <cit.>, in which a perceptual loss is also used to constrain the generated shapes and to stabilize the training.





 §.§ Image generation



  §.§.§ Attentive multistage image generator


As shown in Fig. <ref>, the proposed attentive multistage generative network has two generators (G_0, G_1). The base generator G_0 first generates a low-resolution image x̂_0 conditioned on the global sentence vector and the pre-generated semantic layout. The refiner G_1 then refines details in different regions by paying attention to most relevant words and pre-generated class labels and generates a higher resolution image x̂_1. Specifically,
  
  

  
    h_0    = F_0(z,   e,   Enc(M^0), c^obj, c^lab),   x̂_0 = G_0(h_0),
    
        h_1    = F_1(c^pat, h_0+Enc(M^1), c^obj, c^lab),   x̂_1 = G_1(h_1),

where (i) z is a random vector with standard normal distribution; (ii) Enc(M^0) ( Enc(M^1) ) is the encoding of low-resolution shapes M^0 (higher-resolution shapes M^1); (iii) c^pat= F_attn^grid(e,h_0) are the patch-wise context vectors from the traditional grid attention, (iv) c^obj=F_attn^obj(e, e^g, l^g, M) are the object-wise context vectors from our new object-driven attention, and c^lab=c^lab(l^g, M) are the label context vectors from class labels. We can stack more refiners to the generation process and get higher and higher resolution images. In this paper, we have two refiners (G_1 and G_2) and finally generate images with resolution 256× 256.
 


Compute context vectors via attention. Both patch-wise context vectors c^pat and object-wise context vectors c^obj are attention-driven context vectors for specific image regions, and encode information from the words that are most relevant to that image region. Patch-wise context vectors are for uniform-partitioned image patches determined by the uniform down-sampling/up-sampling structure of CNN, but these patches are not semantically meaningful. Object-wise context vectors are for semantically meaningful image regions specified by bounding boxes, but these regions are at different scales and may have overlaps.

Specifically, the patch-wise context vector c_j^pat ( objective-wise context vector c_t^obj) is a dynamic representation of word vectors relevant to patch j (bounding box B_t), which is calculated by


 
    c_j^pat = ∑_i=1^T_sβ_j,i^pat e_i,   
      c_t^obj = ∑_i=1^T_sβ_t,i^obj e_i.

Here, β_j,i^pat ( β_t,i^obj ) indicates the weight the model attends to the i'th word when generating patch j (bounding box B_t) and is computed by
 

 
    β_j,i^pat = exp(s_j,i^pat)/∑_k=1^T_sexp(s_j,k^pat),        s_j,i^pat = (h_j)^T e_i, 
    β_t,i^obj = exp(s_t,i^obj)/∑_k=1^T_sexp(s_t,k^obj),        s_t,i^obj = (l_t^g)^T e^g_i.

For the traditional grid attention, we use the image region feature h_j, which is one column in the previous hidden layer h ∈^D^pat× N^pat, to query the pre-trained bi-LSTM word vectors e. For the new object-driven attention, we use the GloVe embedding of object class label l_t^g to query the GloVe embedding of the words in the sentence, as illustrated in the lower part of Fig. <ref>.

Feature map concatenation.
The patch-wise context vector c_j^pat can be directly concatenated with the image feature vector h_j in the previous layer. However, the object-wise context vector c_t^obj cannot, because they are associated with bounding boxes instead of pixels in the hidden feature map. We propose to copy the object-wise context vector c_t^obj to every pixel where the t'th object is present, , M_t ⊗ c_t^obj where ⊗ is the vector outer-product, as illustrated in the upper-right part of Fig. <ref>. [This operation can be viewed as an inverse of the pooling operator.]

If there are multiple bounding boxes covering the same pixel, we have to decide whose context vector should be used on this pixel. In this case, we simply do a max-pooling across all the bounding boxes:

    c^obj = max_t : 1≤ t ≤ T M_t ⊗ c_t^obj.

Then c^obj can be concatenated with the feature map h and patch-wise context vectors c^pat for next-stage generation.

Label context vectors.
Similarly, we distribute the class label information to the entire hidden feature map to get the label context vectors, ,

    c^lab = max_t : 1≤ t ≤ T M_t ⊗ e_t^g.

Finally, we concatenate h, c^pat, c^obj and c^lab and pass the concatenated tensor through one up-sampling layer and several residual layers to generate a higher-resolution image.

Grid attention vs. object-driven attention.
The process to compute the patch-wise context vectors above is the traditional grid attention mechanism used in AttnGAN <cit.>. Note that its attention weights β_j,i^pat and context vector c_j^pat are useful only when the hidden feature h_j^pat in the G_0 stage correctly captures the content to be drawn in patch j. This essentially assumes that the generation in the G_0 stage already captures a rough sketch (semantic layout). This assumption is valid for simple datasets like birds <cit.>, but fails for complex datasets like COCO <cit.> where the generated low-resolution image x̂_0 typically does not have a meaningful layout. In this case, the grid attention is even harmful, because patch-wise context vector is attended to a wrong word and thus generate the texture associated with that wrong word. This may be the reason why AttnGAN's generated image contains scattered patches of realistic texture but overall is semantically not meaningful; see Fig. <ref> for example. Similar phenomenon is also observed in DeepDream <cit.>. On the contrary, in our object-driven attention, the attention weights β_t,i^obj and context vector c_t^obj rely on the class label l_t^g of the bounding box and are independent of the generation in the G_0 stage. Therefore, the object-wise context vectors are always helpful to generate images that are consistent with the pre-generated semantic layout. Another benefit of this design is that the context vector c_t^obj can also be used in the discriminator, as we present in  <ref>.



  §.§.§ Discriminators


We design patch-wise and object-wise discriminators to train the attentive multi-stage generator above. Given a patch from uniformly-partitioned image patches determined by the uniform down-sampling structure of CNN, the patch-wise discriminator is trying to determine whether this patch is realistic or not (unconditional) and whether this patch is consistent with the sentence description or not (conditional). Given a bounding box and the class label of the object within it, the object-wise discriminator is trying to determine whether this region is realistic or not (unconditional) and whether this region is consistent with the sentence description and given class label or not (conditional).



Patch-wise discriminators. Given an image-sentence pair x, e (e is the sentence vector), the patch-wise unconditional and text discriminator can be written as

    p^pat,un= D_uncond.^pat(Enc(x)),    p^pat,con = D_text^pat(Enc(x), e),

where Enc is a convolutional feature extractor that extracts patch-wise features, D_uncond. ( D_text^pat ) determine whether the patch is realistic (consistent with the text description) or not.

Shape discriminator. In a similar manner, we have our patch-wise shape discriminator



    p^pix = D^pix(Enc(x, M)),

where we first concatenate the image x and shapes M in the channel dimension, and then extracts patch-wise features by another convolutional feature extractor Enc. The probabilities p^pix determine whether the patch is consistent with the given shape. Our patch-wise discriminators D_uncond.^pat, D_text^pat and D^pix resembles the PatchGAN <cit.> for the image-to-image translation task. Compared with the global discriminators in AttnGAN <cit.>, the patch-wise discriminators not only reduce the model size and thus enable generating higher resolution images, but also increase the quality of generated images; see Table <ref> for experimental evidence.

Object-wise discriminators. Given an image x, bounding boxes of objects B_1:T and their shapes M, we propose the following object-wise discriminators:



    {h_t^obj}_t=1^T =    FastRCNN(x, M, B_1:T), 
    
      p_t^obj,un =D_uncond.^obj(h_t^obj),      p_t^obj,con = D^obj(h_t^obj, e_t^g, c_t^obj).

Here, we first concatenate the image x and shapes M and extract a region feature vector h_t^obj for each bounding box through a Fast R-CNN model <cit.> with an ROI-align layer <cit.>; see Fig. <ref>(a). Then similar to the patch-wise discriminator (<ref>), the unconditional (conditional) probabilities p_t^obj,un ( p_t^obj,con) determine whether the t'th object is realistic (consistent with its class label e_t^g and its text context information c_t^obj) or not; see Fig. <ref>(b). Here, e_t^g is the GloVe embedding of the class label and c_t^obj is its text context information defined in (<ref>).

All discriminators are trained by the traditional cross entropy loss <cit.>.



  §.§.§ Loss function for the image generator

The generator's GAN loss is a weighted sum of these discriminators' loss, ,

    ℒ_GAN(G) = - λ_obj/T∑_t=1^T( log p_t^obj,un_obj uncond. loss + log p_t^obj,con_obj cond. loss)
        - 1/N^pat∑_j=1^N^pat( log p_j^pat,un_uncond. loss + λ_txtlog p_j^pat,con_text cond. loss + λ_pixlog p_j^pix_shape cond. loss).

Here, T is the number of bounding boxes, N^pat is the number of regular patches, (λ_obj, λ_txt, λ_pix) are the weights of the object-wise GAN loss, patch-wise text conditional loss and patch-wise shape conditional loss, respectively. We tried combining our discriminators with the spectral normalized projection discriminator <cit.>, but did not see significant performance improvement. We report performance of the spectral normalized version in  <ref> and provide model architecture details in  <ref>.

Combined with the deep multi-modal attentive similarity model (DAMSM) loss introduced in <cit.>, our final image generator's loss is

    ℒ_G = ℒ_GAN + λ_DAMSMℒ_DAMSM

where λ_damsm is a hyper-parameter to be tuned. Here, the DAMSM loss is a word level fine-grained image-text matching loss computed, which will be elaborated in  <ref>. Based on the experiments on a held-out validation set, we set the hyperparameters in this section as: λ_obj = 0.1, λ_txt = 0.1, λ_pix = 1 and λ_damsm = 100.


Both the patch-wise and object-wise discriminators can be applied to different stages in the generation. We apply the patch-wise discriminator for every stage of the generation, following <cit.>, but only apply the object-wise discriminator at the final stage.




§ EXPERIMENTS


Dataset. We use the COCO dataset <cit.> for evaluation. It contains 80 object classes, where each image is associated with object-wise annotations (, bounding boxes and shapes) and 5 text descriptions. We use the official 2014 train (over 80K images) and validation (over 40K images) splits for training and test stages, respectively.



Evaluation metrics. We use the Inception score <cit.> and Fréchet inception distance (FID) <cit.> score as the quantitative evaluation metrics. In our experiments, we found that Inception score can be saturated, even over-fitted, while FID is a more robust measure and aligns better with human qualitative evaluation. Following <cit.>, we also use R-precision, a common evaluation metric for ranking retrieval results, to evaluate whether the generated image is well conditioned on the given text description. More specifically, given a pre-trained image-to-text retrieval model, we use generated images to query their corresponding text descriptions. First, given generated image x̂ conditioned on sentence s and 99 random sampled sentences {s'_i: 1≤ i ≤ 99}, we rank these 100 sentences by the pre-trained image-to-text retrieval model. If the ground truth sentence s is ranked highest, we count this a success retrieval. For all the images in the test dataset, we perform this retrieval task once and finally count the percentage of success retrievals as the R-precision score.

It is important to point out that none of these quantitative metrics are perfect. Better metrics are required to evaluate image generation qualities in complicated scenes. In fact, the Inception score completely fails in evaluating the semantic layout of the generated images. The R-precision score depends on the pre-trained image-to-text retrieval model it uses, and can only capture the aspects that the retrieval model is able to capture. The pre-trained model we use is still limited in capturing the relations between objects in complicated scenes, so is our R-precision score.

Quantitative evaluation. We compute these three metrics under two settings for the full validation dataset.

Qualitative evaluation. Apart from the quantitative evaluation, we also visualize the outputs of all ablative versions of Obj-GAN and the state-of-the-art methods (, <cit.>) whose pre-trained models are publicly available.





 §.§ Ablation study


In this section, we first evaluate the effectiveness of the object-driven attention. Next, we compare the object-driven attention mechanism with the grid attention mechanism. Then, we evaluate the impact of the spectral normalization for Obj-GAN. We use Fig. <ref> and the higher half of Table <ref> to present the comparison among different ablative versions of Obj-GAN. Note that all ablative versions have been trained with batch size 16 for 60 epochs. In addition, we use the lower half of Table <ref> to show the comparison between Obj-GAN and previous methods. Finally, we validated the Obj-GAN's generalization ability on the novel text descriptions.

Object-driven attention. To evaluate the efficacy of the object-driven attention mechanism, we implement a baseline, named P-AttnGAN w/ Lyt, by disabling the object-driven attention mechanism in Obj-GAN. In essence, P-AttnGAN w/ Lyt can be considered as an improved version of AttnGAN with the patch-wise discriminator (abbreviated as the prefix “P-" in name) and the modules (, shape discriminator) for handling the conditional layout (abbreviated as “Lyt"). Moreover, it can also be considered as a modified implementation of <cit.>, which resembles their two-step (layout-image) generation. Note that there are three key differences between P-AttnGAN w/ Lyt and <cit.>: (i) P-AttnGAN w/ Lyt has a multi-stage image generator that gradually increases the generated resolution and refines the generated images, while <cit.> has a single-stage image generator. (ii) With the help of the grid attentive module, P-AttnGAN w/ Lyt is able to utilize the fine-grained word-level information, while <cit.> conditions on the global sentence information. (iii) The third difference lies in their loss functions: P-AttnGAN w/ Lyt uses the DAMSM loss in (<ref>) to penalize the mismatch between the generated images and the input text descriptions, while <cit.> uses the perceptual loss to penalize the mismatch between the generated images and the ground-truth images. As shown in Table <ref>, P-AttnGAN w/ Lyt yields higher Inception score than <cit.> does.

We compare Obj-GAN with P-AttnGAN w/ Lyt under three settings, with each corresponding to a set of conditional layout input, , the predicted boxes & shapes, the ground-truth boxes & predicted boxes, and the ground-truth boxes & shapes. As presented in Table <ref>, Obj-GAN consistently outperforms P-AttnGAN w/ Lyt on all three metrics. In Fig. <ref>, we use the same layout as the conditional input, and compare the visual quality of their generated images. An interesting phenomenon shown in Fig. <ref> is that both the foreground objects (, airplane and train) and the background (, airport and trees) textures synthesized by Obj-GAN are much richer and smoother than those using P-AttnGAN w/ Lyt. The effectiveness of the object-driven attention for the foreground objects is easy to understand. The benefits for the background textures using the object-driven attention mechanism is probably due to the fact that it implicitly provides stronger signal that distinguishes the foreground. As such, the image generator may have richer guidance and clearer emphasis when synthesizing textures for a certain region.

Grid attention vs. object-driven attention. We compare Obj-GAN with P-AttnGAN herein, so as to compare the effects of the object-driven and the grid attention mechanisms. In Fig. <ref>, we show the generated image of each method as well as the corresponding attention maps aligned on the right side. In a grid attention map, the brightness of a region reflects how much this region attended to the word above the map. As for the object-driven attention map, the word above each attention map is the most attended word by the highlighted object. The highlighted region of an object-driven attention map is the object shape.

As analyzed in  <ref>, the reliability of grid attention weights depends on the quality of the previous layer's image region features. This makes the grid attention unreliable sometimes, especially for complex scenes. For example, the grid attention weights in Fig. <ref> are unreliable because they are scattered (, the attention map for “man") and inaccurate. However, this is not a problem for the object-driven attention mechanism, because its attention weights are directly calculated from embedding vectors of words and class labels and are independent of image features. Moreover, as shown in Fig. <ref> and Equ. (<ref>), the impact region of the object-driven attention context vector is bounded by the object shapes, which further enhances its semantics meaningfulness. As a result, the instance-driven attention significantly improves the visual quality of the generated images, as demonstrated in Fig. <ref>. Moreover, the performance can be further improved if the semantic layout generation is improved. In the extreme case, Obj-GAN based on ground truth layout (Obj-GAN^2) has the best visual quality (the rightmost column of Fig. <ref>) and the best quantitative evaluation (Table <ref>).





Obj-GAN w/ SN vs. Obj-GAN.
We present the comparison between the cases with or without spectral normalization in the discriminators in Table <ref> and Fig. <ref>. We observe that there is no obvious improvement on the visual quality, but slightly worse on the quantitative metrics. We show more results and discussions in  <ref>.

Comparison with previous methods.
To compare Obj-GAN with the previous methods, initialized by the Obj-GAN models in the ablation study, we trained Obj-GAN-SOTA with batch size 64 for 10 more epochs.

In order to evaluate AttnGAN on FID, we conducted the evaluation on the officially released pre-trained model. Note that the Sg2Im <cit.> focuses on generating images from scene graphs and conducted the evaluation on a different split of COCO. However, we still included Sg2Im's results to reflect the broader context of the related topic. As shown in Table <ref>, Obj-GAN-SOTA outperforms all previous methods significantly. We notice that the increment of batch size does boost the Inception score and R-precision, but does not improve FID. The possible explanation is: with a larger batch size, the DAMSM loss (a ranking loss in essence) in (<ref>) plays a more important role and improves Inception and R-precision, but it does not focus on reducing FID between the generated images and the real ones.



Generalization ability.
We further investigate if Obj-GAN just memorizes the scenarios in COCO or it indeed learns the relations between the objects and their surroundings. To this end, we compose several descriptions which reflect novel scenarios that are unlikely to happen in the real-world, , a decker bus is floating on top of a lake, or a cat is catching a frisbee. We use Obj-GAN to synthesize images for these rare scenes. The results in Fig. <ref> further demonstrate the good generalization ability of Obj-GAN.



§ CONCLUSIONS


In this paper, we have presented a multi-stage Object-driven Attentive Generative Adversarial Networks (Obj-GANs) for synthesizing images with complex scenes from the text descriptions. With a novel object-driven attention layer at each stage, our generators are able to utilize the fine-grained word/object-level information to gradually refine the synthesized image. We also proposed the Fast R-CNN based object-wise discriminators, each of which is paired with a conditional input of the generator and provides object-wise discrimination signal for that condition. Our Obj-GAN significantly outperforms previous state-of-the-art GAN models on various metrics on the large-scale challenging COCO benchmark. Extensive experiments demonstrate the effectiveness and generalization ability of Obj-GAN on text-to-image generation for complex scenes.


ieee







§ APPENDIX










 §.§ Obj-GAN vs. the ablative versions


In this section, we show more images generated by our Obj-GAN and its ablative versions on the COCO dataset. In Fig. <ref> and Fig. <ref>, we provide more comparisons as the complementary for Fig. <ref>. It can be found that there are no obvious improvement on the visual quality when using the spectral normalization.



 §.§ Visualization of attention maps


In Fig. <ref>, we visualize attention maps generated by P-AttnGAN and Obj-GAN as the complementary for Fig. <ref>.




 §.§ Results based on the ground-truth layout


We show the results generated by Obj-GAN based on the ground-truth layout in Fig. <ref>, Fig. <ref> and Fig. <ref>.












 §.§ Bi-LSTM text encoder, DAMSM and R-precision



We use the deep attentive multi-modal similarity model (DAMSM) proposed in <cit.>, which learns a joint embedding of the image regions and words of a sentence in a common semantic space. The fine-grained conditional loss enforces the sub-region of the generated image to match the corresponding word in the sentence.

Bi-LSTM text encoder serves as the text encoder for both DAMSM and the box generator (see  <ref>). Bi-LSTM text encoder is a bi-directional LSTM that extracts semantic vectors from the text description. In the Bi-LSTM, each word corresponds to two hidden states, one for each direction. Thus, we concatenate its two hidden states to represent the semantic meaning of a word. The feature matrix of all words is indicated by ė∈ℝ^D× T_s. Its i^th column ė_i is the feature vector for the i^th word. D is the dimension of the word vector and T_s is the number of words. Meanwhile, the last hidden states of the bi-directional LSTM are concatenated to be the global sentence vector, denoted by ê∈ℝ^D. We present the network architectures for the Bi-LSTM text encoder in Table <ref>.

The image encoder 

is a convolutional neural network that maps images to semantic vectors.
The intermediate layers of the CNN model learns local features of different regions of the image, while the later layers learn global features of the image.
More specifically, the image encoder is built upon Inception-v3 model pre-trained on ImageNet.
We first rescale the input image to be 299×299 pixels.
And then, we extract the local feature matrix f ∈ℝ^768×289 (reshaped from 768×17×17) from “mixed_6e” layer of Inception-v3.
Each column of f is the feature vector of a local image region.
768 is the dimension of the local feature vector, and 289 is the number of regions in the image.
Meanwhile, the global feature vector f∈ℝ^2048 is extracted from the last average pooling layer of Inception-v3.
Finally, we convert the image features to the common semantic space of text features by adding a new layer perceptron as shown in Eq. (<ref>),

 
    v = Wf;         v = W f,


where v ∈ℝ^D×289 and its i^th column v_i is the visual
feature vector for the i^th image region; v∈ℝ^D is the visual feature vector for the whole image.
While v_i is the local image feature vector that corresponds to the word embedding, v is the global feature vector that is related to the sentence embedding. D is the dimension of the multimodal (, image and text modalities) feature space.
For efficiency, all parameters in layers built from Inception-v3 model are fixed, and the parameters in newly added layers are jointly learned with the rest of networks.



The fine-grained conditional loss 
is designed to learn the correspondence between image regions and words.
However, it is difficult to obtain manual annotations.

Actually, many words relate to concepts that may not easily be visually defined, such as open or old.
One possible solution is to learn word-image correspondence in a semi-supervised manner, in which the only supervision is the correspondence between the entire image and the whole text description (a sequence of words).

We can first calculate the similarity matrix between all possible pairs of word and image region by Eq. (<ref>),


    s = ė^T v,


where s ∈ℝ^T× 289 and s_i,j means the similarity between the i^th word and the j^th image region.

Generally, a sub-region of the image is described by none or several words of the text description, and it is not likely to be described by the whole sentence.
Therefore, we normalize the similarity matrix by Eq. (<ref>),

 
    s_i,j = exp(s_i,j)/∑_k=0^T-1exp(s_k,j)



Second, we build an attention model to compute a context vector for each word (query).
The context vector c_i is a dynamic representation of image regions related to the i^th word of the text description.
It is computed as the weighted sum over all visual feature vectors,

 
    c_i = ∑_j=0^288α_jv_j,


where we define the weight α_j via Eq. (<ref>),

 
    α_j = exp(γ_1s_i,j)/∑_k=0^288exp(γ_1s_i,k)


Here, γ_1 is a factor that decides how much more attention is paid to features of its relevant regions when computing the context vector for a word.



Finally, we define the relevance between the i^th word and the image using the cosine similarity between c_i and ė_i, , R(c_i, ė_i) = (c_i^T ė_i) / (||c_i|| ||ė_i||).
The relevance between the entire image (Q) and the whole text description (U) is computed by Eq. (<ref>),

 
    R(Q, U) = log(∑_i=1^T-1exp(γ_2 R(c_i, ė_i)) )^1/γ_2,


where γ_2 is a factor that determines how much to magnify the importance of the most relevant word-image pair. When γ_2→∞, R(Q, U) approximates to max_i=1^T-1 R(c_i, ė_i).

For a text-image pair, we can compute the posterior probability
of the text description (U) being matching with the image (Q) via,


 
    P(U|Q) = exp(γ_3 R(Q, U))/∑_U^'∈𝕌exp(γ_3 R(Q, U^')),



where γ_3 is a smoothing factor determined by experiments. 𝕌 denotes a minibatch of M text descriptions, in which only one description U^+ matches the image Q.
Thus, for each image, there are M-1 mismatching text descriptions.
The objective function is to learn the model
parameters Λ by minimizing the negative log posterior probability
that the images are matched with their corresponding text descriptions (ground truth),



    ℒ^w_1(Λ) = -log∏_Q∈ℚ P(U^+|Q),


where `w' stands for “word”.

Symmetrically, we can compute,


    ℒ^w_2(Λ) = -log∏_U∈𝕌 P(Q^+|U),


where P(Q|U) = exp(γ_3 R(Q, U))/∑_Q^'∈ℚexp(γ_3 R(Q^', U)).

If we redefine Eq. (<ref>) by R(Q, U) = (v^T ê) / (||v|| ||ê||) and substitute it to Eq. (<ref>), Eq. (<ref>), Eq. (<ref>), we can obtain loss functions ℒ^s_1 and ℒ^s_2 (where `s' stands for “sentence”) using the sentence embedding ê and the global visual vector v.

The fine-grained conditional loss is defined via Eq. (<ref>),

 
    ℒ_DAMSM = ℒ^w_1 + ℒ^w_2 + ℒ^s_1 + ℒ^s_2


The DAMSM is pre-trained by minimizing ℒ_DAMSM using real image-text pairs.
Since the size of images for pre-training DAMSM is not limited by the size of images that can be generated, real images of size 299×299 are utilized.
Furthermore, the pre-trained DAMSM can provide visually-discriminative word features and a stable fine-grained conditional loss for the attention generative network.


The R-precision score.
The DAMSM model is also used to compute the R-precision score. If there are R relevant documents for a query, we examine the top R ranked retrieval results of a system, and find that r are relevant, and then by definition, the R-precision (and also the precision and recall) is r/R.
More specifically, we use generated images to query their corresponding text descriptions.
First, the image encoder and Bi-LSTM text encoder learned in our pre-trained DAMSM are utilized to extract features of the generated images and the given text descriptions.
Then, we compute cosine similarities between the image features and the text features.
Finally, we rank candidates text descriptions for each image in descending similarity and find the top r relevant descriptions for computing the R-precision.




 §.§ Network architectures for semantic layout generation



Box generator.
We design our box generator by improving the one in <cit.> to be attentive. We denote the bounding box of the t-th object as B_t = (b^x_t, b^y_t, b^w_t, b^h_t, l_t). Then, we formulate the joint probability of sampling B_t from the box generator as

    p(b^x_t, b^y_t, b^w_t, b^h_t, l_t) = p(l_t)p(b^x_t, b^y_t, b^w_t, b^h_t|l_t).

We implement p(l_t) as a categorical distribution, and implement p(b^x_t, b^y_t, b^w_t, b^h_t|l_t) as a mixture of quadravariate Gaussians. As described in <cit.>, in order to reduce the parameter space, we decompose the box coordinate probability as p(b^x_t, b^y_t, b^w_t, b^h_t|l_t) = p(b^x_t, b^y_t|l_t)p(b^w_t, b^h_t|b^x_t, b^y_t,l_t), and approximate it with two bivariate Gasussian mixtures by









    p(b^x_t, b^y_t|l_t)    = ∑_k=1^Kπ_t,k^xy𝒩(b^x_t, b^y_t; μ^xy_t,k, Σ^xy_t,k),
    
        p(b^w_t, b^h_t|b^x_t, b^y_t, l_t)    = ∑_k=1^Kπ_t,k^wh𝒩(b^w_t, b^h_t; μ^wh_t,k, Σ^wh_t,k).

In practice, as in <cit.>, we implement the box generator within a encoder-decoder framework. The encoder is the Bi-LSTM text encoder as mentioned in  <ref>. The Gaussian Mixture Model (GMM) parameters for Eq. (<ref>) are obtained from the decoder LSTM outputs. Given text encoder's final hidden state h^Enc_T_s∈ℝ^D and output H^Enc∈ℝ^T_s × D, we initialize the decoder's initial hidden state h_0 with h^Enc_T_s. As for H^Enc, we use it to compute the contextual input z_t for the decoder:

    z_t   = ∑_i=1^T_sα_i h^Enc_i,  with α_i=W_v· (W_α[h_t-1, h^Enc_i]),

where W_v is a learnable parameter, W_α is the parameter of a linear transformation, and · and [·, ·] represent the dot product and concatenation operation, respectively.

Then, the calculation of GMM parameters are shown as follows:

    [h_t, c_t]   = LSTM([B_t-1, z_t]; [h_t-1, c_t-1]),
      
       l_t   = W^lh_t+𝐛^l,
      
       θ^xy_t   = W^xy[h_t, l_t]+𝐛^xy,
      
       θ^wh_t   = W^wh[h_t, l_t, b_x, b_y]+𝐛^wh,

where θ^·_t = [π^·_t,1:K, μ^·_t,1:K, Σ^·_t,1:K] are the parameters for GMM concatenated to a vector.
We use the same Adam optimizer and training hyperparameters (, learning rate 0.001, β_1=0.9, β_2=0.999) as in <cit.>.

Shape generator. We implement the shape generator in <cit.> with almost the same architecture except the upsample block. In <cit.>, the upsample block is designed as [convtranspose 4 × 4 (pad 1, stride 2) - Instance Normalization - ReLU]. We discovered that the usage of convtranspose would lead to unstable training which is reflected by the frequent severe grid artifacts. To this end, we replace this upsample block with that in our image generator (see Table <ref>) by switching the batch normalization to the instance one.



 §.§ Network architectures for image generation


We present the network architecture for image generators in Table <ref> and the network architectures for discriminators in Table <ref>, Table <ref> and Table <ref>. They are built with basic blocks defined in Table <ref>. We set the hyperparameters of the network structures as: N_g=48, N_d=96, N_c=80, N_e=256, N_l=50, m_0=7, m_1=3, and m_2=3.

We employ an Adam optimizer for the generators with learning rate 0.0002, β_1 = 0.5 and β_2 = 0.999. For each discriminator, we also employ an Adam optimizer with the same hyperparameters.

We design the object-wise discriminators for small objects and large objects, respectively. We specify that if the maximum of width or height of an object is greater than one-third of the image size, then this object is large; otherwise, it is small.



 §.§ Network architectures for spectral normalized projection discriminators


We combine our discriminators above with the spectral normalized projection discriminator in <cit.>. The difference between the object-wise discriminator and the object-wise spectral normalized projection discriminator is illustrated in Figure <ref>. We present detailed network architectures of the spectral normalized projection discriminators in Table <ref>, Table <ref> and Table <ref>, with  basic blocks defined in Table <ref>.

























