Question: Represent the article for finding a single-sentence TLDR

Question: Convolutional neural networks (CNNs) are commonly trained using a fixed spatial image size predetermined for a given model. Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps. In this work, we describe and evaluate a novel mixed-size training regime that mixes several image sizes at training time. We demonstrate that models trained using our method are more resilient to image size changes and generalize well even on small images. This allows faster inference by using smaller images at test time. For instance, we receive a 76.43% top-1 accuracy using ResNet50 with an image size of 160, which matches the accuracy of the baseline model with 2x fewer computations. Furthermore, for a given image size used at test time, we show this method can be exploited either to accelerate training or the final test accuracy. For example, we are able to reach a 79.27% accuracy with a model evaluated at a 288 spatial size for a relative improvement of 14% over the baseline. Figure 1: Test accuracy per image size, models trained on specific sizes (ResNet50, ImageNet). Convolutional neural networks are successfully used to solve various tasks across multiple domains such as visual , audio (van den), language and speech . While scale-invariance is considered important for visual representations , convolutional networks are not scale invariant with respect to the spatial resolution of the image input, as a change in image dimension may lead to a non-linear change of their output. Even though CNNs are able to achieve state-of-theart in many tasks and domains, their sensitivity to the image size is an inherent deficiency that limits practical use cases and requires evaluation inputs to match training image size. For example, demonstrated that networks trained on specific image size, perform poorly on other image sizes at evaluation, as shown in Figure 1. Several works attempted to achieve scale invariance by modifying the network structure . However, the most common method is to artificially enlarge the dataset using a set of label-preserving transformations also known as "data augmentation" . Several of these transformations scale and crop objects appearing within the data, thus increasing the network's robustness to inputs of different scale. Although not explicitly trained to handle varying image sizes, CNNs are commonly evaluated on multiple scales post training, such as in the case of detection and segmentation tasks. In these tasks, a network that was pretrained with fixed image size for classification is used as the backbone of a larger model that is expected to adapt to a wide variety of image sizes. In this work, we will introduce a novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes. The main contributions of the MixSize regime are: • Reducing image size sensitivity. We show that the MixSize training regime can improve model performance on a wide range of sizes used at evaluation. • Faster inference. As our mixed-size models can be evaluated at smaller image sizes, we show up to 2× reduction in computations required at inference to reach the same accuracy as the baseline model. • Faster training vs. high accuracy. We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy. 2 RELATED WORK Deep convolutional networks are traditionally trained using fixed-size inputs, with spatial dimensions H × W and a batch size B. The network architecture is configured such that the spatial dimensions are reduced through strided pooling or convolutions, with the last classification layer applied on a 1 × 1 spatial dimension. Modern convolutional networks usually conclude with a final "global" average pooling (;, that reduces any remaining spatial dimensions with a simple averaging operation. Modifying the spatial size of an input to a convolutional layer by a factor γ, will yield an output with size scaled by the same factor γ. This modification does not require any change to the number of parameters of the given convolutional layer, nor its underlying operation. Small changes in the expected size can occur, however, due to padding or strides performed by the layer. It was observed by practitioners and previous works that a network trained on a specific input dimension can still be used at inference using a modified image size to some extent . Moreover, evaluating with an image size that is larger than used for training can improve accuracy up to a threshold, after which it quickly deteriorates . showed a computational-vs-accuracy trade-off in scaling image size used to train and evaluate with a convolutional network. This finding is consistent with past findings, which demonstrated that training with a larger image size can in a larger classification error . In addition, previous works explored the notion of "progressive resizing" -increasing image size as training progresses to improve model performance and time to convergence. More recently, demonstrated that CNNs can be trained using a fixed small image size and fine-tuned posttraining to a larger size, with which evaluation will be performed. This procedure reduced the traintest discrepancy caused by the change in image size and allowed faster training time and improved accuracy -at the cost of additional fine-tuning procedure and additional computations at inference time. In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes in the image size. Deep neural network training can be distributed across many computational units and devices. The most common distribution method is by "data-parallelism"-computing an average estimate of the gradients using multiple, separably computed data samples. As training NN models is done using batch-SGD method and its variants, scaling this process across more computational devices while maintaining similar utilization for each device inflates the global batch size. Large batch training is known to affect the generalization capabilities of the networks and to require modification of the regime used for its optimization. While several works claimed that large-batch training leads to an inherent "generalization gap" , more recent works demonstrated that this gap is largely caused from an insufficient number of optimization steps performed and can be partly mitigated by hyper-parameter tuning . In order to cope with the changes in the training dynamics of the network, several modifications to the optimization procedure have been proposed such as a linear or a square-root scaling of the learning rate with respect to the batch size growth. Other modifications include per-layer gradient scaling schemes and optimizer modifications . Several works also explored using incremented batch-sizes in order to decrease the number of training iterations required to reach the desired accuracy. Recent work by introduced the notion of "Batch Augmentation" (BA)-increasing the batch size by augmenting several instances of each sample within the same batch. BA aids generalization across a wide variety of models and tasks, with the expense of an increased computational effort per step. A similar method called "Repeated Augmentation" (RA) was proposed by. It was also demonstrated that BA may allow to decrease the number of training steps needed to achieve a similar accuracy and also mitigate I/O throughput bottlenecks . As previous works investigated mostly homogeneous training settings (e.g., using a fixed batch size), an open question still exists on the utility of rapidly varying batch-sizes. We will explore this notion and suggest a new optimizer modification that enables training with multiple varying batch-sizes with limited hyper-parameter tuning. The traditional practice of training convolutional networks using fixed-size images holds several shortcomings. First, CNNs are commonly evaluated using a different size than that used for training (; ;) and it was observed that classification accuracy may degrade above or below a certain size threshold (and Figure 1). To remedy these issues, we suggest a stochastic training regime, where image sizes can change in each optimization step. Motivation. In order to motivate our method, we first evaluate the impact of the image size on the training progress of a CNN -by examining gradient statistics during training 2. Specifically, in Table 1 we measured the correlation of the gradients across image sizes. We see that gradients computed across different scales of the same image have a strong correlation compared to those obtained across different images. This correlation is especially apparent during the first stages of training and decreases as the model converges. This suggests that the small image gradients can be used as an approximation of the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. More specifically, as the gradients of images of different size are highly correlated at the initial steps of training, it may prove beneficial to sacrifice spatial size in favor of batch size that can be increased. To do so, we suggest the following. The MixSize training regime. We suggest "MixSize", a stochastic training regime, where input sizes can vary in each optimization step. In this regime, we modify the spatial dimensions H, W (height and width) of the input image size 3, as well as the batch size. The batch size is changed either by the number of samples used, denoted B, or the number of batch-augmentations for each sample , denoted D ("duplicates"). To simplify our notation and use-cases, we will follow the common practice of training on square images and use S = H = W. Formally, in the MixSize regime, these sizes can be described as random variables sharing a single discrete distribution where ∀i: p i ≥ 0 and i p i = 1. 1.03e −6 1.44e 6.24e 1.95e 6.34e 2.26e As the computational cost of each training step is approximately proportional to S 2 ·B·D, we choose these sizes to reflect an approximately fixed budget for any choice i such that Thus the computational and memory requirements for each step are constant. Benefits and Trade-offs. We will demonstrate that using such a MixSize regime can have a positive impact on the resiliency of trained networks to the image size used at evaluation. That is, mixed-size networks will be shown to have better accuracy across a wide range of sizes. This entails a considerable saving in computations needed for inference, especially when using smaller models. Furthermore, given a fixed budget of computational and time resources (per step), we can now modify our regime along spatial and batch axes. We will explore two trade-offs: • Decrease number of iterations per epoch -by enlarging B at the expense of S. • Improve generalization per epoch -by enlarging D at the expense of S. MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size. This behavior may require hyper-parameter tuning and may also affect size-dependent layers such as batch normalization . To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe two methods we found useful: Gradient Smoothing and Batch-norm calibration. Training with varying batch and spatial sizes inadvertently leads to a change in the variance of the accumulated gradients. For example, in Table 1, the gradient variance is larger when computed over a small image size (unsurprisingly). This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training. This property was explored in previous works concerning large-batch regimes, in which a learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task or across models , with solutions such as a square-root scaling , linear scaling or a fixed norm ratio . Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling. Previous works suggested methods to control the gradient norm by gradient normalization and gradient clipping . These methods explicitly disable or limit the gradient's norm used for each optimization step, but also limit naturally occurring variations in gradient statistics. We suggest an alternative solution to previous approaches, which we refer to as "Gradient smoothing". Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training. We introduce an exponentially moving weighted average of the gradients' normḡ t (scalar) which is updated according toḡ t = αḡ t−1 + (1 − α)g t where We normalize the gradients used for each step by the smoothing coefficient, such that each consecutive step is performed with gradients of similar norm. For example, for the vanilla SGD step, we use a weight update rule of the form This running estimate of gradient norm is similar to the optimizer suggested by , which keeps a per-layer estimate of gradient moments. Gradient smoothing, however, is designed to adapt globally (across all layers) to the batch and spatial size modification and can be used regardless of the optimization method used. We found gradient smoothing to be mostly beneficial in regimes where multiple varying batch sizes are used. Figure 5a in the Appendix demonstrates how gradient smoothing reduces the gap between gradient norms of different sizes. Measuring test error on the same model shows a slight advantage for gradient-smoothing (Appendix Figure 5b). As demonstrated by , using a different image size at evaluation may incur a discrepancy between training and evaluation protocols, caused by using different data pre-processing. suggested a post-training procedure, where a network trained on a specific fixed-size is fine-tuned on another size, later used for evaluation. Their solution required 10s of training epochs, amounting to 1000s of full forward and back-propagation computations, along with parameter updates for batch-norm and classifier layers. In contrast, we surmise that for networks trained with mixed-regimes, discrepancy issues mainly arise from the use of the batch-norm layers and can be solved by targeting them specifically. Batch-norm layers introduce a discrepancy between training and test evaluations , as at inference a running estimate of the mean and variance (of training data) are used instead of the actual mean and variance values. This difference is emphasized further in the use of varying image size, as changing the spatial size of an input map can significantly modify the measured variance of that map. While a fine-tuning process per image size can eliminate this discrepancy , we offer a simpler alternative. For each evaluated size, we calibrate the mean and variance estimates used for that size by computing an average value over a small number of training examples. This calibration requires only a few (100s) feed-forward operations with no back-propagation or parameter update and takes only a few seconds on a single GPU. Interestingly, we highlight the fact that although this process has little or no effect on models trained using a fixed-size input, it does improve our mixed-size models considerably on a wide range of image sizes. CIFAR10/100. First, we examine our method using the common visual datasets CIFAR10/100 As CIFAR datasets are limited in size, we consider the following balanced stochastic regime chosen: The regime was designed to be centered around the mean value of 28. As the original image size used for training is 32 × 32, we are now able to increase either the batch size or number of duplicates for each training step by a factor of 32 2 S 2 such that S 2 ·B ·D is approximately constant. We denote our modified mixed-size regimes as B + for an increased effective batch-size and D + for an increased number of BA duplicates of the same ratio. We used our sampling strategy to train and compare our regime to the baseline . We use the original hyper-parameters without modification. For the B + regime, use our gradient smoothing method, as described in Section 4.1. For each , we measure our final test accuracy on the original 32 × 32 image size. We also perform batch-norm calibration as described in Section 4.2. From Table 2, we see that our MixSize regimes on CIFAR datasets yield two possible improvements: • Reduced number of training steps to achieve a similar test accuracy using B + regime. • Better test accuracy when using D + regime. Training progress on the CIFAR10 using ResNet44 is depicted in Figure 2. Interestingly, although designed only to reduce training time, we can see that our B + regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress. ImageNet. We also perform large scale experiments using the ImageNet dataset to confirm our findings. We used the ResNet-50 model, with the training regime suggested by that consists of base learning rate of 0.1, decreased by a factor of 10 on epochs 30, 60, 80, stopping at epoch 90. We used the base batch size of 256 over 4 devices and L 2 regularization over weights of convolutional layers. We used the standard data augmentation and did not incorporate any additional regularization or augmentation techniques. Additionally, we also used the EfficientNet-B0 model suggested by. We used the same data augmentation and regularization as the original paper, but opted for a shorter training regime with a momentum-SGD optimizer that consisted of a cosine-annealed learning rate over 200 epochs starting from an initial base 0.1 value. For the ImageNet dataset, we use the following stochastic regime found by cross-validation on several alternatives (see Appendix D): While the original training regime consisted of images of size 224×224, our proposed regime makes for an average image size ofS ×S = 144 × 144. This regime was designed so that the reduced spatial size can be used to increase the corresponding batch size or the number of BA duplicates, as described in Section 3. We are first interested in accelerating the time needed for convergence of the tested models using our B + scheme. We enlarge the batch size used for each spatial size by a factor of 224 2 S 2 such that S 2 · B is kept approximately fixed. As the average batch size is larger than B o, which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by by a factor ofB Bo. We note that for the proposed regimes we did not require any learning rate warm-up, due to the use of gradient smoothing. As can be seen in Figure 3, regime B + enables training with approximately 2.7× less training steps, while reaching a better-than-baseline accuracy of 76.61%. As sizes were chosen to reflect in approximately equal computational cost per iteration, B + regime offers a similar improvement in total wall-clock time. Next, we perform a similar experiment with a D + regime, where the number of BA duplicates is similarly increased with respect to D o instead of the batch size. This scaling with an average duplicates ofD = 3. As the computational cost for each step remains approximately constant, as well as the number of required steps per epochs, training a model under this regime requires an equal wall-clock time. However, the increased batch-augmentation improves the final test accuracy to 78.04%, approximately 7% relative improvement over the 76.4% baseline. Next, we examine how MixSize affects the ing model resiliency to changes in the image size during test-time. We evaluated the models by varying the test-time image sizes around the original 224 spatial size: S = 224 + 32 · m, m ∈ {−6, ..., 6}. The common evaluation procedure for ImageNet models first scales the image to a 256 smallest dimension and crops a center 224 × 224 image. We adapt this regime for other image sizes by scaling the smallest dimension to 8 7 S (since 8 7 · 224 = 256) and then cropping the center S × S patch. Models trained with a mixed regime were calibrated to a specific evaluation size by measuring batch-norm statistics for 200 batches of training samples. We note that for original fixed-size regimes this calibration procedure ed with degraded and so we report accuracy without calibration for these models. We did not use any fine-tuning procedure post training for any of the models. As can be seen in Figure 4a, the baseline model trained using a fixed size, reaches 76.4% top-1 accuracy at the same 224 spatial size it was trained on. As observed previously, the model continues to slightly improve beyond that size, to a maximum of 76.8% accuracy. However, it is apparent that the model's performance quickly degrades when evaluating with sizes smaller than 224. We compare these with a D + regime, trained with an average size ofS = 144. As described earlier, this model requires the same time and computational resources as the baseline model. However, due to the decreased average size, we were able to leverage more than 1 duplicates per batch on average, which improved the model's top-1 accuracy to 77.14% at size 224. Furthermore, we find that the model performs much more favorably at image sizes smaller than 224, scoring an improved (over baseline) accuracy of 76.43% at only 160×160 spatial size. We analyzed an alternative regime S, where the average spatial size is larger at 208 × 208 (for more details see Appendix D). The model trained with the S regime offers a similar improvement in accuracy, only across a larger spatial size, as it observed an average size of 208 × 208 during training. Figure 4a demonstrates that while all three models (Fixed with S = 224, S and S ) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes. Specifically, mixed-regime at S = 208 dominates the baseline fixed-size regime at all sizes, while our mixed regime at S = 144 achieves best at sizes smaller than 224. We also compared the classification performance across evaluated image sizes, using networks trained on a variety of fixed sizes and our mixed regimes. As a baseline, we use obtained by (trained with repeated augmentations, without fine-tuning) and compare them with mixed-regime models trained with an equal computational budget, by setting the base number of BA duplicates to D = 2. As can be seen in Figure 4b, mixed-regime trained models offer a wider range of resolutions with close-to-baseline accuracy (within a 2% change) and perform better than their fixed-size counterparts at all sizes. As the number of floating-point operations (flops) grows linearly with the number of pixels, using a mixed regime significantly improves accuracy per compute at evaluation. We further note that our S model reaches a top accuracy of 79.27% at a 288 × 288 evaluation size. In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation (duplicates) in the batch. Stochastic regime benefits are threefold: reduced number of training iterations; or improved model accuracy (generalization) and improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better , as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases. + regime creates two batch sizes: 256 and 2, 048 respectively. Gradient smoothing helps to reduce gap between gradient norms at difference batch sizes and improves final accuracy. We wish to consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension H, we wish to obtain an average size ofH = i p i H i such thatH < H o. We consider three alternatives for image size variations: • Increase image size from small to large, where each image size is used for number of epochs E i = p i E total, where E total is the total number training epochs required. • Using a random image size for each epoch, keeping the epoch number for each size at E i • Sampling image size per training step at probability p i As can be seen in Figure 6, we found that random sampling regimes performed better than scaling image size from small to large . While sampling both at epoch and step time frames performed similarly, replacing sizes on each step seemed to converge faster and to have less noise in measured test accuracy. We note that these behaviours may partly stem from the use of batch-normalization which is sensitive to the image size used at evaluation or insufficient hyper-parameter tuning for each specific size (e.g., spiking error at the end of the small-to-large regime). Considering these findings, we continue to perform our experiments using the third regime -sampling image size per training step. We used alternative size regimes balanced around 224, named S and S. They can be described by the following distributions:

Represent the short summary for finding the corresponding scientific article (long)

Training convnets with mixed image size can improve results across multiple sizes at evaluation

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Deep neural networks trained with data augmentation do not require any other explicit regularization (such as weight decay and dropout) and exhibit greater adaptaibility to changes in the architecture and the amount of training data.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We propose a simple technique for encouraging generative RNNs to plan ahead. We train a ``backward'' recurrent network to generate a given sequence in reverse order, and we encourage states of the forward model to predict cotemporal states of the backward model. The backward network is used only during training, and plays no role during sampling or inference. We hypothesize that our approach eases modeling of long-term dependencies by implicitly forcing the forward states to hold information about the longer-term future (as contained in the backward states). We show empirically that our approach achieves 9% relative improvement for a speech recognition task, and achieves significant improvement on a COCO caption generation task. Recurrent Neural Networks (RNNs) are the basis of state-of-art models for generating sequential data such as text and speech. RNNs are trained to generate sequences by predicting one output at a time given all previous ones, and excel at the task through their capacity to remember past information well beyond classical n-gram models BID6 BID27. More recently, RNNs have also found success when applied to conditional generation tasks such as speech-to-text BID9, image captioning BID61 and machine translation.RNNs are usually trained by teacher forcing: at each point in a given sequence, the RNN is optimized to predict the next token given all preceding tokens. This corresponds to optimizing one-stepahead prediction. As there is no explicit bias toward planning in the training objective, the model may prefer to focus on the most recent tokens instead of capturing subtle long-term dependencies that could contribute to global coherence. Local correlations are usually stronger than long-term dependencies and thus end up dominating the learning signal. The consequence is that samples from RNNs tend to exhibit local coherence but lack meaningful global structure. This difficulty in capturing long-term dependencies has been noted and discussed in several seminal works (; BID6 BID27 BID45 .Recent efforts to address this problem have involved augmenting RNNs with external memory BID14 BID18 BID22, with unitary or hierarchical architectures BID0 BID51, or with explicit planning mechanisms BID23 . Parallel efforts aim to prevent overfitting on strong local correlations by regularizing the states of the network, by applying dropout or penalizing various statistics BID41 BID64 BID15 BID32 BID39 . Figure 1: The forward and the backward networks predict the sequence s = {x 1, ..., x 4} independently. The penalty matches the forward (or a parametric function of the forward) and the backward hidden states. The forward network receives the gradient signal from the log-likelihood objective as well as L t between states that predict the same token. The backward network is trained only by maximizing the data log-likelihood. During the evaluation part of the network colored with orange is discarded. The cost L t is either a Euclidean distance or a learned metric ||g(h DISPLAYFORM0 with an affine transformation g. Best viewed in color. In this paper, we propose TwinNet, 1 a simple method for regularizing a recurrent neural network that encourages modeling those aspects of the past that are predictive of the long-term future. Succinctly, this is achieved as follows: in parallel to the standard forward RNN, we run a "twin" backward RNN (with no parameter sharing) that predicts the sequence in reverse, and we encourage the hidden state of the forward network to be close to that of the backward network used to predict the same token. Intuitively, this forces the forward network to focus on the past information that is useful to predicting a specific token and that is also present in and useful to the backward network, coming from the future (Fig. 1).In practice, our model introduces a regularization term to the training loss. This is distinct from other regularization methods that act on the hidden states either by injecting noise BID32 or by penalizing their norm BID31 BID39, because we formulate explicit auxiliary targets for the forward hidden states: namely, the backward hidden states. The activation regularizer (AR) proposed by BID39, which penalizes the norm of the hidden states, is equivalent to the TwinNet approach with the backward states set to zero. Overall, our model is driven by the intuition (a) that the backward hidden states contain a summary of the future of the sequence, and (b) that in order to predict the future more accurately, the model will have to form a better representation of the past. We demonstrate the effectiveness of the TwinNet approach experimentally, through several conditional and unconditional generation tasks that include speech recognition, image captioning, language modelling, and sequential image generation. To summarize, the contributions of this work are as follows:• We introduce a simple method for training generative recurrent networks that regularizes the hidden states of the network to anticipate future states (see Section 2);• The paper provides extensive evaluation of the proposed model on multiple tasks and concludes that it helps training and regularization for conditioned generation (speech recognition, image captioning) and for the unconditioned case (sequential MNIST, language modelling, see Section 4);• For deeper analysis we visualize the introduced cost and observe that it negatively correlates with the word frequency (more surprising words have higher cost). Given a dataset of sequences S = {s 1, . . ., s n}, where each s k = {x 1, . . ., x T k} is an observed sequence of inputs x i ∈ X, we wish to estimate a density p(s) by maximizing the log-likelihood of the observed data L = n i=1 log p(s i). Using the chain rule, the joint probability over a sequence x 1,..., x T decomposes as: DISPLAYFORM0 This particular decomposition of the joint probability has been widely used in language modeling BID7 BID40 and speech recognition BID5. A recurrent neural network is a powerful architecture for approximating this conditional probability. At each step, the RNN updates a hidden state h f t, which iteratively summarizes the inputs seen up to time t: h DISPLAYFORM1 where f symbolizes that the network reads the sequence in the forward direction, and Φ f is typically a non-linear function, such as a LSTM cell BID27 or a GRU. Thus, h f t forms a representation summarizing information about the sequence's past. The prediction of the next symbol x t is performed using another non-linear transformation on top of h DISPLAYFORM2, which is typically a linear or affine transformation (followed by a softmax when x t is a symbol). The basic idea of our approach is to encourage h f t to contain information that is useful to predict x t and which is also compatible with the upcoming (future) inputs in the sequence. To achieve this, we run a twin recurrent network that predicts the sequence in reverse and further require the hidden states of the forward and the backward networks to be close. The backward network updates its hidden state according to: DISPLAYFORM3 and predicts DISPLAYFORM4 using information only about the future of the sequence. Thus, h f t and h b t both contain useful information for predicting x t, coming respectively from the past and future. Our idea consists in penalizing the distance between forward and backward hidden states leading to the same prediction. For this we use the Euclidean distance (see Fig. 1): DISPLAYFORM5 where the dependence on x is implicit in the definition of h f t and h b t. The function g adds further capacity to the model and comes from the class of parameterized affine transformations. Note that this class includes the identity tranformation. As we will show experimentally in Section 4, a learned affine transformation gives more flexibility to the model and leads to better . This relaxes the strict match between forward and backward states, requiring just that the forward hidden states are predictive of the backward hidden states. The total objective maximized by our model for a sequence s is a weighted sum of the forward and backward log-likelihoods minus the penalty term, computed at each time-step: DISPLAYFORM0 where α is an hyper-parameter controlling the importance of the penalty term. In order to provide a more stable learning signal to the forward network, we only propagate the gradient of the penalty term through the forward network. That is, we avoid co-adaptation of the backward and forward networks. During sampling and evaluation, we discard the backward network. The proposed method can be easily extended to the conditional generation case. The forward hiddenstate transition is modified to h DISPLAYFORM1 where c denotes the task-dependent conditioning information, and similarly for the backward RNN.Bidirectional neural networks BID49 have been used as powerful feature extractors for sequence tasks. The hidden state at each time step includes both information from the past and the future. For this reason, they usually act as better feature extractors than the unidirectional counterpart and have been successfully used in a myriad of tasks, e.g. in machine translation, question answering BID10 and sequence labeling BID37. However, it is not straightforward to apply these models to sequence generation BID65 due to the fact that the ancestral sampling process is not allowed to look into the future. In this paper, the backward model is used to regularize the hidden states of the forward model and thus is only used during training. Both inference and sampling are strictly equivalent to the unidirectional case. Gated architectures such as LSTMs BID27 and GRUs BID13 have been successful in easing the modeling of long term-dependencies: the gates indicate time-steps for which the network is allowed to keep new information in the memory or forget stored information. BID20; Dieng et al. FORMULA1; BID18 effectively augment the memory of the network by means of an external memory. Another solution for capturing long-term dependencies and avoiding gradient vanishing problems is equipping existing architectures with a hierarchical structure BID51. Other works tackled the vanishing gradient problem by making the recurrent dynamics unitary BID0. In parallel, inspired by recent advances in "learning to plan" for reinforcement learning BID52 BID55, recent efforts try to augment RNNs with an explicit planning mechanism BID23 to force the network to commit to a plan while generating, or to make hidden states predictive of the far future BID34.Regularization methods such as noise injection are also useful to shape the learning dynamics and overcome local correlations to take over the learning process. One of the most popular methods for neural network regularization is dropout BID53. Dropout in RNNs has been proposed in BID41, and was later extended in BID50 BID15, where recurrent connections are dropped at random. Zoneout BID32 modifies the hidden state to regularize the network by effectively creating an ensemble of different length recurrent networks. BID31 introduce a "norm stabilization" regularization term that ensures that the consecutive hidden states of an RNN have similar Euclidean norm. Recently, BID39 proposed a set of regularization methods that achieve state-of-the-art on the Penn Treebank language modeling dataset. Other RNN regularization methods include the weight noise BID19, gradient clipping BID45 and gradient noise BID42. We now present experiments on conditional and unconditional sequence generation, and analyze the in an effort to understand the performance gains of TwinNet. First, we examine conditional generation tasks such as speech recognition and image captioning, where the show clear improvements over the baseline and other regularization methods. Next, we explore unconditional language generation, where we find our model does not significantly improve on the baseline. Finally, to further determine what tasks the model is well-suited to, we analyze a sequential imputation task, where we can vary the task from unconditional to strongly conditional. We evaluated our approach on the conditional generation for character-level speech recognition, where the model is trained to convert the speech audio signal to the sequence of characters. The forward and backward RNNs are trained as conditional generative models with softattention. The context information c is an encoding of the audio sequence and the output sequence s is the corresponding character sequence. We evaluate our model on the Wall Street Journal (WSJ) dataset closely following the setting described in BID4. We use 40 mel-filter bank features with delta and delta-deltas with their energies as the acoustic in- We compare the attention model for speech recognition ("Baseline," BID4 ; the regularizer proposed by BID31 ("Stabilizing norm"); penalty on the L2 norm of the forward states BID39 ) ("AR"), which is equivalent to TwinNet when all the hidden states of the backward network are set to zero. We report the of our model ("TwinNet") both with g = I, the identity mapping, and with a learned g. Test CER Valid CER Baseline 6.8 9.0 Baseline + Gaussian noise 6.9 9.1 Baseline + Stabilizing Norm 6.6 9.0 Baseline + AR 6.5 8.9 Baseline + TwinNet (g = I)6.6 8.7 Baseline + TwinNet (learnt g) 6.2 8.4puts to the model, these features are generated according to the Kaldi s5 recipe BID46. The ing input feature dimension is 123.We observe the Character Error Rate (CER) for our validation set, and we early stop on the best CER observed so far. We report CER for both our validation and test sets. For all our models and the baseline, we follow the setup in BID4 and pretrain the model for 1 epoch, within this period, the context window is only allowed to move forward. We then perform 10 epochs of training, where the context window looks freely along the time axis of the encoded sequence, we also perform annealing on the models with 2 different learning rates and 3 epochs for each annealing stage. We use the AdaDelta optimizer for training. We perform a small hyper-parameter search on the weight α of our twin loss, α ∈ {2.0, 1.5, 1.0, 0.5, 0.25, 0.1}, and select the best one according to the CER on the validation set. Results We summarize our findings in TAB0. Our best performing model shows relative improvement of 12% comparing to the baseline. We found that the TwinNet with a learned metric (learnt g) is more effective than strictly matching forward and hidden states. In order to gain insights on whether the empirical usefulness comes from using a backward recurrent network, we propose two ablation tests. For "Gaussian Noise," the backward states are randomly sampled from a Gaussian distribution, therefore the forward states are trained to predict white noise. For "AR," the backward states are set to zero, which is equivalent to penalizing the norm of the forward hidden states BID39. Finally, we compare the model with the "Stabilizing Norm" regularizer BID31, that penalizes the difference of the norm of consecutive forward hidden states. Results shows that the information included in the backward states is indeed useful for obtaining a significant improvement. Analysis The training/validation curve comparison for the baseline and our network is presented in FIG1. 4 The TwinNet converges faster than the baseline and generalizes better. The L2 cost raises in the beginning as the forward and backward network start to learn independently. Later, due to the pressure of this cost, networks produce more aligned hidden representations. FIG2 provides examples of utterances with L2 plotted along the time axis. We observe that the high entropy words produce spikes in the loss for such words as "uzi." This is the case for rare words which are hard to predict from the acoustic information. To elaborate on this, we plot the L2 cost averaged over a word depending on the word frequency. The average distance decreases with the increasing frequency. The histogram comparison FIG1 ) for the cost of rare and frequent words reveal that the not only the average cost is lower for frequent words, but the variance is higher for rare words. Additionally, we plot the dependency of the L2 cost cross-entropy cost of the forward network FIG1 ) to show that the conditioning also plays the role in the entropy of the output, the losses are not absolutely correlated. We evaluate our model on the conditional generation task of image captioning task on Microsoft COCO dataset BID35. The MS COCO dataset covers 82,783 training images and 40,504 images for validation. Due to the lack of standardized split of training, validation and test data, we follow Karpathy's split BID28 BID61. These are 80,000 training images and 5,000 images for validation and test. We do early stopping based on the validation CIDEr scores and we report BLEU-1 to BLEU-4, CIDEr, and Meteor scores. To evaluate the consistency of our method, we tested TwinNet on both encoder-decoder ('Show&Tell', BID59 and soft attention ('Show, Attend and Tell', BID61 image captioning models. We use a Resnet BID25 with 101 and 152 layers pre-trained on ImageNet for image classification. The last layer of the Resned is used to extract 2048 dimensional input features for the attention model BID61 . We use an LSTM with 512 hidden units for both "Show & Tell" and soft attention. Both models are trained with the Adam BID29 optimizer with a Table 2 : Results for image captioning on the MS COCO dataset, the higher the better for all metrics (BLEU 1 to 4, METEOR, and CIDEr). We reimplement both Show&Tell BID59 and Soft Attention BID61 in order to add the twin cost. We use two types of images features extracted either with Resnet-101 or Resnet-152. DISPLAYFORM0 DeepVS BID28 62.5 45.0 32.1 23.0 19.5 66.0 ATT-FCN BID63 70.9 53.7 40.2 30.4 24.3 -Show & Tell BID59 ---27.7 23.7 85.5 Soft Attention BID61 70.7 49.2 34.4 24.3 23.9 -Hard Attention BID61 71.8 50.4 35.7 25.0 23.0 -MSM BID62 73.0 56.5 42.9 32.5 25.1 98.6 Adaptive Attention BID36 74 Table 3: (left) Test set negative log-likelihood for binarized sequential MNIST, where denotes lower performance of our model with respect to the baselines.(right) Perplexity on WikiText-2 and Penn Treebank BID39.AWD-LSTM refers to the model of BID39 trained with the official implementation at http://github.com/salesforce/awd-lstm/.Model MNIST DBN 2hl BID16 ≈84.55 NADE BID57 88.33 EoNADE-5 2hl BID47 84.68 DLGM 8 BID48 ≈85.51 DARN 1hl ≈84.13 DRAW ≤80.97 P-Forcing (3-layer) BID33 79.58 PixelRNN(1-layer) BID44 80.75 PixelRNN(7-layer) BID44 79.20 PixelVAE BID24 79.02 MatNets BID1 78.50Baseline LSTM(3-layers) 79.87 + TwinNet(3-layers) Baseline LSTM(3-layers) + dropout 79.59 + TwinNet(3-layers)79.12 LSTM BID64 82.2 78.4 4-layer LSTM BID38 67.9 65.4 5-layer RHN BID38 64 BID38 78.1 75.6 1-layer LSTM BID38 69.3 65.9 2-layer LSTM BID38 69.1 65.9AWD-LSTM 68.7 65.8 + TwinNet 68.0 64.9 learning rate of 10 −4. TwinNet showed consistent improvements over "Show & Tell" (Table 2). For the soft attention model we observe small but consistent improvements for majority of scores. We investigate the performance of our model in pixel-by-pixel generation for sequential MNIST. We follow the setting described by BID33: we use an LSTM with 3-layers of 512 hidden units for both forward and backward LSTMs, batch size 20, learning rate 0.001 and clip the gradient norms to 5. We use Adam BID29 as our optimization algorithm and we decay the learning rate by half after 5, 10, and 15 epochs. Our are reported at the Table 3 (left). Our baseline LSTM implementation achieves 79.87 nats on the test set. We observe that by adding the TwinNet regularization cost consistently improves performance in this setting by about 0.52 nats. Adding dropout to the baseline LSTM is beneficial. Further gains were observed by adding both dropout and the TwinNet regularization cost. This last model achieves 79.12 nats on test set. Note that this is competitive with deeper models such as PixelRNN (b) (7-layers) and PixelVAE BID24 which uses an autoregressive decoder coupled with a deep stochastic auto-encoder. As a last experiment, we report obtained on a language modelling task using the PennTree Bank and WikiText-2 datasets BID39. We augment the state-of-the-art AWD-LSTM model BID39 with the proposed TwinNet regularization cost. The are reported in Table 3 (right). In this paper, we presented a simple recurrent neural network model that has two separate networks running in opposite directions during training. Our model is motivated by the fact that states of the forward model should be predictive of the entire future sequence. This may be hard to obtain by optimizing one-step ahead predictions. The backward path is discarded during the sampling and evaluation process, which makes the sampling process efficient. Empirical show that the proposed method performs well on conditional generation for several tasks. The analysis reveals an interpretable behaviour of the proposed loss. One of the shortcomings of the proposed approach is that the training process doubles the computation needed for the baseline (due to the backward network training). However, since the backward network is discarded during sampling, the sampling or inference process has the exact same computation steps as the baseline. This makes our approach applicable to models that requires expensive sampling steps, such as PixelRNNs BID44 and WaveNet (a). One of future work directions is to test whether it could help in conditional speech synthesis using WaveNet. We observed that the proposed approach yield minor improvements when applied to language modelling with PennTree bank. We hypothesize that this may be linked to the amount of entropy of the target distribution. In these high-entropy cases, at any time-step in the sequence, the distribution of backward states may be highly multi-modal (many possible futures may be equally likely for the same past). One way of overcoming this problem would be to replace the proposed L2 loss (which implicitly assumes a unimodal distribution of the backward states) by a more expressive loss obtained by either employing an inference network BID30 or distribution matching techniques BID17. We leave that for future investigation.

The paper introduces a method of training generative recurrent networks that helps to plan ahead. We run a second RNN in a reverse direction and make a soft constraint between cotemporal forward and backward states.

Represent the TLDR for finding the corresponding scientific article

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work we propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition. This provides a way to efficiently learn a more accurate generative model of real-world images, and serves as an initial step towards learning corresponding object representations. We evaluate our approach on several multi-object image datasets, and find that the generator learns to identify and disentangle information corresponding to different objects at a representational level. A human study reveals that the ing generative model is better at generating images that are more faithful to the reference distribution. Generative modelling approaches to representation learning seek to recover the process with which the observed data was generated. It is postulated that knowledge about the generative process exposes important factors of variation in the environment (captured in terms of latent variables) that may subsequently be obtained using an appropriate posterior inference procedure. Therefore, the structure of the generative model is critical in learning corresponding representations. Deep generative models of images rely on the expressiveness of neural networks to learn the generative process directly from data BID11 BID24 BID38. Their structure is determined by the inductive bias of the neural network, which steers it to organize its computation in a way that allows salient features to be recovered and ultimately captured in a representation BID6 BID7 BID24. Recently, it has been shown that independent factors of variation, such as pose and lighting of human faces may be recovered in this way BID5.A promising but under-explored inductive bias in deep generative models of images is compositionality at the representational level of objects, which accounts for the compositional nature of the visual world and our perception thereof BID3 BID37. It allows a generative model to describe a scene as a composition of objects (entities), thereby disentangling visual information in the scene that can be processed largely independent of one another. It provides a means to efficiently learn a more accurate generative model of real-world images, and by explicitly Figure 1: A scene (right) is generated as a composition of objects and . considering objects at a representational level, it serves as an important first step in recovering corresponding object representations. In this work we investigate object compositionality for Generative Adversarial Networks (GANs; BID11), and present a general mechanism that allows one to incorporate corresponding structure in the generator. Starting from strong independence assumptions about the objects in images, we propose two extensions that provide a means to incorporate dependencies among objects and . In order to efficiently represent and process multiple objects with neural networks, we must account for the binding problem that arises when superimposing multiple distributed representations BID18. Following prior work, we consider different representational slots for each object BID13 BID34, and a relational mechanism that preserves this separation accordingly.We evaluate our approach 1 on several multi-object image datasets, including three variations of Multi-MNIST, a multi-object variation of CIFAR10, and CLEVR. In particular the latter two mark a significant improvement in terms of complexity, compared to datasets that have been considered in prior work on unconditional multi-object image generation and multi-object representation learning. In our experiments we find that our generative model learns about the individual objects and the of a scene, without prior access to this information. By disentangling this information at a representational level, it generates novel scenes efficiently through composing individual objects and , as can be seen in Figure 1. As a quantitative experiment we compare to a strong baseline of popular GANs (Wasserstein and Non-saturating) with recent state-of-the-art techniques (Spectral Normalization, Gradient Penalty) optimized over multiple runs. A human study reveals that the proposed generative model outperforms this baseline in generating better images that are more faithful to the reference distribution. Generative Adversarial Networks (GANs; BID11) are a powerful class of generative models that learn a stochastic procedure to generate samples from a distribution P (X). Traditionally GANs consist of two deterministic functions: a generator G(z) and a discriminator (or critic) D(x). The goal is to find a generator that accurately transforms samples from a prior distribution z ∼ P (Z) to match samples from the target distribution x ∼ P (X). This can be done by using the discriminator to implement a suitable objective for the generator, in which it should behave adversarial with respect to the goal of the discriminator in determining whether samples x were sampled from P (X) or G(P (Z)) respectively. These objectives can be summarized as a minimax game with the following value function: DISPLAYFORM0 When the generator and the discriminator are implemented with neural networks, optimization may proceed through alternating (stochastic) gradient descent updates of their parameters with respect to. However, in practice this procedure might be unstable and the minimax formulation is known to be hard to optimize. Many alternative formulations have been proposed and we refer the reader to BID30 and BID26 for a comparison. Following the recommendations of BID26 we consider two practical reformulations of in this paper: Non-Saturating GAN (NS-GAN; BID11), in which the generator maximizes the probability of generated samples being real, and Wassertein GAN (WGAN;) in which the discriminator minimizes the Wasserstein distance between G(P (Z)) and P (X). For both formulations we explore two additional techniques that have proven to work best on a variety of datasets and architectures: the gradient penalty from BID15 to regularize the discriminator, and spectral normalization BID33 to normalize its gradient flow. We propose to structure the generator of a GAN to generate images as compositions of individual objects and . In this case it consists of K = 4 object generators (shared weights) that each generate an image from a separate latent vectorẑ i. These are obtained by having each z i ∼ P (Z) participate in a relational stage, which allows each representation to be updated as a function of all others. Alternativelyẑ i = z i if no relations are to be modelled. On the top, a generator (unique weights) generates a image from a separate latent vector DISPLAYFORM1, which optionally participates in the relational stage. The whole system is trained end-to-end as in the standard GAN framework, and the final image is obtained by composing (in this case using alpha compositing) the outputs of all generators. In order to formulate the structure required to achieve object compositionality in neural networks we primarily focus on the corresponding type of generalization behavior that we are interested in. It is concerned with independently varying the different visual primitives (objects) that an image is composed of, requiring these to be identified at a representational level and described in a common format. We account for the binding problem BID18 BID32 BID41 ) that may arise in combining these object representations to arrive at a final image. In the following subsections we initially present structure that assumes strict object independence (Section 3.1), to then relax this assumption by incorporating relational structure (Section 3.2), and finally allow for the possibility of unstructured and occlusion (Section 3.3). If we assume that images in P (X) are composed of objects that are strictly independent of one another then (without loss of generality) we may structure our latent variables accordingly. For images having K objects, we consider K i.i.d. vector-valued random variables Z i that each describe an object at a representational level. K copies of a deterministic generator G(z) transform samples from each Z i into images, such that their superposition in the corresponding scene: DISPLAYFORM0 When each copy of G generates an image of a single object, the ing generative model efficiently describes images in P (X) in a compositional manner. Each object in is described in terms of the same features (i.e. the Z i 's are i.i.d) and the weights among the generators are shared, such that any acquired knowledge in generating a specific object is transferred across all others. Hence, rather than having to learn about all combinations of objects (including their individual variations) that may appear in an image, it suffices to learn about the different variations of each individual object instead. Notice that the generators in cannot communicate, which prevents degenerate solutions from being learned. This comes at a cost in that relations among the objects cannot be modelled in this way. An additional concern is the sum in, which assumes that images only consist of objects, and that their values can be summed in pixel-space. We will address these concerns in the following, using the superposition of generators as a backbone for object compositionality in our approach. In the real world objects are not strictly independent of one another. Certain objects may only occur in the presence of others, or affect their visual appearance in subtle ways (eg. shadows, lighting). In order to incorporate relationships of this kind we introduce a relational stage, in which the representation of an object is updated as a function of all others, before each generator proceeds to generate its image. Following we consider one or more "attention blocks" to compute interactions among the object representations. At its core is Multi-Head Dot-Product Attention (MHDPA; BID40) that performs non-local computation or message-passing BID10 when one associates each object representation with a node in a graph. When specific design choices are made, computation of this kind provides an efficient means to learn about relations between objects and update their representations accordingly BID4.A single head of an attention block updates z i in the following way: DISPLAYFORM0 attention weights DISPLAYFORM1 where d = dim(v i) and each MLP corresponds to a multi-layer perceptron. First, a query vector q i, a value vector v i, and a key vector k i is computed for each z i. Next, the interaction of an object i with all other objects (including itself) is computed as a weighted sum of their value vectors. Weights are determined by computing dot-products between q i and all key vectors, followed by softmax normalization. Finally, the ing update vector a i is projected back to the original size of z i using MLP up before being added. Additional heads (modelling different interactions) use different parameters for each MLP in. In this case their outputs are combined with another MLP to arrive at a final z i. Complex relationships among objects can be modelled by using multiple attention blocks to iteratively update z i. A detailed overview of these computations can be found in Appendix B, and an overview in FIG0. Up until this point we have assumed that an image is entirely composed of objects, which may be prohibitive for complex visual scenes. For example, certain objects that only appear in the "" may not occur frequently enough, nor have a regular visual appearance that allows a model to consider them as such. One could reason that certain visual primitives (those that can be varied independently and re-composed accordingly) will be discovered from the observed data, whereas all other remaining visual information is captured as by another component. However, these are conflicting assumptions as the latent representations z i (and corresponding generator) now need to describe objects that assume a regular visual appearance, as well as that is not regular in its visual appearance at all. Therefore, we consider an additional generator (see FIG0) having its own set of weights to generate the from a separate vector of latent variables z b ∼ P (Z b). We consider two different variations of this addition, one in which z b participates in the relational stage, and one in which it does not. A remaining challenge is in combining objects with and occlusion. A straightforward adaptation of the sum in to incorporate pixel-level weights would require the generator to assign a weight of zero to all pixel locations where objects appear, thereby increasing the complexity of generating the exponentially. Instead, we require the object generators to generate an additional alpha channel for each pixel, and use alpha compositing to combine the outputs of the different generators and through repeated application of: DISPLAYFORM0 4 RELATED WORK Inductive biases aimed at object compositionality have been previously explored, both in the context of generative models and multi-object representation learning. One line of work models an image as a spatial mixture of image patches, utilizing multiple copies of the same function to arrive at a compositional solution. Different implementations consider RBMs , VAEs BID34, or (recurrent) auto-encoders inspired by EM-like inference procedures BID12 to generate these patches. They consider objects at a representational level and recent work has shown a means to efficiently model interactions between them (van BID39 . However, neither of these approaches are capable of modelling complex visual scenes that incorporate unstructured as well as interactions among objects. A conceptually different line of work relies on recurrent neural networks to iteratively model multiple objects in an image, one at a time. BID14 proposes to use attention to arrive at this solution, whereas Eslami et al. FORMULA0 considers objects explicitly. This approach has also been explored in the context of GANs. Im et al. FORMULA0 generates images iteratively by accumulating outputs of a recurrent generator, and BID27 propose to combine these outputs using alpha compositing. BID44 extends this approach further, by considering a separate generator for the , using spatial transformations to integrate a foreground image. They briefly explore multi-object image generation on a dataset consisting of two non-overlapping MNIST digits, yet their approach requires prior knowledge about the size of the objects, and the number of objects to generate. This is information typically unavailable in the real world and not required for our method. A more general concern is the difficulty in modelling relations among objects when they are generated one at a time. Information about the objects must be stored and updated in the memory of the RNN, which is ill-suited for this task without incorporating corresponding relational structure . It prevents relations from being learned efficiently, and requires the RNN to commit to a plan in its first step, without the possibility to revisit this decision. Recent work in GANs is increasingly focusing on incorporating (domain-specific) architectural structure in the generator to generate realistic images. BID29 considers a Spatial Transformer Network as a generator, and proposes an iterative scheme to remove or add objects to a scene. BID22 propose image generation by conditioning on explicit scene graphs to overcome the limitations of standard GANs in generating scenes composed of multiple objects that require relations to be taken into account. BID43 propose a similar approach but condition on a stochastic and-or graph instead. BID1 considers a framework to generate images composed of two objects, conditioned on images of each single object. In our approach we make use of an implicit graph structure (as implemented by our relational mechanism) to model relations among objects, and do not rely on prior information about individual objects (in the form of conditioning). We test different aspects of the proposed structure on several multi-object datasets. We are particularly interested in verifying that images are generated as compositions of objects and that the relational and structure is properly utilized. To that extend, we study how the incorporated structure affects the quality and the content of generated images. Datasets We consider five multi-object datasets. 3 The first three are different variations of Multi-MNIST (MM), in which each image consists of three MNIST digits that were rescaled and drawn randomly onto a 64 × 64 canvas. In Independent MM, digits are chosen randomly and there is no relation among them. The Triplet variation requires that all digits in an image are of the same type, requiring relations among the digits to be considered during the generative process. Similarly RGB Occluded MM requires that each image consist of exactly one red, green, and blue digit. The fourth dataset (CIFAR10 + MM) is a variation of CIFAR10 BID25 in which the digits from RGB Occluded MM are drawn onto a randomly choosen (resized) CIFAR10 image. Our final dataset is CLEVR BID21, which we downsample to 160 × 240 followed by center-cropping to obtain 128 × 128 images. Samples from each dataset can be seen in in Appendix C. Evaluation A popular evaluation metric in comparing generated images by GANs is the Fréchet Inception Distance (FID; BID16). It computes the distance between two empirical distributions of images (one generated, and a reference set) as the Fréchet (Wasserstein-2) distance between two corresponding multivariate Gaussian distributions that were estimated from the Inception-features computed for each image. Although pior work found that FID correlates well with perceived human quality of images on standard image datasets BID16; BID30, we find that FID is of limited use when considering image datasets in which the dominant visual aspects are determined by multiple objects. Our in Section 5.2 suggest that FID can not be used to verify whether image distributions adhere to certain properties, such as the number of objects. We hypothesize that this inability is inherent to the Inception embedding having been trained only for single object classification. 0 Generator 1 Generator 2 Composed Image Generator 0 Generator 1 Generator 2 Composed Image Generator 0 Generator 1 Generator 2 Composed ImageTo compensate for this we conduct two different studies among humans, 1) to compare images generated by our models to a baseline, and 2) to answer questions about the content of generated images. The latter allows us to verify whether generated images are probable samples from our image distribution, eg. by verifying that they have the correct number of objects. As conducting human evaluation of this kind is not feasible for large-scale hyperparameter search we will continue to rely on FID to select the "best" models during hyper-parameter selection. Details of these human studies can be found in Appendix B.Set-up Each model is optimized with ADAM using a learning rate of 10 −4, and batch size 64 for 1M steps. We compute the FID (using 10K samples) every 20K steps, and select the best set of parameters accordingly. On each dataset, we compare GANs that incorporate our proposed structure to a strong baseline that does not. In both cases we conduct extensive grid searches covering on the order of 40-50 hyperparameter configurations for each dataset, using ranges that were previously found good for GAN BID30 BID26. Each configuration is ran with 5 different seeds to be able to estimate its variance. An overview of each hyper-parameter search can be found in Appendix B, and samples of our best models in Appendix C.Composing On Independent MM and Triplet MM we sum the outputs of the object generators as in, followed by clipping. On all other datasets we use alpha compositing with a fixed order. In this case the object generators output an additional alpha channel, except for RGB Occluded MM in which we obtain alpha values by thresholding the output of each object generator for simplicity. Notation In reporting our we will break down the obtained when incorporating structure in GAN across the different structural parts. In particular we will denote k-GAN to describe a generator consisting of K = k components, k-GAN rel. if it incorporates relational structure and k-GAN ind. if it does not. Additionally we will append "bg." when the model includes a separate generator. Since any variation incorporates multiple components, we will use k-GAN to refer to GANs that incorporate any of the proposed structure as a collective. We will use GAN to refer to the collection of GANs with different hyperparameters in our baseline. Utilizing Structure In analyzing the output of each generator for k-GAN, we consistently find that the final image is generated as a composition of images consisting of individual objects and In the case of CLEVR, in which images may have a greater number of objects than the number of components K that we used during training, we find that the generator continues to learn a factored solution. Visual primitives are now made up of multiple objects, examples of which can be seen at the bottom rows in FIG2. A similar tendency was also found when analyzing generated images by k-GAN ind. when k > 3 on Multi-MNIST. The generator decodes part of its latent space as "no digit" as an attempt at generating the correct number of digits. Generator 0 Generator 1 Generator 2 Generator 3 Generator 4 Generator 5 Composed Image Generator 0 Generator 1 Generator 2 Generator 3 Generator 4 Generator 5 Composed Image Generator 0 Generator 1 Generator 2 Generator 3 Generator 4 Generator 5 Composed Image Generator 0 Generator 1 Generator 2 Generator 3 Generator 4 Generator 5 Composed ImageFrom the generated samples in Appendix C we observe that relations among the objects are correctly captured in most cases. In analyzing the generator we find that it sometimes generates a single object together with the . It rarely generates more than one object, confirming that although it is capable, it is indeed more efficient to generate images as compositions of objects. Latent Traversal We explore the degree to which the relational structure affects our initial independence assumption about objects. If it were to cause the latent representations to be fully dependent on one another then our approach would no longer be compositional in the strict sense. Note that although we have a clear intuition in how this mechanism should work, there is no corresponding constraint in the architecture. We conduct an experiment in which we traverse the latent space of a single latent vector in k-GAN rel., by adding a random vector to the original sample with fixed increments and generating an image from the ing latent vectors. Several examples can be seen in Figure 5b. In the first row it can be seen that as we traverse the latent space of a single component the blue digit 9 takes on the shape of a 3, whereas the visual presentation of the others remain unaffected. Similarly in the second and third row the green digits are transformed, while other digits remain fixed. Hence, by disentangling objects at a representational level the underlying representation is more robust to common variations in image space. We observe this behavior for the majority of the generated samples, confirming to a large degree our own intuition of how the relational mechanism should be utilized. When we conduct the same Figure 5: Three generated images by a) GAN and b) 5-GAN rel. bg., when traversing the latent space of a single (object) generator at different increments. On the right it can be seen that in each case only a single digit is transformed, whereas the visual presentation of the others remains unaffected. In the case of GAN (left) the entire scene changes.latent traversal on the latent space of GAN for which the information encoding different objects is entangled, it in a completely different scene (see Figure 5a). FID We train k-GAN and GAN on each dataset, and compare the FID of the models with the lowest average FID across seeds. On all datasets but CLEVR we find that k-GAN compares favorably to our baseline, although typically by a small margin. A break-down of the FID achieved by different variations of k-GAN reveals several interesting observations FIG8 ). In particular, it can be observed that the lowest FID on Independent MM is obtained by 4-GAN without relational structure. This is surprising as each component is strictly independent and therefore 4-GAN ind. is unable to consistently generate 3 digits. Indeed, if we take a look at the generated samples in Figure 11, then we frequently observe that this is the case. It suggests that FID is unable to account for these properties of the generated images, and renders the small FID differences that we observed inconclusive. FIG8 does reveal some large FID differences across the different variations of k-GAN on Triplet MM, and RGB Occluded MM. It can be observed that the lack of a relational mechanism on these datasets is prohibitive (as one would expect), ing in poor FID for k-GAN ind. Simultaneously it confirms that the relational mechanism is properly utilized when relations are present. We asked humans to compare the images generated by k-GAN rel. (k=3,4,5) to our baseline on RGB Occluded MM, CIFAR10 + MM and CLEVR, using the configuration with a generator for the last two datasets. For each model we select the 10 best hyper-parameter configurations (lowest FID), from which we each generate 100 images. We asked up to three raters for each image and report the majority vote or "Equal" if no decision can be reached. FIG4 reports the when asking human raters to compare the visual quality of the generated images by k-GAN to those by GAN. It can be seen that k-GAN compares favorably across all datasets, and in particular on RGB Occluded MM and CIFAR10 + MM we observe large differences. We find that k-GAN performs better even when k > 3, which can be attributed to the relational mechanism, allowing all components to agree on the correct number of digits. In a second study we asked humans to report specific properties of the generated images (number of objects, number of digits, etc.), a complete list of which can be found in Appendix B. Here our goal was to asses if the generated images by k-GAN are more faithful to the reference distribution. The on RGB Occluded MM are summarized in FIG4. It can be seen that k-GAN more frequently generates images that have the correct number of objects, number of digits, and that satisfy all properties simultaneously (color, digit count, shapes). The difference between the correct number of digits and correct number of objects suggests that the generated objects are often not recognizable as digits. This does not appear to be the case from the generated samples in Appendix C, suggesting that the raters may not have been familiar enough with the variety of MNIST digits. On CIFAR10 + MM FIG5 ) it appears that GAN is able to accurately generate the correct number of objects, although the addition of makes it difficult to provide a comparison in this case. On the other hand if we look at the number of digits, then we find that k-GAN outperforms GAN by the same margin, as one would expect compared to the in FIG4.In comparing the generated images by k-GAN and GAN on CLEVR we noticed that the former generated more crowded scenes (containing multiple large objects in the center), and more frequently generated objects with distorted shapes or mixed colors. On the other hand we found cases in which k-GAN generated scenes containing "flying" objects, a by-product of the fixed order in which we apply. We asked humans to score images based on these properties, which confirmed these observations (see FIG5), although some differences are small. The experimental confirm that the proposed structure is beneficial in generating images of multiple objects, and is utilized according to our own intuitions. In order to benefit maximally from this structure it is desirable to be able to accurately estimate the (minimum) number of objects in the environment in advance. This task is ill-posed as it relies on a precise definition of "object" that is generally not available. In our experiments on CLEVR we encounter a similar situation in which the number of components does not suffice the potentially large number of objects in the environment. Here we find that it does not render the proposed structure useless, but instead each component considers "primitives" that correspond to multiple objects. One concern is in being able to accurately determine foreground, and when combining the outputs of the object generators using alpha compositing. On CLEVR we observe cases in which objects appear to be flying, which is the of being unable to route the information content of a "foreground" object to the corresponding "foreground" generator as induced by the fixed order in which images are composed. Although in principle the relational mechanism may account for this distinction, a more explicit mechanism may be preferred BID31.We found that the pre-trained Inception embedding is not conclusive in reasoning about the validity of multi-object datasets. Similarly, the discriminator may have difficulties in accurately judging images from real / fake without additional structure. Ideally we would have a discriminator evaluate the correctness of each object individually, as well as the image as a whole. The use of a patch discriminator BID20, together with the alpha channel of each object generator to provide a segmentation, may serve a starting point in pursuing this direction. We have argued for the importance of compositionality at the representational level of objects in deep generative models of images, and demonstrated how corresponding structure may be incorporated in the generator of a GAN. On a benchmark of multi-object datasets we have shown that the proposed generative model learns about individual objects and in the process of synthesizing samples. A human study revealed that this leads to a better generative model of images. We are hopeful that in disentangling information corresponding to different objects at a representational level these may ultimately be recovered. Hence, we believe that this work is an important contribution towards learning object representations of complex real-world images without any supervision. A EXPERIMENT The generator and discriminator neural network architectures in all our experiments are based on DCGAN BID35. Object Generators k-GAN ind. introduces K = k copies of an object generator (i.e. tied weights, DCGAN architecture) that each generate and image from an independent sample of a 64-dimensional UNIFORM(-1, 1) prior P (Z).Relational Structure When a relational stage is incorporated (k-GAN rel.) each of the z i ∼ P (Z) are first updated, before being passed to the generators. These updates are computed using one or more attention blocks, which integrate Multi-Head Dot-Product Attention (MHDPA; BID40) with a post-processing step. A single head of an attention block updates z i according to.In our experiments we use a single-layer neural network (fully-connected, 32 ReLU) followed by LayerNorm BID2 up. If multiple heads are present, then their outputs are concatenated and transformed by a single-layer neural network (fully-connected, 64 ReLU) followed by LayerNorm to obtain the neŵ z i. If the relational stage incorporates multiple attention blocks that iteratively update z i, then we consider two variations: using unique weights for each MLP in each block, or sharing their weights across blocks. Background Generation When a generator is incorporated (eg. k-GAN rel. bg) it uses the same DCGAN architecture as the object generators, yet maintains its own set of weights. It receives as input its own latent sample z b ∼ P (Z b), again using a UNIFORM(-1, 1) prior, although one may in theory choose a different distribution. We explore both variations in which z b participates in the relational stage, and in which it does not. Composing In order to obtain the final generated image, we need to combine the images generated by each generator. In the case of Independent MM and Triplet MM we simply sum the outputs of the different generators and clip their values to. On RGB Occluded MM we combine the different outputs using alpha compositing, with masks obtained by thresholding the output of each generator at 0.1. On CIFAR10 + MM and CLEVR we require each of the object generators to generate an additional alpha channel by adding an additional feature map in the last layer of the generator. These are then combined with the generated (opaque) using alpha compositing, i.e. through repeated application of. Each model is optimized with ADAM using a learning rate of 0.0001, and batch size 64 for 1 000 000 steps. Each generator step is followed by 5 discriminator steps, as is considered best practice in training GANs. Checkpoints are saved at every 20 000 th step and we consider only the checkpoint with the lowest FID for each hyper-parameter configuration. FID is computed using 10 000 samples from a hold-out set. Baseline We conduct an extensive grid search over 48 different GAN configurations to obtain a strong GAN baseline on each dataset. It is made up of hyper-parameter ranges that were found to be successful in training GANs on standard datasets BID26.

Represent the short summary for finding the corresponding scientific article (long):

We propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.