Question: Represent the article for finding a single-sentence TLDR:

Question: We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn ‘distributional similarity’ in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space. To validate our hypothesis, we focus on the ‘image’ side of image captioning, and vary the input image representation but keep the RNN text generation model of a CNN-RNN constant. We propose a sparse bag-of-objects vector as an interpretable representation to investigate our distributional similarity hypothesis. We found that image captioning models (i) are capable of separating structure from noisy input representations; (ii) experience virtually no significant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together; (iv) are heavily reliant on test sets with a similar distribution as the training set; (v) repeatedly generate the same captions by matching images and ‘retrieving’ a caption in the joint visual-textual space. Our experiments all point to one fact: that our distributional similarity hypothesis holds. We conclude that, regardless of the image representation, image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace. Image description generation, or image captioning (IC), is the task of automatically generating a textual description for a given image. The generated text is expected to describe, in a single sentence, what is visually depicted in the image, for example the entities/objects present in the image, their attributes, the actions/activities performed, entity/object interactions (including quantification), the location/scene, etc. (e.g. "a man riding a bike on the street").Significant progress has been made with end-to-end approaches to tackling this problem, where large-scale parallel image-description datasets such as Flickr30k BID41 and MSCOCO BID4 are used to train a CNN-RNN based neural network IC system BID36 BID17. Such systems have demonstrated impressive performance in the COCO captioning challenge 1 according to automatic metrics, seemingly even surpassing human performance in many instances (e.g. CIDEr score > 1.0 vs. human's 0.85) BID3. However, in reality, the performance of end-to-end systems is still far from satisfactory according to metrics based on human judgement 2. Thus, despite the progress, this task is currently far from being a solved problem. In this paper, we challenge the common assumption that end-to-end IC systems are able to achieve strong performance because they have learned to'understand' and infer semantic information from visual representations, i.e. they can for example deduce that "a boy is playing football" purely by learning directly from mid-level image features and the corresponding textual descriptions in an implicit manner, without explicitly modeling the presence of boy, ball, green field, etc. in the image. It is believed that the IC system has managed to infer that the phrase green field is associated with some'green-like' area in the image and is thus generated in the output description, or that the word boy is generated because of some CNN activations corresponding to a young person. However, there seems to be no concrete evidence that this is the case. Instead, we hypothesize that the apparently strong performance of end-to-end systems is attributed to the fact that they are exploiting the distributional similarity in the multimodal feature space. To our best knowledge, our paper gives the first empirical analysis on visual representations for the task of image captioning. What we mean by'distributional similarity' is that IC systems essentially attempt to match images from the training set that is most similar to a test image, and generate a caption from the most similar training instances (or generate a 'novel' description from a combination of training instances, for example by 'averaging' the descriptions). Previous work has alluded to this observation BID16 BID36, but it has not been thoroughly investigated. This phenomena could also be in part attributed to the fact that the datasets are repetitive and simplistic, with an almost constant and predictable linguistic structure BID18 BID7 BID36.In this paper we investigate the hypothesis of distributional similarity in IC by focusing on the image side of image captioning. Most previous work has concentrated on the text side of image captioning, e.g. by optimizing the language modelling capabilities of the RNN BID27 BID19 to improve its performance on automatic metrics. While there have been efforts on improving IC by utilizing or modeling images more effectively, for example by using attention over mid-level image features and high-level object proposals BID1, in this work we are specifically interested in interpretability and we focus on using a simpler (and faster) model for empirical evaluation. We explore the basic yet effective CNN-RNN model BID17, and investigate the representational contributions while keeping the RNN generator constant. More advanced models can be considered specific variants of BID17.It is worth noting that we are interested in demonstrating the phenomenon of distributional similarity in IC, rather than achieving or improving state-of-the-art performance, As such, we do not resort to fine-tuning or extensive hyperparameter optimization or ensembles. Therefore, our model is not comparable to state-of-the-art models such as BID36, which optimize IC by fine-tuning the image representations, exploring beam size, scheduled sampling, and using ensemble models. Instead, we vary only the image representation to demonstrate that end-to-end IC systems utilize distributional similarity on the image side to generate captions, regardless of the image representation used. Our main contributions are:• An IC experiment where we vary the input image representation but keep the RNN text generation model constant (Section 3). This experiment demonstrates that regardless of the image representation (a continuous image embedding or a sparse, low-dimensional vector), end-to-end IC systems seem to utilize a visual-semantic subspace for IC.• The introduction of a simple, sparse bag-of-objects representation that contains information about the presence of objects in the images. We use this as a tool to investigate the contribution of images in the image captioning framework.• The introduction of pseudo-random vectors derived from object-level representations as a means to evaluate IC systems. Our show that end-to-end models in this framework are remarkably capable of separating structure from noisy input representations.• An experiment where IC models are conditioned on image representations factorized and compresssed to a lower dimensional space (Section 4.1). We show that high dimensional image embeddings that are factorized to a lower dimensional representation and used as input to an IC model in virtually no significant loss in performance, further strengthening our claim that IC models perform similarity matching rather than image understanding.• An analysis of different image representations and their transformed representations (Sections 4.2 and 4.3). We visualize the initial visual subspace and the learned joint visual semantic subspace and observe that the visual semantic subspace has learned to cluster images with similar visual and linguistic information together, further validating our claims of distributional similarity.• An experiment where the IC model is tested on an out-of-domain dataset (Section 4.4), which has a slightly different image distribution. We observe that models, including the state-of-the-art models, show a better performance on test sets that have a similar distribution as the training. However, their performance deteriorates when the distributions are slightly different.• An analysis on the uniqueness of captions generated by IC models using different image representations (Section 4.5). We hypothesize that the captions are often repeated as they are usually generated by matching images in the joint space and retrieving a relevant caption. Our experiments validate this claim. Overall, the study suggests that regardless of the representation used, end-to-end IC models implicitly learn and exploit multimodal similarity spaces rather than performing actual image understanding. This study is in line with the recent work that explore understanding of deep learning models and the representational interpretations BID23 BID32 BID30 and works that have tried to delve into the image captioning task BID7 BID36. To the best of our knowledge, ours is the first work that investigates IC focusing specifically on image representations and their effects. For the experiments in Section 3, we base our implementation on a simple end-to-end approach by BID17. We use the LSTM BID14 based language model as described in BID42.To condition the image information, we first perform a linear projection of the image representation followed by a non-linearity: DISPLAYFORM0 n×d is the linear transformation matrix, σ is the non-linearity. We use Exponential Linear Units BID5 as the non-linear activation in all our experiments. Following BID35, we initialize the LSTM based caption generator with the projected image feature. The caption generator is trained to generate sentences conditioned on the image representation. We train the model by minimizing the cross-entropy, i.e., the sentence-level loss corresponds to the sum of the negative log likelihood of the correct word at each time step: DISPLAYFORM0 where Pr (S|Im f eat ; θ) is the sentence-level loss conditioned on the image feature Im f eat and Pr(w t) is the probability of the word at time step t. This is trained with standard teacher forcing as described in BID31 where the correct word information is fed to the next state in the LSTM.Inference is typically performed with approximation techniques like beam search or sampling BID17 BID35. In this paper, as we are mainly interested in the studying effect of different image representations, we focus on the language output that the models can most confidently produce. Therefore, in order to isolate any other variables from the experiments, we generate captions using a greedy argmax based approach for consistency (unless stated otherwise, we always use greedy decoding). In this section, we verify our hypothesis that a'distributional similarity' space exist in end-to-end IC systems. Such systems attempt to match image representations in order to condition the RNN decoder to generate captions that are similar to the closest images, rather than actually understanding the image in order to describe the image. We keep the IC model constant (Section 2) across experiments, and vary only the image representation used. The following pre-trained CNNs are used:• VGG19 BID29 pre-trained on ILSVRC BID28.• ResNet152 BID13 ) also pre-trained on ILSVRC.• Places365-ResNet152 BID43, a variant of ResNet152 pre-trained on the Places2 dataset BID44. We investigate whether scene-specific categories are useful for IC without the network being trained to classify object-specific categories.• Hybrid1365-ResNet152 BID43, a ResNet152 variant trained on the concatenation of the ILSVRC and Places2 datasets and predicts both object and scene classes. We explore various representations derived from the CNNs above:Penultimate layer (Penultimate): Most previous attempts for IC use the output of the penultimate layer of a CNN pre-trained on ILSVRC. Previous work motivates using'off-the-shelf' feature extractors in the framework of transfer learning BID25 BID8. Such features have often been applied to image captioning BID21 BID17 BID35 BID9 and have been shown to produce state-of-the-art . Therefore, for each image, we extract the fc7 layer of VGG19 (4096D) and the pool5 layer for the ResNet152 variants (2048D).Class prediction vector (Softmax): We also investigate higher-level image representations, where each element in the vector is an estimated posterior probability of object categories. Note that the categories may not directly correspond to the captions in the dataset. While there are alternative methods that fine-tune the image network on a new set of object classes extracted in ways that are directly relevant to the captions BID37, we study the impact of off-the-shelf prediction vectors on the IC task. The intuition is that category predictions from pretrained CNN classifiers may also be beneficial for IC, alongside the standard approach of using midlevel features from the penultimate layer. Therefore, for each image, we use the predicted category posterior distributions of VGG19 and ResNet152 for 1000 object categories), Places365-ResNet152 (365 scene categories), and Hybrid-ResNet152 (1365 object and scene categories). Here we experiment with a method that utilizes the averaged word representations of top-k predicted object classes. We first obtain Softmax predictions using ResNet152 for 1000 object categories (synsets) per image. We then select the objects that have a posterior probability score > 5% and use the 300-dimensional pre-trained word2vec BID22 representations 4 to obtain the averaged vector over all retained object categories. This is motivated by the central observation that averaged word embeddings can represent semantic-level properties and are useful for classification tasks BID2. We also explore representing images using information from object detectors that identifies instances of object categories present in an image, rather than a global, image-level classification. This can potentially provide for a richer and more informative image representation. For this we use:• ground truth (Gold) region annotations for instances of 80 pre-defined categories provided with MSCOCO. It is worth noting that these were annotated independently of the image captions, i.e. people writing the captions had no knowledge of the 80 categories and the annotations. As such, there is no direct correspondence between the region annotations and image captions.• a state-to-the-art object detector YOLO BID26, pre-trained on MSCOCO for 80 categories (YOLO-Coco), and on MSCOCO and ILSVRC for over 9000 categories (YOLO-9k). We explore several representations derived from instance-level object class annotations/detectors above:Bag of objects (BOO): We represent each image as a sparse'bag of objects' vector, where each element represents the frequency of occurrence for each object category in the image (Counts). We also explore an alternative representation where we only encode the presence or absence of the object category regardless of its frequency (Binary), to determine whether it is important to encode object counts in the image. These representations help us examine the importance of explicit object categories and in a sense interactions between object categories (dog and ball) in the image representation. We investigate whether such a sparse and high-level BOO representation is helpful for IC. It is also worth noting that BOO is different from the Softmax representation above as it encodes the number of object occurrences, not the confidence of class predictions at image level. We compare BOO representations derived from the Gold annotations (Gold-Binary and Gold-Counts) and both YOLO-Coco and YOLO-9k detectors (Counts only). To further probe the capacity of the model to discern image representations in an image distributional similarity space, we propose a novel experiment where we examine a type of representation where similar images are represented using similar random vectors, which we term as pseudo-random vectors. We form this representation from BOO Gold-Counts and BOO Gold-Binary. Formally, Im f eat = o∈Objects f × φ o, where φ o ∈ R d is an object-specific random vector and f is a scalar representing counts of the object category. In the case of PseudorandomCounts, f is the frequency counts from Gold-Counts. In the case of Pseudorandom-Binary, f is either 0 or 1 based on Gold-Binary. We use d = 120 for these experiments. Dataset We evaluate image captioning conditioned on different representations on the most widely used dataset for IC, MSCOCO BID4. The dataset consists of 82, 783 images for training, with at least five captions per image, totaling to 413, 915 captions. We perform model selection on a 5000-image development set and report the on a 5000-image test set using standard, publicly available splits 5 of the MSCOCO validation dataset as in previous work BID17.Evaluation Metrics We evaluated system outputs using the standard evaluation metrics for image captioning using the most common metrics: BLEU BID24 which is computed from 1-gram to 4-gram precision scores (B-1 · · · B-4), Meteor BID6 ) (M) and CIDEr BID34 (C) and SPICE BID0 ) (S). All these metrics are based on some form of n-gram overlap between the system output and the reference captions (i.e. no image information is used). For each system-generated caption, we compare against five references. We used the publicly available cocoeval script for evaluation. 6 Model Settings and Hyperparameters We use a single hidden layer LSTM with 128-dimensional word embeddings and 256-dimensional hidden dimensions. As training vocabulary we retain only words that appear at least twice. We report of IC on MSCOCO in Table 1, where the IC model (Section 2) is conditioned on the various image representations described in Section 3.1. As expected, using random image Table 1: Results on the MSCOCO test split, where we vary only the image representation and keep other parameters constant. The captions are generated with beam = 1. We report BLEU, Meteor, CIDEr and SPICE scores.. embeddings clearly does not provide any useful information and performs poorly. The Softmax representations with similar sets of object classes (VGG19, ResNet152, and Hybrid1365-ResNet152) have very similar performance. However, the Places365-ResNet representations perform poorly. We note that the posterior distribution may not directly correspond to captions as there are many words and concepts that are not contained in the set of object classes. Our differ from those by BID37 and BID39 where the object classes have been fine-tuned to correspond directly to the caption vocabulary. We posit that the degradation in performance is due to spurious probability distributions over object classes for similar looking images. The performance of the Pool5 image representations shows a similar trend for VGG19, ResNet152, and Hybrid1365-ResNet152. ResNet152 is slightly better in performance. The Places365-ResNet representation performs poorly. We posit that the representations from the image network trained on object classes rather than scene classes are able to capture more fine-grained image details from the images, whereas the image network trained with scene-based classes captures more coarse-grained information. The performance of the averaged top-k word embeddings is similar to that of the Softmax representation. This is interesting, since the averaged word representational information is mostly noisy: we combine top-k synset-level information into one single vector, however, it still performs competitively. We observe that the performance of the Bag of Objects (BOO) sparse 80-dimensional annotation vector is better than all other image representations judging by the CIDEr score. We remark here again, that this is despite the fact that the annotations may not directly correspond to the semantic information in the image or the captions. The sparse representational information is indicative of the presence of only a subset of potentially useful objects. We notice two distinct patterns, a marked difference with Binary and Count based representations. This takes us back to the motivation that image captioning ideally requires information about objects, interaction between the objects with attribute information. Although our representation is really sparse on the object interactions, it captures the basic concept of the presence of more than one object of the same kind, and thus provides some kind of extra information. A similar trend is observed by BID40, although in their models they further try to learn interactions using a specified object RNN.We also notice that predicted objects using YOLOCoco performs better than the YOLO9k. This is probably expected as the YOLOCoco was trained on the same dataset hence obtaining better object proposals. We also observed that with YOLO9k, we had a significant number of objects being predicted for the test images that were not seen with the training set (around 20%). The most surprising is the performance of the pseudo-random vectors. We notice that both the pseudo-random binary and the pseudo-random count based vectors perform almost as good as the Gold objects. This suggests that the conditioned RNN is able to remove noise and learn some sort of a common'visual-linguistic' semantic subspace. We perform further analysis on the different image representations to gain a further understanding of the representations and demonstrate our distributional similarity hypothesis. In Section 3.3, we observed encouraging from the bag of objects based representations despite being sparse, low-dimensional, and only partially relevant to captions. Interestingly, using pseudo-random vectors derived from bag of objects also gave excellent performance despite the added noise. This leads to the question: are high-dimensional vectors necessary or relevant? To answer this, we evaluate whether the performance of the model significantly degrades if we reduce the dimensionality of a high dimensional representation. We experiment with three exploratory factor analysis based methods -Principal Component Analysis (PCA) BID12, Probabilistic Principal Component Analysis (PPCA) BID33 and Independent Component Analysis (ICA) BID15. In all cases, we obtain 80-dimensional factorized representations on ResNet152 pool5 (2048D) that is commonly used in IC. We summarize our experiment in TAB2. We observe that, the representations obtained by all the factor models seem to retain the necessary representational power to produce appropriate captions equivalent to the original representation. This seems contradictory as we expected a loss in the information content when compressing to arbitrary 80-dimensions. This experiment indicates that the model is not explicitly utilizing the full expressiveness of the full 2048-dimensional representations. We conclude that the model is able to learn from a seemingly weak, structured information and is able to in a performance that is close to one that uses the full representation. In this section, we investigate the distributional similarity hypothesis by inspecting the regularities in the initial representation state for several representations from Section 3.1, using the interpretable bag-of-objects representation. If the representation is informative for IC, then the representations should ideally semantically related images together, and in turn allow for relevant captions to be generated. We compare different image representations with respect to their ability to group and distinguish between semantically related images. For this, we selected three categories from MSCOCO ("dog", "person", "toilet") and also pairwise combinations of these ("dog+person", "dog+toilet", "person+toilet"). Up to 25 images were randomly selected for each of these six groups (single category or pair) such that the images are annotated with only the associated categories. Each group is represented by the average image feature of these images. FIG1 shows the cosine distances between each group, for each of our image representations. The Bag of Objects model clusters these groups the best, as expected (e.g. the average image representation of "dog" correlates with images containing "dog" as a pair like "dog+person" and "dog+toilet"). The Softmax models seem to also to exhibit semantic clusters, although to a lesser extent. This can be observed with "person", where the features are not semantically similar to any other groups. The most likely reason is that there is no "person" category in ILSVRC. Also, Place365 and Hybrid1365 Softmax FIG1 ) also showed very strong similarity for images containing "toilet", where or not they contain "dog" or "person", possibly because they capture scene features. On the other hand, Pool5 features seem to in images that are more similar to each other than Softmax overall. Considering our earlier hypothesis as proposed in Section 3.3 that the conditioned RNN is learning some sort of a common'visual-linguistic' semantic space, we explore the difference in representations in the initial representational space and the transformed representational space. The transformation is learned jointly as a subtask of the image captioning. We posit that image representations in the transformed space will be more semantically coherent with respect to both images and captions. To visualize the two representational spaces, we use Barnes-Hut t-SNE BID20 to compute a 2-dimensional embedding over the test split. In general, we found that images are initially clustered by visual similarity (Pool5) and semantic similarity (Softmax, Bag of Objects). After transformation, we observe that some linguistic information from the captions has ed in different types of clusters. Figure 2 highlights some interesting observations of the changes in clustering across three different representations. For Pool5, images seem to be clustered by their visual appearance, for example snow scenes in FIG2, regardless of the subjects in the images (people or dogs). After transformation, separate clusters seem to be formed for snow scenes involving a single person, groups of people, and dogs. Interestingly, images of dogs in fields and snow scenes are also drawn closer together. Softmax FIG2 ) shows many small, isolated clusters before transformation. After transformation, bigger clusters seem to be formed -suggesting that the captions have again drawn related images together despite being different in the Softmax space. For bag of objects FIG2 ), objects seem to be clustered by co-occurrence of object categories, for example toilets and kitchens are clustered since they share sinks. Toilets and kitchens seem to be further apart in the transformed space. A similar observation was made by BID36 in which the authors observe that endto-end based image captioning models are capable of performing retrieval tasks with comparable performance to the task specific models that are trained with the ranking loss. We further perform a similar analysis on the pseudorandom representations FIG2 ). We observe that the initial representations have very little explicit information and do not cluster. The projected representations however form clusters that mimic the projected space of the bag-of-object cluster. Full sized version of images in FIG2 are presented anonymously in: https://github.com/ anonymousiclr/HJNGGmZ0Z We now demonstrate that end-to-end models are heavily reliant on datasets that have a similar training and test distribution. We posit that an IC system that performs similarity matching will not perform well on a slightly different domain for the same task. Demonstrating this will further validate our hypothesis that IC systems perform image matching to generate image captions. Thus, we evaluate several models trained on MSCOCO on 1000 test image samples from the Flickr30k BID41 ) dataset 7. Like MSCOCO, Flickr30k is an image description dataset; however, unlike MSCOCO, the images have a different distribution and the descriptions are slightly longer and more descriptive. We evaluate the captions generated by our model with Resnet152 pool5 representation and by two other state-of-the-art models pretrained on MSCOCO: a) Self-Critical (SC) BID27 based on self critical sequence training that uses reinforcement learning using metrics, and b) Bottom Up and Top Down (TDBU) BID1 ) based on top-down and bottom-up attention using object region proposals. Both the state-of-the-art models are much more complex than the image conditioned RNN based language model. The are summarized in Table 3.We observe that the scores drop by a large margin. A similar observation was made by BID36, and they alluded the drop in scores to the linguistic mismatch between the datasets. However, we probed the out of training vocabulary words in the Flickr30k test set and observed that it was around 8.6% which seems to be the usual unseen rate. This suggests that there is more to the issue than mere vocabulary mismatch. We observe that while typical sentences on Flickr30k are structurally different and are generally longer, the model is still unable to generate good bigrams or even unigrams as is evident from B-1 and B-2 scores in Table 3.We further investigated the distributions of objects in the images by using YOLO object detector (trained on MSCOCO). We first detect objects on the MSCOCO training set and our MSCOCO test set followed by objects on the Flickr30k test set. It Figure 3 we show the normalized frequency versus the distribution of objects over the MSCOCO train and test sets. We notice that the distributions are very similar and mostly overlap. In FIG3 we show the normalized frequency versus distribution of objects detected over MSCOCO train and Flickr30k test sets. We observe that the two distributions are slightly different, they don't overlap as closely as we see in Figure 3. We hypothesize that the difference in distribution is one of the difference that reflects in the lower performance of a model that is trained on the MSCOCO dataset performing poorly on the Flickr30k test set. We postulate that image captions are often repeated because they are generated by'retrieving' similar images in the joint image-text semantic space and generating the relevant caption at this point in the space. To investigate this, we first show that, regardless of the representation, end-to-end IC systems do not generate unique captions for every distinct image. We use the full validation portion of the MSCOCO dataset (40,504 images) and produce captions with four types of distinct image representations. We report the in Table 4. We observe that in almost all cases, the produced representations are far from unique. In most cases, there is a significant portion of the captions that are repeated. This is also observed by BID7 on different test splits, but using retrieval-based and pipeline based methods for IC.Intrigued by the that almost all the representations end up with similar proportion of unique captions, we further investigate the models using a k-nearest neighbor approach. The key idea is that if the IC systems perform some form of image matching and a complex text retrieval from the training set, then the nearest neighbour (from training) of a test image should have a similar caption to the one generated by the model. We note that the model is clearly not performing text retrieval as the LSTM does generate novel captions, possibly by aggregating or'averaging' the captions of similar images and performing some factorization. Unique (Table 5 : k-nearest neighbor experimentTo perform this experiment, we begin by generating captions for every training image using the bag of objects model (with frequency counts). We then compute the k-nearest training images for each given test image using both the bag of objects representation and its projection (Eq. 2). Finally, we compute the similarity score between the generated caption of the test image against all k-nearest captions. The similarity score measures how well a generated caption matches its nearest neighbour's captions. We expect the score to be high if the IC system generates an image similar to something'summarized' from the training set. We report our in Table 5. We observe that overall the captions seem to closely match the captions of 5 nearest training images. Further analysis showed that 2301 captions had nearest images at a zero distance, i.e., the same exact representation was seen during at least 5 times in training (note that CIDEr gives a score of 10 only if the test caption and all references are the same). We found that among the non-exact image matches, the projected image representation better captures candidates in the training set than the bag of objects. We further analyze the captions and provide details in the appendix. We hypothesized that IC systems essentially exploit a distributional similarity space to'generate' image captions, by attempting to match a test image to similar training image(s) and generate an image caption from these similar images. Our study focused on the image side of image captioning:We varied the image representations while keeping the text generation component of an end-toend CNN-RNN model constant. We found that regardless of the image representation, end-to-end IC systems seem to match images and generate captions in a visual-semantic subspace for IC. We conclude that:• A sparse, low-dimensional bags-of-objects representation can be used as a tool to investigate the contribution of images in IC; we demonstrated that such a vector is sufficient for generating good image captions; • End-to-end IC models are remarkably capable of separating structure from noisy input representations, as demonstrated by pseudo-random vectors; • End-to-end IC models suffer virtually no significant loss in performance when a high dimensional representation is factorized to a lower dimensional space; • End-to-end IC models have learned a joint visual-textual semantic subspace by clustering images with similar visual and linguistic information together; • End-to-end IC models rely on test sets with a similar distribution as the training set for generating good captions; • End-to-end IC models repeatedly generate the same captions by matching images in the joint visual-textual space and'retrieving' a caption in the learned joint space. All the observations above strengthen our distributional similarity hypothesis -that end-to-end IC performs image matching and generates captions for a test image from similar image(s) from the training set -rather than performing actual image understanding. Our findings provide novel insights into what end-to-end IC systems are actually doing, which previous work only suggests or hints at without concretely demonstrating the distributional similarity hypothesis. We believe our findings are important for the IC community to further advance image captioning in a more informed manner. (c) Bag of objects: person, tie Figure 5: Example outputs from our system with different representations, the sub-captions indicate the annotation along with the frequency in braces. We also show the CIDEr score and the difference in CIDEr score relative to the Bag of Objects representation. A ANALYSIS ON GENERATED CAPTIONSHere, we provide a qualitative analysis of different image representations presented and gain some insights into how they contribute to the the IC task. The Bag of Objects representation led to a strong performance in IC despite being extremely sparse and low-dimensional (80 dimensions). Analyzing the test split, we found that each vector consists of only 2.86 non-zero entries on average (standard deviation 1.8, median 2). Thus, with the minimal information being provided to the generator RNN, we find it surprising that it is able to perform so well. We compare the output of the remaining models against the Bag of Objects representation by investigating what each representation adds to or subtracts from this simple, yet strong model. We start by selecting images (from the test split) annotated with the exact same Bag of Objects representation -which should in the same caption. For our qualitative analysis, several sets of one to three MSCOCO categories were manually chosen. For each set, images were selected such that there is exactly one instance of each category in the set and zero for others. We then shortlisted images where the captions generated by the Bag of Objects model produced the five highest and five lowest CIDEr scores (ten images per set). We then compare the captions sampled for each of the other representations. Figure 5 shows some example outputs from this analysis. In Figure 5a, Bag of Objects achieved a high CIDEr score despite only being given "bird" as input, mainly by'guessing' that the bird will be perching/sitting on a branch. The object-based Softmax (VGG and ResNet) models gave an even more accurate description as "owl" is the top-1 prediction of both representations (96% confidence for VGG, 77% for ResNet). Places365 predicted "swamp" and "forest". The Penultimate features on the other hand struggled with representing the images correctly. In Figure 5b, Bag of Objects struggled with lack of information (only "airplane" is given), the Softmax features mainly predicted "chainlink fence", Places365 predicted "kennel" (hence the dog description), and it most likely that Penultimate has captured the fence-like features in the image rather than the plane. In Figure 5c, the Softmax features generally managed to generate a caption describing a woman despite not explicitly containing the'woman' category. This is because other correlated categories were predicted, such as "mask", "wig", "perfume", "hairspray" and in the case of Places365 "beauty salon" and "dressing room". Our model settings were:• LSTM with 128 dimensional word embeddings and 256 dimensional hidden representations Dropout over LSTM of 0.8 • We used Adam for optimization.• We fixed the learning rate to 4e-4We report our by keeping the above settings constant.

This paper presents an empirical analysis on the role of different types of image representations and probes the properties of these representations for the task of image captioning.

Represent the short summary for finding the corresponding scientific article (long)

We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.


Question: Represent the scientific article to find a short TLDR:

Question: We present a sequence-to-action parsing approach for the natural language to SQL task that incrementally fills the slots of a SQL query with feasible actions from a pre-defined inventory. To account for the fact that typically there are multiple correct SQL queries with the same or very similar semantics, we draw inspiration from syntactic parsing techniques and propose to train our sequence-to-action models with non-deterministic oracles. We evaluate our models on the WikiSQL dataset and achieve an execution accuracy of 83.7% on the test set, a 2.1% absolute improvement over the models trained with traditional static oracles assuming a single correct target SQL query. When further combined with the execution-guided decoding strategy, our model sets a new state-of-the-art performance at an execution accuracy of 87.1%. Many mission-critical applications in health care, financial markets, and business process management store their information in relational databases BID10 BID22 BID16. Users access that information using a query language such as SQL. Although expressive and powerful, SQL is difficult to master for non-technical users. Even for an expert, writing SQL queries can be challenging as it requires knowing the exact schema of the database and the roles of various entities in the query. Hence, a long-standing goal has been to allow users to interact with the database through natural language BID0 BID24.The key to achieving this goal is understanding the semantics of the natural language statements and mapping them to the intended SQL. This problem, also known as NL2SQL, was previously understudied largely due to the availability of annotation. Without paired natural language statement and SQL query, a weak supervision approach may be adopted which reduces supervision from annotated SQL queries to answers BID19. This is a more difficult learning problem. Therefore only with recent release of a number of large-scale annotated NL2SQL datasets BID36 BID6, we start to see a surge of interest in solving this problem. Existing NL2SQL approaches largely fall into two categories: sequence-to-sequence style neural "machine translation " systems BID36 BID5 and sets of modularized models with each predicting a specific part of the SQL queries BID32 BID34. The former class suffer from the requirement of labeling a single ground truth query while multiple semantically equivalent queries exist for each intent. For example, as noticed by BID36, the ordering of filtering conditions in a query does not affect execution but affects generation. To account for this, techniques such as reinforcement learning have been used on top of those sequenceto-sequence models. The second class of models employ a sequence-to-set approach: they first predict table columns present in the query and then independently predict the rest for each column. This avoids the ordering issue, but makes it harder to leverage inter-dependencies among conditions. In this work, we develop a sequence-to-action parsing approach (Section 3) for the NL2SQL problem. It incrementally fills the slots of a SQL query with actions from an inventory designed for this task. Taking inspiration from training oracles in incremental syntactic parsing BID8, we further propose to use non-deterministic oracles (Section 4) for training the incremental parsers. These oracles permit multiple correct action continuations from a partial parse, thus are able to account for the logical form variations. Our model combines the advantage of a sequence-to-sequence model that captures inter-dependencies within sequence of predictions and a SELECT`Height (ft)Ẁ HERE Name="Willis Tower" AND Location="Chicago" DISPLAYFORM0 What is the height of Willis Tower in Chicago?Figure 1: Our running example. The input is a natural language question and a table schema, and the output is an executable SQL query. Table contents are shown here, but unknown to our models.modularized model that avoids any standarized linearization of the logical forms. We evaluate our models on the WikiSQL dataset and observe a performance improvement of 2.1% when comparing non-deterministic oracles with traditional static oracles. We further combine our approach and the execution-guided decoding strategy ) and achieve a new state-of-the-art performance with 87.1% test execution accuracy. Experiments on a filtered ATIS dataset in addition confirm that our models can be applied to other NL2SQL datasets. Given an input natural language question, our goal is to generate its corresponding SQL query. In the following and throughout the paper, we use the WikiSQL dataset BID36 as our motivating example. However, it should be noted that our approach is generally applicable to other NL2SQL data, with proper choice of an action inventory and redesign of parser states. Figure 1 shows an example. The SQL structure of the WikiSQL dataset queries is restricted and always follows the template SELECT agg selcol WHERE col op val (AND col op val) *. Here, selcol is a single table column and agg is an aggregator (e.g., COUNT, SUM, empty). The WHERE segment is a sequence of conjunctive filtering conditions. Each op is a filtering operator (e.g., =) and the filtering value val is mentioned in the question. Although the dataset comes with a "standard" linear ordering of the conditions, the order is actually irrelevant given the semantics of AND.Throughout the paper we denote the input to the parser as x. It consists of a natural language question w with tokens w i and a single table schema c with column names c j. A column name c j can have one or more tokens. The parser needs to generate an executable SQL query y as its output. Given an input x, the generation of a structured output y is broken down into a sequence of parsing decisions. The parser starts from an initial state and incrementally takes actions according to a learned policy. Each action advances the parser from one state to another, until it reaches one of the terminal states, where we may extract a complete logical form y. We take a probabilistic approach to model the policy. It predicts a probability distribution over the valid set of subsequent actions given the input x and the running decoding history. The goal of training such an incremental semantic parser is then to optimize this parameterized policy. Formally, we let P θ (y|x) = P θ (a|x), where θ is model parameters. Execution of the action sequence a = {a 1, a 2, . . ., a k} leads the parser from the initial state to a terminal state that contains the parsing y. Here we assume that each y has only one corresponding action sequence a, an assumption that we will revisit in Section 4. The probability of action sequence is further factored as the product of incremental decision probabilities: DISPLAYFORM0 During inference, instead of attempting to enumerate over the entire output space and find the highest scoring a * = arg max a P θ (a|x), our decoder takes a greedy approach: at each intermediate step, it picks the highest scoring action according to the policy: a * i = arg max ai P θ (a i |x, a * <i). Resulting state after taking the action at state p Parameter representation In the following subsections, we define the parser states and the inventory of actions, followed by a description of our encoder-decoder neural-network model architecture. DISPLAYFORM0 We first look at a structured representation of a full parse corresponding to the example in Figure 1: Table 1. The action CONDVAL selects a span of text w i:j from the input question w. In practice, this leads to a large number of actions, quadratic in the length of the input question, so we break down CONDVAL into two consecutive actions, one selecting the starting position w i and the other selecting the ending position w j for the span. At the end of the action sequence, we append a special action END that terminates the parsing process and brings the parser into a terminal state. As an example, the query in Figure 1 translates to an action sequence of {AGG(NONE), SELCOL(c 3), CONDCOL(c 1), CONDOP(=), CONDVAL(w 5:6), CONDCOL(c 2), CONDOP(=), CONDVAL(w 8:8)}. DISPLAYFORM0 The above definitions assume all the valid sequences to have the form of AGG SELCOL (CONDCOL CONDOP CONDVAL) * END. This guarantees that we can extract a complete logical form from each terminal state. For other data with different SQL structure, a redesign of action inventory and parser states is required. We first assume that we have some context-sensitive representations r The main component of our decoder is to model a probability distribution P θ (a|x, a <i) over potential parser actions a conditioned on input x and past actions a <i. It has two main challenges: there is no fixed set of valid parser actions: it depends on the input and the current parser state; the parser decision is context-dependent: it relies on the decoding history and the information embedded in the input question and column headers. We adopt an LSTM-based decoder framework and address the first challenge through individual scoring of actions. The model scores each candidate action a as s a and uses a softmax function to normalize the scores into a probability distribution. At time step i, we denote the current decoder hidden state as h DEC iand model the score of a in the form of a bilinear function: DISPLAYFORM0, where r A a is a vector representation of the action a and is modeled as the concatenation of the action embedding and the parameter representation. The form of the latter is given in Table 1.The dependencies between the parser decisions and the input question and column headers are captured through a dot-product attention mechanism BID20. The input to the first layer of our decoder LSTM at time step i + 1 is a concatenation of the output action representation r A ai from previous time step i, a question attention vector e W i, and a column header attention vector e DISPLAYFORM1 Now we return to the context-sensitive representations r W i and r C j. Ideally, these representations should be both intra-context-sensitive, i.e. aware of information within the sequences, and intersequence-dependent, i.e. utilizing knowledge about the other sequences. These intuitions are reflected in our model design as intra-sequence LSTMs, self-attention and cross-serial attention. Our model architecture is illustrated in FIG1. Each word w i is first mapped to its embedding, and then fed into a bi-directional LSTM (bi-LSTM) that associates each position with a hidden state h W i. For column headers, since each column name can have multiple words, we apply word embedding lookup and bi-LSTM for each column name, and use the final hidden state from the bi-LSTM as the initial representation for the column name. Next, we apply self-attention BID27 to contextualize this initial representation into h C j. After obtaining these intra-contextsensitive representations h W i and h C j, we use cross-serial dot-product attention BID20 to get a weighted average of h Previously, we assumed that each natural language question has a single corresponding SQL query, and each query has a single underlying correct action sequence. However, these assumptions do not hold in practice. One well-observed example is the ordering of the filtering conditions in the WHERE clause. Reordering of those conditions leads to different action sequences. Furthermore, we identify another source of ambiguity in section 4.2, where a question can be expressed by different SQL queries with the same execution . These queries are equivalent from an end-user perspective. For both cases, we obtain multiple correct "reference" transition sequences for each training instance and there is no single target policy for our model to mimic during training. To solve this, we draw inspiration from syntactic parsing and define non-deterministic oracles BID8 ) that allow our parser to explore alternative correct action sequences. In contrast, the training mechanism we discussed in Section 3 is called static oracles. We denote the oracle as O that returns a set of correct continuation actions O(x, a <t) at time step t. Taking any action from the set can lead to some desired parse among a potentially large set of correct . The training objective for each instance L x is defined as: DISPLAYFORM0 where a <i denotes the sequence of actions a 1,..., a i−1 and a i = arg max a∈O(x,a<i) s a, the most confident correct action to take as decided by the parser during training. When O is a static oracle, it always contains a single correct action. In that scenario, Equation 1 is reduced to a naïve crossentropy loss. When O is non-deterministic, the parser can be exposed to different correct action sequences and it is no longer forced to conform to a single correct action sequence during training. Training a text-to-SQL parser is known to suffer from the so-called "order-matters" issue. The filtering conditions of the SQL queries do not presume any ordering. However, an incremental parser must linearize queries and thus impose a pre-defined order. A correct prediction that differs from a golden labeling in its ordering of conditions then may not be properly rewarded. Prior work has tackled this issue through reinforcement learning BID36 ) and a modularized sequenceto-set solution BID32. The former lowers optimization stability and increases training time, while the latter complicates model design to capture inter-dependencies among clauses: information about a predicted filtering condition is useful for predicting the next condition. We leverage non-deterministic oracles to alleviate the "order-matters" issue. Our model combines the advantage of an incremental approach to leverage inter-dependencies among clauses and the modularized approach for higher-quality training signals. Specifically, at intermediate steps for predicting the next filtering condition, we accept all possible continuations, i.e. conditions that have not been predicted yet, regardless of their linearized positions. For the example in Figure 1, in addition to the transition sequence we gave in Section 3.1, our non-deterministic oracles also accept CONDCOL(c 2) as a correct continuation of the second action. If our model predicts this action first, it will continue predicting the second filtering condition before predicting the first. In preliminary experiments, we observed that a major source of parser errors on the development set is incorrect prediction of implicit column names. Many natural language queries do not explicitly mention the column name of the filtering conditions. For example, the question in Figure 1 does not mention the column name "Name". Similarly, a typical question like "What is the area of Canada?" does not mention the word "country". For human, such implicit references make natural language queries succinct, and the missing information can be easily inferred from context. But for a machine learning model, they pose a huge challenge. We leverage the non-deterministic oracles to learn the aforementioned implicit column name mentions by accepting the prediction of a special column name, ANYCOL. During execution, we expand such predictions into disjunction of filtering conditions applied to all columns, simulating the intuition why a human can easily locate a column name without hearing it from the query. For the example in Figure 1, in addition to the action CONDCOL(c 1), we also allow an alternative prediction CONDCOL(ANYCOL). When the latter appears in the query (e.g. ANYCOL="Willis Tower"), we expand it into a disjunctive clause (Rank="Willis Tower" OR Name="Willis Tower" OR ...). With our non-deterministic oracles, when column names can be unambiguously resolved using the filtering values, we accept both ANYCOL and the column name as correct actions during training, allowing our models to predict whichever is easier to learn. In our experiments, we use the default train/dev/test split of the WikiSQL dataset. We evaluate our models trained with both the static oracles and the non-deterministic oracles on the dev and test split. We report both logical form accuracy (i.e., exact match of SQL queries) and execution accuracy (i.e., the ratio of predicted SQL queries that in the same answer after execution). The execution accuracy is the metric that we aim to optimize. We largely follow the preprocessing steps in prior work of BID5. Before the embedding layer, only the tokens which appear at least twice in the training data are retained in the vocabulary, the rest are assigned a special "UNK" token. We use the pre-trained GloVe embeddings BID23, and allow them to be fine-tuned during training. Embeddings of size 16 are used for the actions. We further use the type embeddings for the natural language queries and column names following BID34: for each word w i, we have a discrete feature indicating whether it appears in the column names, and vice versa for c j. These features are embedded into 4-dimensional vectors and are concatenated with word embeddings before being fed into the biLSTMs. The encoding bi-LSTMs have a single hidden layer with size 256 (128 for each direction). The decoder LSTM has two hidden layers each of size 256. All the attention connections adopt the dot-product form as described in Section 3.2.For the training, we use a batch size of 64 with a dropout rate of 0.3 to help with the regularization. We use Adam optimizer BID14 with the default initial learning rate of 0.001 for the parameter update. Gradients are clipped at 5.0 to increase stability in training. The main are presented in TAB4. Our model trained with static oracles achieves comparable with the current state-of-the-art Coarse2Fine BID5 and MQAN models. On top of this strong model, using non-deterministic oracles during training leads to a large improvement of 2.1% in terms of execution accuracy. The significant drop in the logical form accuracy is expected, as it is mainly due to the use of ANYCOL option for the column choice: the ing SQL query may not match the original annotation. We further separate the contribution of "order-matters" and ANYCOL for the non-deterministic oracles. When our non-deterministic oracles only address the "order-matters" issue as described in Section 4.1, the model performance stays roughly the same compared with the static-oracle model. We hypothesize that it is because the ordering variation presented in different training instances is already rich enough for a vanilla sequence-to-action model to learn well. Adding ANYCOL to the oracle better captures the implicit column name mentions and has a significant impact on the performance, increasing the execution accuracy from 81.8% to 83.7%.Our incremental parser uses a greedy strategy for decoding, i.e. picking the highest scoring action predicted by the policy. A natural extension is to expand the search space using beam search decoding. We further incorporate the execution-guided strategy Table 3: Execution accuracy (%) and decoding speed of our models on the test set of WikiSQL, with varying decoding beam size. The notation "+ EG (k)" is as in TAB4.errors and empty . The key insight is that a partially generated output can already be executed using the SQL engine against the database, and the execution can be used to guide the decoding. The decoder maintains a state for the partial output, which consists of the aggregation operator, selection column and the completed filtering conditions until that stage in decoding. After every action, the execution-guided decoder retains the top-k scoring partial SQL queries free of runtime exceptions and empty output. At final stage, the query with the highest likelihood is chosen. With k = 5, the execution-guided decoder on top of our previous best-performing model achieves an execution accuracy of 87.1% on the test set, setting a new state of the art. We also report the performance of the static oracle model with execution-guided decoding in Table 3. It comes closely to the performance of the non-deterministic oracle model, but requires a larger beam size, which translates to an increase in the decoding time. To test whether our model can generalize to other datasets, we perform experiments with the ATIS dataset BID25 BID3. ATIS has more diverse SQL structures, including queries on multiple tables and nested queries. To be compatible with our task setting, we only retain examples in the ATIS dataset that are free of nested queries, containing only AND operations and no INNER JOIN operators. We perform table joins and create a single table to be included in the input to our models along with the natural language question. The reduced dataset consists of 933 examples, with 714/93/126 examples in the train/dev/test split, respectively. Our models trained with the static and non-deterministic oracles (without ANYCOL) achieve accuracy of 67.5% and 69.1% on the test set, respectively. The improvement gained from using nondeterministic oracles during training validates our previous hypothesis: ATIS is a much smaller dataset compared with WikiSQL, therefore explicitly addressing "order-matters" helps here. We didn't apply ANYCOL due to the nature of ATIS data. WikiSQL, introduced by BID36, is the first large-scale dataset with annotated pairs of natural language queries and their corresponding SQL forms on a large selection of table schemas. While its coverage of SQL syntax is weaker than previous datasets such as ATIS BID25 BID3 and GeoQuery BID35, WikiSQL is highly diverse in its questions, BID29 BID32 BID34 BID5 BID21.NL2SQL is a special case of semantic parsing. The task of semantic parsing maps natural language to a logical form representing its meaning, and has been studied extensively by the natural language processing community (see Liang 2016 for a survey). The choice of meaning representation is usually task-dependent, including lambda calculus BID31, lambda dependency-based compositional semantics (, λ-DCS), and SQL BID36. Neural semantic parsing, on the other hand, views semantic parsing as a sequence generation problem. It adapts deep learning models such as those introduced by BID26; BID1; BID28. Combined with data augmentation BID13 BID12 or reinforcement learning BID36, sequence-to-sequence with attention and copying has already achieved state-of-the-art on many datasets including WikiSQL.The meaning representation in semantic parsing usually has strict grammar syntax, as opposed to target sentences in machine translation. Thus, models are often constrained to output syntactically valid . BID4 propose models that generate tree outputs through hierarchical decoding and models that use sketches to guide decoding, but they do not explicitly deal with grammar constraints. In contrast, BID33 and directly utilize grammar productions during decoding. Training oracles have been extensively studied for the task of syntactic parsing, where incremental approaches are common BID8. For syntactic parsing, due to the more structurally-constrained nature of the task and clearly-defined partial credits for evaluation, dynamic oracles allow the parsers to find optimal subsequent actions even if they are in some sub-optimal parsing states BID7 BID9 BID2. In comparison, non-deterministic oracles are defined for the optimal parsing states that have potential to reach a perfect terminal state. To the best of our knowledge, our work is the first to explore non-deterministic training oracles for incremental semantic parsing. In this paper, we introduce a sequence-to-action incremental parsing approach for the NL2SQL task. With the observation that multiple SQL queries can have the same or very similar semantics corresponding to a given natural language question, we propose to use non-deterministic oracles during training. On the WikiSQL dataset, our model trained with the non-deterministic oracles achieves an execution accuracy of 83.7%, which is 2.3% higher than the current state of the art. We also discuss using execution-guided decoding in combination with our model. This leads to a further improvement of 3.4%, achieving a new state-of-the-art 87.1% execution accuracy on the test set. To the best of our knowledge, our work is the first to use non-deterministic oracles for training incremental semantic parsers. Designing such non-deterministic oracles requires identification of multiple correct transition sequences for a given training instance, and an algorithm that decides the possible continuations for any intermediate state that will lead to one of the desired terminal states. We have shown promising for WikiSQL and filtered ATIS dataset and it would be interesting to extend our work to other more complex NL2SQL tasks and to other semantic parsing domains.

Represent the short summary for finding the corresponding scientific article (long)

We design incremental sequence-to-action parsers for text-to-SQL task and achieve SOTA results. We further improve by using non-deterministic oracles to allow multiple correct action sequences.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the scientific article to find a short tldr

Question: We propose Efficient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods. In ENAS, a controller learns to discover neural network architectures by searching for an optimal path within a larger model. The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected path is trained to minimize the cross entropy loss. On the Penn Treebank dataset, ENAS can discover a novel architecture thats achieves a test perplexity of 57.8, which is state-of-the-art among automatic model design methods on Penn Treebank. On the CIFAR-10 dataset, ENAS can design novel architectures that achieve a test error of 2.89%, close to the 2.65% achieved by standard NAS . Most importantly, our experiments show that ENAS is more than 10x faster and 100x less resource-demanding than NAS. Neural architecture search (NAS) has been applied successfully to design model architectures for image classification and language modeling BID0 BID3 BID6. NAS however is computationally expensive and time consuming: for example, use 450 GPUs and train for 3-4 days. Meanwhile, using less resources tends to produce less compelling BID31 BID0.The main computational bottleneck of NAS is the training of each child model to convergence to measure its accuracy. We believe that it is very inefficient and wasteful to train every child model and then throw away all the trained weights even though the child models have much in common. The graph represents the entire search space while the red arrows define a model in the search space, which is decided by a controller. Here we assume that node 1 is the input to the model whereas nodes 3, 5, and 6 are the outputs of the model. The goal of this work is to remove this inefficiency by enabling more sharing between the child models. This idea is similar to the concept of weight inheritance in neuro-evolution (e.g., BID33). To understand our method, we first need to understand the standard NAS. In standard NAS BID0, an RNN controller is trained by policy gradient to search for a good architecture, which is basically a computational graph. Our observation is that all of the graphs, that NAS has iterated over, can be viewed as sub-graphs of a larger graph. In other words, we can represent the space of these graphs as a single directed acyclic graph (DAG). As illustrated in FIG0, a neural network architecture can be found by taking a subset of edges in this DAG. This design is advantageous because it enables sharing parameters among all architectures in the search space. There is growing interest in improving the efficiency of neural architecture search. Concurrent to our work are the promising ideas of using learning curve prediction to skip bad models BID1, predicting the accuracies of models before training BID7, using iterative search method for architectures of growing complexity BID25 ), or using hierarchical representation of architectures BID26. Our method is also inspired by the concept of weight inheritance in neuroevolution, which has been demonstrated to have positive effects at scale BID33.Closely related to our method are other recent approaches that avoid training each architecture from scratch, such as convolutional neural fabrics -ConvFabrics BID34 and SMASH BID5. These methods are more computationally efficient than standard NAS. However, the search space of ConvFabrics is not flexible enough to include novel architectures, e.g. architectures with arbitrary skip connections as in. Meanwhile, SMASH can design interesting architectures but requires a hypernetwork to generate the weights, conditional on the architectures. While a hypernetwork can efficiently rank different architectures, as shown in the paper, the real performance of each network is different from its performance with parameters generated by a hypernetwork. Such discrepancy in SMASH can cause misleading signals for reinforcement learning. Even more closely related to our method is PathNet BID10, which uses evolution to search for a path inside a large model for transfer learning between Atari games. In the following, we will first present our search space to design recurrent cells for recurrent networks. We will then explain how to train, and infer with the controller. Finally, we will explain our search space to design convolutional architectures. To facilitate our discussion of ENAS, we first describe how we employ ENAS to design a recurrent cell. The search space of ENAS is a Directed Acyclic Graph as mentioned in Section 1. The DAG has N nodes where the edges represent the flow of information between these N nodes. Similar to NAS, ENAS has a controller RNN, which decides which edges are activated and which computations are performed at each node. To create a recurrent cell, the controller RNN samples N blocks of decisions. Here we illustrate the ENAS mechanism via a simple example for a recurrent cell with N = 4 computational nodes. Let x t be the input signal for a recurrent cell (e.g., word embedding), and h t−1 be the output from the previous time step. The example cell, which we visualize in FIG1, is sampled as follows.1. At node 1: The controller first samples an activation function. In our example in FIG1, it chooses the tanh activation function, which means that node 1 of the recurrent cell should compute DISPLAYFORM0 ). 2. At node 2: The controller then samples a previous index and an activation function. In our example, it chooses the previous index 1 and the activation function ReLU. Thus, node 2 of the recurrent cell should compute h 2 = ReLU(h 1 · W2,1). 3. At node 3: The controller again samples a previous index and an activation function. In our example, it chooses the previous index 2 and the activation function ReLU. Therefore, h 3 = ReLU(h 2 · W (h) 3,2 ). 4. At node 4: The controller again samples a previous index and an activation function. In our example, it chooses the previous index 1 and the activation function tanh, leading to h 4 = tanh (h 1 · W4,1). 5. For the output, we simply average all the loose ends, which are the nodes that are not input to any other nodes. In our example, since the indices 3 and 4 were never sampled to be the input for any node, the recurrent cell uses their average (h 3 + h 4)/2 as its output. In the example above, we note that for each pair of nodes j <, there is an independent parameter matrix W (h),j. As shown in the example, the controller decides which parameter matrices are used, by choosing the previous indices. Therefore, in ENAS, all recurrent cells in a search space share the same set of parameters. Our search space includes an exponential number of configurations. Specifically, if the recurrent cell has N nodes and we allow 4 activation functions (namely tanh, ReLU, identity, and sigmoid), then the search space has 4 N ×N! configurations. In our experiments, N = 12, which means there are approximately 10 15 models in our search space. Our controller network is a two-layer LSTM BID16 which samples decisions via softmax classifiers. The controller network samples these decisions in an autoregressive fashion: the decision in the previous step is fed as input embedding into the next step. At the first step, the controller network receives an empty embedding as input. In ENAS, there are two sets of learnable parameters: the parameters of the controller LSTM, denoted by θ, and the shared parameters of the child models, denoted by ω. The training procedure of ENAS consists of two alternating phases. The first phase trains ω, the shared parameters of the child models, on a whole pass through the training data set, and the second phase trains θ, the parameters of the controller LSTM, for a fixed number of steps. These two phases are alternated during the training of ENAS. In our experiments with the Penn Treebank dataset, for each phase of training ω, we train ω for 450 steps with SGD, each on a batch of 64 examples, where gradients are computed using back-propagation through time, truncated to 35 time steps. Meanwhile, for each phase of training θ, we train it for 2000 steps with the Adam optimizer and REINFORCE. More details of their training are as follows. Training the shared parameters ω of the child models. In this step, we fix the policy π(m; θ) and perform stochastic gradient descent (SGD) updates on ω to minimize the expected loss function DISPLAYFORM0 Here, L(m; ω) is the standard cross-entropy loss, computed on a minibatch of training data, with a model m sampled from π(m; θ). The gradient is computed via the Monte Carlo estimate DISPLAYFORM1 Note that Eqn 1 provides an unbiased estimate of the gradient DISPLAYFORM2 Therefore, according to Chapter 3.3.2 of BID4, with an appropriate learning schedule, the SGD updates of ω converge almost surely. While convergence is guaranteed, these updates on ω have an inherently larger variance than SGD performed on a fixed model m. Nevertheless, we find that M = 1 works just fine, i.e. we can update ω using the gradient from any single model m sampled from π(m; θ). As mentioned, we train ω for a whole pass through the training data set. Training the parameters of θ of the controller LSTM. In this step, we fix ω and update the policy parameters θ, aiming to maximize the expected reward DISPLAYFORM3 We employ the Adam optimizer BID20, for which the gradient is computed using REINFORCE BID37, with a moving average baseline to reduce variance. We compute the reward R(m, ω) on the validation set, rather than the training set to encourage ENAS, to select the model that generalizes well rather than the model that overfits the training set well. In our experiments with language modeling, the reward function is c/valid ppl, where the perplexity is computed on a minibatch, also sampled from the validation set. Later, in our experiments with image classification, the reward function is the classification accuracy on a minibatch of images sampled from the validation set. Inference. We discuss how to derive novel architectures from a trained ENAS model. Following, we sample several models from the trained policy π (m, θ). For each sampled model, we compute its reward on a single minibatch sampled from the validation set. We then take the model with the highest reward to re-train from scratch. We now discuss the search space for convolutional architectures. Recall that in the search space of the recurrent cell, the controller RNN samples two decisions at each decision block: 1) what previous node to connect to and 2) what activation function to use. In the search space for convolutional models, the controller RNN also samples two sets of decisions at each decision block: 1) what previous node to connect to and 2) what computation operation to use. The decision of what previous node to connect to allows the model to form skip connections BID15; whereas the decision of what computation operation to use sets a particular layer into convolution or average pooling or max pooing. These decisions help construct a layer in the convolutional model. To be even more flexible, instead of deciding if a particular layer is convolution or average pooling or max pooling, we change the search space to blend all these choices together. To achieve this, we treat all operations such as convolution, average pooling, and max pooling as channels, and allow the controller to select a mask over these channels. For example, in our experiments, we allow the controller to choose a mask over how many conv1x1, conv3x3, conv5x5, conv7x7, average pooling, max pooling operations to use (see FIG2 . Each operation at each layer in our network has its own convolutional parameters. During each data pass, only the parameters corresponding to the active channels are used and updated. The network of N layers, where each layer has 6 channels as described. Bottom: A block of the controller network, which consists of 6 binary masks, followed by the steps that sample skip connections. For the skip connections, at layer k, up to k − 1 mutually distinct previous indices are sampled. For example, at layer k = 5, let's suppose the controller samples {2, 4}, then the outputs of layer 2 and layer 4 will be concatenated and sent to layer 5 via skip connections. In our experiments with the CIFAR-10 dataset, the training of the controller LSTM and the child models are also alternating. For each phase of training θ, we train it for 2000 steps with the Adam optimizer and REINFORCE. Meanwhile, for each phase of training the shared parameter ω, we train it for 450 minibatches, each has 100 images, using Nesterov Momentum BID32. An alternative to designing the entire convolutional network is to design smaller modules and then fit repeats of them together BID40. FIG3 illustrates this approach, where the convolutional cell and reduction cell architectures are to be designed. We now discuss how to use ENAS to search for the architecture of these cells. Following, we sample both our convolutional cell and our reduction cell using an RNN controller. Specifically, at each decision block, the controller RNN samples for two sets of decisions: 1) two previous nodes to be used as inputs to the current block and 2) two operations to respectively apply to the two sampled nodes. We allow the following 5 operations: identity, separable convolution with kernel size 3x3 and 5x5, and average pooling and max pooling with kernel size 3x3. Note that each cell receives two input nodes, indexed by node 1 and node 2, corresponding to the outputs of the two previous cells in the entire network. FIG4 depicts an example run of our controller in the search space with 4 nodes. As with our other search spaces, each operation in each cell has its own parameters. During each data pass, only the relevant parameters are used and trained with their corresponding gradients. In the following, we will show our experimental with ENAS to design recurrent cells on the Penn Treebank dataset and to design convolutional architectures on the CIFAR-10 dataset. We then present an ablation study which shows the role of ENAS in discovering novel architectures, as well as details regarding the efficiency of ENAS. Dataset. We first apply ENAS to the task of language modeling, whose goal is predict next words in a text given a history of previous words, by fitting a probabilistic model over sentences. We use the Penn Treebank dataset (PTB) BID28, a well-studied language modeling benchmark. In this experiment, we show that ENAS discovers a novel recurrent cell which, without extensive hyper-parameters tuning, can outperform models with the same or more parameters. Training details. Our controller is trained using Adam , with a learning rate of 0.001. We use a tanh constant of 2.5 and a temperature of 5.0 for the sampling logits BID2. We also add the controller's sample entropy to the reward, with a weight of 10 −4. Additionally, we augment the simple transformations between the constructed recurrent cell's nodes with highway connections BID41. For instance, instead of having h 2 = ReLU(h 1 · W (h) 2,1 ) as shown in the example from Section 3.1, we have DISPLAYFORM0 2,1 ). More details can be found in Appendix A. A novel RNN cell found in our search space is shown in FIG5.The shared parameters of the child models ω are trained using SGD with a learning rate of 0.2, decayed by a factor of 0.9 after every 3 epochs starting at epoch 15, for a total of 150 epochs. During the architecture search process, following BID29, we randomly reset the starting state with probability of 0.001. We also tie the model's word embeddings matrix with its softmax matrix BID18. When retraining the architecture recommended by the controller, however, we use variational dropout BID11, an 2 regularization with weight decay of 10 −7, and a state slowness regularization of 0.0005 BID30.Results. Running on a single Nvidia GTX 1080Ti GPU, ENAS finds the recurrent cell in less than 10 hours. This cell is depicted in FIG5. Table 1 presents our in comparison with other methods. The ENAS cell, with 24M parameters, outperforms the NAS cell and has a similar performance to the LSTM model that uses extensive hyperparameters tuning BID29 ), which we did not do. Our ENAS cell has a few interesting properties. First, all non-linearities in the cell are either ReLU or tanh, even though the search space also has two other functions: identity and sigmoid. We suspect this cell is a local optimum, similar to the observations by. When we randomly pick some nodes and switch the non-linearity into identity or sigmoid, the perplexity increases up to 8 points. When we randomly, swith some ReLU nodes into tanh or vice versa, the perplexity also increases, but only up to 3 points. Dataset. The CIFAR-10 dataset BID22 ) consists of 50,000 training images and 10,000 test images. We use the standard data pre-processing and augmentation techniques, i.e., subtracting the mean and dividing the standard deviation from each channel computed on the training images, centrally padding the training images to 40 × 40 and randomly cropping them back to 32 × 32, and randomly flipping them horizontally. Search spaces. In Section 3.3, we present a search space for convolutional architectures in which the controller can make decisions over skip connections and the mask over the channels. To improve our , we additionally explore two restricted versions of this search space: one where the controller only needs to make decisions of the mask over the channels and one where the controller only needs to make decisions of the skip connections. More details are available in Appendix B.1. Searching for the masks over channels. We fix a pattern of skip connections and search for the masks at each branch and each layer in a 12-layer network. The pattern that we use is the dense pattern BID17. 2. Searching for skip connections. We force all the convolutions to have the filter size of 3 × 3, and only search for the skip connections.3. Searching for convolutional and reduction cells. We search for both cells as discussed in Section 3.4.Training details. The shared parameters ω are trained with Nesterov momentum , where the learning rate follows the cosine schedule with l max = 0.05, l min = 0.001, T 0 = 10 and T mul = 2 BID27. Each architecture search is run for 10 + 20 + 40 + 80 + 160 = 310 epochs. The parameter ω is initialized from a scaled Gaussian as described in BID14. We also apply an 2 weight decay of 10 −4. The same settings are employed to train the architectures recommended by the controller. The policy parameters θ are initialized uniformly in [−0.1, 0.1], and trained with the Adam optimizer at a learning rate of 10 −3. We additionally utilize three techniques to prevent the premature convergence of REINFORCE. First, we apply a temperature τ = 5.0 and a tanh constant c = 2.5 to the controller's logits. Second, we add to the reward the entropy term of the controller's samples weighted by λ ent = 0.1, which discourages convergence BID38. Lastly, we enforce the sparsity in the skip connections by adding to the reward the Kullback-Leibler divergence between: 1) the skip connection probability between any two layers and 2) our chosen probability ρ = 0.4, which represents the prior belief of a skip connection being formed. The KL divergence term is weighted by λ kl = 0.5.Tricks to stabilize and improve training. We find the following tricks crucial for achieving good performance with ENAS.• Structure of Convolutional Layers. Each convolutional operation in our method is followed by a batch normalization BID19 and then a ReLU layer. We find the alternate setting of batch norm-conv-ReLU to have worse .• Stabilizing Stochastic Skip Connections. If a layer receives skip connections from multiple layers before it, then these layers' outputs are concatenated in their depth dimension, and then a convolution of filter size 1 × 1 (followed by a batch normalization layer and a ReLU layer) is performed to ensure the number of output channels is still equal to C.• Global Average Pooling. After the final convolutional layer, we average all the activations of each channel and then pass them to the Softmax layer. This trick was introduced by BID24, with the purpose of reducing the number of parameters in the dense connection to the Softmax layer to avoid overfitting. The last two tricks are extremely important, since the gradient updates of the shared parameters ω, as described in Eqn 1, have a very high variance. In fact, we find that without these last two tricks, the training of ENAS is very unstable. Results. Table 2 summarizes the test errors of ENAS and other approaches. As can be seen from the table, ENAS successfully found several architectures that outperform other automatic model designing approaches with the same usage of computing resource. In particular, in the general search space, ENAS takes 15.6 hours to find a model that achieves 4.23% error rate on CIFAR-10. This model outperforms all but one model reported by, while taking 30x less time and using 800x less computing resource to discover. In the restricted search space over the masks, ENAS takes 11.6 hours to find a model that achieves the test error of 4.35%. The ing model, depicted in FIG6 -Top Left, almost always has 64 or 96 channels at each branch and each layer, indicating that the controller does not choose to activate all blocks. This is the desired behavior, as activating all channels would over-parametrize the model and in overfitting. Moreover, the fact that the model found in a restricted search space has similar performance to the model found in the general search space indicates that ENAS can discover skip connection patterns that are comparable to the dense pattern, which is the state-of-the-art human-designed architecture on CIFAR-10 BID17.In the restricted search space over skip connections, ENAS takes 12.4 hours to discover the pattern of skip connections depicted in FIG6 BID34 16 21.2 7.43 Macro NAS with Q-Learning BID0 11 11.2 6.92 Net Transformation BID6 17 19.7 5.70 FractalNet BID23 21 38.6 4.60 SMASH BID5 211 16.0 4.03 NAS 39 7.1 4.47 NAS + more filters Table 2: Classification error rates of ENAS and other methods on CIFAR-10. In this table, the first block presents the state-of-the-art models, all of which are designed by human experts. The second block presents various approaches that design the entire network. ENAS outperforms all these methods but NAS, which requires much more computing resource and time. The last block presents techniques that design modular cells which are used to build a large model. ENAS outperforms MicroNAS, which uses 32 GPUs to search, and achieves similar performance with NASNet-A.that skip connections are formed much more densely at higher layers than at lower layers, where most connections are only between consecutive layers. The model has the test error of 5.04%, which is slightly worse than the one found in the restricted search space over masks. However, if we increase the number of output channels from 256 to 512, then the network achieves the test error of 3.87%.In the search space over cells, ENAS takes 11.5 hours to discover the convolution cell and the reduction cell, which are shown in FIG6 -Bottom. With the convolutional cell replicated for 6 times, ENAS achieves 3.54% test error, which is on par with the 3.41% error of NASNet-A. With CutOut , ENAS's error decreases to 2.89%, compared to 2.65% by NASNet-A. However, as discussed in Appendix B, our ENAS cells are trained for only 310 epochs, compared to 600 epochs by NAS. Sanity Check and Ablation Study. To understand the role of ENAS, we carry out two control experiments. In the first study, we uniformly at random pick a configuration of channels and skip connections and just train a model. As a , about half of the channels and skip connections are selected, ing in a model with 47.1M parameters and an error rate of 5.86%. This error rate is significantly worse than the models designed by ENASand has many more parameters. In the second study, we only train ω and do not update the controller. The effect is similar to dropout with a rate of 0.5 on both the channels and the skip connections. At convergence, the model has the error rate of 11.92%. On the validation set, the ensemble of 250 Monte Carlo configurations of the trained model could only reach 8.99% test error rate. We therefore conclude that the appropriate training of the ENAS controller is crucial for good performance. Neural Architecture Search (NAS) is an important advance that allows faster architecture design for neural networks. However, the computational expense of NAS prevents it from being widely adopted. In this paper, we presented ENAS, an alternative method to NAS, that requires three orders of magnitude less resources×time. The key insight of our method is to share parameters across child models during architecture search. This insight is implemented by having NAS search for a path within a larger model. We demonstrate empirically that the method works well on both CIFAR-10 and Penn Treebank datasets. The shared parameters ω between different recurrent cells thus consist of all the matrices DISPLAYFORM0,j, and W (h),j. The controller decides the connection j and the activation function f, for each ∈ {2, 3, ..., N}. The layers that are never selected by any subsequent layers are averaged and sent to a softmax head, or to higher recurrent layers. As in the case of convolutional models, to stabilize the training of ω, we add a batch normalization layer after the average of the layers that are not selected. B Details for CIFAR-10 Search Spaces B.1 Details on Search Space 1: ChannelsWe use a block size of S = 32, ing in C/S = 256/32 = 8 blocks per branch per layer. Each branch configuration has its own embedding and softmax head. To elaborate, this means that a time step in the controller RNN that predicts the configuration for any branch should have a softmax matrix of size H × (2 C/S − 1), where H = 64 is the hidden dimension of the RNN, and 2 C/S − 1 = 255 is the number of possible binary masks for that branch. Each branch also has an embedding matrix of size (2 C/S − 1) × H, from which the row corresponding to the sampled binary mask is selected and sent to the next time step. Layers 4 and 8 of our 12-layer network are max pooling layers with a kernel size of 2 × 2 and a stride of 2, and reduce each spatial dimension of the layers' outputs by a factor of 2. Within each group of 3 layers where the spatial dimensions of the layers remain constant, we connect each layer to all layers before it BID17. We use 3 × 3 convolutions with 48 output channels at all layers. The controller RNN for this search space has the same form as the controller RNN that is depicted in FIG2, with two modifications. First, each block has only 2 time steps (as opposed to 7 in the general search space): the time step that predicts the mask for the convolution of filter size 3 × 3, which we force to always activate all channels; and the anchor time step, which we allow to sample multiple previous indices. Second, our controller is thus allowed to form skip connections between arbitrary layers, but forming such connections between layers with different spatial dimensions would in compilation failures. To circumvent, after each max pooling in the network, we centrally pad the output so that its spatial dimensions remain unchanged. We perform a 3×3 convolution on the image to create the outputs for the first convolutional cell. After that, in the first 6 convolution cells, each separable convolution has 32 output channels. Each reduction cell is applied in the same way as the convolutional cell, with the only modification being each operation is applied with a stride of 2. When the cells are found, the final models (both with and without CutOut) are trained for 310 epochs, using the cosine learning schedule, where the reset cycle is originally set to 10 and then doubled after each reset. Additionally, we insert an auxiliary head at the layer immediately before the second application of the reduction cell BID35.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

An approach that speeds up neural architecture search by 10x, whilst using 100x less computing resource.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.