Question: Conventional deep learning classifiers are static in the sense that they are trained on a predefined set of classes and learning to classify a novel class typically requires re-training. In this work, we address the problem of Low-shot network-expansion learning. We introduce a learning framework which enables expanding a pre-trained (base) deep network to classify novel classes when the number of examples for the novel classes is particularly small. We present a simple yet powerful distillation method where the base network is augmented with additional weights to classify the novel classes, while keeping the weights of the base network unchanged. We term this learning hard distillation, since we preserve the response of the network on the old classes to be equal in both the base and the expanded network. We show that since only a small number of weights needs to be trained, the hard distillation excels for low-shot training scenarios. Furthermore, hard distillation avoids detriment to classification performance on the base classes. Finally, we show that low-shot network expansion can be done with a very small memory footprint by using a compact generative model of the base classes training data with only a negligible degradation relative to learning with the full training set. In many real life scenarios, a fast and simple classifier expansion is required to extend the set of classes that a deep network can classify. For example, consider a cleaning robot trained to recognize a number of objects in a certain environment. If the environment is modified with an additional novel object, it is desired to be able to update the classifier by taking only a few images of that object and expand the robot classifier. In such a scenario, the update should be a simple procedure, based on a small collection of images captured in a non-controlled setting. Furthermore, such a low-shot network update should be fast and without access the entire training set of previously learned data. A common solution to classifier expansion is fine-tuning the network BID6. However fine-tuning requires keeping a large amount of base training data in memory, in addition to collecting sufficient examples of the novel classes. Otherwise, fine-tuning can lead to degradation of the network accuracy on the base classes, also known as catastrophic forgetting BID0. In striking contrast, for some tasks, humans are capable of instantly learning novel categories. Using one or only a few training examples humans are able to learn a novel class, without compromising previously learned abilities or having access to training examples from all previously learned classes. We consider the classifier expansion problem under the following constraints:1. Low-shot: very few samples of the novel classes are available. 2. No forgetting: preserving classification performance on the base classes. 3. Small memory footprint: no access to the base classes training data. In this work we introduce a low-shot network expansion technique, augmenting the capability of an existing (base) network trained on base classes by training additional parameters that enables to classify novel classes. The expansion of the base network with additional parameters is performed in the last layers of the network. To satisfy low-shot along with no-forgetting constraints, we present a hard distillation framework. Distillation in neural networks BID5 is a process for training a target network to imitate another network. A loss function is added to the target network so that its output matches the output of the mimicked network. In standard soft distillation the trained network is allowed to deviate from the mimicked network. Whereas hard distillation enforces that the output of the trained network for base classes matches the output of the mimicked network as a hard constraint. We achieve hard distillation by keeping the weights of the base network intact, and learn only the newly added weights. Network expansion with hard distillation yields a larger network, distilling the knowledge of the base network in addition to augmented capacity to classify novel classes. We show that in the case of low-shot (only 1-15 examples of a novel class), hard distillation outperforms soft distillation. Moreover, since the number of additional parameters in the expanded network is small, the inference time of the new network is nearly identical to the base network. To maintain a small memory footprint, we refrain from saving the entire training set. Instead, we present a compact generative model, consisting of a collection of generative models fitted in the feature space to each of the base classes. We use a Gaussian Mixture Model (GMM) with small number of mixtures, and show it inflicts a minimal degradation in classification accuracy. Sampling from the generative GMM model is fast, reducing the low-shot training time and allowing fast expansion of the network. We define a benchmark for low-shot network expansion. The benchmark is composed of a series of tests of increasing complexity, ranging from simple tasks where base and novel classes are from different domains and to difficult tasks where base and novel classes are from the same domain and shares objective visual similarities. We perform a comprehensive set of experiments on this challenging benchmark, comparing the performance of the proposed to alternative methods. To summarize, the main contributions of the paper are:1. A novel hard-distillation solution to a low-shot classifier expansion problem 2. GMM as a sufficient generative model to represent base classes in a feature space 3. A new benchmark for the low-shot classifier expansion problem 2 RELATED WORKS A common solution to the class-incremental learning problem is to use a Nearest-Neighbors (NN) based classifier in feature space. A significant advantage of a NN-based classifier is that it can be easily extended to classify a novel class, even when only a single example of the class is available (one-shot learning). However NN-based classifiers require keeping in the memory significant amount of training data from the base classes. BID7 proposed to use Nearest Class Mean (NCM) classifier, where each class is represented by a single prototype example which is the mean feature vector of all class examples. One major disadvantage of NCM and NN-based methods is that they are based on a fixed feature representation of the data. To overcome this problem BID7 proposed to learn a new distance function in the feature space using metric learning. The ideas of metric learning combined with the NN classifier resonate with recent work by on Matching Networks for one-shot learning, where both feature representation and the distance function are learned end-to-end with attention and memory augmented networks. The problem we consider in this paper is different from the one discussed by. We aim to expand existing deep classifier trained on large dataset to classify novel classes, rather than to create a general mechanism for one-shot learning. BID3 presented an innovative low-shot learning mechanism, where they proposed a Squared Gradient Magnitude regularization technique for an improved fixed feature representation learning designed for low-shot scenarios. They also introduced techniques to hallucinate additional training examples for novel data classes. In contrast, we present a method which aims to maximize performance in low-shot network expansion given a fixed representation, allowing expanding the representation based on novel low-shot data. Furthermore, in our work, we demonstrate the ability to expand the network without storing the entire base classes training data. Recently, BID9 proposed iCaRL -(Incremental Classifier and Representation Learning), to solve the class-incremental learning problem. iCaRL is based on Nearest-Mean-of-Exemplars classifier, similar to the NCM classifier of BID7. In the iCaRL method, the feature representation is updated and the class means are recomputed from a small stored number of representative examples of the base classes. During the feature representation update, the network parameters are updated by minimizing a combined classification and distillation loss. The iCaRL method was introduced as a class-incremental learning method for large training sets. In Section 4 we discuss its adaptation to low-shot network expansion and compare it to our method. BID11 proposed the Progressive Network for adding new tasks without affecting the performance of old tasks. They propose freezing the parameters that were trained on old tasks and expand the network with a additional layers when training a new task. BID15 proposed the Progressive learning technique which solves the problem of online sequential learning in extreme learning machines paradigm (OS-ELM). The purpose of their work is to incrementally learn the last fully-connected layer of the network. When a sample from a novel class arrives, the last layer is expanded with additional parameters. The Progressive learning solution updates the last layer only sequentially and only works in the ELM framework (does not update internal layers of the network). In another work BID14 proposed an incremental learning technique which augments the base network with additional parameters in last fully connected layer to classify novel classes. Similar to iCaRL, they perform soft distillation by learning all parameters of the network. Instead of keeping historical training data, they propose phantom sampling -hallucinating data from past distribution modeled with Generative Adversarial Networks. In this work we propose a solution that borrows ideas from freeze-and-expand paradigm, improved feature representation learning, network distillation and modeling past data with a generative model. We propose to apply expansion to the last fully connected layer of a base network to enable classification on novel classes, and to deeper layers to extend and improve the feature representation. However, in contrast to other methods BID9; BID15, we do not retrain the base network parameters, but only the newly introduced weights of the expansion. Moreover, the extended feature representation is learned from samples of base and novel classes. In contrast to BID3, where the improved feature representation is learned from simulating low-shot scenarios on the base classes only, before the actual novel data is available. Finally, in order to avoid keeping all historical training data, we use Gaussian Mixture Model of the feature space as a generative model for base classes. Assume a deep neural network is trained on K base classes with the full set of training data. This base network can be partitioned into two subnetworks: a feature extraction network and a classification network. The feature extraction network f rep maps an input sample x into a feature representation v ∈ R N. The classification network f cls maps feature vectors v into a vector of approximated class posterior probabilities P (k|v) which correspond to each one of K classes. The whole network can be represented as composition of two networks f net (x) = f cls (f rep (x)). For example, if the classification network consists of the last fully connected layer (FC) followed by softmax, then DISPLAYFORM0 N is class's k weights vector, and Z is the normalization factor. In the following we discuss how the pre-learned feature representation of feature extraction network can be leveraged to classify additional classes in low-shot scenario with only relatively minor changes to the classification subnetwork. First, we discuss how to expand the classification network to classify one additional class. We can expand f cls from a K-class classifier into K + 1 class classifier by adding a new weight vector w K+1 ∈ R N to the last FC layer. Thus, the K + 1 class probability is DISPLAYFORM0 v, where Z is a new normalization factor for K + 1 classes. We would like to preserve classification accuracy on the base classes to avoid catastrophic forgetting. To that end, during training we constrain to optimize of the w K+1 weights, while the vectors {w i} K i=1 are kept intact. We refer to this paradigm as hard distillation. By preserving the base classes weight vectors, we guarantee that as a of the last classification layer expansion the only new errors that can appear are between the novel class and the base classes, but not among the base classes. Moreover, the small number of newly learned parameters helps avoid over-fitting, which is especially important in low-shot scenarios. Similarly, we can expand the classification network to classify more than one novel class. are the number of input feature to F C2 before the expansion. Due to the small memory footprint constraint, we are unable to keep the entire training data of the base classes. As an alternative, we can use a generative model of the base classes and during training draw samples from the model. There are various approaches to this task, such as , , Pixel CNN van den , or conventional methods of non-parametric kernel density estimation BID4. However, it is usually hard to generate accurate samples from past learned distributions in the image domain, and these methods still require a significant amount of memory to store the model network parameters. Furthermore, since training typically requires thousands of samples, we prefer a generative model that allows fast sampling to reduce the low-shot phase training time. In our work, we use the Gaussian Mixture Model (GMM) density estimator as an approximate generative model of the data from the base classes. However, instead of approximating the generative distribution of the image data, we approximate a class conditional distribution of its feature representation. Thus, we model a GMM DISPLAYFORM0 where M is the number of mixtures for each bass class. In order to satisfy the small memory footprint constraint, we use a GMM which assumes feature independence, i.e., the covariance matrix Σ i of each Gaussian mixture is diagonal. We denote this model as Deep Feature GMM. If we have K classes, and the feature vectors dimensionality is N, the memory requirements for storing information about base classes is O(M KN). The feature representation v, which we learn a generative model for, can be from the last fully connected layer or from deeper layers. In Section 4.5, we evaluate the effectiveness of the use of the Deep Features GMM, showing that despite its compact representation, there is a minimal degradation in accuracy when training a classifier based only on data that is generated from the Deep Features GMM, compared to the accuracy obtained on the full training data. We apply standard data augmentation (random crop, horizontal flip, and color noise) to the input samples of the novel classes and create 100 additional samples variants from each of the novel class samples. These samples are passed through the feature extraction network f rep to obtain their corresponding feature representation. Note that new samples and their augmented variants are passed through f rep only once. As described in Section 3.1, we expand the classification subnetwork f cls and train the expanded network to classify novel classes in addition to the base classes. FIG0 illustrates the proposed method in the case where f cls is the last fully connected layer. As mentioned above, we only learn the N dimensional vector w K+1, which augments the K × N weight matrix of the FC layer. Each training batch is composed of base classes feature vectors drawn from the Deep Features GMM models learned from the base classes training data and the available samples of a novel class. The training batch is balanced to have equal number of generations/samples per class. Since the forward and backward passes are carried out by only the last FC layers, each iteration can be done very rapidly. We use SGD with gradient dropout (see below) to learn w K+1. More specifically, the weights update is done by: DISPLAYFORM0 where µ is the momentum factor, α is the learning rate and M is a binary random mask with probability p of being 1 (M is randomly generated throughout the low-shot training). That is, the gradient update is applied to a random subset of the learned weights. This SGD with gradient dropout is our heuristic variant for Noisy SGD proposed by BID1 which provably helps to escape saddle points. In Section 4.3 we demonstrate the contribution of the gradient dropout when only a few novel labeled samples are available. The procedure described in the previous subsections expands the last classification layer, but does not change the feature representation space. In some cases, especially in those which the novel classes are similar to the base classes, it is desirable to update and expand the feature representation. To expand the feature representation, we add new parameters to deeper layers of the network. This of course requires an appropriate expansion of all subsequent layers. To satisfy the hard distillation constraints, we enforce that the feature representation expansion does not affect the network output for the base classes. All weights in subsequent layers which connects the expanded representation to the base classes are set to zero and remain unchanged during learning. In FIG0 (b) we demonstrate an expansion of two last fully connected layers. The F C2 weight matrix is zero padded to adjust to the new added weights in F C1. Only the expansion to F C2 uses the new added features in F C1. The details of the representation learning expansion can be found in Appendix D. In this section, we evaluate the proposed low-shot network expansion method on several classification tasks. We design a benchmark which measures the performance of several alternative low-shot methods in scenarios that resemble real-life problems, starting with easier tasks (Scenario 1) to harder tasks (Scenario 2 & 3). In each experiment, we use a standard dataset that is partitioned to base classes and novel classes. We define three scenarios:Scenario 1 Generic novel classes: unconstrained novel and base classes which can be from different domains. Scenario 2 Domain specific with similar novel classes: base and novel classes are drawn from the same domain and the novel classes share visual similarities among themselves. Scenario 3 Domain specific with similar base and novel classes: base and novel classes are drawn from the same domain and each novel class shares visual similarities with one of the base classes. In each scenario we define five base classes (learned using the full train set) and up to five novel classes, which should be learned from up to 15 samples only. We compare the proposed method to several alternative methods for low-shot learning described in Section 4.2. Dataset for Scenario 1 For the task of generic classification of the novel classes we use the ImageNet dataset BID10, such that the selected classes were not part of the ILSVRC2012 1000 classes challenge. Each class have at least 1000 training images and 250 test images per class. The randomly selected 5 partition of 5 base classes and 5 novel classes. Dataset for Scenario 2 and Scenario 3 For these scenarios we use the UT-Zappos50K BID17 shoes dataset for fine-grained classification. We choose 10 classes representing different types of shoes each having more than 1,000 training images and 250 test images. To define similarity between the chosen classes, we fine-tune the base network (VGG-19) on the selected classes with the full dataset, and we use the confusion matrix as a measure of similarity between classes. Using the defined similarities, we randomly partition the 10 classes to 5 base and 2 novel classes, where for Scenario 2 we enforce similarity between novel classes, and for Scenario 3 we enforce similarity between novel and base classes. The confusion matrix representing the visual similarities and an illustration of the similarities between the base and novel classes is presented in FIG0 in Appendix C. In the proposed method we use the VGG-19 network BID12 trained on ImageNet ILSVRC2012 BID10 1000 classes as a feature extraction subnetwork f rep. In all three scenarios for training the classification subnetwork f cls on the base classes, we fine-tune the last two fully-connected layers of VGG-19 on the 5 selected base classes, while freezing the rest of the layers of f rep.We denote the method proposed in Section 3 as Generative Low-Shot Network Expansion: Gen-LSNE. We compare our proposed method to NCM BID7, and to the Prototype-kNN method which is an extension of NCM and the soft distillation based method inspired by iCaRL method , adapted for the low-shot scenario. We compare the proposed method to NCM classifier proposed by BID7. Additionally, we extend the NCM classifier by using multiple prototypes for each class, as in the Prototype-kNN classifier BID4. Both NCM and Prototype-kNN are implemented in a fixed feature space of the FC2 layer of the VGG-19 network. In our implementation of the Prototype-kNN, we fit a Deep Features GMM model with 20 mixtures for each of the base classes. We extract feature representation of all of the available samples from the novel classes. The Deep Features GMM centroids of the base feature vectors and the novel feature vectors of the samples are considered as a prototypes of each class. We set k for Prototype-kNN classifier to be the smallest number of prototypes per class (the number of prototypes in the novel classes is lower than the number of mixture in the base classes). The Prototype-kNN classification rule is the majority vote among k nearest neighbors of the query sample. If the majority vote is indecisive, that is, there are two or more classes with the same number of prototypes among the k nearest neighbors of query image, we repeat classification with k = 1. We want to measure the benefit of the hard distillation constraint in low-shot learning scenario. Thus, we formulate a soft distillation based method inspired by iCaRL BID9 and methods described by BID14 and BID15 as an alternative to the proposed method. In the iCaRL method, feature representation is updated by re-training the whole representation network. Since in low-shot scenario we have only a small number of novel class samples, updating the whole representation network is infeasible. Using the soft distillation method, we adapt to the low-shot scenario by updating only the last two fully connected layers F C1, F C2, but still use a combination of distillation and classification loss as in the iCaRL method. The iCaRL method stores a set of prototype images and uses the Nearest Mean Exemplar (NME) classifier at the final classification stage. In order to provide a fair comparison with the hard distillation method and uphold our memory restriction, we avoid storing prototypes in image domain, and use the proposed Deep-Features GMM as a generative model for the base-classes. Using NME classifier with prototypes of the base classes is in fact a special case of Prototype-kNN with k = 1. Therefore, in soft distillation method instead of NME we use a learned expanded network with additional parameters in last fully connected layers, which aligns with BID14 and BID15, and in our proposed hard-distillation method in Section 3.1. To summarize, soft distillation applies a distillation loss and allows the F C1, F C2 layers to adjust to the new data, while the proposed hard-distillation freezes F C1, F C2 and trains only the new (expanded) parameters without using a distillation loss. We denote the soft distillation based methods as Soft-Dis in the presented . In Section 3.3 we proposed using gradient dropout regularization on SGD as a technique to improve convergence and overcome overfitting in a low-shot scenario. We perform ablation experiments to assess the importance of the gradient dropout and train using both soft distillation (Soft-Dis) and proposed hard distillation (Gen-LSNE) with and without gradient dropout regularization. Scenario 1: Generic novel classes In this experiment, the base classification network is trained on five base classes and then expanded to classify two novel classes chosen at random. For each of the five class partitions (Section 4.1), we perform five trials by randomly drawing two novel classes from five novel classes available in the partition. The are an average of 25 trials. The of this experiment are presented in Figure 2. In Figure 6 in Appendix B we present detailed of the test error on the base and novels classes apart. Prototype-kNN and the Soft-Dis methods perform better on the base classes. However, our method is significantly better on the novel classes and the overall test error is considerably improved, particularly when the number of samples is small. In addition, we see the significant gain in accuracy delivered by the gradient dropout when the number of novel samples is lower than 3 samples. Furthermore, gradient dropout also improves the of the Soft-Dis method. The Prototype-kNN method is unable to effectively utilize the new available novel samples, however it best preserves the base class accuracy when the number of novel samples and base class prototypes is high (above 5, see Figure 6). Since the number of prototypes used is equal to the number of novel samples used, the addition of a novel samples/base class prototypes generally has a greater impact on the preservation of the base class accuracy. We assume that since the spread of the base class prototypes is better than that of the novel classes, then some novel samples are misclassified as some similar base class. NCM generally performs considerably better than Prototype-kNN, despite the use of less information from the base classes. However, NCM is unable to effectively utilize more novel samples when they are available. Gen-LSNE significantly outperforms NCM with a single novel sample, and overall outperforms all the tested method with nine and below samples per novel class. In this section we explore the effect of expansion of deeper layers, as described in Section 3.4. We partition the datasets as defined in 4.1 to five base and five novel classes, and we test a 10 classes classification task. We expand the feature representation which is obtained after F C1 layer with 5 new features. The size of the feature representation after the FC1 layer of VGG-19 is of dimension 4k. Thus, F C1 is expanded with 4k · 5 new weights. The are averaged over the 5 trails. FIG3 shows the obtained, we denote +5Inner as the experiments with the additional five shared representation features. We see a marginal gain in Scenario 1. However, we observe a significant gain in Scenario 2 and 3 when the number of samples increases (especially Scenario 2). Observe that Gen-LSNE significantly outperforms the alternative tested method in almost all of the tested cases. Note that the addition of the five shared additional representation features has no observable effect on Soft-Dis. In the Deep-features GMM evaluation experiment, we feed the full training data to the base network and collect the feature vectors before F C1, i.e., two FC layers before the classification output. We fit a GMM model to the feature vectors of each of the base classes with a varying number of mixture. We train the two last FC layers of the base network from randomly initialized weights, where the training is based on generating feature vectors from the fitted GMM. We measure the top-1 accuracy on the test set of the networks trained with GMM models and the base network trained with full training data on the datasets defined in 4.1. The difference in top-1 accuracy between the network trained with full data and the networks trained with GMM models represent degradation caused by compressing the data into simple generative model. The of the experiment presented in the TAB3 demonstrate that learning with samples from GMM models cause only a negligible degradation relative to learning with a full training set. We observe that the degradation in accuracy is not monotonic in the number of GMM mixtures, and that for many practical purposes a small number of mixture may be sufficient. We have introduced Gen-LSNE, a technique for low-shot network expansion. The method is based on hard-distillation, where pre-trained base parameters are kept intact, and only a small number of parameters are trained to accommodate the novel classes. We presented and evaluated the advantages of hard-distillation: (i) it gains significant increased accuracy (up to 20%) on the novel classes, (ii) it minimizes forgetting: less than 3% drop in accuracy on the base classes, (iii) small number of trained parameters avoids overfitting, and (iv) the training for the expansion is fast. We have demonstrated that our method excels when only a few novel images are provided, rendering our method practical and efficient for a quick deployment of the network expansion. We have also presented Deep-Features GMM for effective base class memorization. This computationally and memory efficient method allows training the network from a generative compact feature-space representation of the base classes, without storing the entire training set. In basic setting of a hard distillation, novel class classification is based on the base representation. We have also presented an extended version of network expansion, where the representation is updated by training the last deeper layers. We have shown that when more images of the novel classes are available, the adjusted representation is effective. Generally speaking, the representation can be improved as more samples are available, but at the expense of longer training, larger memory footprint, and risking forgetting. In the future we would like to continue exploring hard-distillation methods, in extremely low-shot classifier expansion, where only one or a handful of images of the novel class are provided. An additional research direction would be to maximize the information gain that is obtained from each novel image, aspiring towards human low-shot understanding. The fully connected layer FC1 is parametrized with weight matrix W ∈ R N ×V, where V is the dimensionality of input feature vector ν and N is the dimensionality of the output feature representation vector v. As was described in Section 3.4 we want to extend the feature representation v ∈ R N with E additional dimensionsṽ ∈ R E. We expand the weight matrix W with E × V additional weights as shown in FIG0. We denote the expanded weights as W exp ∈ R E×V. The expanded weights are to be learned from the novel data. We draw a random set of S novel examples. Let DISPLAYFORM0 denote the mean response of the FC1 to the set of novel examples. Let DISPLAYFORM1 be the E indexes of maximum elements of R. We initialize the expansion to FC1 layer W exp in the following manner: DISPLAYFORM2 where w j ∈ R V is the j th row of matrix W, ∼ N (0, std(W)), and α is a weight constant (in our experiments α = 0.25). This initialization allows the expansion of the feature representationṽ to have non zero responses (after ReLU) with respect to the novel samples. Since we operate in a Low-Shot scenario, where only few samples of novel classes are available, this weights initialization plays crucial role in convergence of FC1 extended weights. We initialize the subsequent layer FC2 in the following manner: let W ∈ R K×N be the weight matrix of FC2, where N is the dimensionality of the feature representation vector v and K is the number of base classes. Since v was expanded with additional E featuresṽ, and we want to allow classification of L novel classes, the dimension of expanded W will be (K + L) × (N + E). As was mentioned in Section 3.4 and illustrated in FIG0, the hard distillation constraint requires that W will be zero-padded with K × E zeros to avoid influence of the expanded featuresṽ on the output of the base network. In contrast, the expansion of W which we denote W exp ∈ R L×(N +E) should be encouraged to produce larger responses toṽ to improve learning. We initialize the expansion of FC2 layer W exp in the following manner: DISPLAYFORM3 where u ∈ {−1, 1} with probability 0.5 and Γ is an amplification parameter. In our experiments we used Γ = 2. We found that this initialization technique is crucial in assuring convergence of the added weights and the ability of the new weights to improve classification in low-shot setting.

Represent the TLDR for finding the corresponding scientific article

In this paper, we address the problem of Low-shot network-expansion learning

We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.


Question: Represent the article for finding a single-sentence TLDR

Question: People ask questions that are far richer, more informative, and more creative than current AI systems. We propose a neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decoder based deep neural network. From extensive experiments using an information-search game, we show that our method can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings. We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data. People can ask rich, creative questions to learn efficiently about their environment. Question asking is central to human learning yet it is a tremendous challenge for computational models. There is always an infinite set of possible questions that one can ask, leading to challenges both in representing the space of questions and in searching for the right question to ask. Machine learning has been used to address aspects of this challenge. Traditional methods have used heuristic rules designed by humans , which are usually restricted to a specific domain. Recently, neural network approaches have also been proposed, including retrieval methods which select the best question from past experience and encoder-decoder frameworks which map visual or linguistic inputs to questions (; ; ;). While effective in some settings, these approaches do not consider settings where the questions are asked about partially unobservable states. Besides, these methods are heavily data-driven, limiting the diversity of generated questions and requiring large training sets for different goals and contexts. There is still a large gap between how people and machines ask questions. Recent work has aimed to narrow this gap by taking inspiration from cognitive science. For instance, incorporates aspects of "theory of mind" in question asking by simulating potential answers to the questions, but the approach relies on imperfect agents for natural language understanding which may lead to error propagation. Related to our approach, proposed a powerful question-asking framework by modeling questions as symbolic programs, but their algorithm relies on hand-designed program features and requires expensive calculations to ask questions. We use "neural program generation" to bridge symbolic program generation and deep neural networks, bringing together some of the best qualities of both approaches. Symbolic programs provide a compositional "language of thought" for creatively synthesizing which questions to ask, allowing the model to construct new ideas based on familiar building blocks. Compared to natural language, programs are precise in their semantics, have clearer internal structure, and require a much smaller vocabulary, making them an attractive representation for question answering systems as well (; ;). However, there has been much less work using program synthesis for question asking, which requires searching through infinitely many questions (where many questions may be informative) rather than producing a single correct answer to a question. Deep neural networks allow for rapid question-synthesis using encoder-decoder modeling, eliminating the need for the expensive symbolic search and feature evaluations in. Together, the questions can be synthesized quickly and evaluated formally for quality groundtruth board partly revealed board example questions How long is the red ship? (size Red) Is purple ship horizontal? (== (orient Purple) H) Do all three ships have the same size? (=== (map (λ x (size x)) (set AllShips))) Figure 1: The Battleship task. Blue, red, and purple tiles are ships, dark gray tiles are water, and light gray tiles are hidden. The agent can see a partly revealed board, and should ask a question to seek information about the hidden board. Example questions and translated programs are shown on the right. We recommend viewing the figures in color. (e.g. the expected information gain), which as we show can be used to train question asking systems using reinforcement learning. In this paper, we develop a neural program generation model for asking questions in an informationsearch game similar to "Battleship" used in previous work (; ; . The model uses a convolutional encoder to represent the game state, and a Transformer decoder for generating questions. Building on the work of , the model uses a grammar-enhanced question asking framework, such that questions as programs are formed through derivation using a context free grammar. Importantly, we show that the model can be trained from human demonstrations of good questions using supervised learning, along with a data augmentation procedure that leverages previous work to produce additional human-like questions for training. Our model can also be trained without such demonstrations using reinforcement learning. We evaluate the model on several aspects of human question asking, including reasoning about optimal questions in synthetic scenarios, density estimation based on free-form question asking, and creative generation of genuinely new questions. To summarize, our paper makes three main contributions: 1) We propose a neural network for modeling human question-asking behavior, 2) We propose a novel reinforcement learning framework for generating creative human-like questions by exploiting the power of programs, and 3) We evaluate different properties of our methods extensively through three different experiments. Question generation has attracted attention from the machine learning community. Early research mostly explored rule-based methods which strongly depend on human-designed rules . Recent methods for question generation adopt deep neural networks, especially using the encoder-decoder framework, and can generate questions without handcrafted rules. These methods are mostly data-driven, which use pattern recognition to map inputs to questions. Researchers have worked on generating questions from different types of inputs such as knowledge base facts , pictures , and text for reading comprehension . However aspects of human question-asking remain beyond reach, including the goal-directed and flexible qualities that people demonstrate when asking new questions. This issue is partly addressed by some recent papers which draw inspiration from cognitive science. Research from and generate questions by sampling from a candidate set based on goal-oriented metrics. This paper extends the work of to overcome the limitation of the candidate set, and generate creative, goal-oriented programs with neural networks. Our work also builds on neural network approaches to program synthesis, which have been applied to many different domains (; ;). Those approaches often draw inspiration from computer architecture, using neural networks to simulate stacks, memory, and controllers in differentiable form . Other models incorporate Deep Reinforcement Learning (DRL) to optimize the generated programs in a goal oriented environment, such as generating SQL queries which can correctly perform a specific database processing task , translating strings in Microsoft Excel sheets , understanding and constructing 3D scenes and objects . Recent work has also proposed ways to incorporate explicit grammar information into the program synthesis process. design a special module to capture the grammar information as a prior, which can be used during generation. Some recent papers (; Figure (a) shows the network architecture. The board is represented as a grid of one-shot vectors and is embedded with a convolutional neural network. The board embedding and a sequence of symbols are inputted to a Transformer decoder to generate output vectors (details in section 4). PE means positional embeddings, and WE means word embeddings. (b) shows the derivation steps for program "(> (size Blue) 3)" using CFG. Non-terminals are shown as bold-faced, and terminals are shown in italic. The production rules used are shown next to each arrow. ) encode grammar with neural networks and use DRL to explicitly encourage the generation of semantically correct programs. Our work differs from these in two aspects. First, our goal is to generate informative human-like questions in the new domain instead of simply correct programs. Second, we more deeply integrate grammar information in our framework, which directly generates programs based on the grammar. In this paper, we work with a task used in previous work for studying human information search as well as question asking . The task is based on an information search game called "Battleship", in which a player aims to resolve the hidden layout of the game board based on the revealed information (Figure 1). There are three ships with different colors (blue, red, and purple) placed on a game board consisting of 6 × 6 grid of tiles. Each ship can be either horizontal or vertical, and takes 2, 3 or 4 tiles long. All tiles are initially turned over (light grey in Figure 1), and the player can flip one tile at a time to reveal an underlying color (either a ship color, or dark grey for water). The goal of the player is to determine the configuration of the ships (positions, sizes, orientations) in the least number of flips. In the modified version of this task studied in previous work (;, the player is presented with a partly revealed game board, and is required to ask a natural language question to gain information about the underlying configuration. As shown in Figure 1, the player can only see the partly revealed board, and might ask questions such as "How long is the red ship?" In this paper, we present this task to our computational models, and ask the models to generate questions about the game board. designed a powerful context free grammar (CFG) to describe the questions in the Battleship domain. The grammar represents questions in a LISP program-like format, which consists of a set of primitives (like numbers, colors, etc.) and a set of functions over primitives (like arithmetic operators, comparison operators, and other functions related to the game board). Another research by shows that it captures the full range of questions that people asked in an extensive dataset, mainly because the majority of this grammar is general functions which make it flexible enough. The grammar is able to generate an infinite set of other possible questions beyond collected human questions, capturing key notions of compositionality and computability. Figure 1 provides some examples of produced programs. The full grammar is provided in Appendix C. The neural network we use includes a Convolutional Neural Network (CNN) for encoding the input board, and a Transformer decoder for estimating the symbol distribution or selecting actions in different settings described below. The input x ∈ {0, 1} 6×6×5 is a binary representation of the 6x6 game board with five channels, one for each color to be encoded as a one-hot vector in each grid location. A simple CNN maps the input x to the encoder output e ∈ R 6×6×M, where M is the length of encoded vectors. Then a Transformer decoder takes e and a sequence of length L as input, and outputs a sequence of vectors where N o is the output size. As shown later, the input sequence and output vectors can be interpreted differently in different settings. The model is shown in Figure 2 (a), and details are described in Appendix A. Our model is compatible with both supervised and reinforcement training. Supervised training. In the supervised setting, the goal is to model the distribution of questions present in the training set. Each output y i ∈ R No is a symbol at position i in the program, where N o is the number of different symbols in the grammar. The model is trained with symbol-level cross entropy loss, and can be used to calculate the log-likelihood of a given sequence, or to generate a question symbol-by-symbol from left to right. Generation works as follows. Suppose at step t, a sequence of length t along with the encoded board is presented to the decoder. The model predicts the vector y t which represents the probability of each symbol to be chosen as next. Then we sample a symbol at location t + 1 and execute the decoder again with the new sequence, until an <eos> symbol is generated or maximum length is reached. Sequence-based RL. The framework can be adapted to generate a sequence of symbols without stepwise supervision, such that reward is provided only after the entire question is generated. Grammar-enhanced RL. Finally, the framework can be used with a novel grammar-enhanced RL training procedure. Figure 2 (b) illustrates the process of generating a program from the context-free grammar specified in. Beginning from the start symbol "A", at each step a production rule is chosen and applied to one of the non-terminals in the current string. The choice of rule is modeled as a Markov Decision Process, and we solve it with DRL. Each state is a partially derived string passed to the decoder, and we use the first output y 1 ∈ R No to represent the probability of selecting each production rule from all possible N o rules. After the rule is applied, the new string is passed back into the decoder, repeating until only terminals are contained in the sequence. We adopt the leftmost derivation here to avoid the ambiguity of parsing order, so at each step the left-most non-terminal will be replaced. In the first experiment, we designed three tasks to evaluate whether the model can learn simple compositional rules and reasoning strategies. These tasks include counting the number of visible ship tiles, locating a missing ship tile, and generalizing both strategies to unseen scenario types. Figure 3 illustrates the three tasks we designed in this experiment by providing examples of each task. Counting task. Models must select the ship color with the least number of visible tiles on the board. Each board has a unique answer, and models respond by generating a program "(topleft (coloredTiles X))" where X is a ship color. 4000 examples are used for training, and another 1000 examples are used for testing. Missing tile task. Models must select the ship that is missing a tile and identify which tile is missing. All ships are completely revealed except one, which is missing exactly one tile. Models respond by generating "(== (color Y) X)" where X is a color and Y is a location on the board. The number of training and test examples are the same as the counting task. Compositionality task. Models must combine both of the above strategies to find the missing tile of the ship with the least visible tiles. Outputs are produced as "(Z (coloredTiles X))" where X is a color and Z is either topleft or bottomright. Each board has a unique answer. This task further evaluates compositionality by withholding question types from training. With three values for X and two for Z, there are six possible question types and one is picked as the "held out" type. The other five "known" question types have 800 training examples. For the held out question type, the number of training examples is varied from 0 to 800, to test how much data is needed for generalization. Another 200 new boards of each question type is used for evaluation. More information about the model hyperparameters and training procedures are provided in Appendix B.1. We train our model in a fully supervised fashion. Accuracy for the counting and missing tile tasks is shown in Figure 3. The full neural program generation model shows strong reasoning abilities, achieving high accuracy for both the counting and missing tile tasks, respectively. We also perform ablation analysis of the encoder filters of the model, and provide the in Appendix D. The for the compositionality task are summarized in Table 1. When no training data regarding the held out question type is provided, the model cannot generalize to situations systematically different from training data, exactly as pointed out in previous work on the compositional skills of encoder-decoder models . However, when the number of additional training data increases, the model quickly incorporates the new question type while maintaining high accuracy on the familiar question tasks. On the last row of Table 1, we compare our model with another version where the decoder is replaced by two linear transformation operations which directly classify the ship type and location (details in Appendix B.1). This model has 33.0% transfer accuracy on compositional scenarios never seen during training. This suggests that the model has the potential to generalize to unseen scenarios if the task can be decomposed to subtasks and combined together. In this experiment, we examine if the neural network has the capability of capturing the distribution of human questions as a conditioned language model. To train the model, we need to construct a training set of many paired game boards and questions. Instead of laboriously collecting a large number of real human questions, and translating them into programs by hand, we construct the dataset by sampling from a previous computational model of human question asking . More precisely, we randomly generate a large number of game boards and sample K questions given each board. For generating the boards, we first uniformly sample the configuration of three ships, and randomly cover arbitrary number of tiles, with the restriction that at least one ship tile is observable. Next we randomly sample K programs for each board with importance sampling based on the cognitive model proposed by , which models the probability of a question under a given context as where ε(·) is a parameterized energy function for estimating the likelihood of a question being asked by human, which considers multiple features such as question informativeness, complexity, answer type, etc. Z is a normalization constant. We also randomly generate a larger set of questions to pretrain the decoder component of the model as a "language model" over questions, enabling it to better capture the grammatical structure of possible questions. Details regarding the model hyperparameters, training procedure, and pre-training procedure are provided in Appendix B.2. We evaluate the log-likelihood of reference questions generated by our full model as well as some lesioned variants of the full model, including a model without pretraining, a model with the Transformer decoder replaced by an LSTM decoder, a model with the convolutional encoder replaced by a simple MLP encoder, and a model that only has a decoder (unconditional language model). Though the method from also works on this task, here we cannot compare with their method for two reasons. One is that our dataset is constructed using their method, so the likelihood of their method should be an upper bound in our evaluation setting. Additionally, they can only approximate the log-likelihood due to an intractable normalizing constant, and thus it difficult to directly compare with our methods. Two different evaluation sets are used, one is sampled from the same process on new boards, the other is a small set of questions collected from human annotators. In order to calculate the log-likelihood of human questions, we use translated versions of these questions that were used in previous work , and filtered some human questions that score poorly according to the generative model used for training the neural network (Appendix B.2). A summary of the is shown in Table 2a. The full model performs best on both datasets, suggesting that pretraining, the Transformer decoder, and the convolutional encoder are all important components of the approach. However, we find that the model without an encoder performs reasonably well too, even out-performing the full model with a LSTM-decoder on the human-produced questions. This suggests that while contextual information from the board leads to improvements, it is not the most important factor for predicting human questions. To further investigate the role of contextual information and whether or not the model can utilize it effectively, we conduct another analysis. Intuitively, if there is little uncertainty about the locations of the ships, observing the board is critical since there are fewer good questions to ask. To examine this factor, we divide the scenarios based on the entropy of the hypothesis space of possible ship locations into a low entropy set (bottom 30%), medium entropy set (40% in the middle), and high entropy set (top 30%). We evaluate different models on the split sets of sampled data and report the in Table 2b. When the entropy is high, it is easier to ask a generally good question like "how long is the red ship" without information of the board, so the importance of the encoder is reduced. If entropy is low, the models with access to the board has substantially higher log-likelihood than the model without encoder. Also, the first experiment (section 5.1) would be impossible without an encoder. Together, this implies that our model can capture important context-sensitive characteristics of how people ask questions. In this experiment, we evaluate our reinforcement learning framework proposed in Section 4 on its ability of generating novel questions from scratch, without providing a large training set. The reward for training the reinforcement agent is calculated based on the energy value of the generated question q. We transform the energy value to a proper range for reward by −ε(q)/10 and clamp it between −1 and 1. The model is optimized with the REINFORCE algorithm . A baseline for REINFORCE is established simply as the average of the rewards in a batch. In order to produce higher-quality questions, we manually tune the information-related parameter of the energy function from to make it more information-seeking in this experiment. This process is described in Appendix B.2. We compare the models on their ability to generate diverse questions with high expected information gain (EIG), which is defined as the expected reduction in entropy, averaged across all possible answers to a question x. where I(·) is the Shannon entropy. The terms p(h) and p(h|d; x) are the prior and posterior distribution of a possible ship configuration h given question x and answer d ∈ A x. We compare our program-based framework with a simple text-based model, which has the same architecture but is trained with supervision on the text-based question dataset collected by . We also compare with the supervised program-based model from the last experiment. Finally, we implement a sequence-based reinforcement learning agent that specifies the program without direct access to the grammatical rules. For this alternative RL agent, we find it necessary to pretrain for 500 epochs with stepwise supervision. The models are evaluated on 2000 randomly sampled boards, and the are shown in Table 3. Note that any ungrammatical questions are excluded when we calculate the number of unique questions. First, when the text-based model is evaluated on new contexts, 96.3% of the questions it generates were included in the training data. We also find that the average EIG and the ratio of EIG>0 is worse than the supervised model trained on programs. Some of these deficiencies are due to the very limited text-based training data, but using programs instead can help overcome these limitations. With the program-based framework, we can sample new boards and questions to create a much larger dataset with executable program representations. This self-supervised training helps to boost performance, especially when combined with grammar-enhanced RL. From the table, the grammar-enhanced RL model is able to generate informative and creative questions. It can be trained from scratch without examples of human questions, and produces many novel questions with high EIG. In contrast, the supervised model rarely produces new questions beyond the training set. The sequence-level RL model is also comparatively weak at generating novel questions, perhaps because it is also pre-trained on human questions. It also more frequently generates ungrammatical questions. We also provide examples in Figure 4 to show the diversity of questions generated by the grammar enhanced model, and more in the supplementary materials. Figure 4a shows novel questions the model produces, which includes clever questions such as " Where is the bottom right of all the purple and blue tiles?" or "What is the size of the blue ship minus the purple ship?", while it can also sometimes generates meaningless questions such as "Is the blue ship shorter than itself?" Additional examples of generated questions are provided in Appendix B. Is any ship two tiles long? (> (++ (map (lambda x (== (size x) 2)) (set AllShips))) 0) Are there any ships in row 1? (> (++ (map (lambda y (and (== (rowL y) 1) (not (== (color y) Water)))) (set AllTiles))) 0) Is part of a ship on tile 4-6? (not (== (color 4-6) Water)) What is the size of the blue ship? (setSize (coloredTiles Blue)) What is the size of the purple ship? (size Purple) Which column is the first part of the blue ship? (colL (topleft (coloredTiles Blue))) What is the orientation of the blue ship? With the grammar enhanced framework, we can also guide the model to ask different types of questions, consistent with the goal-directed nature and flexibility of human question asking. The model can be queried for certain types of questions by providing different start conditions to the model. Instead of starting derivation from the start symbol "A", we can start derivation from a intermediate state such as "B" for Boolean questions or a more complicated "(and B B)" for composition of two Boolean questions. In Figure 4b, we show examples where the model is asked to generate four specific types of questions: true/false questions, number questions, location-related questions, and compositional true/false questions. We see that the model can flexibly adapt to new constraints and generate meaningful questions. In Figure 4c, we compare the model generated questions with human questions, each randomlysampled from the model outputs and the human dataset. These examples again demonstrate that our model is able to generate clever and human-like questions. However, we also find that people sometimes generate questions with quantifiers such as "any" and "all", which are operationalized in program form with lambda functions. These questions are complicated in representation and not favored by our model, showing a current limitation in our model's capacity. We introduce a neural program generation framework for question asking task under partially unobservable settings, which is able to generate creative human-like questions with human question demonstrations by supervised learning or without demonstrations by grammar-enhanced reinforcement learning. Programs provide models with a "machine language of thought" for compositional thinking, and neural networks provide an efficient means of question generation. We demonstrate the effectiveness of our method in extensive experiments covering a range of human question asking abilities. The current model has important limitations. It cannot generalize to systematically different scenarios, and it sometimes generates meaningless questions. We plan to further explore the model's compositional abilities in future work. Another promising direction is to model question asking and question answering jointly within one framework, which could guide the model to a richer sense of the question semantics. Besides, allowing the agent to iteratively ask questions and try to win the game is another interesting future direction. We would also like to use our framework in dialog systems and open-ended question asking scenarios, allowing such systems to synthesize informative and creative questions. Encoder. A simple CNN with one layer of filters is used to encode the board. Intuitively, many questions are related to specific positions, thus the position information should be recoverable from the encoding. On the other hand, some features of the board are translation-invariant, such as whether a ship is blocked by another ship. In order to capture the position-sensitive information as well as the translation-invariant patterns, three convolution operations with different filter sizes (1 × 1, 3 × 3, and 5 × 5) are performed in parallel on the same input. The inputs are padded accordingly to make sure the feature maps have the same width and height. Then three feature maps are concatenated together along the dimension of output channels, and passed through a linear projection. Formally, the outputs of the convolutions c can be obtained by where Conv k denotes a convolution operation on a k × k filter, ReLU(·) means applying a ReLU activation, and [A; B] means the concatenation of matrices A and B. Then c ∈ R 6×6×3Cout is projected to the encoder output e ∈ R 6×6×M by matrix W e o ∈ R 3Cout,M, where C out is the number of out channels of each convolution, and M is the length of encoded vectors. Decoder. We use the decoder from the Transformer model . With an input sequence of length L, the decoder computes the hidden states through several stacked Decoder Attention Layers. Each layer is composed by three sub-layers, a self-attention module, an attention over the encoded board, and a fully connected feed-forward network. Residual connections are employed around each sub-layer, followed by a layer normalization . After N layers of attention modules, a final output layer transforms the hidden states to the output vectors y i ∈ R No at every position i from 1 to L, where N o is the output size. Given the output from encoder e, and the hidden representation h n−1 from Decoder Attention Layer n − 1, each layer computes the hidden representation as where LN(·) means layer normalization , FC(·) is a fully connected layer, ATT(·) and Self-ATT(·) are multi-head attention mechanisms, which computes the attention over the output of encoder e, and the attention over the input h n−1 itself, respectively. They are defined as follows Multi-ATT(·) is the multi-head attention mechanism described in the paper by , which is a concatenation of multiple standard attention mechanisms with inputs projected using different matrices. A multi-head attention with n heads is defined as where is the scaled dot-product attention operation. Q, K, V are a set of vectors called queries, keys, and values, respectively, and d k is the dimension of queries and keys. After N layers, we apply a linear projection and a softmax activation to h N to get the output vectors y 1,..., y L. In this experiment, we use C out = 10, L = 50 for the model encoder. Each word is embedded with 50 dimension vectors in the decoder. The decoder has 2 layers, each multi-head attention module has 4 heads, and The model is trained for 500 epochs using Adam optimizer with initial learning rate at 0.001 and a batch size is set as 32. To further examine the model's ability on compositionality task, we evaluate another version of the model which replaces the decoder with two linear transformations to directly predict y l ∈{topleft, bottomright}, and y c ∈{Blue, Red, Color}. With the hidden representation of the encoder c in equation 1, y l and y c are calculated as where W l ∈ R |c|×2, W c ∈ R |c|×3, andc is the flattened vector of c. In this experiment, the model encoder has the same hyper-parameters as in the first experiment. We increase the size of the decoder by setting number of layers to 3, number of heads to 6, and set The model is also trained for 500 epochs using Adam optimizer with the same initial learning rate at 0.001. In order to better familiarize the model with grammar, we also pretrain the decoder for 500 epochs on a larger set of question programs. This pretraining corpus is first uniformly sampled from the PCFG grammar defining questions, then we calculate the average energy value of each program on 10, 000 boards, and keep the top 2500 unique questions. For the evaluation set of human questions, we found that some simple questions become complicated in program form. For example, question "How many ship pieces are there in the second column?" will be translated to "(++ (map (lambda y (and (== (colL y) 2) (not (== (color y) Water)))) (set All_Tiles)))". Such complicated programs score very poorly according to the energy function, so they do not appear in the training set. As a , the average log-likelihood is extremely skewed by these unseen questions. For a more robust evaluation, we remove the last 10% human questions with low energy values according to. The neural model for this experiment has the same hyper-parameters as in the last experiment, and is optimized by REINFORCE algorithm with initial learning rate 0.0001 and batch size 24. To encourage the exploration of the model, we also apply an -greedy strategy with set to 0.5 at the beginning, and gradually decreased to 0.01 as training continues. This model is trained for 500 epochs, within each epoch the model passes 10, 000 different boards. From some preliminary experiments, we find that the models have a strong preference of generating similar programs of relatively low complexity, with the original energy values as rewards. Thus, we tune two parameters of the energy model as mentioned in section 5.3, which are the two parameters corresponding to information seeking features (denoted as f EIG and f EIG=0 in the original paper). We increase this two parameters from 0.1 until the reinforcement learning models are able to generate a diverse set of questions. The sequence RL baseline which directly generates sequence with the decoder is trained with MIXER algorithm , which is a variant of REINFORCE algorithm widely used in sequence generation tasks. MIXER provides a smooth transition from supervised learning to reinforcement learning. This model is pretrained for 500 epochs, and trained with RL for 300 epochs. The grammar is provided in table 4 and 5. Table 4: Part 1 of the grammatical rules. Rules marked with b have a reference to the Battleship game board (e.g., during evaluation the function orient looks up the orientation of a ship on the game board) while all other rules are domain-general (i.e., can be evaluated without access to a game board). True if all elements in set of numbers are equal B → (any setB) True if any element in set of booleans is True B → (all setB) True if all elements in set of booleans are True True if the two ships are touching (diagonal does not count) B → (isSubset setL setL) True if the first set of locations is subset of the second set of locations Number of True elements in set of booleans Column number of location L N → (setSize setL) Number of elements in set of locations The most left of the most top location in the set of locations L → (bottomright setL) The most right of the most bottom location in the set of locations All locations with color C setL → (setDifference setL setL) Remove second set from first set setL → (union setL setL) Combine both sets setL → (intersection setL setL) Elements that exist in both sets setL → (unique setL) Unique elements in set We also perform an ablation test on the neural network in the first experiment (section 5.1). Accuracy for the counting and missing tile tasks is summarized in Table 6a for the full model and lesioned variants. The full neural program generation model shows strong reasoning abilities, achieving an accuracy of 99.80% and 95.50% for the counting and missing tile tasks, respectively. The full model is compared to weakened variants with only one filter size in the encoder, either "3x3" and "1x1 conv only," and the performance of the weakened models drop dramatically on the missing tile task. To better understand the role of different filter sizes, Table 6b breaks down the errors in the missing tile task on whether the question can pick the right ship (color acc.) and whether it can select the right location (location acc.). The 3 × 3 convolution filters can accurately select the correct color, but often fail to choose the right tile. The model with 1 × 1 convolution filters has poor performance for both color and location. In the current architecture, predicting the correct location requires precise information that seems to be lost without filters of different sizes. For the experiment on estimating the distribution of human questions (Experiment 5.2), Table 7 provides a full table of log-likelihood of different models on evaluation set of different uncertainty level. Here we provide more examples of questions generated by our models in the generation experiment (Experiment 5.3). Figure 5, 6 and 7 provides more examples for the same settings as shown in figure 4 in the main text. Figure 8 shows generated examples of the text-based model. What is the size of blue plus the number of the row which contains the beginning of the red ship? (+ (size Blue) (colL (topleft (coloredTiles Red)))) 2.46 Energy: -5.11 Figure 5: Novel questions generated by the grammar enhanced model.

Represent the TLDR for finding the corresponding scientific article!

We introduce a model of human question asking that combines neural networks and symbolic programs, which can learn to generate good questions with or without supervised examples.

Represent the short summary for finding the corresponding scientific article (long)

We use reinforcement learning to train an agent to solve a set of visual arithmetic tasks using provided pre-trained perceptual modules and transformations of internal representations created by those modules.


Question: Represent the scientific article to find a short TLDR:

Question: The classification of images taken in special imaging environments except air is the first challenge in extending the applications of deep learning. We report on an UW-Net (Underwater Network), a new convolutional neural network (CNN) based network for underwater image classification. In this model, we simulate the visual correlation of attention with image understanding for special environments, such as fog and underwater by constructing an inception-attention (I-A) module. The experimental demonstrate that the proposed UW-Net achieves an accuracy of 99.3% on underwater image classification, which is significantly better than other image classification networks, such as AlexNet, InceptionV3, ResNet and Se-ResNet. Moreover, we demonstrate the proposed IA module can be used to boost the performance of the existing object recognition networks. By substituting the inception module with the I-A module, the Inception-ResnetV2 network achieves a 10.7% top1 error rate and a 0% top5 error rate on the subset of ILSVRC-2012, which further illustrates the function of the attention in the image classifications. Underwater images and videos contain a lot of valuable information for many underwater scientific researches . However, the image analysis systems and classification algorithms designed for natural images cannot be directly applied to the underwater images due to the complex distortions existed in underwater images (e.g., low contrast, blurring, non-uniform brightness, non-uniform color casting and noises) and there is, to the best of our knowledge, no model for underwater image classification. Except for the inevitable distortions exhibited in underwater images, there are other three key problems for the classification of underwater images: the in underwater images taken in different environments are various; the salient objects such as ruins, fish, diver exist not only in underwater environment, but also in air. The features extracted from the salient objects cannot be relied on primarily in the classification of underwater images; and since the classification of underwater images is only a dualistic classification task, the structure of the designed network should be simple to avoid the over-fitting. Increasing the depth and width of a CNN can usually improve the performance of the model, but is more prone to cause over-fitting when the training dataset is limited, and needs more computational resource . To remit this issue, proposed the inception module, which simultaneously performs the multi-scale convolution and pooling on a level of CNN to output multi-scale features. In addition, the attention mechanism is proposed and applied in the recent deep models which takes the advantage that human vision pays attention to different parts of the image depending on the recognition tasks; ). Although these strategies play an important role in advancing the field of image classifications, we find that the large-scale features such as the area play a more important role in the visual attention mechanism when people understanding of underwater images, which is unlike the attention mechanism applied in natural scene image classification . In this paper, we propose an underwater image classification network, called UW-Net. The overview network structure is shown in Fig. 1. Unlike other models, the UW-Net pays more attention to the Figure 1: The structure of the UW-Net. The bottom part is the output of the eighth layer in the I-A module. The red area represents a higher response of features for the underwater image classification. As shown, our I-A module concerns more about the regions of underwater images. features of images by construct the inception-attention (I-A) modules and thus achieves better performance. The contributions of this paper are as follows: (i) to the best of our knowledge, it is the first CNN-based model for underwater image classification; (ii) an inception-attention module is proposed, which joints the multi-dimensional inception module with the attention module to realize the multiple weighting of the output of various scales of features; (iii) this work is a first attempt to simulate the visual correlation between understanding images and areas through I-A modules. The rest of the paper is organized as follows: Section 2 introduces the related work. The proposed UW-Net is described in Section 3. Section 4 illustrates the experimental and analysis, and we summarize this paper in Section 5. As mentioned before, there is less work focusing on underwater image classification. Thus, we mainly introduce the classification models designed for natural scenes and the recent attention mechanism which is incorporated in our network in this section. won the ImageNet Large Scale Visual Recognition Competition (ILSVRC) , CNNs become more and more popular in the application of image recognition tasks. Many CNNs pursue better performance by means of superimposing more convolution layers, but the number of the parameters is concomitantly increased. As the depth increases , the gradient of the network will disappear or explode in the training process. proposed the "GoogleNet" model, in which the inception architecture is first proposed. The 1 × 1 convolution is also applied as a dimension reduction technique and to extract more nonlinear characteristics of the features. By fusing the features extracted from multi-scale convolutions, better image representation is obtained in the deep layers of the network. Based on the initial inception structure, multiple network structures such as Inception V3 and Inception V4 are further proposed. proposed the Resnet, in which the residual modules are proved to be effective in solving the gradient disappearance problem of deep convolution networks. An important feature of human vision is that people usually focus more on a certain area of the whole scene while ignoring other areas ), which is called the attention mechanism (AM). proposed the SeNet by constructing the attention mechanism of feature channels, and won the championship of ILSVRC2017. proposed the residual attention network (RAN), which also combined the attention mechanism with the residual modules, and achieves 4.8% top-5 error rate on ILSVRC 2012 dataset. These works are all evidences of the effectiveness of the attention mechanism in deep learning models on one hand, and on the other, none of these studies on attention mechanism focus on features. Although the performance of the existing image classification models has exceeded that of human beings in some classification tasks , most of the existing classification models assume that images have legible texture and uniform features. However, when light propagating through the water, the absorption and scattering determined by the internal optical property (IOP) of the water affect the process of underwater imaging. Not only the water body, but also the dissolved organic matter and small floating particles (called sea snow), whose concentration and species vary greatly, also affect the underwater image quality . With the depth increases in water, the color of light disappears according to their wavelengths. Artificial lighting often in uneven lighting, creates bright spots in the image, and makes the scattering of suspended matter worse . These challenges make the design of an effective underwater image classification algorithm difficult. Moreover, human recognition of underwater images is often based on the features of the images. In the next section, we use the inception module with multiple sizes of receptive fields to extract features and simulate the human attention mechanism by combining an attention module emphasizing the features of underwater images. The network structure of UW-Net is constructed based on multiple I-A modules, as shown in Fig. 2. The network can be extended with more I-A modules for other complex visual classification tasks. An underwater image is first forwarded to a convolutional layer with 7 × 7 size of kernels to obtain large reception fields. An auxiliary classification branch is added after two I-A modules, and the output of the auxiliary branch will be used to optimize the network as part of the loss function. As follows, we will introduce the key components in the UW-Net, i.e., I-A module including inception module and attention module, and classification branch in details. The classical inception models (; are constructed by multiple sizes of convolutional kernels. The feature maps are processed by different convolution kernels in one inception module, and then merged and forwarded to the next inception module directly. However, not all the extracted features of every convolution kernel are positively related to the current image classification task. For example, the features extracted by a convolution kernel with large size tend to describe the global information, which has little effect on a fine-grained image classification task even though they can be transferred to the deeper layers, and will in a certain degree of waste of computational resource. Underwater images exhibit the characteristics of large intra-class differences. Furthermore, the positions and proportions of the areas vary in underwater images, and the recognition of underwater image is based more on the global features of the image. In addition, the quality of most underwater images is poor due to the effects of lighting absorption and scattering. In view of these characteristics of the underwater images, we adopt convolution kernels with larger size and the average pooling to reduce the impact of local features of the image on the final classification . In the experiment, we find that the best classification can be obtained by using convolution kernel sizes of 1 × 1, 5 × 5 and 7 × 7 in the inception module. The structure of the UW-Net. After two I-A modules, one branch serves as an auxiliary classification branch and one branch serves as the backbone of the network. The final model has two outputs, in which the output of the auxiliary branch is used to compute the loss function. We construct the attention module based on soft attention, as shown in Fig. 3, which consists of a trunk branch and a mask branch to simulate the recognition of underwater image by humans beings . Inspired by the residual network , the trunk branch takes the output of the previous layer as input directly so that the basic features of the image can be transmitted to the deep layers of the network, and the gradient disappearance and the gradient explosion can be remitted. On the other hand, the down-sampling operation is firstly performed in the mask branch, and then up-sampling by bilinear interpolation is used at the last step to keep the same size of the feature map with the input. The activation function of the first and second convolutional layers are Relu and Sigmoid , respectively. The adaptive weight N (x) for a point x of the original features map P (x), in the range of, can be learned after the mask branch. The output of the attention module F (x) can be expressed as: For a N (x) approximating 1, F (x) will be near twice the value of the original feature P (x), which means that for the features that are valid for the current classification, more attention will be given. On the contrary, for a N (x) approximating 0, the output of the attention module will approximate the original feature P (x). In the proposed UW-Net, an auxiliary classification branch is introduced to the output of the second I-A module to reduce the risk of over-fitting. The convergence curve of the UW-Net is shown in Fig. 4. It is obvious that adding auxiliary classifier can not only accelerate the convergence, but also improve the accuracy on the test set. The loss function J of the UW-Net can be expressed as follows: where J 0 is the cross entropy of the final output of the model and the real label of the image, J 1 is the cross entropy of the output of the model's auxiliary classification branch and the actual label, α is the weight attenuation coefficient of the network, and L is the L2 regularization term. The UW-Net is constructed by two I-A modules proposed above, and the detailed blocks and parameter settings are reported in Table 1. Before the first I-A module, the size of the image is reduced to 36 × 36, and the channel of the input image is increased to 32 by two convolutional layers and one pooling operation. After the I-A module, the size of the feature map is reduced and the channel of the feature map is increased. The final prediction of the model is obtained after an average pooling (Avgpool) and a fully connected (FC) layer. In this section, a series of experiments are conducted to demonstrate the performance of the UW-Net in underwater image classification. Additionally, the effectiveness of the proposed I-A module in improving the performance of existing inception based networks is investigated. We compare the UW-Net with several typical image classification models and inception model based networks. The figure 2 36 × 36 × 32 1×I-A module As in figure 2 36 × 36 × 128 1×Inception As in Section 3.1.1 17 × 17 × 512 models for comparison are re-trained in the same dataset without changing the structure, only the parameters are optimized. To ensure the diversity of underwater images, we collect more than 4,000 underwater images from ImageNet dataset 1, JAMSTEC dataset 2, underwater rock image dataset 3 and online underwater images. These images are labeled as underwater images. In addition, more than 5,000 non-underwater images from the ImageNet including birds, cars, food, airplanes, cats, etc. are selected and labeled as non-underwater images. The proportion of the samples used in training and testing is 70% and 30%, respectively. To reduce the risk of over-fitting, we augment the data by random clipping and flipping. We use the initialization method proposed by to initialize weights and train the UW-Net by using the SGD (Stochastic Gradient Descent) with a mini-batch size of 32. The weight decay, momentum and initial learning rate are set to 0.001, 0.9 and 0.001, respectively. The learning rate is decreased ten times of its original value at 1k and 2k iterations. The training end at 3k iterations. The loss curve is shown in Fig. 5. Figure 6: Visualized examples of regional contributions to the final classification. From up to down are: the output of the fifth layer in Alexnet, the output of the thirteenth layer in VGG16, the output of the twelfth layer in Googlenet, the output of the seventeenth layer in Resnet-50, the output of the seventeenth layer in SE-Resnet-50 and the output of the eighth layer in the UW-Net. The UW-Net achieves 100% and 99.3% accuracy on the training and testing datasets, respectively. We also report the comparisons with the AlexNet, VGG16, InceptionV3, Resnet-50 and SE-Resnet-50 on the testing dataset in Table 2, and class activation maps (CAMs) produced by these models for underwater image classification are shown in Fig. 6. The darker red color in the CAMs represent more importance of the regions to the final classification. As shown in Fig. 6, the interesting areas of the UW-Net locate more in the areas compared to the competing models. The data in Table 2 shows that the UW-Net is superior to the competing models for the task of underwater image classification by fewer computation units and parameters, and higher accuracy. By adopting the I-A model in the UW-Net, we achieve higher accuracy with a smaller depth of network. At the same time, the UW-Net requires lower computation than the other inception based models. The I-A module designed in this work can not only be applied in the UW-Net, but also to other common image classification networks. To further verify the generalization ability of the I-A module in boosting the performance of related models, we embed the proposed I-A module with the max-pooling in the down-sampling of the mask branch into several image classification models including GoogleNet, InceptionV3, InceptionV4 and Inception-ResnetV2. One hundred categories of images are selected from the ILSVRC-2012 dataset, including ships, sharks, dogs, cocks, etc. Each category contains about 1300 images, and a total number of 130,000 images is included. The training and testing sets consist of 125,000, and 5000 images respectively. The size of the images is 299 × 299. All the networks are tested and tuned on this dataset in the same way. The experimental are shown in Table 3. It can be seen that the error rates of Top-1 obtained by substituting the inception module with the I-A module are significantly decreased. Such a indicates the generalization and effectiveness of the proposed I-A module, also gives another evidence that attention weighted large-scale image features simulate better the visual understanding mechanism in image classifications. A new underwater image classification network UW-Net is proposed in this work, wherein an inception-attention module is constructed. In this model, we simulate the visual correlation between understanding images and areas through I-A modules, which joint the multidimensional inception module with the attention module to realize the multiple weighting of the output of various scales of features. The 100% accuracy on the training set and 99.3% accuracy on the testing set of the UW-Net is achieved benefiting from the refinement of the usefulness of multiscale features by the I-A module. In the future, we will try to improve the performance of other underwater image visual analysis models by introducing the proposed I-A module.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A visual understanding mechanism for special environment

Represent the TLDR for finding the corresponding scientific article

This paper aims to leverage good properties of robust visual features like SIFT to renovate CNN architectures towards better accuracy and robustness.