Question: Sequence-to-sequence (seq2seq) neural models have been actively investigated for abstractive summarization. Nevertheless, existing neural abstractive systems frequently generate factually incorrect summaries and are vulnerable to adversarial information, suggesting a crucial lack of semantic understanding. In this paper, we propose a novel semantic-aware neural abstractive summarization model that learns to generate high quality summaries through semantic interpretation over salient content. A novel evaluation scheme with adversarial samples is introduced to measure how well a model identifies off-topic information, where our model yields significantly better performance than the popular pointer-generator summarizer. Human evaluation also confirms that our system summaries are uniformly more informative and faithful as well as less redundant than the seq2seq model. Automatic text summarization holds the promise of alleviating the information overload problem BID13. Considerable progress has been made over decades, but existing summarization systems are still largely extractive-important sentences or phrases are identified from the original text for inclusion in the output BID22. Extractive summaries thus unavoidably suffer from redundancy and incoherence, leading to the need for abstractive summarization methods. Built on the success of sequence-to-sequence (seq2seq) learning models BID35, there has been a growing interest in utilizing a neural framework for abstractive summarization BID28 BID20 BID41 BID36 BID5.Although current state-of-the-art neural models naturally excel at generating grammatically correct sentences, the model structure and learning objectives have intrinsic difficulty in acquiring semantic interpretation of the input text, which is crucial for summarization. Importantly, the lack of semantic understanding causes existing systems to produce unfaithful generations. BID3 report that about 30% of the summaries generated from a seq2seq model contain fabricated or nonsensical information. Furthermore, current neural summarization systems can be easily fooled by off-topic information. For instance, FIG0 shows one example where irrelevant sentences are added into an article about "David Collenette's resignation". Both the seq2seq attentional model BID20 and the popular pointer-generator model BID31 are particularly susceptible to unfaithful generation, partially because these models tend to rely on sentences at the beginning of the articles for summarization while being ignorant about their content. Therefore, we design a novel adversarial evaluation metric to measure the robustness of each summarizer against small amounts of randomly inserted topic-irrelevant information. The intuition is that if a summarization system truly understands the salient entities and events, it would ignore unrelated content.32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada. Article Snippet: For years Joe DiMaggio was always introduced at Yankee Stadium as "baseball's greatest living player." But with his memory joining those of Babe Ruth, Lou Gehrig, Mickey Mantle and Miller Huggins. Canada's Minister of Defense resigned today, a day after an army official testified that top military officials had altered documents to cover up responsibility for the beating death of a Somali teen-ager at the hands of Canadian peacekeeping troops in 1993. Defense minister David Collenette insisted that his resignation had nothing to do with the Somalia scandal. Ted Williams was the first name to come to mind, and he's the greatest living hitter.... Seq2seq: George Vecsey sports of The Times column on New York State's naming of late baseball legend Joe DiMaggio as "baseball's greatest living player," but with his memory joining those of Babe Ruth, Lou Gehrig, Mickey Mantle and Miller dens. Pointer-generator: Joe DiMaggio is first name to come to mind, and Ted Williams is first name to come to mind, and he's greatest living hitter; he will be replaced by human resources minister, Doug Young, and will keep his Parliament seat for governing Liberal Party. Our Model: Former Canadian Defense Min David Collenette resigns day after army official testifies that top military officials altered documents to cover up responsibility for beating death of Somali teen-ager at hands of Canadian peacekeeping troops in 1993. To address the above issues, we propose a novel semantic-aware abstractive summarization model, inspired by the human process of writing summaries-important events and entities are first identified, and then used for summary construction. Concretely, taking an article as input, our model first generates a set of summary-worthy semantic structures consisting of predicates and corresponding arguments (as in semantic parsing), then constructs a fluent summary reflecting the semantic information. Both tasks are learned under an encoder-decoder architecture with new learning objectives. A dual attention mechanism for summary decoding is designed to consider information from both the input article and the generated predicate-argument structures. We further present a novel decoder with a segment-based reranking strategy to produce diverse hypotheses and reduce redundancy under the guidance of generated semantic information. Evaluation against adversarial samples shows that while performance by the seq2seq attentional model and the pointer-generator model is impacted severely by even a small addition of topic-irrelevant information to the input, our model is significantly more robust and consistently produces more on-topic summaries (i.e. higher ROUGE and METEOR scores for standard automatic evaluation). Our model also achieves significantly better ROUGE and METEOR scores than both models on the benchmark dataset CNN/Daily Mail BID11. Specifically, our model's summaries use substantially fewer and shorter extractive fragments than the comparisons and have less redundancy, alleviating another common problem for the seq2seq framework. Human evaluation demonstrates that our model generates more informative and faithful summaries than the seq2seq model. To discourage the generation of fabricated content in neural abstractive models, a pointer-generator summarizer BID31 ) is proposed to directly reuse words from the input article via a copying mechanism BID9. However, as reported in their work BID31 and confirmed by our experiments, this model produces nearly extractive summaries. While maintaining their model's rephrasing ability, here we improve the faithfulness and informativeness of neural summarization models by enforcing the generation of salient semantic structures via multi-task learning and a reranking-based decoder. Our work is in part inspired by prior abstractive summarization work, where the summary generation process consists of a distinct content selection step (i.e., what to say) and a surface realization step (i.e., how to say it) BID40 BID26. Our model learns to generate salient semantic roles and a summary in a single end-to-end trained neural network. Our proposed model also leverages the recent successes of multi-task learning (MTL) as applied to neural networks for a wide array of natural language processing tasks BID18 BID33 BID25 BID27 BID12. Most recent work BID23 leverages MTL to jointly improve performance on summarization and entailment generation. Instead of treating the tasks equally, we employ semantic parsing for the sake of facilitating more informative and faithful summary generation. Figure 2: Our semantic-aware summarization model with dual attention over both input and generated salient semantic roles. In both shared and separate decoder models, the semantic roles are generated first, followed by the summary construction. Best viewed in color. In the standard seq2seq model, a sequence of input tokens x = {x 1, ..., x n} is encoded by a recurrent neural network (RNN) with hidden states h i at every timestep i. The final hidden state of the encoder is treated as the initial state of the decoder, another RNN (h n = s 0). The decoder will produce the output sequence y = {y 1, ..., y T}, with hidden states s t at every timestep t. Our model outputs two sequences: the sequence of semantic role tokens y s = {y is calculated as the summation of the encoder hidden states, weighted by the attention distribution a t at every timestep t: DISPLAYFORM0 where v, W s, W h are learnable parameters. The context vector, along with the decoder hidden state, is used to produce the vocabulary distribution via softmax: The loss is defined as the negative log likelihood of generating summary y over the training set D using model parameters θ: DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 The log probability for each training sample is the average log likelihood across decoder timesteps. Encoder. In our models, our encoder is a single-layer bidirectional long short-term memory (LSTM) unit BID7, where the hidden state is a concatenation of the forwards and backwards LSTMs: DISPLAYFORM4 Decoders. We propose two different decoder architectures-separate decoder and shared decoderto handle semantic information. In the separate decoder model (See Figure 2c), the semantic decoder and the summary decoder are each implemented as their own single-layer LSTM. This setup, inspired by the one-to-many multi-task learning framework of BID18, encourages each decoder to focus more on its respective task. Decoder output and attention over the input are calculated using Eqs. We further study a shared decoder model (See Figure 2a) for the purpose of reducing the number of parameters as well as increasing the summary decoder's exposure to semantic information. One single-layer LSTM is employed to sequentially produce the important semantic structures, followed by the summary. Our output thus becomes y = [y s y a], and the first timestep of the summary decoder is the last timestep of the semantic decoder. Attention is calculated as in Eqs. 1-5.For both models, the loss becomes the weighted sum of the semantic loss and the summary loss: DISPLAYFORM5 In our experiments, we set α as 0.5 unless otherwise specified. We also investigate multi-head attention BID37 over the input to acquire different language features. To our knowledge, we are the first to apply it to the task of summarization. As shown later in the section, this method is indeed useful for summarization, with different heads learning different features. In fact, the multi-head attention is particularly well-suited for our shared decoder model, as some heads learn to attend semantic aspects and others learn to attend summary aspects. Target Semantic Output. We use semantic role labeling (SRL) as the target semantic representation, which identifies predicate-argument structures. To create the target data, articles in the training set are parsed in the style of PropBank BID14 using the DeepSRL parser from BID10. We then choose up to five SRL structures that have the most overlap with the reference summary. Here we first consider matching headwords of predicates and arguments. If no match is found, we consider all content words. Note that the semantic labels are only used for model training. At test time, no external resource beyond the article itself is used for summary decoding. To further leverage semantic information, a dual attention mechanism is used to attend over the generated semantic output in addition to the input article during summary decoding. Although attending over multiple sources of information has been studied for visual QA BID21 and sentence-level summarization BID3 ), these are computed over different encoder states. In contrast, our dual attention mechanism considers decoding from the semantic output. Although our attention mechanism may appear close to intra-or self-attention BID30, the function of our dual attention is to attend over a specific portion of the previously generated content that represents its own body of information, whileas traditional self-attention is predominantly used to discourage redundancy. The context vector attending over the semantic decoder hidden states is calculated as follows:, segment-reranking BID32, and hamming loss BID42. Here, we discuss our own segment-reranking beam search decoder, which encourages both beam diversity and semantic coverage. Our reranker is applied only during summary decoding, where we rely on the generated semantic roles for global guidance. The only modification made to the semantic decoder is a de-duplication strategy, where generated predicates seen in previous output are eliminated. DISPLAYFORM0 Reranking. Regular beam search chooses the hypotheses for the next timestep solely based on conditional likelihood (i.e., p(y a | x)). In our proposed summary decoder, we also leverage our generated semantic information and curb redundancy during beam selection. Reranking is performed every R timesteps based on the following scorer, where hypotheses with less repetition (r) and covering more content words from the generated semantics (s) are ranked higher: DISPLAYFORM1 where LRS is the longest repeating substring in the current hypotheses. We define a repeating substring as a sequence of three or more tokens that appears in that order more than once in the hypothesis, with the intuition that long repeating fragments (ex. "the Senate proposed a tax bill; the Senate proposed a tax bill.") should be penalized more heavily than short ones (ex. "the Senate proposed a tax bill; Senate proposed."). s measures the percentage of generated semantic words reused by the current summary hypothesis, contingent on the predicate of semantic structure matching. At every other timestep, we rank the hypotheses based on conditional likelihood and a weaker redundancy handler, r, that considers unigram novelty (i.e., percentage of unique content words): score = log (p(y a | x)) + α r. We use R = 10, α = 0.4, β = 0.1, α = 0.1 in our experiments. Beam Diversity. In the standard beam search algorithm, all possible extensions to the beams are considered, ing in a comparison of B × D hypotheses, where B is the number of beams and D is the vocabulary size. In order to make the best use of the reranking algorithm, we develop two methods to enforce hypothesis diversity during non-reranking timesteps. Beam Expansion. Inspired by BID32, we rank only the K highest scoring extensions from each beam, where K < B. This ensures that at least two unique hypotheses from the previous timestep will carry on to the next timestep. Beam Selection. We further reinforce hypothesis diversity through the two-step method of likelihood selection and dissimilarity selection. In likelihood selection, we accept N hypotheses (where N < B) from our pool of B × K based solely on conditional probabilities, as in traditional beam search. From the remaining hypotheses pool, we select B − N hypotheses on the basis of dissimilarity. We choose the hypotheses with the highest dissimilarity score ∆([h] N, h ), where h is a candidate and [h] N are the hypotheses chosen during likelihood selection. In our experiments, we use token-level Levenshtein edit distance BID15 as the dissimilarity metric, where (Lev(h, h) ). In experiments, we use B = 12, K = 6, N = 6. DISPLAYFORM2 Datasets. We experiment with two popular large datasets of news articles paired with human-written summaries: the CNN/Daily Mail corpus BID11 (henceforth CNN/DM) and the New York Times corpus BID29 ) (henceforth NYT). For CNN/DM, we follow the experimental setup from BID31 and obtain a dataset consisting of 287,226 training pairs, 13,368 validation pairs, and 11,490 test pairs. For NYT, we removed samples with articles of less than 100 words or summaries of less than 20 words. We further remove samples with summaries containing information outside the article, e.g., "[AUTHOR]'s movie review on..." where the author's name does not appear in the article. NYT consists of 280,146 training pairs and 15,564 pairs each for validation and test. Training Details and Parameters. For all experiments, a vocabulary of 50k words shared by input and output is used. Model parameters and learning rate are adopted from prior work BID31 for comparison purpose. All models are also trained in stages of increasing maximum token lengths to expedite training. The models trained on the NYT dataset use an additional final training stage, where we optimized only on the summary loss (i.e., α = 1 in Eq. 7). During decoding, unknown tokens are replaced with the highest scoring word in the corresponding attention distribution. Baselines and Comparisons. We include as our extractive baselines TEXTRANK BID19 and LEAD-2, the first 2 sentences of the input article, simulating the average length of the target summaries. We consider as abstractive comparison models vanilla seq2seq with attention (SEQ2SEQ) and pointer-generator BID31 (POINT-GEN), which is trained from scratch using the released code. Results for variants of our model are also reported.1 Automatic Evaluation Metrics. For automatic evaluation, we first report the F1 scores of ROUGE-1, 2, and L BID17, and METEOR scores based on exact matching and full matching that considers paraphrases, synonyms, and stemming BID6.We further measure two important aspects of summary quality: extractiveness-how much the summary reuses article content verbatim, and redundancy-how much the summary repeats itself. For the first aspect, we utilize the density metric proposed by BID8 that calculates "the average length of extractive fragments": DISPLAYFORM0 where A represents the article, S represents the summary, and F (A, S) is a set of greedily matched extractive fragments from the article-summary pair. Based on density, we propose a new redundancy metric: DISPLAYFORM1 where F (S) contains a set of fragments at least three tokens long that are repeated within the summary, and #f is the repetition frequency for fragment f. Intuitively, longer fragments and more frequent repetition should be penalized more heavily. Adversarial Evaluation. Our pilot study suggests that the presence of minor irrelevant details in a summary often hurts a reader's understanding severely, but such an error cannot be captured or penalized by the recall-based ROUGE and METEOR metrics. In order to test a model's ability to discern irrelevant information, we design a novel adversarial evaluation scheme, where we purposely mix a limited number of off-topic sentences into a test article. The intuition is that if a summarization system truly understands the salient entities and events, it would ignore unrelated sentences. We randomly select 5,000 articles from the CNN/DM test set with the "news" tag in the URL (mainly covering international or domestic events). For each article, we randomly insert one to four sentences from articles in the test set with the "sports" tag. For NYT, we randomly select 5,000 articles from the test set with government related tags ("U.S.", "Washington", "world") and insert one to four sentences from articles outside the domain ("arts", "sports", "technology"). We ensure that sentences containing a pronoun in the first five words are not interrupted from the previous sentence so that discourse chains are unlikely to be broken. the content of the article, it would ignore unrelated sentences. We took 15,000 articles from the NYT corpus with government related tags ("u.s.", "washington", "world"), and for each article, randomly inserted up to four sentences from articles outside the domain ("arts", "sports", "technology"). We ensured sentences that contained a pronoun in the first five words were not interrupted from the previous sentence so as not to break discourse or coreference chains. DISPLAYFORM0 Figure 3: ROUGE-L and full METEOR scores on adversarial samples, where n irrelevant sentences are inserted into original test articles. Our models (shared decoder for NYT, shared+MHA for CNN/DM) are sturdier against irrelevant information than seq2seq and pointer-generator. separate decoder, and with or without multi-head attention (MHA). In addition to ROUGE and METEOR, we also display extractive density (Dens.) and redundancy (Red.) (lower scores are preferred). Best performing amongst our models are in bold. All our ROUGE scores have a 95% confidence interval of at most ±0.04. Our models all statistically significantly outperform seq2seq for ROUGE and METEOR (approximate randomization test, p < 0.01). Our shared decoder with MHA model statistically significantly outperforms pointer-generator in ROUGE-1 and ROUGE-L on CNN/DM. DISPLAYFORM1 The adversarial evaluation on seq2seq, pointer-generator, and our shared decoder model are shown in Figure 3, where our model consistently yields significantly better ROUGE-L and METEOR scores than the comparisons, and is less affected as more irrelevant sentences are added. Sample summaries for an adversarial sample are shown in FIG0. We find that our semantic decoder plays an important role in capturing salient predicates and arguments, leading to higher quality summaries. Automatic Evaluation. The main are displayed in Table 1. On CNN/DM, all of our models significantly outperform seq2seq across all metrics, and our shared decoder model with multi-head attention yields significantly better ROUGE (R-1 and R-L) scores than the pointer-generator on the same dataset (approximate randomization test, p < 0.01). Despite the fact that both ROUGE and METEOR favor recall and thus reward longer summaries, our summaries that are often shorter than comparisons still produce significantly better ROUGE and METEOR scores than seq2seq on the NYT dataset. Furthermore, our system summaries, by all model variations, reuse fewer and shorter phrases from the input (i.e., lower density scores) than the pointer-generator model, signifying a potentially stronger ability to rephrase. Note that the density scores for seq2seq are likely deflated due to its inability to handle out-of-vocabulary words. Our models also produce less redundant summaries than their abstractive comparisons. Human Evaluation. We further conducted a pilot human evaluation on 60 samples selected from the NYT test set. For each article, the summaries by the seq2seq model, our shared decoder model, and the human reference were displayed in a randomized order. Two human judges, who are native or fluent English speakers, were asked to read the article and rank the summaries against each other based on non-redundancy, fluency, faithfulness to input, and informativeness (whether the summary delivers the main points of the article). Ties between system outputs were allowed but discouraged. Table 2, our model was ranked significantly higher than seq2seq across all metrics. Surprisingly, our model's output was ranked higher than the reference summary in 26% of the samples for informativeness and fluency. We believe this is due to the specific style of the NYT reference summaries and the shorter lengths of our summaries: the informal style of the reference summaries (e.g., frequently dropping subjects and articles) negatively affects its fluency rating, and readers find our shorter summaries to be more concise and to the point. NON-RED. FLUENCY FAITH. INFORM. HUMAN 1.4 ± 0.7 1.5 ± 0.7 1.5 ± 0.8 1.6 ± 0.8 SEQ2SEQ 2.0 ± 0.8 2.4 ± 0.7 2.1 ± 0.8 2.4 ± 0.7 OURS 1.6 ± 0.7 2.1 ± 0.8 1.9 ± 0.7 2.0 ± 0.7 Table 2: Human ranking on non-redundancy, fluency, faithfulness of summaries, and informativeness. The mean (± std. dev.) for the rankings is shown (lower is better). Across all metrics, the difference between our model (shared decoder) and human summary, as well as between our model and seq2seq is statistically significant (one-way ANOVA, p < 0.05). Usage of Semantic Roles in Summaries. We examine the utility of the generated semantic roles. Across all models, approximately 44% of the generated predicates are part of the reference summary, indicating the adequacy of our semantic decoder. Furthermore, across all models, approximately 65% of the generated predicates are reused by the generated summary, and approximately 53% of the SRL structures are reused by the system using a strict matching constraint, in which the predicate and head words for all arguments must match in the summary. When gold-standard semantic roles are used for dual attention in place of our system generations, ROUGE scores increase by about half a point, indicating that improving semantic decoder in future work will further enhance the summaries. Coverage. We also conduct experiments using a coverage mechanism similar to the one used in BID31. We apply our coverage in two places: over the input to handle redundancy, and over the generated semantics to promote its reuse in the summary. However, no significant difference is observed. Our proposed reranker handles both issues in a more explicit way, and does not require the additional training time used to learn coverage parameters. Alternative Semantic Representation. Our summarization model can be trained with other types of semantic information. For example, in addition to using the salient semantic roles from the input article, we also explore using SRL parses of the reference abstracts as training signals, but the higher level of abstraction required for semantic generation hurts performance by two ROUGE points for almost all models, indicating the type of semantic structure matters greatly for the ultimate summarization task. For future work, other semantic representation along with novel model architecture will be explored. For instance, other forms of semantic representation can be considered, such as frame semantics BID1 or Abstract Meaning Representation (AMR) BID2. Although previous work by has shown that seq2seq models are able to successfully generate linearized tree structures, we may also consider generating semantic roles with a hierarchical semantic decoder BID34. We presented a novel semantic-aware neural abstractive summarization model that jointly learns summarization and semantic parsing. A novel dual attention mechanism was designed to better capture the semantic information for summarization. A reranking-based decoder was proposed to promote the content coverage. Our proposed adversarial evaluation demonstrated that our model was more adept at handling irrelevant information compared to popular neural summarization models. Experiments on two large-scale news corpora showed that our model yielded significantly more informative, less redundant, and less extractive summaries. Human evaluation further confirmed that our summaries were more informative and faithful than comparisons.

We propose a semantic-aware neural abstractive summarization model and a novel automatic summarization evaluation scheme that measures how well a model identifies off-topic information from adversarial samples.

Represent the TLDR for finding the corresponding scientific article

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The recent work of Super Characters method using two-dimensional word embedding achieved state-of-the-art in text classification tasks, showcasing the promise of this new approach. This paper borrows the idea of Super Characters method and two-dimensional embedding, and proposes a method of generating conversational response for open domain dialogues. The experimental on a public dataset shows that the proposed SuperChat method generates high quality responses. An interactive demo is ready to show at the workshop. And code will be available at github soon. Dialogue systems are important to enable machine to communicate with human through natural language. Given an input sentence, the dialogue system outputs the response sentence in a natural way which reads like humantalking. Previous work adopts an encoder-decoder architecture BID9, and also the improved architectures with attention scheme added BID0 BID10 BID11. In architectures with attention, the input sentence are encoded into vectors first, and then the encoded vectors are weighted by the attention score to get the context vector. The concatenation of the context vector and the previous output vector of the decoder, is fed into the decoder to predict the next words iteratively. Generally, the encoded vectors, the context vector, and the decoder output vector are all one-dimensional embedding, i.e. an array of real-valued numbers. The models used in decoder and encoder usually adopt RNN networks, such as bidirectional GRU BID0 Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. BID3, LSTM BID6, and bidirectional LSTM BID11. However, the time complexity of the encoding part is very expensive. The recent work of Super Characters method has obtained state-of-the-art for text classification on benchmark datasets in different languages, including English, Chinese, Japanese, and Korean. The Super Characters method is a two-step method. In the first step, the characters of the input text are drawn onto a blank image. Each character is represented by the two-dimensional embedding, i.e. an matrix of real-valued numbers. And the ing image is called a Super Characters image. In the second step, Super Characters images are fed into a twodimensional CNN models for classification. Examples of two-dimensional CNN models are used in Computer Vison (CV) tasks, such as VGG BID7, ResNet BID4, SE-net BID5 and etc. in ImageNet BID1.In this paper, we propose the SuperChat method for dialogue generation using the two-dimensional embedding. It has no encoding phase, but only has the decoding phase. The decoder is fine-tuned from the pretrained two-dimensional CNN models in the ImageNet competition. For each iteration of the decoding, the image of text through two-dimensional embedding of both the input sentence and the partial response sentence is directly fed into the decoder, without any compression into a concatenated vector as done in the previous work. The proposed SuperChat method is motivated by the two-dimensional embedding used in the Super Characters method. If the Super Characters method could keep the same good performance when the number of classes in the text classification problem becomes even larger, e.g. the size of dialogue vocabulary, then the Super Characters method should be able to address the task of conversational dialogue generation. This can be done by treating the input sentence and the partial response sentence as one combined text input. Figure 1. SuperChat method illustration. The input Chinese sentence means "Wher are you?" in English, and the output given by the proposed SuperChat method is "I am within your heart". Figure 1 illustrates the proposed SuperChat method. The response sentence is predicted sequentially by predicting the next response word in multiple iterations. During each iteration, the input sentence and the current partial response sentence are embedded into an image through twodimentional embedding. The ing image is called as a SuperChat image. And then this SuperChat image is fed into a CNN model to predict the next response word. In each SuperChat image, the upper portion corresponses to the input sentence, and the lower portion corresponses to the partial response sentence. At the beggining of the iteration, the partial response sentence is initiallized as null. The prediction of the first response word is based on the SuperChat image with only the input sentence embedded, and then the predicted word is added to the current partial response sentence. This iteration continues until End Of Sentence (EOS) appeared. Then, the final output would be a concatenation of the sequential output. The CNN model used in this method is fine-tuned from pretrained ImageNet models to predict the next response word with the generated SuperChat image as input. It can be trained end-to-end using large dialogue corpus. Thus the problem of predicting the next response word in dialogue generation is converted into an image classification problem. The training data is generated by labeling each SuperChat image as an example of the class indicated by its next response word. EOS is labeled to the SuperChat image if the response sentence is finished. The cut-length of sentences is high-related to the font size of each character. For fixed image size, the larger cutlength means smaller font size for each character, and vice versa. On the one hand, we want to cover long sentences, which means the cut-length should be big, so there will be variety in both the input dialogue and the response dialogue. On the other hand, if we set the cutlength too big, the font size of each character will be small, and there could be large blank area for short sentences, which is a waste of the space on the image. The cut-length should be configured according to the sentence length distribution. It should be also emphasized that the split of the image into input and response part could be not even. Depending on the statistics of the training data, maybe larger or smaller size could be assigned to response and input text. Also, the font size for each part does not need to be the same. Although the examples used in Figure 1 is illustrated with Chinese sentences, however, it can be also applied to other languages. For example, Asian languages such as Japanese and Korean, which has the same square shaped characters as in Chinese. For Latin languages where words may have variant length, SEW method BID8 could be used to convert the Latin languages also into the squared shape before applying the SuperChat method to generate the dialogue response. Beam search BID2 ) could be also used. In that case, instead of hard prediction for the first character, a soft prediction will be used to output all the possible sentences and one of the best will be selected as the final output. The dataset used is Simsimi 1. This is a Chinese chitchat database. This data set contains 454,561 dialogue pairs. There are totally 5,523 characters used in the response sentences, of which 4996 are characters with frequency less than 1000. The top five frequency characters are ",", "我" (I), "你" (you), "的" (of), and "是" (is).Based on the distribution of the sentence length, we set cut length for input sentence at 18, and response cut length also at 18. So, altogether we have 36 characters within one SuperChat image, which could be a layout of 6 rows by 6 columns of characters. The input sentence takes the upper 3 rows, and the response sentence takes the lower 3 rows. For simplicity, we removed all the emoticons in the data set. In order to get enough samples for training, only characters whose frequency is not less than 1000 appearances are selected in the list of characters to predict. After this filtering, the remaining set is composed by the sentences with both input and response sentence length less than 18 characters, and all its characters in the list of the 528 frequent characters (including EOS). The ing set is 178,192 pairs of dialogues, and a total of 989,087 SuperCharacter images are generated. We set our image size at three channels of 224x224 grey image, in order to use the pretrained models on ImageNet. We also added a margin area for the four edges in the SuperChat image, which means the first character will not start from the pixel location of, but from (m, m) instead. Here m is the we set for the four edges. In this experiment, we set m = 16, which in the remaining area is the square of 224−16x2 = 192 pixels. If we set same font size for both input and response sentence, it in a font size of 192/6 = 32 pixels. That means, each character takes an area of 32x32 pixels. The fonts used is the "simhei". For each character, we split its labeled data into 75% for training and 25% for testing. Resulting in 739,289 training samples and 249,798 testing samples. SE-net-154 is used in this experiment. The pretrained model on ImageNet is used as initialization 2. This model is fine-tuned on the generated SuperChat images just as an image classification task. The fine-tuning is done by simply modifying the last layer to 528, which is the size of the subset of the response vocabulary. The learning curve on the test data is shown in FIG0.We can see that at the begining of the training, the curve climbs up quickly, and after it achieved 60%, the slope goes slowly and almost saturate to 64%. The x-axis is in unit of every 500 iterations, so totally it is one million iterations, with batch size of 5, wich accounts to about 7 epochs over the training data. In this paper, we propose the SuperChat method for dialogue response generation. It has no encoding, but only decodes the two-dimensional embedding of the input sentence and partial response sentence to predict the next response word iteratively. The pretrained two-dimensional CNN model is fine-tuned with the generated SuperChat images. The experimental shows high quality response. An interactive demonstration is to show at the workshop TAB0 Submission and Formatting Instructions for ICML 2019 TAB0. Sample response sentences generated by the SuperChat method on the Simsimi data set. Response Sentence 1 "你有男朋友吗?" "没有,我只有老公" "Do you have boyfriend?" "No,I only have a husband" 2 "哼!你不想我我就不和你玩" "我很好玩的跟我玩嘛跟我玩嘛" "Hmm! I won't play with you if you didn't miss me" "I am funny, please play with me, please" 3 "我看到你第一眼,就喜欢你了" "我也喜欢你" "The first time I saw you, I liked you" "I like you too" 4 "是什么意思" "你自己百度去" "What does that mean" "Go Baidu it yourself" 5 "我去睡觉了" "晚安，好梦，明天叫你起床哦" "I am going sleep now" "Good dream, wake you up tomorrow" 6 "你什么时候睡觉" "我等你睡了再睡" " When will you sleep" "I will wait untill after you sleep" 7 "无聊中" "我陪你聊天呗" "I am bored" "Let's chitchat" 8 "我好难过,哭的都好累" "不哭不哭，哭了不好看了" "I am sad, and cried a lot" "Don't cry, not looking good when crying" 9 "问你家哪的" "我是火星人" " Where are you from" "I am from Mars" 10 "对啊你吃什么" "你喜欢吃什么，我就喜欢吃什么" "BTW what do you eat" "I eat whatever you like to eat" 11 "为你付出那种伤心你永远不了解" "我不会骗你。现在不会、以后也不会。" " You won't know my heartbroken for you" "I won't lie to you. Not now, neither future." 12 "那你是美女不" "我是美女！" "So you are a beauty" "I am a beauty!" 13 "睡了吗" "没有啊，我在等你呢。" "Are you asleep" "Nop, I am waiting for you." 14 "你会打鸣不" "喔喔喔！" "Can you do rooster crowing" "Cockadoodledoo!" 15 "你累么" "你累吗？累了早点睡啦。乖乖" "Are you tied" "You tired? Then go sleep. Honey" 16 "我想她了,怎么办" "去找她啊！" "I miss her, what should I do" "Go see her!" 17 "你什么都懂噢" "我不知道,但有些东西我刚好知道" "You seem to know everything" "I don't konw, but it happens I know something" 18 "我昨天发状态找你。你都不给回" "我错了" "I sent message to you. But you didn't reply" "I apologize" 19 "讨厌" "我讨厌一切讨厌我的人" "Dislike you" "I dislike anyone who dislikes me" 20 "猜猜我是什么星座" "双子座" "Guess what is my constellation""Gemini"

Represent the short summary for finding the corresponding scientific article (long)

Print the input sentence and current response sentence onto an image and use fine-tuned ImageNet CNN model to predict the next response word.

represent the short summary for finding the corresponding scientific article (long)

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR

Question: Convolutional neural networks (CNNs) have been generally acknowledged as one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes very hard to obtain, and it is desirable to be able to learn them from few examples. In this work, we address these limitations of CNNs by developing novel, simple, and interpretable models for few-shot learn- ing. Our models are based on the idea of encoding objects in terms of visual concepts, which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of visual concepts to the few-shot setting, and then uncover two key properties of feature encoding using visual concepts, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being much more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using visual concepts helps expose the natural capability of CNNs for few-shot learning. After their debut BID13 BID13 have played an ever increasing role in computer vision, particularly after their triumph BID11 on the ImageNet challenge BID3. Some researchers have even claimed that CNNs have surpassed human-level performance BID8, although other work suggests otherwise. Recent studies also show that CNNs are vulnerable to adversarial attacks BID6. Nevertheless, the successes of CNNs have inspired the computer vision community to develop more sophisticated models BID9 BID21.But despite the impressive achievements of CNNs we only have limited insights into why CNNs are effective. The ever-increasing depth and complicated structures of CNNs makes them very difficult to interpret while the non-linear nature of CNNs makes it very hard to perform theoretical analysis. In addition, CNNs traditionally require large annotated datasets which is problematic for many real world applications. We argue that the ability to learn from a few examples, or few-shot learning, is a characteristic of human intelligence and is strongly desirable for an ideal machine learning system. The goal of this paper is to develop an approach to few-shot learning which builds on the successes of CNNs but which is simple and easy to interpret. We start from the intuition that objects can be represented in terms of spatial patterns of parts which implies that new objects can be learned from a few examples if they are built from parts that are already known, or which can be learned from a few examples. We recall that previous researchers have argued that object parts are represented by the convolutional layers of CNNs BID29 BID14 provided the CNNs are trained for object detection. More specifically, we will build on recent work BID23 which learns a dictionary of Visual Concepts (VCs) from CNNs representing object parts, see FIG0. It has been shown that these VCs can be combined to detect semantic parts BID24 and, in work in preparation, can be used to represent objects using VC-Encoding (where In general, these patches roughly correspond to semantic parts of objects, e.g., the cushion of a sofa (a), the side windows of trains (b) and the wheels of bicycles (c). All VCs are referred to by their indices (e.g., VC 139). We stress that VCs are learned in an unsupervised manner and terms like"sofa cushion" are inferred by observing the closest image patches and are used to describe them informally.objects are represented by binary codes of VCs). This suggests that we can use VCs to represent new objects in terms of parts hence enabling few-shot learning. But it is not obvious that VCs, as described in BID24, can be applied to few-shot learning. Firstly, these VCs were learned independently for each object category (e.g., for cars or for airplanes) using deep network features from CNNs which had already been trained on data which included these categories. Secondly, the VCs were learned using large numbers of examples of the object category, ranging from hundreds to thousands. By contrast, for few-shot learning we have to learn the VCs from a much smaller number of examples (by an order of magnitude or more). Moreover, we can only use deep network features which are trained on datasets which do not include the new object categories which we hope to learn. This means that although we will extract VCs using very similar algorithms to those in BID23 our motivation and problem domain are very different. To summarize, in this paper we use VCs to learn models of new object categories from existing models of other categories, while BID23 uses VCs to help understand CNNs and to perform unsupervised part detection. In Section 3, we will review VCs in detail. Briefly speaking, VCs are extracted by clustering intermediate-level raw features of CNNs, e.g., features produced by the Pool-4 layer of VGG16 BID19. Serving as the cluster centers in feature space, VCs divide intermediate-level deep network features into a discrete dictionary. We show that VCs can be learned in the few-shot learning setting and they have two desirable properties when used for image encoding, which we call category sensitivity and spatial patterns. More specifically, we develop an approach to few-shot learning which is simple, interpretable, and flexible. We learn a dictionary of VCs as described above which enables us to represent novel objects by their VC-Encoding. Then we propose two intuitive models: (i) nearest neighbor and (ii) a factorizable likelihood model based on the VC-Encoding. The nearest neighbor model uses a similarity measure to capture the difference between two VC-Encodings. The factorizable likelihood model learns a likelihood function of the VC-Encoding which, by assuming spatial independence, can be learned form a few examples. We emphasize that both these models are very flexible, in the sense that they can be applied directly to any few-shot learning scenarios. This differs from other approaches which are trained specifically for scenarios such as 5-way 5-shot (where there are 5 new object categories with 5 examples of each). This flexibility is attractive for real world applications where the numbers of new object categories, and the number of examples of each category, will be variable. Despite their simplicity, these models achieve comparable to the state-of-theart few-shot learning methods (using only the simplest versions of our approach), such as learning a metric and learning to learn. From a deeper perspective, our show that CNNs have the potential for few-shot learning on novel categories but to achieve this potential requires studying the internal structures of CNNs to re-express them in simpler and more interpretable terms. Overall, our major contributions are two-fold: We show that VCs can be learned in the few-shot setting using CNNs trained on other object categories. By encoding images using VCs, we observe two desirable properties, i.e., category sensitivity and spatial patterns. Based on these properties, we present two simple, interpretable, and flexible models for fewshot learning. These models yield competitive compared to the state-of-the-art methods on specific few-shot learning tasks and can also be applied directly, without additional training, to other few-shot scenarios. Our work lies at the intersection of attempts to understand the internal representations of neural networks and research on few-shot learning. Therefore, we review here the previous literature on these two topics. Recently, there have been numerous studies aimed at understanding the behavior of neural networks and, in particular, to uncover the internal representations within CNNs. Some researchers try to visualize internal representations by sampling BID28, generating BID20 BID15 or by backpropagating BID14 images in order to maximize the activations of the hidden units. A particularly relevant work by BID29 shows that object and object parts detectors emerge in CNNs. Conversely, other works investigate the discriminative power of the hidden features of CNNs by assessing them on specific problems BID18 BID2 BID0 BID27. The overall findings suggest that deep networks have internal representations of object parts. The most relevant work to our paper is the study of VCs which discovered mid-level visual cues in the internal features of CNNs and showed relationships between these visual cues and semantic parts BID23 ). This work is described in detail in Section 3 There have been growing attempts to perform few-shot learning motivated by attempts to mimic human abilities and to avoid some of the limitations of conventional data-demanding learning. An early attempt was made building on probabilistic program induction BID12 and another attempt exploited object parts BID25. More recent efforts at few-shot learning can be broadly categorized into two classes. The first is to design methods to embed the inputs into a feature space friendly to few-shot settings BID10 BID22. Their goal is to find a good similarity measure (e.g., using Siamese networks) that can be applied rapidly to novel categories. The second is meta-learning which efficiently trains an ordinary model with the budget of few examples BID17 BID5 ). An alternative approach by BID16 performs few-shot learning by estimating parameters of the prediction layer using regression from previously learned objects. We emphasize that the approach in our paper differs from these works, most of which are tailored for a few specific few-shot learning scenarios (i.e., test and train conditions must match), while our methods are simple and flexible, so that they work both in normal and almost all few-shot settings. In BID23, VCs were discovered as internal representations within deep networks which roughly correspond to mid-level semantic visual cues. These VCs play a core role in our work on understanding properties of CNNs and developing our interpretable few-shot learning models. In this section, we review how VCs were defined and learned in BID23. We will describe later in Section 4.1 how we modify VCs for few-shot learning. We first summarize the formalization of VCs, which are illustrated in FIG1. CNNs contain a hierarchy of lattices L l, where l ∈ {0, 1, . . .} stands for the layer of the lattice. In particular, the input image is defined over the lattice L 0 and the lattice on which we derive VCs is specified by DISPLAYFORM0 DISPLAYFORM1 In other words, the responses of all the channels in position p constitute the feature vector f n p. Then we have F k = ∪ n F n k from all images of interest. Note that by collecting feature vectors into F k, all spatial and image identity information is removed. Since layer k is usually pre-selected for different network architectures for VCs applications (e.g., BID23 typically studied layer k = 4 for VGG16 net in their work), the subscript will be omitted in the remainder of the paper for simplicity. Now we describe how VCs are extracted. The approach assumes that the VCs are represented by a population code of the CNN feature vectors. They are extracted using an unsupervised clustering algorithm. Since we first normalize the feature vectors into unit length, instead of using K-means as proposed in BID23, we assume that the feature vectors are generated by a mixture of von Mises-Fisher distributions (vMFM) BID7 and learn this mixture by the EM algorithm BID1. The goal is to maximize the likelihood function DISPLAYFORM2 where M is the total feature vector count (i.e., the number of all feature vectors collected from all images of interest) and V is the predefined VC (cluster) number. V d (·) is the density function of the vMF distribution. f m denotes each feature vector we get from the intermediate layer of a CNN (without image identity or spatial information). α, µ, and κ are vMFM parameters and represent the mixing proportion, mean direction, and concentration values respectively. We define the set of VCs by V = {µ v : v = 1, . . ., V} (i.e., by the mean directions of the learned vMFM distribution). Alternatively, since the {µ v} have the same dimensionality as the {f m}, we denote a specific VC center by DISPLAYFORM3 To help understand the VCs, we compute the cosine distances from the original feature vectors to the VCs as follows: DISPLAYFORM4 where d n p,v denotes the distance between feature vector f n p and the VC v in the n-th image at position p, and we call them VC distances. We select those feature vectors with the smallest distances to each VC and trace them back to the original input image using π k →0. This yields "visualization patches" of VCs, shown in FIG0. We observe that these patches roughly correspond to the semantic parts of objects, which justifies our assertion that VCs are semantic visual cues. In the previous studies of VCs BID23, the CNNs that were used to generate feature vectors were trained for a large scale object classification task that included the object categories of interest. Moreover, they extracted VCs using hundreds of images within a specific category of object, which ed to category specific visual cues that were useful for interpreting CNN behaviors and building novel models for semantic part detections. In more recent work (in preparation) VCs were used to encode semantic parts and objects using VC-Encoding that could be applied to detection tasks in the presence of occlusion. VC-Encoding is described in the next section. We emphasize that none of this prior work on VCs addressed few-shot learning and, by contrast, only addressed situations where there were many training examples. This section describes the technical ideas of our paper. In Section 4.1, we introduce how we learn VCs in the few-shot setting. In Section 4.2, we introduce VC-Encoding and show its two desirable properties for few-shot classification tasks. Then in Section 4.3 and Section 4.4, we propose two simple, interpretable models for few-shot learning based on VC-Encoding. It is not obvious that VCs can be applied to few-shot learning tasks where only few examples are available for each novel category. It is not possible to train the CNNs on the objects (as was done in BID23) and also there may not be enough data to get good VC clusters. Hence we modify the way VCs are learned: we learn VCs from small number of examples of novel object categories using features from CNNs trained on other object categories. This is similar to how metric-learning and meta-learning are trained on large datasets which do not include the novel categories. This ensures that the CNN used for feature extraction has never seen the categories on which we will perform few-shot classification. To extract VCs for the novel categories which only have few examples each, we pool feature vectors from the different categories together and perform the clustering algorithm on all of them. This gives us a little more data and encourages VC sharing between different categories. This improves data efficiency and also makes it easier to apply our VC models to multiple novel categories. By the two modifications described above, we obtain few-shot VCs, i.e., VCs that are suitable for few-shot learning. This is critical for our application and differentiates this work from previous studies of VCs. Surprisingly, we find that we only need a few images (e.g., five images per category) to extract high quality VCs (see visualizations in Figure 3a) which, when used for VC-Encoding, possess similar desirable properties as the traditional VCs and hence are suitable for few-shot object classification task. We assume that objects can be decomposed into semantic parts. From the perspective of VCs, this means that most {f p} should be assigned to a single VC. This requires specifying an explicit relationships between the {f p} and the VCs. A natural choice is to compute the distances d p,v between the f p and the v-th VC and threshold it to produce a binary value DISPLAYFORM0.., V } as the VC-Encoding. Note the image index n is omitted here since the operations are identical for all images of interest. We use two criteria to specify a good encoding, coverage and f irerate, defined as following: DISPLAYFORM1 The choice of the encoding threshold T is a trade-off between requiring sufficient coverage and a firing rate that is close to one. In practice, we choose T for each testing trial by a grid-search with step size 0.001 which outputs the smallest threshold ensuring that the average coverage >= 0.8 for all few-shot training images. This yields the final VC-Encoding B used in our models, with the following desirable properties:Category Sensitivity Despite the fact that the VCs are learned from a mix of images with different category labels, the first insight is that many VCs tend to fire (b ·,v = 1) intensively for one or a few specific object categories. In Figure 3a, we calculate the occurrence distributions of several VCs for 6 object categories out of the 12 in PASCAL3D+ BID26. In each column that represents a specific VC, the occurrence frequencies tend to be high for one or two object categories and low for the others. This suggests that VC identities can provide useful informations for object classification. Moreover, the corresponding visualized patches on the top of Figure 3a support our understanding that VCs have this category sensitivity because they capture the semantic parts that are specific for object categories. Spatial Patterns The spatial pattern of VC firings is also indicative of the object category. Although spatial information is ignored during feature clustering, the learned VCs give binary maps that contain regular spatial patterns for images of the same category with relatively similar viewpoints (as shown in Figure 3b). This is consistent with the more general conjecture that the spatial patterns of semantic parts play a vital role in object recognition, and shows again that the VC-Encoding can capture the spatial patterns of the semantic parts to a certain extend. Next, we build two simple few-shot learning models based on VC-Encoding learned from few examples. First, we propose a simple template matching method which is similar to traditional nearest neighbor algorithms. The only novelty is that we use a similarity metric between VC-Encodings which is spatially "fuzzy" so that it can tolerate small spatial shifts of the semantic parts in images. Formally, the similarity metric takes the following form: FIG4 ). During testing, we classify an image to the category of the training example with the largest similarity. DISPLAYFORM0 One motivation for this method is that sequential convolutional operations carried in a neural network can be considered as embedding input images into a hierarchy of feature spaces. Each convolutional layer can be treated as a different level of decomposition of the inputs since the convolution along with non-linear activation, which take the form Y = σ(W · X + B), is composed of matching templates W and using non-linear function σ to filter out patterns based on the threshold B. In light of this interpretation, and the two properties described earlier, it is reasonable that VC-Encoding will yield an explicit semantic decomposition. Apart from the intuitive nearest neighbor method, we present a second method which models the likelihood of the VC-Encoding. We observe that we can specify a distribution over the VC-Encoding b p,v using a bernoulli distribution with probability θ p,v. Following Naïve Bayes, we assume all the elements of the VC-Encoding b are independent (making it possible to learn the distribution from a very small number of examples). Hence we can express the likelihood of b as following: DISPLAYFORM0 For each object category y, we derive a probabilistic distribution θ y from the training examples. Thus the prediction of object category given the VC-Encoding b is given by: DISPLAYFORM1 Note that by doing this, we are in fact implementing a discriminative model obtained from a generative distribution. We smooth each distribution θ y using a Gaussian filter to guard against unlikely events. Few-Shot learning is a very challenging task where humans perform much better than current algorithms. It requires the ability, or efficiency, to learn generalizable knowledge from strictly limited examples, such as a few training images. Nevertheless, Section 4.1 suggests that a few images may be enough for learning object models when represented by VC-Encodings. Indeed our experiments show that both our two VCs-based few-shot learning models are competitive in performance with existing methods designed specifically for few-shot learning such as BID17. In addition, while previous few-shot methods are trained to work in specific few-shot scenarios, such as 5-way classifications, our methods can be applied to a large range of few-shot scenarios without additional training. The experimental show that trained CNNs have the potential to recognize novel objects from few examples by exploiting VC-Encoding. To assess the capability of our few-shot methods, we first evaluate them on a common few-shot learning benchmark, namely Mini-ImageNet. The Mini-ImageNet dataset was first proposed by BID22 as a benchmark for evaluating few-shot learning methods. It selects 100 categories out of 1000 categories in ImageNet with 600 examples per category. We use the split proposed by BID17 consisting of 64 training categories, 16 validation categories and 20 testing categories. In accordance with the convention for Mini-ImageNet, we perform numerous trials of few-shot learning during testing. In each trial, we randomly sample 5 unseen categories from a preserved testing set. Each category is composed of 5 training images for the 5-shot setting and 1 training image for the 1-shot setting. During evaluation, we randomly select 15 images for each category following BID17.As Table 1 illustrates, we compare our methods against two baselines in line with the ones in BID17. In addition, we present the performances of state-of-the-art few-shot learning 45.61 ± 1.14% 63.07 ± 1.02% 45.11 ± 0.66% Table 1: Average classification accuracies on Mini-ImageNet with 95% confidence intervals. Evaluations of Baseline-finetune and Baseline-nearest-neighbor are from BID17. Pool3-nearest-neighbor stands for a nearest neighbor method based on raw Pool-3 features from the same VGG-13 as our methods. At the bottom are our factorizable likelihood method and nearest neighbor method based on VCs. Marked in bold at the top are the best published for each scenario. Marked in bold at the bottom are our best for the corresponding set-up. At the right is an extended setting for variance in the number of categories. Note in the last column we use the same models as in the middle column and we omit those that cannot be directly applied to this setting. We adopt the for Matching Network from BID17.methods including matching network BID22, Meta-Learner BID17 and MAML BID5. Regarding our methods, we train a VGG-13 on the training and validation set. The network is trained with the objective of cross entropy. We preserve 10% images in each category to validate our network. We extract 200 VCs from the Pool-3 layer. The reason for choosing Pool-3 features is that a grid in Pool-3 lattices L 3 correspond to a 36 × 36 patch in the original 84 × 84 image, which is a plausible size for a semantic part. For the gaussian filter used to smooth the factorizable likelihood model, we use a σ of 1.2. To directly examine the impact of the VCs, we also include the of nearest neighbor matching using raw features from the pool-3 layer (referred as Pool3-Nearest-Neighbor in Table 1). Moreover, we attempt to evaluate the fewshot learning ability with the variance in the number of categories (as shown in the last column of Table 1) and further extend the few-shot learning evaluation with other randomly selected settings (as shown in Table 2), where we need to use exactly the same model as the middle columns in Table 1 trained once on the training and validation set. The show that our VCs-based methods compare well with current methods which were specifically designed for few-shot learning. If we contrast with meta-learning-based methods, we achieve higher accuracy than Meta-Learner both in the 1-shot and the 5-shot set-ups, while just slightly behind MAML. Compared with metric-based methods, which are more similar to ours, we marginally outperform the matching network, which is the state-of-the-art method of this category. These confirm our assumption that low level visual cues within trained CNNs can naturally perform few-shot learning. We observe that on the 5-shot scenario, our likelihood model is significantly better than our nearest neighbor model. A possible explanation is that the likelihood model Table 2: Average classification accuracies on Mini-ImageNet with 95% confidence intervals under randomly selected few-shot settings. All models used here and in Table 1 are the same set of models trained only once on the training set. Like the last column of Table 1, we omit those models which cannot be directly applied to various few-shot settings. Step I: Transform input to Pool-4 Raw Features. Step III: Given a set of VC distribution, calculate Pixel-wise Likelihood. Step IV: Aggregate Pixel-wise Likelihood into Class-wise Likelihood. Step V: Choose the class with highest likelihood. Step II: Transform Raw Features into VC-Encoding. Figure 5: Visualizing the inference procedure of the factorizable likelihood model (using 1 VC for example). The original image is processed by part of VGG-16, and represented by its VC-Encoding. After calculating the likelihood for each pixel using distributions derived from a few examples, we obtain the pixel-wise likelihood. Then we use the likelihood to make the final decision. For better visualization, we rescale the variance of aggregated likelihood to 1. For all the visualizations, we use the same Universal Color Bar. This figure is best viewed in color.combines several training examples into a distribution while nearest neighbor can only use examples individually. For instance, if a front wheel of cars appears in one training example and a rare wheel occurs in another example, the likelihood model can combine these two wheels into a distribution while nearest neighbor can only match testing examples with either the front wheel or the rare wheel. Finally, using the same model in the middle columns of Table 1, some previous methods like Meta-Learner LSTM are not applicable to various extended settings in Table 2 and the last column in Table 1. In fact, these methods can directly deal with only changes in the number of shots but cannot deal easily with changes in the number of categories. But unlike these methods, our few-shot learning methods based on VC-Encoding can be easily extended (with minimal re-training) to any number of shots and any number of categories. To delve deeper into our methods, we apply them to PASCAL3D+, a dataset with larger high quality images than Mini-ImageNet. PASCAL3D+ BID26 ) is a dataset augmenting 12 rigid categories of the PASCAL VOC 2012 (Everingham et al.). It was originally tailored for 3D object detection and pose estimation. We choose PASCAL3D+ as our testbed since it provides high quality images with comparable image size to ImageNet. We interpret our few-shot recognition mainly by visualizing every step of the inference. With input images of sufficient sizes, we can obtain large VC-Encoding distribution maps whose visualizations are easy for humans to interpret. The simplicity of our methods makes the inference process of few-shot recognition very transparent. On PASCAL3D+, we first qualitatively analyze this procedure based on VCs. In Figure 5, we visualize every step of our method based on an example VC. Among the closest patches, the corner of the TV Monitor occurs most frequently. So we assume this VC relate to the corner of TV Monitor. The distributions of this VC suggest it mainly responds to the upper right of TV Monitors since only the upper right corner of TV Monitor's distribution map shows high frequency (see the distributions in Figure 5). Using this VC, we convert the original deep network features into VC-Encoding. The VC-Encoding implies this VC fires on the upper right corner of the input (see the VC-Encoding map in Figure 5). After calculating the pixel-wise likelihood using the distributions from a few images, it is clear that except for the TV Monitor, each category has low likelihood in the area of the corner TAB6: Average classification accuracies on PASCAL3D+. At the top is the group of baseline methods including nearest neighbor and Exemplar-SVM based on Pool-4 features from the same VGG-16 used in our methods. In the middle are our factorizable likelihood models using different number of VCs. At the bottom are our VCs-based nearest neighbor models. Marked in bold are the best within each group for each scenario.(dark parts of pixel-wise likelihood maps in Figure 5). Finally, we aggregate the likelihood and make the correct classification decision. Meanwhile, we quantitatively evaluate our methods on PASCAL3D+. More specifically, we employ PASCAL3D+ as our testing dataset. For training, we use the ImageNet BID3 classification dataset without object categories related to 12 rigid categories (956 categories left). We train an ordinary VGG-16 as our starting point which achieves 71.27% top-1 accuracy. For testing, we crop the objects out using annotated bounding boxes provided by BID26 and resize them into 224 × 224. Then we use Pool-4 features produced by the VGG-16 to implement our few-shot methods instead of the Pool-3 features used in 5.1. The main reason for this change is the increased input image size of 224 in PASCAL3D+, which suggests that Pool-4 features will be better for capturing the semantic parts. As a comparison, we propose 2 baseline models. One (referred to as Baseline-nearest-neighbor in TAB6) is a nearest neighbor method based on raw Pool-4 features using the cosine distance metric. The other (referred to as Baseline-SVM in TAB6) is an Exemplar-SVM trained using hinge loss. Both of these baselines use the same pre-trained VGG-16 as our methods. During evaluation, we set 20 trials of both 5-shot and 1-shot learning over 12 categories on PASCAL3D+. We also assess our methods using different numbers of VCs to see impacts of the number of VCs. The are shown in TAB6.In light of our testing , we conclude that VC-Encoding is a useful semantic decomposition of images into parts. In general, our methods based on VCs significantly outperform two baselines. In particular, the difference between our nearest neighbor methods and the baseline nearest neighbor methods is due to the use of VCs (e.g., by first transferring the raw feature vectors into VC distances, and by thresholding to get the VC-Encoding). Thus, we claim that decomposing fuzzy features (i.e., deep network features) into explicit semantic cues (i.e., the VCs) improves both interpretability and performance. We also notice that our methods are not sensitive to the number of VCs since changes of the number of VCs only cause slight differences among accuracies. In this paper we address the challenge of developing simple interpretable models for few-shot learning exploiting the internal representations of CNNs. We are motivated by VCs BID23 which enable us to represent objects in terms of VC-Encodings. We show that VCs can be adapted to the few-shot learning setting where the VCs are extracted from a small set of images of novel object categories using features from CNNs trained on other object categories. We observe two properties of VC-Encoding, namely category sensitivity and spatial pattern, which leads us to propose two novel, but closely related, methods for few-shot learning which are simple, interpretable, and flexible. Our methods show comparable performances to the current state-of-the-art methods which are specialized for specific few-shot learning scenarios. We demonstrate the flexibility of our two models by showing that they can be applied to a range of different few-shot scenarios with minimal re-training. In summary, we show that VCs and VC-Encodings enable ordinary CNNs to perform few-shot learning. We emphasize that in this paper we have concentrated on developing the core ideas of our two few-shot learning models and that we have not explored variants of our ideas which could lead to better performance by exploiting standard performance enhancing tricks, or by specializing to specific few-shot challenges. Future work includes improving the quality of the extracted VCs and extending our approach to few-shot detection. Although they were not tailored for specific few-shot learning scenarios, our VC-Based methods have comparable performance to previous methods while being more flexible and interpretable. Since we are proposing a new method for few-shot learning, we carried out ablation studies to understand the influences of different model components on the effectiveness of our approach. Encoding Against Distance. In Section 4.2, we transformed real-valued distances from VCs into binary VC-Encoding by a threshold, which was dynamically determined to satisfy our requirement for coverage and f irerate. We claim that this encoding is beneficial to learning with limited budgets, e.g., few shots. First, it enables us to implement the Factorizable Likelihood Model, which relies on binary encoding and shows better than nearest neighbor methods. Second, by thresholding the distances, we are filtering out the noises of model estimations based on few training examples. This also helps the nearest neighbor model. In Table 1, we compare our models based on VC-Encoding with a model based on real-valued distances (referred to as VC-Distance in Table 1), which uses the cosine distances as the metric. The show that while despite being better than the original CNN features (Pool3-nearest-neighbor in Table 1), the distance-based model is not as effective as binary coded models. Table 1: VC models using distances and VC-Encoding on Mini-ImageNet. VC-distance stands for a nearest neighbor model using distances regarding VCs and the cosine distance metric. VC-nearestneighbor and VC-likelihood are our proposed models based on VC-Encoding. Pool3-nearestneighbor is a nearest neighbor model using original CNN features. Marked in bold are the best for each few-shot set-up. Clustering Methods. VCs are obtained without supervision by clustering features extracted by CNNs. In our proposed models, we use the clustering method based on a mixture of von MisesFisher distribution. By learning this distribution through the EM algorithm, we can obtain cluster centers that remain on the unit sphere. This is mathematically reasonable since we expect the features to distribute on the unit sphere. In Table. 2, we compare the using von Mises-Fisher distribution with those using K-Means clustering on the 5-Shot 5-category learning task. The show that for both VC-Likelihood and VC-nearest-neighbor K-Means is just slightly behind the von Mises-Fisher distribution. Hence, we state that VC-based models are robust to different clustering methods. Clustering Likelihood Nearest Neighbor K-Means 62.51 ± 0.89% 59.60 ± 1.04% von Mises-Fisher 63.07 ± 1.02%58.85 ± 1.12% Table 2: Different clustering for each model we proposed on the 5-shot 5-category setting on MiniImageNet. Likelihood refers to the Factorizable Likelihood Model. Marked in bold are the best for each model. Scale of Semantic Parts. While our VCs are extracted without any supervision, we find that they are related to semantic parts of objects. However, semantic parts can have various scales in images. To keep the simplicity of our model, we choose a single scale of semantic parts. In terms of the CNN we used, we choose to use Pool-3 features of VGG-13. In Table. 3, we test different scales of VCs, i.e., features from different layers, under the 5-Shot 5-Category few-shot learning setting. We see that our proposed scale is the most effective of the different scales. In addition, we notice that while the performance of the baseline method, which uses the original features for nearest neighbor, drops drastically as scale changes, the performances our VC-based models, especially our Likelihood model, decay more gradually. This also reflects the robustness of the VC-based models. In this section we illustrate more VCs (in FIG0 and their distributions over object categories (in TAB8).

We enable ordinary CNNs for few-shot learning by exploiting visual concepts which are interpretable visual cues learnt within CNNs.

Querying a black-box neural network reveals a lot of information about it; we propose novel "metamodels" for effectively extracting information from a black box.