Question: Represent the article for finding a single-sentence TLDR

Question: Dense word vectors have proven their values in many downstream NLP tasks over the past few years. However, the dimensions of such embeddings are not easily interpretable. Out of the d-dimensions in a word vector, we would not be able to understand what high or low values mean. Previous approaches addressing this issue have mainly focused on either training sparse/non-negative constrained word embeddings, or post-processing standard pre-trained word embeddings. On the other hand, we analyze conventional word embeddings trained with Singular Value Decomposition, and reveal similar interpretability. We use a novel eigenvector analysis method inspired from Random Matrix Theory and show that semantically coherent groups not only form in the row space, but also the column space. This allows us to view individual word vector dimensions as human-interpretable semantic features. Understanding words has a fundamental impact on many natural language processing tasks, and has been modeled with the Distributional Hypothesis BID0. Dense d-dimensional vector representations of words created from this model are often referred to as word embeddings, and have successfully captured similarities between words, such as word2vec and GloVe BID1 BID2. They have also been applied to downstream NLP tasks as word representation features, ranging from sentiment analysis to machine translation BID3 BID4.Despite their widespread popularity in usage, the dimensions of these word vectors are difficult to interpret BID5. Consider w president = [0.1, 2.4, 0.3] as the 3-dimensional vector of "president" from word2vec. In this 3-dimensional space (or the row space), semantically similar words like "minister" and "president" are closely located. However, it is unclear what the dimensions represent, as we do not know the meaning of the 2.4 in w president. It is difficult to answer questions like'what is the meaning of high and low values in the columns of W' and'how can we interpret the dimensions of word vectors'. To address this problem, previous literature focused on the column space by either training word embeddings with sparse and non-negative constraints BID6 BID7 BID8, or post-processing pre-trained word embeddings BID5 BID9 BID10. We instead investigate this problem from a random matrix perspective. In our work, we analyze the eigenvectors of word embeddings obtained with truncated Singular Value Decomposition (SVD) BID11 BID12 of the Positive Pointwise Mutual Information (PPMI) matrix BID13. Moreover, we compare this analysis with the row and column space analysis of Skip Gram Negative Sampling (SGNS), a model used to train word2vec BID14. From the works of BID15 proving that both SVD and SGNS factorizes and approximates the same matrix, we hypothesize that a study of the principal eigenvectors of the PPMI matrix reflects the information contained in SGNS.Contributions: Without requiring any constraints or post-processing, we show that the dimensions of word vectors can be interpreted as semantic features. In doing so, we also introduce novel word embedding analysis methods inspired by the literature of eigenvector analysis techniques from Random Matrix Theory. Recently, there have been several works that have shown similar in semantic grouping among the column values. Several of these algorithms proposed to train non-negative sparse interpretable word vectors BID6 BID7 BID8 BID16.Furthermore, BID5 also proposed methods to post-process pre-trained word vectors with non-negativity and sparsity constraints. However, their vectors were optionally binarized, which is difficult to interpret intensity than real-values. BID9 has proposed to overcome these limitations by simply training a rotation matrix to transform pre-trained word2vec and GloVe, without being sparse or binary. Finally, BID10 post-trained the pre-trained word embeddings with k-sparse autoencoders with similar constraints to BID5.While these methods were able to successfully achieve interpretability in the column space evaluated with word intrusion detection tests, they either enforced sparsity and non-negativity constraints, or required extensive post-processing. Furthermore, they focused less on the analysis and discussion on the actual meanings of the columns despite their pursuit of interpretable dimensions. Hence, in our work, we put more emphasis on such implications with conventional algorithms without any extra constraints or post-processing steps. We define the Positive Pointwise Mutual Information (PPMI) matrix as M PPMI, the set of unique words as vocabulary V, and word embedding matrices created from SVD and SGNS as W SVD and W SGNS. The k-th largest eigenvalue and corresponding eigenvector of M PPMI are denoted as λ k and u k ∈ R |V |, and the k-th column of W SGNS as v k ∈ R |V |. The word vectors are denoted w SVD word or w SVD word, but when context is clear or does not matter, we simply use w word. Note that we often use the term "eigen" when and "singular" interchangeably because M PPMI is defined as a square matrix. Each entry of a co-occurrence matrix M represents the co-occurrence counts of words w i and c j in all documents in the corpus. However, raw co-occurrence counts have been known to underperform than other transformed variants BID15. Pointwise Mutual Information (PMI) BID13 instead transforms matrix by measuring the log ratio between the joint probability of w and c when assuming independence of the two and not. DISPLAYFORM0 The problem of this association measure is when dealing with never observed pairs which in PMI(w, c) = log 0. To cope with such, Positive Pointwise Mutual Information has been used to map all negative values to 0 from the intuition that positive associations are often more informative in downstream NLP tasks BID15.PPMI(w, c) = max(PMI(w, c), 0) Truncated SVD (we will further refer this as simply SVD), which is equivalent to maximum variance Principal Component Analysis (PCA) and has been popularized by Latent Semantic Analysis (LSA) BID12, factorizes the PPMI matrix as M PPMI = U · S · V T and truncates to d dimensions. Following the works of BID17, the word embedding matrix is taken as W = U d, instead of the more "standard" eigenvalue weighting W = U d · S. We discuss the effect of this in Section 6.2. Unlike PPMI and SVD which gives exact solutions, the word2vec Skip-Gram model, proposed by BID1, trains two randomly initialized word embedding matrices W and C with a neural network. DISPLAYFORM0 where DISPLAYFORM1 The intuition is to basically maximize the dot product between "similar" word and context pairs, and minimize the dot product between wrong pairs. The Softmax function is simply a generalized version of the logistic function to multi-class scenario. However, the normalization constant which computes the exponentials of all context words, is very computationally expensive when the vocabulary size is large. Hence, BID14 proposed Skip Gram with Negative Sampling (SGNS) to simplify the objective using negative sampling. We analyze the distributions of eigenvectors, calculate the Inverse Participation Ratios (IPR) to quantify the ratio of significant elements and measure structural sparsity, and qualitatively interpret the significant elements. The empirical distribution of eigenvector elements u k is compared with a Normal distribution N(µ u k, σ 2 u k) to measure normality of the eigenvectors, where µ u k, σ 2 u k refer to the mean and variance of u k. BID18 have shown that eigenvectors deviating from Gaussian contain genuine correlation between stocks, while also revealing a global bias that represented newsbreaks influencing all stocks. We search for similar patterns in Section 5.1.Inverse Participation Ratio: The Inverse Participation Ratio (IPR) of u k, denoted as I k, quantifies the inverse ratio of significant elements in the eigenvector u k BID18 BID19 BID20. DISPLAYFORM0 where u k i is the i-th element of u k. The intuition of IPR can be illustrated with two extreme cases. First, if all elements of u k have same values 1/ |V |, then I k is simply 1/|V |, with reciprocal 1/I k being |V |. This means that all |V | elements contribute similarly. On the other hand, a one-hot vector with only one element as one, and the rest as zero, u k will have an IPR value of one (also same for reciprocal). Hence, the reciprocal, 1/I k, measures the ratio of significant participants in u k. In short, the larger the I k, the smaller the ratio of participation, and the sparser the vector, in turn, reflecting structural sparsity of u k. Furthermore, as 1/I k ∈ [1, |V |], dividing this reciprocal with |V | will yield the sparsity of a given vector u k ∈ R |V |.Visualization of Top Eigenvector elements: As u k, v k ∈ R |V |, we can map each index of the vectors to a word in the vocabulary V. Hence, we investigate the dimensions and their indices (or words) with the largest absolute values and search for semantic coherence. Similar approaches with financial data have shown to group stocks from same industries or nearby regions BID18, and with genetic data, revealed important co-evolving genes in gene co-expression networks BID19. BID1, which has also been used by BID1. Removing most of the noisy non-alphanumerics, such as XML tags, the dataset size effectively reduced from approximately 66GB to 25GB, containing around 3.4B tokens. The vocabulary size is approximately 346K as we only consider words with at least 100 occurrences. SGNS and SVD We adapt the code from the hyperwords 3 released by BID17 to train both W SVD and W SGNS. Our code is publicly available online BID3. For W SGNS, we set negative sampling as 5. For both, we set a context window size of 2 (taking 5 words as context) and embedding dimension d = 500. From FIG0, we can see that eigenvectors corresponding to the larger eigenvalues such as u 1 or u 2 clearly deviate from a Gaussian distribution, and so do u 100 and u 500, but less. This shows us that the eigenvectors are not random and contain meaningful correlations. It is expected to see such pattern because these vectors are the principal eigenvectors. On a more interesting note, u 1 not only significantly deviates from a normal distribution, but also has only non-zero negative values as its elements, and no other eigenvectors have shown this behavior. This suggests that this particular eigenvector could represent a common bias that affects all "words", as it captured the effect of news outbreaks for stock prices in BID18. We revisit the interpretation of this observation in Section 6.1. FIG1 illustrates the IPR of u k plotted against the corresponding eigenvalue λ k, and vice versa for v k. From the plot, we can clearly see that the eigenvectors of W SVD have approximately 10x higher IPR values than those of W SGNS, meaning that the vectors are much sparser for W SVD. From FIG1, we can see that the largest eigenvector has the smallest IPR of 0.000006, and the reciprocal 1/I k divided by |V |, yields 48%, while the same for the largest I k gave around 4.7%. The mean value of 1/I k divided by |V |, across all eigenvectors was 27.5% indicating that there exists some sparse structure within the eigenvectors of W SVD. On the other hand, FIG1 shows that mean for v k was around 36%, meaning that column vectors of W SGNS are generally denser and less structured. Such discrepancy in structural sparsity motivates us to analyze the eigenvectors of W SVD in depth.6 Analysis and Discussion Based on the of previous sections, we further examine the top elements of the eigenvectors by sorting their absolute values in decreasing order. Table 3: Top participants of the salient columns of the word vector for "airport."baseball related words. Some words from u 121 initially seem irrelevant to baseball. However, "buehrle" is a baseball player, "rbis" stand for "Run Batted Ins", and "astros" is a baseball team name from Houston. Meanwhile, the words grouped in u 1, the largest eigenvector, could explain the bias we mentioned in Section 5.1. The significant participants tend to be strong transition words that are used often for dramatic effects, such as "importantly" or "crucially". Evidently, these words increase the intensity of the context. Moreover, while it was originally hypothesized that the largest principal eigenvectors would capture some semantic relationship, the 121th vector u 121 show surprisingly focused and narrow semantic grouping related to baseball. Further investigation reveals that u 121 has one of the highest IPR values, hence being one of the most sparse vectors. We verify similar trends in other eigenvectors with high IPR values as shown in TAB3. An interesting pattern arises here, in which the sparser eigenvectors tend to capture more distinct and rare features such as foreign names or languages, or topics like baseball. Furthermore, we compare the column space analysis on W SVD and W SGNS. Consider the word vector w airport for the word "airport. " We choose the salient dimensions, which are the largest elements, of w airport, and investigate the significant elements of those chosen dimensions (columns). Table 3 shows that the columns from W SVD display semantic coherence while those from W SGNS seem random. u 53 groups words that are related to the location of the airports. For example, one could say "Trindade station connects with the airport. " Similarly, u 337 groups famous airline companies together, while "fiumicino" is a famous airport in Italy. Sections 5.1 and 5.2 revealed that the eigenvectors contain genuine correlation and structure in the column space. We further show in Section 6.1 that semantically coherent words form groups of significant participants in each eigenvector. Now we can answer the questions we asked earlier. What is the meaning of high and low values in the columns of W? If word vector w from W SVD has a high absolute value in column k, it means that the word is relevant to the semantic group formed in u k. For example, the words from FIG2 have highest values in column k = 121, in which u 121 represents a semantic group related to baseball, as shown in TAB1.How can we interpret the dimensions of word vectors? The answer to this question follows naturally. As the salient dimensions represent relevant semantic groups, we can view the dimensions of w as semantic features. This view is in line with the Topic Modeling literature, in which words and documents are clustered into distinct latent topics. Hence, we can also see the word embedding dimensions as latent topics that can be interpretable. It can be easily seen from FIG2 that similar words do not show any interpretable similarity in their W SGNS representations, despite being nearest neighbors in the row space. On the other hand, it is very clear from FIG2 that similar words have similar representations, or feature vectors. We thus empirically verify that the dimensions of the row vectors can be viewed as semantic or syntactic features. Finally, the structural sparsity discovered with the IPR is further confirmed by contrasting FIG2. It is clearly visible that the the vectors from SVD are much sparser than from SGNS.Effect of Eigenvalue Weighting: As mentioned in Section 3, weighting with the eigenvalues essentially scales each feature column by the corresponding eigenvalues. Such process can be viewed as simply incorporating a prior, and does not hurt the interpretability. However, as BID17 showed that eigenvalue weighting decreases the performance of downstream NLP tasks, we can assume that either the prior is wrong, or too strong. In fact, in many cases, the largest eigenvalues are often order of magnitude larger than others, which can explain why not weighting the word embeddings with their corresponding eigenvalues would work better. In this work, we analyzed the eigenvectors, or the column space, of the word embeddings obtained from the Singular Value Decomposition of PPMI matrix. We revealed that the significant participants of the eigenvectors form semantically coherent groups, allowing us to view each word vector as an interpretable feature vector composed of semantic groups. These can be very useful in error analysis in downstream NLP tasks, or cherry-picking useful feature dimensions to easily create compressed and efficient task-specific embeddings. Future work will proceed in this direction on applying interpretability to practical usage.

Without requiring any constraints or post-processing, we show that the salient dimensions of word vectors can be interpreted as semantic features.

represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent this text

Question: Neural networks in the brain and in neuromorphic chips confer systems with the ability to perform multiple cognitive tasks. However, both kinds of networks experience a wide range of physical perturbations, ranging from damage to edges of the network to complete node deletions, that ultimately could lead to network failure. A critical question is to understand how the computational properties of neural networks change in response to node-damage and whether there exist strategies to repair these networks in order to compensate for performance degradation. Here, we study the damage-response characteristics of two classes of neural networks, namely multilayer perceptrons (MLPs) and convolutional neural networks (CNNs) trained to classify images from MNIST and CIFAR-10 datasets respectively. We also propose a new framework to discover efficient repair strategies to rescue damaged neural networks. The framework involves defining damage and repair operators for dynamically traversing the neural networks loss landscape, with the goal of mapping its salient geometric features. Using this strategy, we discover features that resemble path-connected attractor sets in the loss landscape. We also identify that a dynamic recovery scheme, where networks are constantly damaged and repaired, produces a group of networks resilient to damage as it can be quickly rescued. Broadly, our work shows that we can design fault-tolerant networks by applying on-line retraining consistently during damage for real-time applications in biology and machine learning. In this paper, inspired by the powerful paradigms introduced by deep learning, we attempt to understand the computational and mathematical 23 principles that impact the ability of neural networks to tolerate damage and be repaired. We characterize the response of two classes 24 of neural networks, namely multilayer perceptrons (MLP's) and convolutional neural nets (CNN's) to node-damage and propose a new 25 framework that identifies strategies to efficiently rescue damaged networks in a principled fashion. Our key contribution is the introduction of a framework that conceptualizes damage and repair of networks as operators of a dynamical 27 system in the high-dimensional parameter space of a neural network. The damage and repair operators are used to dynamically traverse the 28 landscape with the goal of mapping local geometric features (like, fixed points, limit-cycles or point/line-attractors) of the neural 29 networks' loss landscape. The framework led us to discovering that the iterative application of damage and repair operators in 30 networks that are highly resilient to node-deletions as well as guides us to uncover the presence of geometric features that resemble a 31 path-connected attractors set, in many respects, in the neural networks' loss landscape. Attractor-like geometric features in the networks' 32 loss landscape explains why the iterative damage-repair strategy always in the rescue of damaged networks within a small number of 33 training cycles. 2 Susceptibility of neural networks to damage The first question we ask in this paper is how do neural networks respond to physical perturbations and how does it affect their functional 36 performance. We characterize the impact of neural damage on'cognitive' performance of neural networks by tracking the performance of 37 two classes of artificial neural networks, namely MLPs and CNNs, to deletion of neural units from the network. The MLPs and CNNs were 38 trained to perform simple cognitive tasks like image classification on MNIST and CIFAR-10 datasets respectively before the networks were 39 perturbed. To damage a node i in the hidden layer of an MLP or in the fully connected layer of a CNN, we zero all connections between node i and the 41 rest of the network. And, to damage a node j in the convolutional layer of a CNN, we zero the entire feature map. In this paper, we are 42 specifically interested in node-damage as our perturbation because of its similarity in phenomena to neuron death in biological networks 43 and node-failures in neuromorphic hardware. We observe a steep increase in the rate of decline of functional performance as we incrementally delete nodes from either an MLP with 1 hidden layer (Fig-1a), an MLP with 2 hidden layers (Fig-1b) or a CNN with 2 convolutional layers, a pooling layer and 2 fully connected 46 layers (Fig-1c). We refer to this discrete jump in the rate of decline of performance as a phase transition. The existence of a phase transition shows that neural nets (MLP's and CNN's) damaged above their respective critical thresholds are not 48 resilient to any further perturbation. We are interested in deciphering strategies that enable the quick rescue of damaged neural nets and also want to identify networks that are more resilient to perturbation. 3 Can we rescue these damaged networks? We ask whether it is fundamentally possible to rescue damaged networks in order to compensate for their performance degradation. To do The plots in figure-2 show that damaged neural networks can be rescued to regain their original functional performance when re-trained 57 via both strategies 1 and 2. However, they require a large number of training cycles (epochs) to be effectively rescued (figure-2c). The 58 requirement of a large number of training cycles for the effective rescue of a neural network reduces the feasibility of either strategy as it 59 isn't ideal for both, living neural networks in the brain or artificial networks implemented on neuromorphic hardware to be re-trained for 60 extended periods of time to recover from small damages to its network.'space' in the networks' loss manifold that contains high performing, more resilient, sparser networks. As the iterative process of damage and repair always enabled the fast recovery of a damaged network (irrespective of the number of damaged units), this was surprising to us and we were interested in 74 determining if the loss landscape manifold had'special' geometric features that enabled this rescue. To map geometric features of a neural networks' loss landscape, we formally conceptualize the iterative 76 damage-repair paradigm as a dynamical system that involves the application of a damage and repair 77 operator (r) on a neural network (w). We define w to be a feed-forward neural network with n nodes and N total connections. Here, w i is the set of connections made by node i with the previous layer in the network. By definition, w i = φ, if node i is in the first layer. We also have: Dim(w i) = N and w ∈ R N To damage a neural network, we define a damage operator D i, that damages node i in the network. To repair a neural network, we define a rescue operator r {i,j}. Here {i, j} refers to the set of damaged nodes. The rescue operator forces the 83 network to descend the loss manifold, while fixing nodes within the set and their connections to zero. Rescue of the network is achieved by 84 performing a constrained gradient descent on the networks' loss manifold. where, η is the gradient step-size and ∂L ∂ w k is the gradient of the loss function of the neural network along w k A damage-repair sequence involves the application of a damage operator followed by a repair operator. A stochastic damage-repair sequence involves the random sampling of a damage operator from D, followed by the application of an 88 appropriate repair operator (ensuring that gradient descent is performed on remaining undamaged nodes). We define a random variable D to sample an operator D i from the set of all possible damage operators = {D i : i ∈ {1, ..., n}}. An iterative 90 damage-repair sequence is the repeated application of a random damage operator D coupled with a deterministic repair operator r {i,j,k,...}, that ensures all damaged nodes maintain a zero edge-weight, while other weights are plastic. Here, we show the long-hand and short-hand 92 notation for the iterative application of damage-repair operators. We hypothesize that ∃ an open set of networks U, that constitutes an invariant set, where: For any two points, w 1 and w 2 ∃γ: −→ U, such that: Our numerical strongly suggests the presence of an invariant, path-connected topological space U in the neural networks' loss 97 manifold. In our experiments, the invariant, path-connected set is a collection of trained networks, whose image corresponding to the 98 application of a damage and repair operator lies in the same set, visualized by the thick black arc (as shown in figure-4) obtained by tSNE 99 embeddings of the high-dimensional network (w). We observe that iterative application of the damage-repair operator on a network sampled 100 from U in a series of networks that belong to the same set U. This is observed in fig-4b & fig-4d. The red lines indicate damage of 101 network, while the green lines correspond to repair of damaged networks. This hints at the possibility that U is an invariant set. We also 102 interpolated between all pairs of networks sampled from U and observed that all the interpolated networks were present in U as well. In this paper, we address a pertinent question of how neural networks in the brain, or in engineered systems respond to damage of their units 105 and whether there exists efficient strategies to repair damaged networks. We observe a phase transition behavior as we incrementally delete 106 nodes from the neural network as the rate of decline of performance steeply increases after crossing a critical number of node deletions. We discover that damaged networks can be rescued and the iterative damage-rescue strategy produces networks that are highly resilient 108 to perturbations, and can be rescued within a small number of training cycles. This is enabled by the putative presence of an invariant, path-connected set in the networks' loss manifold. Although we have shown numerical that strongly suggest the presence of invariant 110 sets in the loss manifold, our future work will focus on analytically proving the presence of these topological spaces in the loss manifold, through the formalization presented in the paper, and the use of the Koopman operator machinery, amongst others.

Represent the TLDR for finding the corresponding scientific article

strategy to repair damaged neural networks

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Automatic question generation from paragraphs is an important and challenging problem, particularly due to the long context from paragraphs. In this paper, we propose and study two hierarchical models for the task of question generation from paragraphs. Specifically, we propose (a) a novel hierarchical BiLSTM model with selective attention and (b) a novel hierarchical Transformer architecture, both of which learn hierarchical representations of paragraphs. We model a paragraph in terms of its constituent sentences, and a sentence in terms of its constituent words. While the introduction of the attention mechanism benefits the hierarchical BiLSTM model, the hierarchical Transformer, with its inherent attention and positional encoding mechanisms also performs better than flat transformer model. We conducted empirical evaluation on the widely used SQuAD and MS MARCO datasets using standard metrics. The demonstrate the overall effectiveness of the hierarchical models over their flat counterparts. Qualitatively, our hierarchical models are able to generate fluent and relevant questions. Question Generation (QG) from text has gained significant popularity in recent years in both academia and industry, owing to its wide applicability in a range of scenarios including conversational agents, automating reading comprehension assessment, and improving question answering systems by generating additional training data. Neural network based methods represent the stateof-the-art for automatic question generation. These models do not require templates or rules, and are able to generate fluent, high-quality questions. Most of the work in question generation takes sentences as input (; ; ;). QG at the paragraph level is much less explored and it has remained a challenging problem. The main challenges in paragraph-level QG stem from the larger context that the model needs to assimilate in order to generate relevant questions of high quality. Existing question generation methods are typically based on recurrent neural networks (RNN), such as bi-directional LSTM. Equipped with different enhancements such as the attention, copy and coverage mechanisms, RNN-based models (; ;) achieve good on sentence-level question generation. However, due to their ineffectiveness in dealing with long sequences, paragraph-level question generation remains a challenging problem for these models. proposed a paragraph-level QG model with maxout pointers and a gated self-attention encoder. To the best of our knowledge this is the only model that is designed to support paragraph-level QG and outperforms other models on the SQuAD dataset . One straightforward extension to such a model would be to reflect the structure of a paragraph in the design of the encoder. Our first attempt is indeed a hierarchical BiLSTM-based paragraph encoder (HPE), wherein, the hierarchy comprises the word-level encoder that feeds its encoding to the sentence-level encoder. Further, dynamic paragraph-level contextual information in the BiLSTM-HPE is incorporated via both word-and sentence-level selective attention. However, LSTM is based on the recurrent architecture of RNNs, making the model somewhat rigid and less dynamically sensitive to different parts of the given sequence. Also LSTM models are slower to train. In our case, a paragraph is a sequence of sentences and a sentence is a sequence of words. The Transformer is a recently proposed neural architecture designed to address some deficiencies of RNNs. Specifically, the Transformer is based on the (multi-head) attention mechanism, completely discarding recurrence in RNNs. This design choice allows the Transformer to effectively attend to different parts of a given sequence. Also Transformer is relatively much faster to train and test than RNNs. As humans, when reading a paragraph, we look for important sentences first and then important keywords in those sentences to find a concept around which a question can be generated. Taking this inspiration, we give the same power to our model by incorporating word-level and sentence-level selective attention to generate high-quality questions from paragraphs. In this paper, we present and contrast novel approachs to QG at the level of paragraphs. Our main contributions are as follows: • We present two hierarchical models for encoding the paragraph based on its structure. We analyse the effectiveness of these models for the task of automatic question generation from paragraph. • Specifically, we propose a novel hierarchical Transformer architecture. At the lower level, the encoder first encodes words and produces a sentence-level representation. At the higher level, the encoder aggregates the sentence-level representations and learns a paragraphlevel representation. • We also propose a novel hierarchical BiLSTM model with selective attention, which learns to attend to important sentences and words from the paragraph that are relevant to generate meaningful and fluent questions about the encoded answer. • We also present attention mechanisms for dynamically incorporating contextual information in the hierarchical paragraph encoders and experimentally validate their effectiveness. Question generation (QG) has recently attracted significant interests in the natural language processing (NLP) (; ; ;) and computer vision (CV) ) communities. Given an input (e.g., a passage of text in NLP or an image in CV), optionally also an answer, the task of QG is to generate a natural-language question that is answerable from the input. Existing text-based QG methods can be broadly classified into three categories: (a) rule-based methods, (b) template-base methods, and (c) neural network-based methods. Rule based methods perform syntactic and semantic analysis of sentences and apply fixed sets of rules to generate questions. They mostly rely on syntactic rules written by humans and these rules change from domain to domain. On the other hand, template based methods use generic templates/slot fillers to generate questions. More recently, neural networkbased QG methods (; ;) have been proposed. They employ an RNN-based encoder-decoder architecture and train in an end-to-end fashion, without the need of manually created rules or templates. were the first to propose a sequence-to-sequence (Seq2Seq) architecture for QG. proposed to augment each word with linguistic features and encode the most relevant pivotal answer in the text while generating questions. encode ground-truth answers (given in the training data), use the copy mechanism and additionally employ context matching to capture interactions between the answer and its context within the passage. They encode ground-truth answer for generating questions which might not be available for the test set. recently proposed a Seq2Seq model for paragraph-level question generation, where they employ a maxout pointer mechanism with a gated self-attention encoder. contrast recurrent and non-recurrent architectures on their effectiveness in capturing the hierarchical structure. In Machine Translation, non-recurrent model such as a Transformer that does not use convolution or recurrent connection is often expected to perform better. However, Transformer, as a non-recurrent model, can be more effective than the recurrent model because it has full access to the sequence history. Our findings also suggest that LSTM outperforms the Transformer in capturing the hierarchical structure. In contrast, report settings in which attention-based models, such as BERT are better capable of learning hierarchical structure than LSTM-based models. We propose a general hierarchical architecture for better paragraph representation at the level of words and sentences. This architecture is agnostic to the type of encoder, so we base our hierarchical architectures on BiLSTM and Transformers. We then present two decoders (LSTM and Transformer) with hierarchical attention over the paragraph representation, in order to provide the dynamic context needed by the decoder. The decoder is further conditioned on the provided (candidate) answer to generate relevant questions. Notation: The question generation task consists of pairs (X X X, y y y) conditioned on an encoded answer z, where X X X is a paragraph, and y is the target question which needs to be generated with respect to the paragraph.. Let us denote the i-th sentence in the paragraph by x x x i, where x i,j denotes the j-th word of the sentence. We assume that the first and last words of the sentence are special beginningof-the-sentence < BOS > and end-of-the-sentence < EOS > tokens, respectively. Our hierarchical paragraph encoder (HPE) consists of two encoders, viz., a sentence-level and a word-level encoder; (c.f. Figure 1). The lower-level encoder WORDENC encodes the words of individual sentences. This encoder produces a sentence-dependent word representation r r r i,j for each word x i,j in a sentence x x x i, i.e., r r r i = WORDENC (x x x i). This representation is the output of the last encoder block in the case of Transformer, and the last hidden state in the case of BiLSTM. Furthermore, we can produce a fixed-dimensional representation for a sentence as a function of r i r i r i, e.g., by summing (or averaging) its contextual word representations, or concatenating the contextual representations of its < BOS > and < EOS > tokens. We denote the ing sentence representation bys s s i for a sentence x x x i. Sentence-Level Encoder: At the higher level, our HPE consists of another encoder to produce paragraph-dependent representation for the sentences. The input to this encoder are the sentence representations produced by the lower level encoder, which are insensitive to the paragraph context. In the case of the transformer, the sentence representation is combined with its positional embedding to take the ordering of the paragraph sentences into account. The output of the higher-level encoder is contextual representation for each set of sentences s s s = SENTENC (s s s), where s s s i is the paragraphdependent representation for the i-th sentence. In the following two sub-sections, we present our two hierarchical encoding architectures, viz., the hierarchical BiLSTM in Section 3.2) and hierarchical transformer in Section 3.3). In this first option, c.f., Figure 1, we use both word-level attention and sentence level attention in a Hierarchical BiLSTM encoder to obtain the hierarchical paragraph representation. We employ the attention mechanism proposed in at both the word and sentence levels. We employ the BiLSTM (Bidirectional LSTM) as both, the word as well as the sentence level encoders. We concatenate forward and backward hidden states to obtain sentence/paragraph representations. Subsequently, we employ a unidirectional LSTM unit as our decoder, that generates the target question one word at a time, conditioned on (i) all the words generated in the previous time steps and (ii) on the encoded answer. The methodology employed in these modules has been described next. We use the LSTM decoder's previous hidden state and the word encoder's hidden state to compute attention over words (Figure 1 hidden states of the BiLSTM encoder to obtain the final hidden state representation (h t) at time step t. Representation (h t) is calculated as: h h h t = WORDENC (h t−1, [e e e t, f f f Sentence-level Attention: We feed the sentence representationss s s to our sentence-level BiLSTM encoder (c.f. Figure 1). Similar to the word-level attention, we again the compute attention weight over every sentence in the input passage, using (i) the previous decoder hidden state and (ii) the sentence encoder's hidden state. As before, we concatenate the forward and backward hidden states of the sentence level encoder to obtain the final hidden state representation. The hidden state (g g g t) of the sentence level encoder is computed as:, where f f f s t is the embedded feature vector denoting whether the sentence contains the encoded answer or not. The selective sentence level attention (a a a s t) is computed as: a a a ), where, K is the number of sentences, u The context vector c c c t is fed to the decoder at time step t along with embedded representation of the previous output. In this second option (c.f. Figure 2), we make use of a Transformer decoder to generate the target question, one token at a time, from left to right. For generating the next token, the decoder attends to the previously generated tokens in the question, the encoded answer and the paragraph. We postulate that attention to the paragraph benefits from our hierarchical representation, described in Section 3.1. That is, our model identifies firstly the relevance of the sentences, and then the relevance of the words within the sentences. This in a hierarchical attention module (HATT) and its multi- head extension (MHATT), which replace the attention mechanism to the source in the Transformer decoder. We first explain the sentence and paragragh encoders (Section 3.3.1) before moving on to explanation of the decoder (Section 3.3.2) and the hierarchical attention modules (HATT and MHATT in Section 3.3.3). The sentence encoder transformer maps an input sequence of word representations x = (x 0, · · ·, x n) to a sequence of continuous sentence representations r = (r 0, · · ·, r n). Paragraph encoder takes concatenation of word representation of the start word and end word as input and returns paragraph representation. Each encoder layer is composed of two sub-layers namely a multihead self attention layer (Section 3.3.3) and a position wise fully connected feed forward neural network (Section 3.3.4). To be able to effectively describe these modules, we will benefit first from a description of the decoder (Section 3.3.2). The decoder stack is similar to encoder stack except that it has an additional sub layer (encoderdecoder attention layer) which learn multi-head self attention over the output of the paragraph encoder. The output of the paragraph encoder is transformed into a set of attention vectors K encdec and V encdec. Encoder-decoder attention layer of decoder takes the key K encdec and value V encdec. Decoder stack will output a float vector, we can feed this float vector to a linear followed softmax layer to get probability for generating target word. Let us assume that the question decoder needs to attend to the source paragraph during the generation process. To attend to the hierarchical paragraph representation, we replace the multi-head attention mechanism (to the source) in Transformer by introducing a new multi-head hierarchical attention module M HAT T (q s, K s, q w, K w, V w) where q s is the sentence-level query vector, q w is the word level query vector, K s is the key matrix for the sentences of the paragraph, K w is the key matrix for the words of the paragraph, and V w is the value matrix fr the words of the paragraph. The vectors of sentence-level query q s and word-level query q s are created using non-linear transformations of the state of the decoder h h h t−1, i.e. the input vector to the softmax function when generating the previous word w t−1 of the question. The matrices for the sentence-level key K s and word-level key K w are created using the output. We take the input vector to the softmax function h h h t−1, when the t-th word in the question is being generated. Firstly, this module attends to paragraph sentences using their keys and the sentence query vector: a a a = sof tmax(q s K s /d). Here, d is the dimension of the query/key vectors; the dimension of the ing attention vector would be the number of sentences in the paragraph. Secondly, it computes an attention vector for the words of each sentence: w is the key matrix for the words in the i-th sentences; the dimension of the ing attention vector b b b i is the number of tokens in the i-th sentence. Lastly, the context vector is computed using the word values of their attention weights based on their sentence-level and word-level attentions: Attention in MHATT module is calculated as: Where Attention(Q w, V w, K w) is reformulation of scaled dot product attention of . For multiple heads, the multihead attention z = Multihead(Q w, K w, V w) is calculated as: where z is fed to a position-wise fully connected feed forward neural network to obtain the final input representation. Output of the HATT module is passed to a fully connected feed forward neural net (FFNN) for calculating the hierarchical representation of input (r) as: r = F F N N (x) = (max(0, xW 1 + b1))W 2 + b 2, where r is fed as input to the next encoder layers. The final representation r from last layer of decoder is fed to the linear followed by softmax layer for calculating output probabilities. We performed all our experiments on the publicly available SQuAD and MS MARCO datasets. SQuAD contains 536 Wikipedia articles and more than 100K questions posed about the articles by crowd-workers. We split the SQuAD train set by the ratio 90%-10% into train and dev set and take SQuAD dev set as our test set for evaluation. We take an entire paragraph in each train/test instance as input in all our experiments. MS MARCO contains passages that are retrieved from web documents and the questions are anonimized versions of BING queries. We take a subset of MS MARCO v1.1 dataset containing questions that are answerable from atleast one paragraph. We split train set as 90%-10% into train (71k) and dev (8k) and take dev set as test set (9.5k). Our split is same but our dataset also contains (para, question) tuples whose answers are not a subspan of the paragraph, thus making our task more difficult. For evaluating our question generation model we report the standard metrics, viz., BLEU and ROUGE-L . We performed human evaluation to further analyze quality of questions generated by all the models. We analyzed quality of questions generated on a) syntactic correctness b) semantic correctness and c) relevance to the given paragraph. We compare QG of our hierarchical LSTM and hierarchical Transformer with their flat counterparts. We describe our models below: Seq2Seq + att + AE is the attention-based sequence-to-sequence model with a BiLSTM encoder, answer encoding and an LSTM decoder. HierSeq2Seq + AE is the hierarchical BiLSTM model with a BiLSTM sentence encoder, a BiL-STM paragraph encoder and an LSTM decoder conditioned on encoded answer. TransSeq2Seq + AE is a Transformer-based sequence-to-sequence model with a Transformer encoder followed by a Transformer decoder conditioned on encoded answer. Table 3: Human evaluation (column "Score") as well as inter-rater agreement (column "Kappa") for each model on the SQuAD test set. The scores are between 0-100, 0 being the worst and 100 being the best. Best for each metric (column) are bolded. The three evaluation criteria are: syntactically correct (Syntax), semantically correct (Semantics), and relevant to the text (Relevance). In Table 1 and Table 2 we present automatic evaluation of all models on SQuAD and MS MARCO datasets respectively. We present human evaluation in Table 3 and Table 4 respectively. A number of interesting observations can be made from automatic evaluation in Table 1 and Table 4: Human evaluation (column "Score") as well as inter-rater agreement (column "Kappa") for each model on the MS MARCO test set. The scores are between 0-100, 0 being the worst and 100 being the best. Best for each metric (column) are bolded. The three evaluation criteria are: syntactically correct (Syntax), semantically correct (Semantics), and relevant to the text (Relevance). • Overall, the hierarchical BiLSTM model HierSeq2Seq + AE shows the best performance, achieving best on BLEU2-BLEU4 metrics on both SQuAD dataset, whereas the hierarchical Transformer model TransSeq2Seq + AE performs best on BLEU1 and ROUGE-L on the SQuAD dataset. • Compared to the flat LSTM and Transformer models, their respective hierarchical counterparts always perform better on both the SQuAD and MS MARCO datasets. • On the MS MARCO dataset, we observe the best consistent performance using the hierarchical BiLSTM models on all automatic evaluation metrics. • On the MS MARCO dataset, the two LSTM-based models outperform the two Transformer-based models. Interestingly, human evaluation , as tabulated in Table 3 and Table 4, demonstrate that the hierarchical Transformer model TransSeq2Seq + AE outperforms all the other models on both datasets in both syntactic and semantic correctness. However, the hierarchical BiLSTM model HierSeq2Seq + AE achieves best, and significantly better, relevance scores on both datasets. From the evaluation , we can see that our proposed hierarchical models demonstrate benefits over their respective flat counterparts in a significant way. Thus, for paragraph-level question generation, the hierarchical representation of paragraphs is a worthy pursuit. Moreover, the Transformer architecture shows great potential over the more traditional RNN models such as BiLSTM as shown in human evaluation. Thus the continued investigation of hierarchical Transformer is a promising research avenue. In the Appendix, in Section B, we present several examples that illustrate the effectiveness of our Hierarchical models. In Section C of the appendix, we present some failure cases of our model, along with plausible explanations. We proposed two hierarchical models for the challenging task of question generation from paragraphs, one of which is based on a hierarchical BiLSTM model and the other is a novel hierarchical Transformer architecture. We perform extensive experimental evaluation on the SQuAD and MS MARCO datasets using standard metrics. Results demonstrate the hierarchical representations to be overall much more effective than their flat counterparts. The hierarchical models for both Transformer and BiLSTM clearly outperforms their flat counterparts on all metrics in almost all cases. Further, our experimental validate that hierarchical selective attention benefits the hierarchical BiLSTM model. Qualitatively, our hierarchical models also exhibit better capability of generating fluent and relevant questions. We implement 1 all our models in PyTorch 2. In HierSeq2Seq we used 2-layer BiLSTM sentence and paragraph encoders and a single-layer LSTM decoder. We set RNN hidden size to 600. For TransSeq2Seq and HierTransSeq2Seq we used 4-layer Transformer encoder and decoder with model dimension set to 256 and inner hidden dimension set to 2048. For both the above settings we use a vocabulary of size 45k, shared across source and target of train and dev sets. We prune out all words with frequency lower than 5 in the entire dataset (train and dev). We use pretrained GloVe embeddings of 300 dimensions to represent the words in vocabulary. We use beam search decoder with beam size 5 to decode questions. To train HierSeq2seq model, we use SGD with momentum for optimization. We set learning rate to 0.1 at the beginning, with the value being halved at every even epochs starting from epoch 8. We train our models for 20 epochs and select the model with lowest perplexity as the final model. we train HierTransSeq2seq using Adam optimizer, we set initial learning rate to 1e-7 and warmup steps to 2000. We use standard inverse square root scheduler for scheduling the learning rate. B EXAMPLE QUESTIONS GENERATED BY OUR BEST MODEL (HIERSEQ2SEQ + AE) ON MS MARCO TEST SET

represent the tldr for finding the corresponding scientific article

Automatic question generation from paragraph using hierarchical models

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the article for finding a single-sentence TLDR

Question: Many deployed learned models are black boxes: given input, returns output. Internal information about the model, such as the architecture, optimisation procedure, or training data, is not disclosed explicitly as it might contain proprietary information or make the system more vulnerable. This work shows that such attributes of neural networks can be exposed from a sequence of queries. This has multiple implications. On the one hand, our work exposes the vulnerability of black-box neural networks to different types of attacks -- we show that the revealed internal information helps generate more effective adversarial examples against the black box model. On the other hand, this technique can be used for better protection of private content from automatic recognition models using adversarial examples. Our paper suggests that it is actually hard to draw a line between white box and black box models. Black-box models take a sequence of query inputs, and return corresponding outputs, while keeping internal states such as model architecture hidden. They are deployed as black boxes usually on purpose -for protecting intellectual properties or privacy-sensitive training data. Our work aims at inferring information about the internals of black box models -ultimately turning them into white box models. Such a reverse-engineering of a black box model has many implications. On the one hand, it has legal implications to intellectual properties (IP) involving neural networks -internal information about the model can be proprietary and a key IP, and the training data may be privacy sensitive. Disclosing hidden details may also render the model more susceptible to attacks from adversaries. On the other hand, gaining information about a black-box model can be useful in other scenarios. E.g. there has been work on utilising adversarial examples for protecting private regions (e.g. faces) in photographs from automatic recognisers BID12. In such scenarios, gaining more knowledge on the recognisers will increase the chance of protecting one's privacy. Either way, it is a crucial research topic to investigate the type and amount of information that can be gained from a black-box access to a model. We make a first step towards understanding the connection between white box and black box approaches -which were previously thought of as distinct classes. We introduce the term "model attributes" to refer to various types of information about a trained neural network model. We group them into three types: architecture (e.g. type of non-linear activation), optimisation process (e.g. SGD or ADAM?), and training data (e.g. which dataset?). We approach the problem as a standard supervised learning task applied over models. First, collect a diverse set of white-box models ("meta-training set") that are expected to be similar to the target black box at least to a certain extent. Then, over the collected meta-training set, train another model ("metamodel") that takes a model as input and returns the corresponding model attributes as output. Importantly, since we want to predict attributes at test time for black-box models, the only information available for attribute prediction is the query input-output pairs. As we will see in the experiments, such input-output pairs allow to predict model attributes surprisingly well. In summary, we contribute: Investigation of the type and amount of internal information about the black-box model that can be extracted from querying; Novel metamodel methods that not only reason over outputs from static query inputs, but also actively optimise query inputs that can extract more information; Study of factors like size of the meta-training set, quantity and quality of queries, and the dissimilarity between the meta-training models and the test black box (generalisability); Empirical verification that revealed information leads to greater susceptibility of a black-box model to an adversarial example based attack. There has been a line of work on extracting and exploiting information from black-box learned models. We first describe papers on extracting information (model extraction and membership inference attacks), and then discuss ones on attacking the network using the extracted information (adversarial image perturbations (AIP)).Model extraction attacks either reconstruct the exact model parameters or build an avatar model that maximises the likelihood of the query input-output pairs from the target model BID19 BID15. BID19 have shown the efficacy of equation solving attacks and the avatar method in retrieving internal parameters of non-neural network models. BID15 have also used the avatar approach with the end goal of generating adversarial examples. While the avatar approach first assumes model hyperparameters like model family (architecture) and training data, we discriminatively train a metamodel to predict those hyperparameters themselves. As such, our approach is complementary to the avatar approach. Membership inference attacks determine if a given data sample has been included in the training data BID0 BID16. In particular, BID0 also trains a decision tree metamodel over a set of classifiers trained on different datasets. This work goes far beyond only inferring the training data by showing that even the model architecture and optimisation process can be inferred. Using the obtained cues, one can launch more effective, focused attacks on the black box. We use adversarial image perturbations (AIPs) as an example of such attack. AIPs are small perturbations over the input such that the network is mislead. Research on this topic has flourished recently after it was shown that the needed amount of perturbation to completely mislead an image classifier is nearly invisible BID18 BID2 BID10.Most effective AIPs require gradients of the target network. Some papers proposed different ways to attack black boxes. They can be grouped into three approaches. Approximate gradients by numerical gradients BID11. The caveat is that thousands and millions of queries are needed to compute a single AIP, depending on the image size. Use the avatar approach to train a white box network that is supposedly similar to the target BID14 a; BID3. We note again that our metamodel is complementary to the avatar approach -the avatar network hyperparemters can be determined by the metamodel. Exploit transferability of adversarial examples; it has been shown that AIPs generated against one network can also fool other networks BID10. in particular have shown that generating AIPs against an ensemble of networks make it more transferable. We show in this work that the AIPs transfer better within an architecture family (e.g. ResNet or DenseNet) than across, and that such a property can be exploited by our metamodel for generating more targetted AIPs. Figure 1: Overview of our approach. We want to find out the type and amount of internal information about a black-box model that can be revealed from a sequence of queries. We approach this by first building metamodels for predicting model attributes, and then evaluating their performance on black-box models. Our main approach, metamodel, is described in figure 1. In a nutshell, the metamodel is a classifier of classifiers. Specifically, The metamodel submits n query inputs DISPLAYFORM0 to a black box model f; the metamodel takes corresponding model outputs f (DISPLAYFORM1 as an input, and returns predicted model attributes as output. As we will describe in detail, the metamodel not only learns to infer model attributes from query outputs from a static set of inputs, but also searches for query inputs that are designed to extract greater amount of information from the target models. In this section, our main methods are introduced in the context of MNIST digit classifiers. While MNIST classifiers are not fully representative of generic learned models, they have a computational edge: it takes only five minutes to train each of them with reasonable performance. We could thus prepare a diverse set of 11k MNIST classifiers within 40 GPU days for the meta-training and evaluation of our metamodels. We stress, however, that the proposed approach is generic with respect to the task, data, and the type of models. We also focus on 12 model attributes (table 1) that cover hyperparameters for common neural network MNIST classifiers, but again the range of predictable attributes are not confined to this list. We need a dataset of classifiers to train and evaluate metamodels. We explain how MNIST-NETS has been constructed, a dataset of 11k MNIST digit classifiers; the procedure is task and data generic. Every model in MNIST-NETS shares the same convnet skeleton architecture: "N conv blocks → M fc blocks → 1 linear classifier". Each conv block has the following structure: "ks × ks convolution → optional 2 × 2 max-pooling → non-linear activation", where ks (kernel size) and the activation type are to be chosen. Each fc block has the structure: " linear mapping → non-linear activation → optional dropout" This convnet structure already covers many LeNet BID8 In order to learn generalisable features, the metamodel needs to be trained over a diverse set of models. The base architecture described above already has several free parameters like the number of layers (N and M), the existence of dropout or maxpooling layers, or the type of nonlinear activation. Apart from the architectural hyperparameters, we increase diversity along two more axes -optimisation process and the training data. Along the optimisation axis, we vary optimisation algorithm (SGD, ADAM, or RMSprop) and the training batch size. We also consider training MNIST classifiers on either on the entire MNIST training set (All 0, 60k), one of the two disjoint halves (Half 0/1, 30k), or one of the four disjoint quarters (Quarter 0/1/2/3, 15k).See table 1 for the comprehensive list of 12 model attributes altered in MNIST-NETS. The number of trainable parameters (#par) and the training data size (size) are not directly controlled but derived from the other attributes. We also augment MNIST-NETS with ensembles of classifiers (ens), whose procedure will be described later. The number of all possible combinations of controllable options in table 1 is 18, 144. We also select random seeds that control the initialisation and training data shuffling from {0, · · ·, 999}, ing in 18, 144, 000 unique models. Training such a large number of models is intractable; we have sampled (without replacement) and trained 10, 000 of them. All the models have been trained with learning rate 0.1 and momentum 0.5 for 100 epochs. It takes around 5 minutes to train each model on a GPU machine (GeForce GTX TITAN); training of 10k classifiers has taken 40 GPU days. In order to make sure that MNIST-NETS realistically represents commonly used MNIST classifiers, we have pruned low-performance classifiers (validation accuracy< 98%), ing in 8, 582 classifiers. Ensembles of trained classifiers have been constructed by grouping the identical classifiers (modulo random seed). Given t identical ones, we have augmented MNIST-NETS with 2, · · ·, t combinations. The ensemble augmentation has ed in 11, 282 final models. See appendix table 6 for statistics of attributes -due to large sample size all the attributes are evenly covered. Attribute prediction can get arbitrarily easy by including the black-box model (or similar ones) in the meta-training set. We introduce multiple splits of MNIST-NETS with varying requirements on generalization. Unless stated otherwise, every split has 5, 000 training (meta-training), 1, 000 testing (black box), and 5, 282 leftover models. The Random (R) split randomly (uniform weights) assigns training and test splits, respectively. Under the R split, the training and test models come from the same distribution. We introduce harder Extrapolation (E) splits. We separate a few attributes between the training and test splits. They are designed to simulate more difficult domain gaps when the meta-training models are significantly different from the black box. Specific examples of E splits will be shown in §4. The metamodel predicts the attribute of a black-box model g in the test split by submitting n query inputs and observing the outputs. It is trained over meta-training models f in the training split (f ∼ F). We propose three approaches for the metamodels -we collectively name them kennen 2. See figure 2 for an overview. kennen-o first selects a fixed set of queries [x i] i=1···n from a dataset. Both during training and testing, always these queries are submitted. kennen-o learns a classifier m θ to map from the order-sensitively concatenated n query outputs, [f (x i)] i=1···n (n × 10 dim for MNIST), to the simultaneous prediction of 12 attributes in f. The training objective is: DISPLAYFORM0 where F is the distribution of meta-training models, y a is the ground truth label of attribute a, and L is the cross-entropy loss. With the learned parameterθ, m DISPLAYFORM1 gives the prediction of attribute a for the black box g. In our experiments, we model the classifier m θ via multilayer perceptron (MLP) with two hidden layers with 1000 hidden units. The last layer consists of 12 parallel linear layers for a simultaneous prediction of the attributes. In our preliminary experiments, MLP has performed better than the linear classifiers. The optimisation problem in equation 1 is solved via SGD by approximating the expectation over f ∼ F by an empirical sum over the training split classifiers for 200 epochs. For query inputs, we have used a random subset of n images from the validation set (both for MNIST and ImageNet experiments). The performance is not sensitive to the choice of queries (see appendix §C). Next methods (kennen-i/io) describe how to actively craft query inputs, potentially outside the natural image distribution. Note that kennen-o can be applied to any type of model (e.g. non-neural networks) with any output structure, as long as the output can be embedded in an Euclidean space. We will show that this method can effectively extract information from f even if the output is a top-k ranking. DISPLAYFORM2 kennen-i crafts a single query inputx over the meta-training models that is trained to repurpose a digit classifier f into a model attribute classifier for a single attribute a. The crafted input drives the classifier to leak internal information via digit prediction. The learned input is submitted to the test black-box model g, and the attribute is predicted by reading off its digit prediction g(x). For example, kennen-i for max-pooling layer prediction crafts an input x that is predicted as "1" for generic MNIST digit classifiers with max-pooling layers and "0" for ones without. See figure 3 for visual examples. drop pool ks 77.0%94.8% 88.5%Figure 3: Inputs designed to extract internal details from MNIST digit classifiers. E.g. feeding the middle image reveals the existence of a maxpooling layer with 94.8% chance. We describe in detail how kennen-i learns this input. The training objective is: DISPLAYFORM3 where f (x) is the 10-dimensional output of the digit classifier f. The condition x: image ensures the input stays a valid image DISPLAYFORM4 The loss L, together with the attribute label y a of f, guides the digit prediction f (x) to reveal the attribute a instead. Note that the optimisation problem is identical to the training of digit classifiers except that the ground truth is the attribute label rather than the digit label, that the loss is averaged over the models instead of the images, and that the input x instead of the model f is optimised. With the learned query inputx, the attribute for the black box g is predicted by g(x). In particular, we do not use gradient information from g. We initialise x with a random sample from the MNIST validation set (random noise or uniform gray initialisation gives similar performances), and run SGD for 200 epochs. For each iteration x is truncated back to D to enforce the constraint. While being simple and effective, kennen-i can only predict a single attribute at a time, and cannot predict attributes with more than 10 classes (for digit classifiers). kennen-io introduced below overcomes these limitations. kennen-i may also be unrealistic when the exploration needs to be stealthy: it submits unnatural images to the system. Also unlike kennen-o, kennen-i requires end-to-end differentiability of the training models f ∼ F, although it still requires only black-box access to test models g. DISPLAYFORM5 We overcome the drawbacks of kennen-i that it can only predict one attribute at a time and that the number of predictable classes by attaching an additional interpretation module on top of the output. Our final method kennen-io combines kennen-i and kennen-o approaches: both input generator and output interpreters are used. Being able to reason over multiple query outputs via MLP layers, kennen-io supports the optimisation of multiple query inputs as well. Specifically, the kennen-io training objective is given by: DISPLAYFORM6 Note that the formulation is identical to that for kennen-o (equation 1), except that the second minimisation problem regarding the query inputs is added. With learned parametersθ and the attribute a for the black box g is predicted by m DISPLAYFORM7 DISPLAYFORM8. Again, we require end-to-end differentiability of meta-training models f, but only the black-box access for the test model g. To improve stability against covariate shift, we initialise m θ with kennen-o for 200 epochs. Afterwards, gradient updates of DISPLAYFORM9 and θ alternate every 50 epochs, for 200 additional epochs. We have introduced a procedure for constructing a dataset of classifiers (MNIST-NETS) as well as novel metamodels (kennen variants) that learn to extract information from black-box classifiers. In this section, we evaluate the ability of kennen to extract information from black-box MNIST digit classifiers. We measure the class-balanced attribute prediction accuracy for each attribute a in the list of 12 attributes in table 1. See table 2 for the main of our metamodels, kennen-o/i/io, on the Random split. Unless stated otherwise, metamodels are trained with 5, 000 training split classifiers. Given n = 100 queries with probability output, kennen-o already performs far above the random chance in predicting 12 diverse attributes (73.4% versus 34.9% on average); neural network output indeed contains rich information about the black box. In particular, the presence of dropout (94.6%) or max-pooling (94.9%) has been predicted with high precision. As we will see in §4.3, outputs of networks trained with dropout layers form clusters, explaining the good prediction performance. It is surprising that optimisation details like algorithm (71.8%) and batch size (50.4%) can also be predicted well above the random chance (33.3% for both). We observe that the training data attributes are also predicted with high accuracy (71.8% and 90.0% for size and split). TAB1 shows the comparison of kennen-o/i/io. kennen-i has a relatively low performance (average 52.7%), but kennen-i relies on a cheap resource: 1 query with single-label output. kennen-i is also performant at predicting the kernel size (88.5%) and pooling (94.8%), attributes that are closely linked to spatial structure of the input. We conjecture kennen-i is relatively effective for such attributes. kennen-io is superior to kennen-o/i for all the attributes with average accuracy 80.1%. DISPLAYFORM0 We examine potential factors that contribute to the successful prediction of black box internal attributes. We measure the prediction accuracy of our metamodels as we vary the number of meta-training models, the number of queries, and the quality of query output. Figure 4: kennen-o performance of against the size of meta-training set (left), number of queries (middle), and quality of queries (right). Unless stated otherwise, we use 100 probability outputs and 5k models to train kennen-o. Each curve is linearly scaled such that random chance (0 training data, 0 query, or top-0) performs 0%, and the perfect predictor performs 100%. We have trained kennen-o with different number of the meta-training classifiers, ranging from 100 to 5, 000. See figure 4 (left) for the trend. We observe a diminishing return, but also that the performance has not saturated -collecting larger meta-training set will improve the performance. See figure 4 (middle) for the kennen-o performance against the number of queries with probability output. The average performance saturates after ∼ 500 queries. On the other hand, with only ∼ 100 queries, we already retrieve ample information about the neural network. Many black-box models return top-k ranking (e.g. Facebook face recogniser), or single-label output. We represent top-k ranking outputs by assigning exponentially decaying probabilities up to k digits and a small probability to the remaining. See table 2 for the kennen-o performance comparison among 100 probability, top-10 ranking, bottom-1, and top-1 outputs, with average accuracies 73.4%, 69.7%, 54.4%, and 39.5%, respectively. While performance drops with coarser outputs, when compared to random chance (34.9%), 100 single-label bottom-1 outputs already leak a great amount of information about the black box (54.4%). It is also notable that bottom-1 outputs contain much more information than do the top-1 outputs; note that for high-performance classifiers top-1 predictions are rather uniform across models and thus have much less freedom to leak auxiliary information. Figure 4 (right) shows the interpolation from top-1 to top-10 (i.e. top-9) ranking. We observe from the jump at k = 2 that the second likely predictions (top-2) contain far more information than the most likely ones (top-1). For k ≥ 3, each additional output label exhibits a diminishing return. So far we have seen on the Random (R) split. In realistic scenarios, the meta-training model distribution may not be fully covering possible black box models. We show how damaging such a scenario is through Extrapolation (E) split experiments. EVALUATION E-splits split the training and testing models based on one or more attributes (§3.1). For example, we may assign shallower models (#layers ≤ 10) to the training split and deeper ones (#layers ¿10) to the testing split. In this example, we refer to #layers as the splitting attribute. Since for an E-split, some classes of the splitting attributes have zero training examples, we only evaluate the prediction accuracies over the non-splitting attributes. When the set of splitting attributes isÃ, a subset of the entire attribute set A, we define E-split accuracy or E.Acc(Ã) to be the mean prediction accuracy over the non-splitting attributes A \Ã. For easier comparison, we report the normalised accuracy (N.Acc) that shows the how much percentage of the R-split accuracy is achieved in the E-split setup on the non-splitting attributes A \Ã. Specifically: DISPLAYFORM0 where R.Acc(Ã) and Chance(Ã) are the means of the R-split and Chance-level accuracies over A \Ã. Note that N.Acc is 100% if the E-split performance is at the level of R-split and 0% if it is at chance level. Table 3: Normalised accuracies (see text) of kennen-o and kennen-io on R and E splits. We denote E-split with splitting attributes attr1 and attr2 as "E-attr1-attr2". Splitting criteria are also shown. When there are two splitting attributes, the first attribute inherits the previous row criteria. The normalised accuracies for R-split and multiple E-splits are presented in table 3. We consider three axes of choices of splitting attributes for the E-split: architecture (#conv and #fc), optimisation (alg and bs), and data (size). For example, "E-#conv-#fc" row presents when metamodel is trained on shallower nets (2 or 3 conv/fc layers each) compared to the test black box model (4 conv and fc layers each).Not surprisingly, E-split performances are lower than R-split ones (N.Acc < 100%); it is advisable to cover all the expected black-box attributes during meta-training. Nonetheless, E-split performances of kennen-io are still far above the chance level (N.Acc ≥ 70% 0%); failing to cover a few attributes during meta-training is not too damaging. Comparing kennen-o and kennen-io for their generalisability, we observe that kennen-io consistently outperforms kennen-o under severe extrapolation (around 5 pp better N.Acc). It is left as a future work to investigate the intriguing fact that utilising out-of-domain query inputs improves the generalisation of metamodel. It is surprising that metamodels can extract inner details with great precision and generalisability. This section provides a glimpse of why and how this is possible via metamodel input and output analyses. Full answers to those questions is beyond the scope of the paper. We analyse the inputs to our metamodels (i.e. query outputs from black-box models) to convince ourselves that the inputs do contain discriminative features for model attributes. As the input is high dimensional (1000 when the number of queries is n = 100), we use the t-SNE (van der Maaten & Hinton, Nov 2008) visualisation method. Roughly speaking, t-SNE embeds high dimensional data points onto the 2-dimensional plane such that the pairwise distances are best respected. We then colour-code the embedded data points according to the model attributes. Clusters of same-coloured points indicate highly discriminative features. The visualisation of input data points are shown in Appendix figures 9 and 10 for kennen-o and kennen-io, respectively. For experimental details, see Appendix §D. In the case of kennen-o, we observe that some attributes form clear clusters in the input space -e.g. Tanh in act, binary dropout attribute, and RMSprop in alg. For the other attributes, however, it seems that the clusters are too complicated to be represented in a 2-dimensional space. For kennen-io (figure 10), we observe improved clusters for pool and ks. By submitting crafted query inputs, kennen-io induces query outputs to be better clustered, increasing the chance of successful prediction. We show confusion matrices of kennen-o/io to analyse the failure modes. See Appendix figures 11 and 12. For kennen-o and kennen-io alike, we observe that the confusion occurs more frequently with similar classes. For attributes #conv and #fc, more confusion occurs between or than between. A similar trend is observed for #par and bs. This is a strong indication that there exists semantic attribute information in the neural network outputs (e.g. number of layers, parameters, or size of training batch) and the metamodels learn semantic information that can generalise, as opposed to merely relying on artifacts. This observation agrees with a of the extrapolation experiments in §4.2: the metamodels generalise. Compared to those of kennen-o, kennen-io confusion matrices exhibit greater concentration of masses both on the correct class (diagonals) and among similar attribute classes (1-off diagonals for #conv, #fc, #par, bs, and size). The former re-confirms the greater accuracy, while the latter indicates the improved ability to extract more semantic and generalisable features from the query outputs. This, again, agrees with §4.2: kennen-io generalises better than kennen-o. We have verified through our novel kennen metamodels that black-box access to a neural network exposes much internal information. We have shown that only 100 single-label outputs already reveals a great deal about a black box. When the black-box classifier is quite different from the metatraining classifiers, the performance of our best metamodel -kennen-io-decreases; however, the prediction accuracy for black box internal information is still surprisingly high. While MNIST experiments are computationally cheap and a massive number of controlled experiments is possible, we provide additional ImageNet experiments for practical implications on realistic image classifiers. In this section, we use kennen-o introduced in §3 to predict a single attribute of black-box ImageNet classifiers -the architecture family (e.g. ResNet or VGG?). In this section, we go a step further to use the extracted information to attack black boxes with adversarial examples. It is computationally prohibitive to train O(10k) ImageNet classifiers from scratch as in the previous section. We have resorted to 19 PyTorch 3 pretrained ImageNet classifiers. The 19 classifiers come from five families: Squeezenet, VGG, VGG-BatchNorm, ResNet, and DenseNet, each with 2, 4, 4, 5, and 4 variants, respectively BID6 BID17 BID7 BID4 BID5. See Appendix table 7 for the the summary of the 19 classifiers. We observe both large intra-family diversity and small inter-family separability in terms of #layers, #parameters, and performances. The family prediction task is not as trivial as e.g. simply inferring the performance. We predict the classifier family (S, V, B, R, D) from the black-box query output, using the method kennen-o, with the same MLP architecture (§3). kennen-i and kennen-io have not been used for computational reasons, but can also be used in principle. We conduct 10 cross validations (random sampling of single test network from each family) for evaluation. We also perform 10 random sampling of the queries from ImageNet validation set. In total 100 random tries are averaged. Results: compared to the random chance (20.0%), 100 queries in high kennen-o performance (90.4%). With 1, 000 queries, the prediction performance is even 94.8%. In this section we attack ImageNet classifiers with adversarial image perturbations (AIPs). We show that the knowledge about the black box architecture family makes the attack more effective. AIPs are carefully crafted additive perturbations on the input image for the purpose of misleading the target model to predict wrong labels BID2. Among variants of AIPs, we use efficient and robust GAMAN BID12. See appendix figure 7 for examples of AIPs; the perturbation is nearly invisible. Typical AIP algorithms require gradients from the target network, which is not available for a black box. Mainly three approaches for generating AIPs against black boxes have been proposed: numerical gradient, avatar network, or transferability. We show that our metamodel strengthens the transferability based attack. We hypothesize and empirically show that AIPs transfer better within the architecture family than across. Using this property, we first predict the family of the black box (e.g. ResNet), and then generate AIPs against a few instances in the family (e.g. ResNet101, ResNet152). The generation of AIPs against multiple targets has been proposed by, but we are the first to systemically show that AIPs generalise better within a family when they are generated against multiple instances from the same family. We first verify our hypothesis that AIPs transfer better within a family. Within-family: we do a leave-one-out cross validation -generate AIPs using all but one instances of the family and test on the holdout. Not using the exact test black box, this gives a lower bound on the within-family performance. Across-family: still leave out one random instance from the generating family to match the generating set size with the within-family cases. We also include the use-all case (Ens): generate AIPs with one network from each family. See table 4 for the . We report the misclassification rate, defined as 100−top-1 accuracy, on 100 random ImageNet validation images. We observe that the within-family performances dominate the across-family ones (diagonal entries versus the others in each row); if the target black box family is identified, one can generate more effective AIPs. Finally, trying to target all network ("Ens") is not as effective as focusing resources (diagonal entries). We empirically show that the reverse-engineering enables more effective attacks. We consider multiple scenarios. " White box" means the target model is fully known, and the AIP is generated specifically for this model. " Black box" means the exact target is unknown, but we make a distinction when the family is known ("Family black box"). See table 5 for the misclassification rates (MC) in different scenarios. When the target is fully specified (white box), MC is 100%. When neither the exact target nor the family is known, AIPs are generated against multiple families (82.2%). When the reverse-engineering takes place, and AIPs are generated over the predicted family, attacks become more effective (85.7%). We almost reach the family-oracle case (86.2%). Our metamodel can predict architecture families for ImageNet classifiers with high accuracy. We additionally show that this reverse-engineering enables more focused attack on black-boxes. We have presented first on the inference of diverse neural network attributes from a sequence of input-output queries. Our novel metamodel methods, kennen, can successfully predict attributes related not only to the architecture but also to training hyperparameters (optimisation algorithm and dataset) even in difficult scenarios (e.g. single-label output, or a distribution gap between the metatraining models and the target black box). We have additionally shown in ImageNet experiments that reverse-engineering a black box makes it more vulnerable to adversarial examples. We show the statistics of MNIST-NETS, our dataset of MNIST classifiers, in table 6. We complement the kennen-o in the main paper (figure 4) with kennen-io . See figure 5. Similarly for kennen-o, kennen-io shows a diminishing return as the number of training models and the number of queries increase. While the performance saturates with 1, 000 queries, it does not fully saturate with 5, 000 training samples. kennen-o selects a random set of queries from MNIST validation set (§3.2). We measure the sensitivity of kennen-o performance with respect to the choice of queries, and discuss the possibility to optimise the set of queries. With 1, 10, or 100 queries, we have trained kennen-o with 100 independent samples of query sets. The mean and standard deviations are shown in figure 6. The sensitivity is greater for smaller number of queries, but still minute (1.2 pp standard deviation).Instead of solving the combinatorial problem of finding the optimal set of query inputs from a dataset, we have proposed kennen-io that efficiently solves a continuous optimisation problem to find a set of query inputs from the entire input space. We have compared kennen-io against kennen-o with multiple query samples in figure 6. We observe that kennen-io is better than kennen-o with all 100 query set samples at each level. We remark that there exists a trade-off between detectability and effectiveness of exploration. While kennen-io extracts information from target model more effectively, it increases the detectability of attack by submitting out-of-domain inputs. If it is possible to optimise or sample the set of natural queries from a dataset or distribution of natural inputs, it will be a strong attack; developing such a method would be an interesting future work. We describe the detailed procedure for the metamodel input visualisation experiment (discussed in §4.3). First, 1000 test-split (Random split) black-box models are collected. For each model, 100 query images are passed (sampled at random from MNIST validation set), ing in 100 × 10 dimensional input data points. We have used t-SNE(van der Maaten & Hinton, Nov 2008) to embed the data points onto the 2-dimensional plane. Each data point is coloured according to each attribute class. The for kennen-o and kennen-io are shown in figures 9 and 10. Since t-SNE is sensitive to initialisation, we have run the embedding ten times with different random initialisations; the qualitative observations are largely identical. In this section, we show examples of AIPs. See figure 7 for the examples of AIPs and the perturbed images. The perturbation is nearly invisible to human eyes. We have also generated AIPs with respect to a diverse set of architecture families (S, V, B, R, D, SVBRD) at multiple L 2 norm levels. See figure 8; the same image in a diverse set of patterns depending on the architecture family.

Represent the short summary for finding the corresponding scientific article (long)

Querying a black-box neural network reveals a lot of information about it; we propose novel "metamodels" for effectively extracting information from a black box.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.