Question: represent the text to find a very short summary (1-2 sentences max)

Question: We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks. Linguistic theories generally regard natural language as consisting of two part: a lexicon, the complete set of all possible words in a language; and a syntax, the set of rules, principles, and processes that govern the structure of sentences BID46. To generate a proper sentence, tokens are put together with a specific syntactic structure. Understanding a sentence also requires lexical information to provide meanings, and syntactical knowledge to correctly combine meanings. Current neural language models can provide meaningful word represent BID0 BID41. However, standard recurrent neural networks only implicitly model syntax, thus fail to efficiently use structure information BID53.Developing a deep neural network that can leverage syntactic knowledge to form a better semantic representation has received a great deal of attention in recent years BID50 BID53 BID11. Integrating syntactic structure into a language model is important for different reasons: 1) to obtain a hierarchical representation with increasing levels of abstraction, which is a key feature of deep neural networks and of the human brain BID1 BID31 BID47; 2) to capture complex linguistic phenomena, like long-term dependency problem BID53 and the compositional effects BID50; 3) to provide shortcut for gradient back-propagation BID11.A syntactic parser is the most common source for structure information. Supervised parsers can achieve very high performance on well constructed sentences. Hence, parsers can provide accurate information about how to compose word semantics into sentence semantics BID50, or how to generate the next word given previous words BID56. However, only major languages have treebank data for training parsers, and it request expensive human expert annotation. People also tend to break language rules in many circumstances (such as writing a tweet). These defects limit the generalization capability of supervised parsers. Unsupervised syntactic structure induction has been among the longstanding challenges of computational linguistic BID23 BID25 BID2. Researchers are interested in this problem for a variety of reasons: to be able to parse languages for which no annotated treebanks exist BID35; to create a dependency structure to better suit a particular NLP application BID56; to empirically argue for or against the poverty of the stimulus BID12 BID10; and to examine cognitive issues in language learning BID51.In this paper, we propose a novel neural language model: Parsing-Reading-Predict Networks (PRPN), which can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to form a better language model. With our model, we assume that language can be naturally represented as a tree-structured graph. The model is composed of three parts:1. A differentiable neural Parsing Network uses a convolutional neural network to compute the syntactic distance, which represents the syntactic relationships between all successive pairs of words in a sentence, and then makes soft constituent decisions based on the syntactic distance.2. A Reading Network that recurrently computes an adaptive memory representation to summarize information relevant to the current time step, based on all previous memories that are syntactically and directly related to the current token.3. A Predict Network that predicts the next token based on all memories that are syntactically and directly related to the next token. We evaluate our model on three tasks: word-level language modeling, character-level language modeling, and unsupervised constituency parsing. The proposed model achieves (or is close to) the state-of-the-art on both word-level and character-level language modeling. The model's unsupervised parsing outperforms some strong baseline models, demonstrating that the structure found by our model is similar to the intrinsic structure provided by human experts. The idea of introducing some structures, especially trees, into language understanding to help a downstream task has been explored in various ways. For example, BID50; BID53 learn a bottom-up encoder, taking as an input a parse tree supplied from an external parser. There are models that are able to infer a tree during test time, while still need supervised signal on tree structure during training. For example, BID49; BID59, etc. Moreover, BID55 did an in-depth analysis of recursive models that are able to learn tree structure without being exposed to any grammar trees. Our model is also able to infer tree structure in an unsupervised setting, but different from theirs, it is a recurrent network that implicitly models tree structure through attention. Apart from the approach of using recursive networks to capture structures, there is another line of research which try to learn recurrent features at multiple scales, which can be dated back to 1990s (e.g. El BID15 BID48 ; BID32). The NARX RNN BID32 is another example which used a feed forward net taking different inputs with predefined time delays to model long-term dependencies. More recently, BID27 also used multiple layers of recurrent networks with different pre-defined updating frequencies. Instead, our model tries to learn the structure from data, rather than predefining it. In that respect, BID11 relates to our model since it proposes a hierarchical multi-scale structure with binary gates controlling intralayer connections, and the gating mechanism is learned from data too. The difference is that their gating mechanism controls the updates of higher layers directly, while ours control it softly through an attention mechanism. In terms of language modeling, syntactic language modeling can be dated back to BID7. BID6; BID44 have also proposed language models with a top-down parsing mechanism. Recently; have introduced neural networks into this space. It learns both a discriminative and a generative model with top-down parsing, trained with a supervision signal from parsed sentences in the corpus. There are also dependency-based approaches using neural networks, including BID4; BID16; BID54.Parsers are also related to our work since they are all inferring grammatical tree structure given a sentence. For example, SPINN is a shift-reduce parser that uses an LSTM as its composition function. The transition classifier in SPINN is supervisedly trained on the Stanford PCFG Parser BID24 output. Unsupervised parsers are more aligned with what our model is doing. BID25 presented a generative model for the unsupervised learning of dependency structures. BID23 is a generative distributional model for the unsupervised induction of natural language syntax which explicitly models constituent yields and contexts. We compare our parsing quality with the aforementioned two papers in Section 6.3. Figure 1: Hard arrow represents syntactic tree structure and parent-to-child dependency relation, dash arrow represents dependency relation between siblings Suppose we have a sequence of tokens x 0,..., x 6 governed by the tree structure showed in FIG4.The leafs x i are observed tokens. Node y i represents the meaning of the constituent formed by its leaves x l(yi),..., x r(yi), where l(·) and r(·) stands for the leftmost child and right most child. Root r represents the meaning of the whole sequence. Arrows represent the dependency relations between nodes. The underlying assumption is that each node depends only on its parent and its left siblings. Directly modeling the tree structure is a challenging task, usually requiring supervision to learn BID53. In addition, relying on tree structures can in a model that is not sufficiently robust to face ungrammatical sentences BID19. In contrast, recurrent models provide a convenient way to model sequential data, with the current hidden state only depends on the last hidden state. This makes models more robust when facing nonconforming sequential data, but it suffers from neglecting the real dependency relation that dominates the structure of natural language sentences. In this paper, we use skip-connection to integrate structured dependency relations with recurrent neural network. In other words, the current hidden state does not only depend on the last hidden state, but also on previous hidden states that have a direct syntactic relation to the current one. FIG0 shows the structure of our model. The non-leaf node y j is represented by a set of hidden states y j = {m i} l(yj)≤i≤r(yj), where l(y j) is the left most descendant leaf and r(y j) is the right most one. Arrows shows skip connections built by our model according to the latent structure. Skip connections are controlled by gates g t i. In order to define g t i, we introduce a latent variable l t to represent local structural context of x t:• if x t is not left most child of any subtree, then l t is the position of x t's left most sibling.• if x t is the left most child of a subtree y i, then l t is the position of the left most child that belongs to the left most sibling of y i.and gates are defined as: DISPLAYFORM0 Given this architecture, the siblings dependency relation is modeled by at least one skip-connect. The skip connection will directly feed information forward, and pass gradient backward. The parentto-child relation will be implicitly modeled by skip-connect relation between nodes. The model recurrently updates the hidden states according to: DISPLAYFORM1 and the probability distribution for next word is approximated by: DISPLAYFORM2 where g t i are gates that control skip-connections. Both f and h have a structured attention mechanism that takes g t i as input and forces the model to focus on the most related information. Since l t is an unobserved latent variable, We explain an approximation for g t i in the next section. The structured attention mechanism is explained in section 5.1. In this section we give a probabilistic view on how to model the local structure of language. A detailed elaboration for this section is given in Appendix B. At time step t, p(l t |x 0, ..., x t) represents the probability of choosing one out of t possible local structures. We propose to model the distribution by the Stick-Breaking Process: DISPLAYFORM0 The formula can be understood by noting that after the time step i+1,..., t−1 have their probabilities assigned, t−1 j=i+1 α t j is remaining probability, 1 − α t i is the portion of remaining probability that we assign to time step i. Variable α t j is parametrized in the next section. As shown in Appendix B, the expectation of gate value g t i is the Cumulative Distribution Function (CDF) of p(l t = i|x 0, ..., x t). Thus, we can replace the discrete gate value by its expectation: DISPLAYFORM1 With these relaxations, Eq.2 and 3 can be approximated by using a soft gating vector to update the hidden state and predict the next token. Inferring tree structure with Syntactic Distance In Eq.4, 1 − α t j is the portion of the remaining probability that we assign to position j. Because the stick-breaking process should assign high probability to l t, which is the closest constituent-beginning word. The model should assign large 1 − α t j to words beginning new constituents. While x t itself is a constituent-beginning word, the model should assign large 1 − α t j to words beginning larger constituents. In other words, the model will consider longer dependency relations for the first word in constituent. Given the sentence in FIG4, at time step t = 6, both 1 − α 6 2 and 1 − α 6 0 should be close to 1, and all other 1 − α 6 j should be close to 0.In order to parametrize α t j, our basic hypothesis is that words in the same constituent should have a closer syntactic relation within themselves, and that this syntactical proximity can be represented by a scalar value. From the tree structure point of view, the shortest path between leafs in same subtree is shorter than the one between leafs in different subtree. To model syntactical proximity, we introduce a new feature Syntactic Distance. For a sentence with length K, we define a set of K real valued scalar variables d 0,..., d K−1, with d i representing a measure of the syntactic relation between the pair of adjacent words (x i−1, x i). x −1 could be the last word in previous sentence or a padding token. For time step t, we want to find the closest words x j, that have larger syntactic distance than d t. Thus α t j can be defined as: DISPLAYFORM0 where hardtanh(x) = max(−1, min(1, x)). τ is the temperature parameter that controls the sensitivity of α t j to the differences between distances. The Syntactic Distance has some nice properties that both allow us to infer a tree structure from it and be robust to intermediate non-valid tree structures that the model may encounter during learning. In Appendix C and D we list these properties and further explain the meanings of their values. Parameterizing Syntactic Distance BID45 shows that it's possible to identify the beginning and ending words of a constituent using local information. In our model, the syntactic distance between a given token (which is usually represented as a vector word embedding e i) and its previous token e i−1, is provided by a convolutional kernel over a set of consecutive previous tokens e i−L, e i−L+1,..., e i. This convolution is depicted as the gray triangles shown in FIG2. Each triangle here represent 2 layers of convolution. Formally, the syntactic distance d i between token e i−1 and e i is computed by Convolving h and d on the whole sequence with length K yields a set of distances. For the tokens in the beginning of the sequence, we simply pad L − 1 zero vectors to the front of the sequence in order to get K − 1 outputs. Similar to Long Short-Term Memory-Network (LSTMN) BID9, the Reading Network maintains the memory states by maintaining two sets of vectors: a hidden tape H t−1 = (h t−Nm, ..., h t−1), and a memory tape C t−1 = (c t−L, ..., c t−1), where N m is the upper bound for the memory span. Hidden states m i is now represented by a tuple of two vectors (h i, c i). The Reading Network captures the dependency relation by a modified attention mechanism: structured attention. At each step of recurrence, the model summarizes the previous recurrent states via the structured attention mechanism, then performs a normal LSTM update, with hidden and cell states output by the attention mechanism. DISPLAYFORM1 Structured Attention At each time step t, the read operation attentively links the current token to previous memories with a structured attention layer: DISPLAYFORM2 where, δ k is the dimension of the hidden state. Modulated by the gates in Eq.5, the structured intra-attention weight is defined as: DISPLAYFORM3 This yields a probability distribution over the hidden state vectors of previous tokens. We can then compute an adaptive summary vector for the previous hidden tape and memory denoting byh t and c t: DISPLAYFORM4 Structured attention provides a way to model the dependency relations shown in FIG4.Recurrent Update The Reading Network takes x t,c t andh t as input, computes the values of c t and h t by the LSTM recurrent update BID20. Then the write operation concatenates h t and c t to the end of hidden and memory tape. Predict Network models the probability distribution of next word x t+1, considering on hidden states m 0,..., m t, and gates g t+1 0,..., g t+1 t.Note that, at time step t, the model cannot observe x t+1, a temporary estimation of d t+1 is computed considering on x t−L,..., x t: DISPLAYFORM0 From there we compute its corresponding {α t+1} and {g t+1 i} for Eq.3. We parametrize f (·) function as: DISPLAYFORM1 Figure 4: Syntactic distance estimated by Parsing Network. The model is trained on PTB dataset at the character level. Each blue bar is positioned between two characters, and represents the syntactic distance between them. From these distances we can infer a tree structure according to Section 4.2.where h l:t−1 is an adaptive summary of h lt+1≤i≤t−1, output by structured attention controlled by g t+1 0,..., g t+1 t−1.f (·) could be a simple feed-forward MLP, or more complex architecture, like ResNet, to add more depth to the model. We evaluate the proposed model on three tasks, character-level language modeling, word-level language modeling, and unsupervised constituency parsing. From a character-level view, natural language is a discrete sequence of data, where discrete symbols form a distinct and shallow tree structure: the sentence is the root, words are children of the root, and characters are leafs. However, compared to word-level language modeling, character-level language modeling requires the model to handle longer-term dependencies. We evaluate a character-level variant of our proposed language model over a preprocessed version of the Penn Treebank (PTB) and Text8 datasets. When training, we use truncated back-propagation, and feed the final memory position from the previous batch as the initial memory of next one. At the beginning of training and test time, the model initial hidden states are filled with zero. Optimization is performed with Adam using learning rate lr = 0.003, weight decay w decay = 10 −6, β 1 = 0.9, β 2 = 0.999 and σ = 10 −8. We carry out gradient clipping with maximum norm 1.0. The learning rate is multiplied by 0.1 whenever validation performance does not improve during 2 checkpoints. These checkpoints are performed at the end of each epoch. We also apply layer normalization to the Reading Network and batch normalization to the Predict Network and parsing network. For all of the character-level language modeling experiments, we apply the same procedure, varying only the number of hidden units, mini-batch size and dropout rate. Penn Treebank we process the Penn Treebank dataset BID34 by following the procedure introduced in BID38. For character-level PTB, Reading Network has two recurrent layers, Predict Network has one residual block. Hidden state size is 1024 units. The input and output embedding size are 128, and not shared. Look-back range L = 10, temperature parameter τ = 10, upper band of memory span N m = 20. We use a batch size of 64, truncated backpropagation with 100 timesteps. The values used of dropout on input/output embeddings, between recurrent layers, and on recurrent states were (0, 0.25, 0.1) respectively. In Figure 4, we visualize the syntactic distance estimated by the Parsing Network, while reading three different sequences from the PTB test set. We observe that the syntactic distance tends to be higher between the last character of a word and a space, which is a reasonable breakpoint to separate between words. In other words, if the model sees a space, it will attend on all previous step. If the model sees a letter, it will attend no further then the last space step. The model autonomously discovered to avoid inter-word attention connection, and use the hidden states of space (separator) tokens to summarize previous information. This is strong proof that the model can understand the latent structure of data. As a our model achieve state-of-the-art performance and significantly Model BPC Norm-stabilized RNN BID28 1.48 CW-RNN BID27 1.46 HF-MRNN BID38 1.41 MI-RNN BID57 1.39 ME n-gram BID38 1.37 BatchNorm LSTM BID13 1.32 Zoneout RNN BID29 1.27 HyperNetworks BID18 1.27 LayerNorm HM-LSTM BID11 1.24 LayerNorm HyperNetworks BID18 1.23 PRPN 1.202 Table 1: BPC on the Penn Treebank test set outperform baseline models. It is worth noting that HM-LSTM BID11 ) also unsupervisedly induce similar structure from data. But discrete operations in HM-LSTM make their training procedure more complicated then ours. Comparing to character-level language modeling, word-level language modeling needs to deal with complex syntactic structure and various linguistic phenomena. But it has less long-term dependencies. We evaluate the word-level variant of our language model on a preprocessed version of the Penn Treebank (PTB) BID34 and Text8 BID33 dataset. We apply the same procedure and hyper-parameters as in character-level language model. Except optimization is performed with Adam with β 1 = 0. This turns off the exponential moving average for estimates of the means of the gradients BID36. We also adapt the number of hidden units, mini-batch size and the dropout rate according to the different tasks. Penn Treebank we process the Penn Treebank dataset BID38 by following the procedure introduced in BID39. For word-level PTB, the Reading Network has two recurrent layers and the Predict Network do not have residual block. The hidden state size is 1200 units and the input and output embedding sizes are 800, and shared BID21 BID43. Look-back range L = 5, temperature parameter τ = 10 and the upper band of memory span N m = 15. We use a batch size of 64, truncated back-propagation with 35 time-steps. The values used of dropout on input/output embeddings, between recurrent layers, and on recurrent states were (0.7, 0.5, 0.5) respectively. Model PPL RNN-LDA + KN-5 + cache BID38 92.0 LSTM BID58 78.4 Variational LSTM BID22 78.9 CharCNN BID22 78.9 Pointer Sentinel-LSTM BID37 70.9 LSTM + continuous cache pointer BID17 72.1 Variational LSTM (tied) + augmented loss 68.5 Variational RHN (tied) BID61 65.4 NAS Cell (tied) BID62 62.4 4-layer skip connection LSTM (tied) BID36 58.3 PRPN 61.98 Table 3: Ablation test on the Penn Treebank. "-Parsing Net" means that we remove Parsing Network and replace Structured Attention with normal attention mechanism; "-Reading Net Attention" means that we remove Structured Attention from Reading Network, that is equivalent to replace Reading Network with a normal 2-layer LSTM; "-Predict Net Attention" means that we remove Structured Attention from Predict Network, that is equivalent to have a standard projection layer; "Our 2-layer LSTM" is equivalent to remove Parsing Network and remove Structured Attention from both Reading and Predict Network. Text8 dataset contains 17M training tokens and has a vocabulary size of 44k words. The dataset is partitioned into a training set (first 99M characters) and a development set (last 1M characters) that is used to report performance. As this dataset contains various articles from Wikipedia, the longer term information (such as current topic) plays a bigger role than in the PTB experiments BID42. We apply the same procedure and hyper-parameters as in character-level PTB, except we use a batch size of 128. The values used of dropout on input/output embeddings, between Recurrent Layers and on recurrent states were (0.4, 0.2, 0.2) respectively. Model PPL LSTM-500 BID42 156 SCRNN BID42 161 MemNN BID52 147 LSTM-1024 BID17 121 LSTM + continuous cache pointer BID17 99.9 PRPN 81.64 Table 4: PPL on the Text8 valid setIn TAB1, our are comparable to the state-of-the-art methods. Since we do not have the same computational resource used in BID36 to tune hyper-parameters at large scale, we expect that our model could achieve better performance after an aggressive hyperparameter tuning process. As shown in Table 4, our method outperform baseline methods. It is worth noticing that the continuous cache pointer can also be applied to output of our Predict Network without modification. Visualizations of tree structure generated from learned PTB language model are included in Appendix A. In Table 3, we show the value of test perplexity for different variants of PRPN, each variant remove part of the model. By removing Parsing Network, we observe a significant drop of performance. This stands as empirical evidence regarding the benefit of having structure information to control attention. The unsupervised constituency parsing task compares hte tree structure inferred by the model with those annotated by human experts. The experiment is performed on WSJ10 dataset. WSJ10 is the 7422 sentences in the Penn Treebank Wall Street Journal section which contained 10 words or less after the removal of punctuation and null elements. Evaluation was done by seeing whether proposed constituent spans are also in the Treebank parse, measuring unlabeled F1 (UF 1) of unlabeled constituent precision and recall. Constituents which could not be gotten wrong (those of span one and those spanning entire sentences) were discarded. Given the mechanism discussed in Section 4.2, our model generates a binary tree. Although standard constituency parsing tree is not limited to binary tree. Previous unsupervised constituency parsing model also generate binary trees BID23 BID2. Our model is compared with the several baseline methods, that are explained in Appendix E.Different from the previous experiment setting, the model treat each sentence independently during train and test time. When training, we feed one batch of sentences at each iteration. In a batch, shorter sentences are padded with 0. At the beginning of the iteration, the model's initial hidden states are filled with zero. When testing, we feed on sentence one by one to the model, then use the gate value output by the model to recursively combine tokens into constituents, as described in Appendix A.Model UF 1 LBRANCH 28.7 RANDOM 34.7 DEP-PCFG BID5 48.2 RBRANCH 61.7 CCM BID23 71.9 DMV+CCM BID26 77.6 UML-DOP BID2 82.9 PRPN 70.02 UPPER BOUND 88.1 Table 5: Parsing Performance on the WSJ10 dataset Table 5 summarizes the . Our model significantly outperform the RANDOM baseline indicate a high consistency with human annotation. Our model also shows a comparable performance with CCM model. In fact our parsing network and CCM both focus on the relation between successive tokens. As described in Section 4.2, our model computes syntactic distance between all successive pair of tokens, then our parsing algorithm recursively assemble tokens into constituents according to the learned distance. CCM also recursively model the probability whether a contiguous subsequences of a sentence is a constituent. Thus, one can understand how our model is outperformed by DMV+CCM and UML-DOP models. The DMV+CCM model has extra information from a dependency parser. The UML-DOP approach captures both contiguous and non-contiguous lexical dependencies BID2. In this paper, we propose a novel neural language model that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. We introduce a new neural parsing network: Parsing-Reading-Predict Network, that can make differentiable parsing decisions. We use a new structured attention mechanism to control skip connections in a recurrent neural network. Hence induced syntactic structure information can be used to improve the model's performance. Via this mechanism, the gradient can be directly backpropagated from the language model loss function into the neural Parsing Network. The proposed model achieve (or is close to) the state-of-the-art on both word/character-level language modeling tasks. Experiment also shows that the inferred syntactic structure highly correlated to human expert annotation. A INFERRED TREE STRUCTURE The tree structure is inferred from the syntactic distances yielded by the Parsing Network. We first sort the d i's in decreasing order. For the first d i in the sorted sequence, we separate sentence into constituents ((x <i), (x i, (x >i))). Then we separately repeat this operation for constituents (x <i) and (x >i). Until the constituent contains only one word. In this section we give a probabilistic view on how to model the local structure of language. Given the nature of language, sparse connectivity can be enforced as a prior on how to improve generalization and interpretability of the model. At time step t, p(l t |x 0, ..., x t) represents the probability of choosing one out of t possible local structures that defines the conditional dependencies. If l t = t, it means x t depends on all the previous hidden state from m t to m t (t ≤ t).A particularly flexible option for modeling p(l t |x 0, ..., x t) is the Dirichlet Process, since being nonparametric allows us to attend on as many words as there are in a sentence; i.e. number of possible structures (mixture components) grows with the length of the sentence. As a , we can write the probability of l t+1 = t as a consequence of the stick breaking process 1: DISPLAYFORM0 for 1 ≤ t < t − 1, and DISPLAYFORM1 where α j = 1 − β j and β j is a sample from a Beta distribution. Once we sample l t from the process, the connectivity is realized by a element-wise multiplication of an attention weight vector with a masking vector g t defined in Eq. 1. In this way, x t becomes functionally independent of all x s for all s < l t. The expectation of this operation is the CDF of the probability of l, since DISPLAYFORM2 By telescopic cancellation, the CDF can be expressed in a succinct way: DISPLAYFORM3 for t < t, and P(l t ≤ t) = 1. However, being Bayesian nonparametric and assuming a latent variable model require approximate inference. Hence, we have the following relaxations 1. First, we relax the assumption and parameterize α t j as a deterministic function depending on all the previous words, which we will describe in the next section.2. We replace the discrete decision on the graph structure with a soft attention mechanism, by multiplying attention weight with the multiplicative gate: DISPLAYFORM4 With these relaxations, Eq. can be approximated by using a soft gating vector to update the hidden state h and the predictive function f. This approximation is reasonable since the gate is the expected value of the discrete masking operation described above. In this appendix, we show that having no partial overlapping in dependency ranges is an essential property for recovering a valid tree structure, and PRPN can provide a binary version of g t i, that have this property. The masking vector g t i introduced in Section 4.1 determines the range of dependency, i.e., for the word x t we have g t i = 1 for all l t ≤ i < t. All the words fall into the range l t ≤ i < t is considered as x t's sibling or offspring of its sibling. If the dependency ranges of two words are disjoint with each other, that means the two words belong to two different subtrees. If one range contains another, that means the one with smaller range is a sibling, or is an offspring of a sibling of the other word. However, if they partially overlaps, they can't form a valid tree. While Eq.5 and Eq.6 provide a soft version of dependency range, we can recover a binary version by setting τ in Eq.6 to +∞. The binary version of α t j corresponding to Eq. 6 becomes: DISPLAYFORM0 which is basically the sign of comparing d t and d j+1, scaled to the range of 0 and 1. Then for each of its previous token the gate value g t i can be computed through Eq.5. Now for a certain x t, we have DISPLAYFORM1 where DISPLAYFORM2 Now all the words that fall into the range t ≤ i < t are considered as either sibling of x t, or offspring of a sibling of x t FIG2 ). The essential point here is that, under this parameterization, the dependency range of any two tokens won't partially overlap. Here we provide a terse proof:Proof. Let's assume that the dependency range of x v and x n partially overlaps. We should have g u i = 1 for u ≤ i < v and g n i = 1 for m ≤ i < n. Without losing generality, we assume u < m < v < n so that the two dependency ranges overlap in the range [m, v]. The second property comes from τ. The hyperparameter τ has an interesting effect on the tree structure: if it is set to 0, then for all t, the gates g t i will be open to all of e t's predecessors, which will in a flat tree where all tokens are direct children of the root node; as τ becomes larger, the number of levels of hierarchy in the tree increases. As it approaches + inf, the hardtanh(·) becomes sign(·) and the dependency ranges form a valid tree. Note that, due to the linear part of the gating mechanism, which benefits training, when τ has a value in between the two extremes the truncation range for each token may overlap. That may sometimes in vagueness in some part of the inferred tree. To eliminate this vagueness and ensure a valid tree, at test time we use τ = + inf. Under this framework, the values of syntactic distance have more intuitive meanings. If two adjacent words are siblings of each other, the syntactic distance should approximate zero; otherwise, if they belong to different subtrees, they should have a larger syntactic distance. In the extreme case, the syntactic distance approaches 1 if the two words have no subtree in common. In FIG2 we show the syntactic distances for each adjacent token pair which in the tree shown in FIG4. Our model is compared with the same baseline methods as in BID26. RANDOM chooses a binary tree uniformly at random from the set of binary trees. This is the unsupervised baseline. LBRANCH and RBRANCH choose the completely left-and right-branching structures, respectively. RBRANCH is a frequently used baseline for supervised parsing, but it should be stressed that it encodes a significant fact about English structure, and an induction system need not beat it to claim a degree of success. UPPER BOUND is the upper bound on how well a binary system can do against the Treebank sentences. Because the Treebank sentences are generally more flat than binary, limiting the maximum precision which can be attained, since additional brackets added to provide a binary tree will be counted as wrong. We also compared our model with other unsupervised constituency parsing methods. DEP-PCFG is dependency-structured PCFG BID5. CCM is constituent-context model BID23. DMV is an unsupervised dependency parsing model. DMV+CCM is a combined model that jointly learn both constituency and dependency parser BID25.

In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: Unsupervised embedding learning aims to extract good representations from data without the use of human-annotated labels. Such techniques are apparently in the limelight because of the challenges in collecting massive-scale labels required for supervised learning. This paper proposes a comprehensive approach, called Super-AND, which is based on the Anchor Neighbourhood Discovery model. Multiple losses defined in Super-AND make similar samples gather even within a low-density space and keep features invariant against augmentation. As a , our model outperforms existing approaches in various benchmark datasets and achieves an accuracy of 89.2% in CIFAR-10 with the Resnet18 backbone network, a 2.9% gain over the state-of-the-art. Deep learning and convolutional neural network have become an indispensable technique in computer vision (; ;). Remarkable developments, in particular, were led by supervised learning that requires thousands or more labeled data. However, high annotation costs have become a significant drawback in training a scalable and practical model in many domains. In contrast, unsupervised deep learning that requires no label has recently started to get attention in computer vision tasks. From clustering analysis , and self-supervised model to generative model (; ;), various learning methods came out and showed possibilities and prospects. Unsupervised embedding learning aims to extract visually meaningful representations without any label information. Here "visually meaningful" refers to finding features that satisfy two traits: (i) positive attention and (ii) negative separation (; c;). Data samples from the same ground truth class, i.e., positive samples, should be close in the embedding space (Fig. 1a); whereas those from different classes, i.e., negative samples, should be pushed far away in the embedding space (Fig. 1b). However, in the setting of unsupervised learning, a model cannot have knowledge about whether given data points are positive samples or negative samples. Several new methods have been proposed to find'visually meaningful' representations. The sample specificity method considers all data points as negative samples and separates them in the feature space . Although this method achieves high performance, its decisions are known to be biased from learning only from negative separation. One approach utilizes data augmentation to consider positive samples in training , which efficiently reduces any ambiguity in supervision while keeping invariant features in the embedding space. Another approach is called the Anchor Neighborhood Discovery (AND) model, which alleviates the complexity in boundaries by discovering the nearest neighbor among the data points . Each of these approaches overcomes different limitations of the sample specificity method. However, no unified approach has been proposed. This paper presents a holistic method for unsupervised embedding learning, named Super-AND. Super-AND extends the AND algorithm and unifies various but dominant approaches in this domain with its unique architecture. Our proposed model not only focuses on learning distinctive features across neighborhoods, but also emphasizes edge information in embeddings and maintains the unchanging class information from the augmented data. Besides combining existing techniques, we newly introduce Unification Entropy loss (UE-loss), an adversary of sample specificity loss, which is able to gather similar data points within a low-density space. Extensive experiments are conducted on several benchmark datasets to verify the superiority of the model. The show the synergetic advantages among modules of Super-AND. The main contributions of this paper are as follows: • We effectively unify various techniques from state-of-the-art models and introduce a new loss, UE-loss, to make similar data samples gather in the low-density space. • Super-AND outperforms all baselines in various benchmark datasets. It achieved an accuracy of 89.2% in the CIFAR-10 dataset with the ResNet18 backbone network, compared to the state-of-the-art that gained 86.3%. • The extensive experiments and the ablation study show that every component in Super-AND contributes to the performance increase, and also indicate their synergies are critical. Our model's outstanding performance is a step closer to the broader adoption of unsupervised techniques in computer vision tasks. The premise of data-less embedding learning is at its applicability to practical scenarios, where there exists only one or two examples per cluster. Codes and trained data for Super-AND are accessible via a GitHub link. Generative model. This type of model is a powerful branch in unsupervised learning. By reconstructing the underlying data distribution, a model can generate new data points as well as features from images without labels. Generative adversarial network has led to rapid progress in image generation problems ). While some attempts have been made in terms of unsupervised embedding learning , the main objective of generative models lies at mimicking the true distribution of each class, rather than discovering distinctive categorical information the data contains. Self-supervised learning. This type of learning uses inherent structures in images as pseudo-labels and exploits labels for back-propagation. For example, a model can be trained to create embeddings by predicting the relative position of a pixel from other pixels or the degree of changes after rotating images . Predicting future frames of a video can benefit from this technique . proposed the sample specificity method that learns feature representation from capturing apparent discriminability among instances. All of these methods are suitable for unsupervised embedding learning, although there exists a risk of false knowledge from generated labels that weakly correlate with the underlying class information. Learning invariants from augmentation. Data augmentation is a strategy that enables a model to learn from datasets with an increased variety of instances. Popular techniques include flipping, scaling, rotation, and grey-scaling. These techniques do not deform any crucial features of data, but only change the style of images. Some studies hence use augmentation techniques and train models Clustering analysis. This type of analysis is an extensively studied area in unsupervised learning, whose main objective is to group similar objects into the same class. Many studies either leveraged deep learning for dimensionality reduction before clustering or trained models in an end-to-end fashion . proposed a concept called deep cluster, an iterative method that updates its weights by predicting cluster assignments as pseudo-labels. However, directly reasoning the global structures without any label is error-prone. The AND model, which we extend in this work, combines the advantages of sample specificity and clustering strategy to mitigate the noisy supervision via neighborhood analysis . Problem definition. Assume that there is an unlabeled image set I, and a batch set B with n images: B = {x 1, x 2, ..., x n} ⊂ I. Our goal is to get a feature extractor f θ whose representations (i.e., v i = f θ (x i)) are "visually meaningful," a definition we discussed earlier. LetB = {x 1,x 2, ...,x n} be the augmentation set of input batches B. Super-AND projects images x i,x i from batches B,B to 128 dimensional embeddings v i,v i. During this process, the Sobelprocessed images are also used, and feature vectors from both images are concatenated to emphasize edge information in embeddings (see the left side in Fig. 2). Then, the model computes the probability of images p i,p i being recognized as its own class with a nonparametric classifier (see the right side in Fig. 2). A temperature parameter (τ < 1) was added to ensure a label distribution with low entropy. To reduce the computational complexity in calculating embeddings from all images, we set up a memory bank M to save instance embeddings m i accumulated from the previous steps, as similarly proposed by. The memory bank M is updated by exponential moving average . The probability vector p i is defined in Eq 1, where the superscript j in vector notation (i.e., v j) represents the j-th component value of a given vector. We define the neighborhood relationship vectors r i,r i, and compute these vectors by the cosine similarity between the embedding vectors v i,v i, and the memory bank M (Eq 2). The extracted vectors are used to define the loss term that detects a discrepancy between neighborhoods. The loss term also enforces features v to remain unchanged even after data augmentation. The loss term is written as where N is the set of progressively selected pairs discovered by the nearest neighbor algorithm, and V, R,R, P are matrices of concatenated embedded vectors v i, r i,r i, p i from the batch image set, respectively. w(t) is the hyper-parameter that controls the weights of UE-loss. The algorithm below describes how to train Super-AND. Algorithm 1: Main algorithm for training Super-AND. Input: Unlabeled image set I, encoder f θ to train, the number of total rounds for training: Rounds, and the number of total epochs for training: Compute gradient and update weights by backpropagation 14 end 15 end Existing clustering methods like train networks to find an optimal mapping. However, their learned decisions are unstable due to initial randomness, and some overfitting can occur during the training period (a). To tackle these limitations, the AND model suggests a finer-grained clustering focusing on'neighborhoods'. By regarding the nearest-neighbor pairs as local classes, AND can separate data points that belong to different neighborhood sets from those in the same neighborhood set. We adopt this neighborhood discovery strategy in our Super-AND. The AND algorithm has three main steps: neighborhood discovery, progressive neighborhood selection with curriculum, and neighborhood supervision. For the first step, the k nearest neighborhood (k-NN) algorithm is used to discover all neighborhood pairs (Eq 7 and Eq 8), and these pairs are progressively selected for curriculum learning. We choose a small part of neighborhood pairs at the first round, and gradually increase the amount of selection for training (Current/Total rounds × 100%). Since we cannot assure that every neighborhood is visually similar, this progressive method helps provide a consistent view of local class information for training at each round. When selecting candidate neighborhoods for local classes, the entropy of probability vector H(x i) is utilized as a criterion (Eq 9). Probability vector p i, obtained from softmax function (Eq 1), shows the visual similarity between training instances in a probabilistic manner. Data points with low entropy represent they reside in a relatively low-density area and have only a few surrounding neighbors. Neighborhood pairs containing such data points likely share consistent and easily distinguishable features from other pairs. We select neighborhood set N fromÑ that is in a lower entropy order. The AND-loss function is defined to distinguish neighborhood pairs from one another. Data points from the same neighborhoods need to be classified in the same class (i.e., the left-hand term in Eq 10). If any data point is present in the selected pair, it is considered to form an independent class (i.e., the right-hand term in Eq 10). Existing sample specificity methods consider every single data point as a prototype for a class. They use the cross-entropy loss to separate all data points in the L2-normalized embedding space. Due to its confined space by normalization, data points cannot be placed far away from one another, and this space limitation induces an effect that leads to a concentration of positive samples, as shown in Fig. 1a. The unification entropy loss (UE-loss) is able to even strengthen the concentration-effect above. We define the UE-loss as the entropy of the probability vectorp i. Probability vectorp i is calculated from the softmax function and represents the similarity between instances except for instance itself (Eq 11). By excluding the class of one's own, minimizing the loss makes nearby data points attract each other -a concept that is contrary to minimizing the sample specificity loss. Employing both AND-loss and the UE-loss will enforce similar neighborhoods to be positioned close while keeping the overall neighborhoods as separated as possible. This loss is calculated as in Eq 12. Unsupervised embedding learning aims at training encoders to extract visually meaningful features that are consistent with ground truth labels. Such learning cannot use any external guidance on features. Several previous studies tried to infer which features are substantial in a roundabout way; data augmentation is one such solution (; ; ;). Since augmentation does not deform the underlying data characteristics, invariant features learned from the augmented data will still contain the class-related information. Naturally, a training network based on these features will show performance gain. We define the Augmentation-loss to learn invariant image features. Assume that there is an image along with its augmented versions. We may regard every augmentation instance as a positive sample. The neighborhood relationship vectors, which show the similarity between all instances stored in memory, should also be similar to initial data points than other instances in the same batch. In Eq 13, the probability of an augmented instance that is correctly identified as class-i is denoted asp i i; and that of i-th original instance that is wrongly identified as class-j (j = i),p j i. The Augmentation-loss is then defined to minimize misclassification over instances in all batches (Eq 14). The evaluation involved extensive experiments. We enumerated the model with different backbone networks on two kinds of benchmarks: coarse-grained and fine-grained. Our ablation study helps speculate which components of the model are critical in performance. Finally, the proposed model is compared to the original AND from different perspectives. Datasets. A total of six image datasets are utilized, where three are coarse-grained datasets: is used for qualitative analysis. Training. We used AlexNet and ResNet18 as the backbone networks. Hyper-parameters were tuned in the same way as the AND algorithm. We used SGD with Nesterov momentum 0.9 for the optimizer. We fixed the learning rate as 0.03 for the first 80 epochs, and scaled-down 0.1 every 40 epochs. The batch size is set as 128, and the model was trained in 5 rounds and 200 epochs per round. Weights for UE-loss w(t) (Eq 6) are initialized from 0 and increased 0.2 every 80 epochs. For Augmentation-loss, we used four types: Resized Crop, Grayscale, ColorJitter, and Horizontal Flip. Horizontal Flip was not used in the case of the SVHN dataset because the SVHN dataset is digit images. Update momentum of the exponential moving average for memory bank was set to 0.5. Evaluation. Following the method from , we used the weighted k-NN classifier for making prediction. Top k-nearest neighbors N top were retrieved and used to predict the final outcome in a weighted fashion. We set k = 200 and the weight function for each class c as, where c i is the class index for i-th instance. Top-1 classification accuracy was used for evaluation. Baseline models. We adopt six state-of-the-art baselines for comparison. They are SplitBrain (b), Counting , DeepCluster , Instance , ISIF , and AND . For fair comparison, the same backbone networks were used. Coarse-grained evaluation. Table 1 describes the object classification performance of seven models, including the proposed Super-AND on three coarse-grained datasets: CIFAR-10, CIFAR-100, and SVHN. Super-AND surpasses state-of-the-art baselines on all datasets except for one case, where the model underperforms marginally on CIFAR-100 with AlexNet. One notable observa- Table 1: k-NN Evaluation on coarse-grained datasets. Results that are marked as * are borrowed from the previous works (; tion is that the difference between previous models and super-AND is mostly larger in the case of ResNet18 than the AlexNet backbone network. These reveal that our model is superior to other methods and may indicate that our methodology can give more benefits to stronger CNN architectures. Fine-grained evaluation. We perform evaluations on fine-grained datasets that require the ability to discriminate subtle differences between classes. Table 2 shows that Super-AND achieves an outstanding performance compared to three baselines with the ResNet18 backbone network. We excerpted the of Instance and AND model from the previous work. Backbone network. We tested the choice of backbone networks in terms of classification performance. AlexNet, ResNet18, and ResNet101 are used and evaluated on CIFAR-10, as shown in Table 3 . From the , we can infer that the stronger the backbone network our model has, the better the performance model can produce. Ablation study. To verify every component does its role and has some contribution to the performance increase, an ablation study was conducted. Since Super-AND combines various mechanisms based on AND algorithm, we study the effect of removing each component: Super-AND without the UE-loss, Super-AND without the Sobel filter, Super-AND without the Augmentation-loss. Table 4 displays the evaluation based on the CIFAR-10 dataset and the ResNet18 backbone network. We found that every component contributes to the performance increase, and a particularly dramatic decrease in performance occurs when removing the Augmentation-loss. Initialization. Instead of running the algorithm from an arbitrary random model, we can pre-train the network with "good" initial data points to discover consistent neighborhoods. We investigate two different initialization methods and check whether the choice is critical. Three models were compared: a random model, an initialized model with instance loss from AND, and an initialized model with multiple losses from Super-AND. Table 5 shows that the choice of initialization is not significant, and solely using the instance loss even has an adverse effect on performance. This finding implies that Super-AND is robust to random initial data points, yet the model will show an unexpected outcome if initialization uses ambiguous knowledge. Embedding quality analysis. Super-AND leverages the synergies from learning both similarities in neighborhoods and invariant features from data augmentation. Super-AND, therefore, has a high capability of discovering cluster relationships, compared to the original AND model that only uses the neighborhood information. Fig. 3 exploits t-SNE to visualize the learned representations of three of the selected classes based on the two algorithms in CIFAR-10. The plot demonstrates that Super-AND discovers consistent and discriminative clusters. We investigated the embedding quality by evaluating the class consistency of selected neighborhoods. Cheat labels are used to check whether neighborhood pairs come from the same class. Since both algorithms increase the selection ratio every round when gathering the part of discovered neighborhoods, the consistency of selected neighborhoods will naturally decrease. This relationship is drawn in Fig. 4. The reduction for Super-AND, nonetheless, is not significant compared to AND: our model maintains high-performance throughout the training rounds. Qualitative study. Fig. 5 illustrates the top-5 nearest retrievals of AND (i.e., upper rows) and Super-AND (i.e., lower rows) based on the STL-10 dataset. The example queries shown are dump trucks, airplanes, horses, and monkeys. Images with red frames, which indicate negative samples, appear more frequently for AND than Super-AND. This finding implies that Super-AND excels in capturing the class information compared to AND. Its clusters are robust to misleading color information and well recognize the shape of objects within images. For example, in the case of the airplane query, pictures retrieved from Super-AND are consistent in shape while AND confuse a cruise picture as an airplane. The color composition in Super-AND is also more flexible and can find a red dump truck or a spotted horse, as shown in the examples. This paper presents Super-AND, a holistic technique for unsupervised embedding learning. Besides the synergetic advantage combining existing methods brings, the newly proposed UE-loss that groups nearby data points even in a low-density space while maintaining invariant features via data augmentation. The experiments with both coarse-grained and fine-grained datasets demonstrate our model's outstanding performance against the state-of-the-art models. Our efforts to advance unsupervised embedding learning directly benefit future applications that rely on various image clustering tasks. The high accuracy achieved by Super-AND makes the unsupervised learning approach an economically viable option where labels are costly to generate.

represent the tldr for finding the corresponding scientific article

We proposed a comprehensive approach for unsupervised embedding learning on the basis of AND algorithm.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.


Question: represent this text

Question: Analysis of histopathology slides is a critical step for many diagnoses, and in particular in oncology where it defines the gold standard. In the case of digital histopathological analysis, highly trained pathologists must review vast whole-slide-images of extreme digital resolution (100,000^2 pixels) across multiple zoom levels in order to locate abnormal regions of cells, or in some cases single cells, out of millions. The application of deep learning to this problem is hampered not only by small sample sizes, as typical datasets contain only a few hundred samples, but also by the generation of ground-truth localized annotations for training interpretable classification and segmentation models. We propose a method for disease available during training. Even without pixel-level annotations, we are able to demonstrate performance comparable with models trained with strong annotations on the Camelyon-16 lymph node metastases detection challenge. We accomplish this through the use of pre-trained deep convolutional networks, feature embedding, as well as learning via top instances and negative evidence, a multiple instance learning technique fromatp the field of semantic segmentation and object detection. Histopathological image analysis (HIA) is a critical element of diagnosis in many areas of medicine, and especially in oncology, where it defines the gold standard metric. Recent works have sought to leverage modern developments in machine learning (ML) to aid pathologists in disease detection tasks, but the majority of these techniques require localized annotation masks as training data. These annotations are even more costly to obtain than the original diagnosis, as pathologists must spend time to assemble pixel-by-pixel segmentation maps of diseased tissue at extreme resolution, thus HIA datasets with annotations are very limited in size. Additionally, such localized annotations may not be available when facing new problems in HIA, such as new disease subtybe classification, prognosis estimation, or drug response prediction. Thus, the critical question for HIA is: can one design a learning architecture which achieves accurate classification with no additional localized annotation? A successful technique would be able train algorithms to assist pathologists during analysis, and could also be used to identify previously unknown structures and regions of interest. Indeed, while histopathology is the gold standard diagnostic in oncology, it is extremely costly, requiring many hours of focus from pathologists to make a single diagnosis BID21 BID30. Additionally, as correct diagnosis for certain diseases requires pathologists to identify a few cells out of millions, these tasks are akin to "finding a needle in a haystack." Hard numbers on diagnostic error rates in histopathology are difficult to obtain, being dependent upon the disease and tissue in question as well as self-reporting by pathologists of diagnostic errors. However, as reported in the review of BID25, false negatives in cancer diagnosis can lead not only to catastrophic consequences for the patient, but also to incredible financial risk to the pathologist. Any tool which can aid pathologists to focus their attention and effort to the must suspect regions can help reduce false-negatives and improve patient outcomes through more accurate diagnoses BID8. Medical researchers have looked to computer-aided diagnosis for decades, but the lack of computational resources and data have prevented wide-spread implementa-tion and usage of such tools BID11. Since the advent of automated digital WSI capture in the 1990s, researchers have sought approaches for easing the pathologist's workload and improve patient outcomes through image processing algorithms BID11 BID22. Rather than predicting final diagnosis, many of these procedures focused instead on segmentation, either for cell-counting, or for the detection of suspect regions in the WSI. Historical methods have focused on the use of hand-crafted texture or morphological BID5 features used in conjunction with unsupervised techniques such as K-means clustering or other dimensionality reduction techniques prior to classification via k-Nearest Neighbor or a support vector machine. Over the past decade, fruitful developments in deep learning BID19 have lead to an explosion of research into the automation of image processing tasks. While the application of such advanced ML techniques to image tasks has been successful for many consumer applications, the adoption of such approaches within the field of medical imaging has been more gradual. However, these techniques demonstrate remarkable promise in the field of HIA. Specifically, in digital pathology with whole-slide-imaging (WSI) BID33 BID26, highly trained and skilled pathologists review digitally captured microscopy images from prepared and stained tissue samples in order to make diagnoses. Digital WSI are massive datasets, consisting of images captured at multiple zoom levels. At the greatest magnification levels, a WSI may have a digital resolution upwards of 100 thousand pixels in both dimensions. However, since localized annotations are very difficult to obtain, datasets may only contain WSI-level diagnosis labels, falling into the category of weakly-supervised learning. The use of DCNNs was first proposed for HIA in BID3, where the authors were able to train a model for mitosis detection in H&E stained images. A similar technique was applied for WSI for the detection of invasive ductal carcinoma in BID4. These approaches demonstrated the usefulness of learned features as an effective replacement for hand-crafted image features. It is possible to train deep architectures from scratch for the classification of tile images BID29 BID13. However, training such DCNN architectures can be extremely resource intensive. For this reason, many recent approaches applying DCNNs to HIA make use of large pre-trained networks to act as rich feature extractors for tiles BID15 BID17 BID21 BID32 BID27. Such approaches have found success as aggregation of rich representations from pre-trained DCNNs has proven to be quite effective, even without from-scratch training on WSI tiles. In this paper, we propose CHOWDER 1, an approach for the interpretable prediction of general localized diseases in WSI with only weak, whole-image disease labels and without any additional expert-produced localized annotations, i.e. per-pixel segmentation maps, of diseased areas within the WSI. To accomplish this, we modify an existing architecture from the field of multiple instance learning and object region detection BID9 to WSI diagnosis prediction. By modifying the pre-trained DCNN model BID12, introducing an additional set of fully-connected layers for context-aware classification from tile instances, developing a random tile sampling scheme for efficient training over massive WSI, and enforcing a strict set of regualrizations, we are able to demonstrate performance equivalent to the best human pathologists. Notably, while the approach we propose makes use of a pre-trained DCNN as a feature extractor, the entire procedure is a true end-to-end classification technique, and therefore the transferred pre-trained layers can be fine-tuned to the context of H&E WSI. We demonstrate, using only whole-slide labels, performance comparable to top-10 ranked methods trained with strong, pixel-level labels on the Camelyon-16 challenge dataset, while also producing disease segmentation that closely matches ground-truth annotations. We also present for diagnosis prediction on WSI obtained from The Cancer Genome Atlas (TCGA), where strong annotations are not available and diseases may not be strongly localized within the tissue sample. While approaches using localized annotations have shown promise for HIA, they fail to address the cost associated with the acquisition of hand-labeled datasets, as in each case these methods require access to pixel-level labels. As shown with ImageNet BID6 ), access to data drives innovation, however for HIA hand-labeled segmentation maps are costly to produce, often subject to missed diseased areas, and cannot scale to the size of datasets required for truly effective deep learning. Because of these considerations, HIA is uniquely suited to the weakly supervised learning (WSL) setting. Here, we define the WSL task for HIA to be the identification of suspect regions of WSI when the training data only contains image-wide labels of diagnoses made by expert pathologists. Since WSI are often digitally processed in small patches, or tiles, the aggregation of these tiles into groups with a single label (e.g. "healthy", "cancer present") can be used within the framework of multiple instance learning (MIL) BID7 BID0 BID31. In MIL for binary classification, one often makes the standard multi-instance (SMI) assumption: a bag is classified as positive iff at least one instance (here, a tile) in the bag is labelled positive. The goal is to take the bag-level labels and learn a set of instance-level rules for the classification of single instances. In the case of HIA, learning such rules provides the ability to infer localized regions of abnormal cells within the large-scale WSI.In the recent work of BID13 for WSI classification in the WSL setting, the authors propose an EM-based method to identify discriminative patches in high resolution images automatically during patch-level CNN training. They also introduced a decision level fusion method for HIA, which is more robust than max-pooling and can be thought of as a Count-based Multiple Instance (CMI) learning method with two-level learning. While this approach was shown to be effective in the case of glioma classification and obtains the best , it only slightly outperforms much simpler approaches presented in BID13, but at much greater computational cost. In the case of natural images, the WELDON and WILDCAT techniques of BID9 and BID10, respectively, demonstrated state-of-the-art performance for object detection and localization for WSL with image-wide labels. In the case of WELDON, the authors propose an end-to-end trainable CNN model based on MIL learning with top instances BID20 as well as negative evidence, relaxing the SMI assumption. Specifically, in the case of semantic segmentation, BID20 argue that a target concept might not exist just at the subregion level, but that the proportion of positive and negative samples in a bag have a larger effect in the determination of label assignment. This argument also holds for the case of HIA, where pathologist diagnosis arises from a synthesis of observations across multiple resolution levels as well as the relative abundance of diseased cells. In Sec. 2.3, we will detail our proposed approach which makes a number of improvements on the framework of BID9, adapting it to the context of large-scale WSI for HIA. Tissue Detection. As seen in FIG0, large regions of a WSI may contain no tissue at all, and are therefore not useful for training and inference. To extract only tiles with content relevant to the task, we use the same approach as BID29, namely, Otsu's method BID24 applied to the hue and saturation channels of the image after transformation into the HSV color space to produce two masks which are then combined to produce the final tissue segmentation. Subsequently, only tiles within the foreground segmentation are extracted for training and inference. Color Normalization. According to, stain normalization is an important step in HIA since the of the H&E staining procedure can vary greatly between any two slides. We utilize a simple histogram equalization algorithm consisting of left-shifting RGB channels and subsequently rescaling them to, as proposed in BID23. In this work, we place a particular emphasis on the tile aggregation method rather than color normalization, so we did not make use of more advanced color normalization algorithms, such as BID16.Tiling. The tiling step is necessary in histopathology analysis. Indeed, due to the large size of the WSI, it is computationally intractable to process the slide in its entirety. For example, on the highest resolution zoom level, denoted as scale 0, for a fixed grid of non-overlapping tiles, a WSI may possess more than 200,000 tiles of 224×224 pixels. Because of the computational burden associated with processing the set of all possible tiles, we instead turn to a uniform random sampling from the space of possible tiles. Additionally, due to the large scale nature of WSI datasets, the computational burden associated with sampling potentially overlapping tiles from arbitrary locations is a prohibitive cost for batch construction during training. Instead, we propose that all tiles from the non-overlapping grid should be processed and stored to disk prior to training. As the tissue structure does not exhibit any strong periodicity, we find that sampling tiles along a fixed grid without overlapping provides a reasonably representative sampling while maximizing the total sampled area. Given a target scale ∈ {0, 1, . . ., L}, we denote the number of possible tiles in WSI indexed by i ∈ {1, 2, . . ., N} as M T i,. The number of tiles sampled for training or inference is denoted by M S i, and is chosen according to DISPLAYFORM0 DISPLAYFORM1 is the empirical average of the number of tiles at scale over the entire set of training data. Feature Extraction. We make use of the ResNet-50 BID12 architecture trained on the ImageNet natural image dataset. In empirical comparisons between VGG or Inception architectures, we have found that the ResNet architecture provides features more well suited for HIA. Additionally, the ResNet architecture is provided at a variety of depths. However, we found that ResNet-50 provides the best balance between the computational burden of forward inference and richness of representation for HIA.In our approach, for every tile we use the values of the ResNet-50 pre-output layer, a set of P = 2048 floating point values, as the feature vector for the tile. Since the fixed input resolution for ResNet-50 is 224 × 224 pixels, we set the resolution for the tiles extracted from the WSI to the same pixel resolution at every scale. Given a WSI, extracting tile-level features produces a bag of feature vectors which one attempts to use for classification against the known image-wide label. The dimension of these local descriptors is M S × P, where P is the number of features output from the pre-trained image DCNN and M S is the number of sampled tiles. Approaches such as Bag-of-visual-words (BoVW) or VLAD BID14 could be chosen as a baseline aggregation method to generate a single image-wide descriptor of size P ×1, but would require a huge computational power given the dimensionality of the input. Instead, we will try two common approaches for the aggregation of local features, specifically, the MaxPool and MeanPool and subsequently apply a classifier on the aggregated features. After applying these pooling methods over the axis of tile indices, one obtains a single feature descriptor for the whole image. Other pooling approaches have been used in the context of HIA, including Fisher vector encodings BID27 and p−norm pooling BID32. However, as the reported effect of these aggregations is quite small, we don't consider these approaches when constructing our baseline approach. After aggregation, a classifier can be trained to produce the desired diagnosis labels given the global WSI aggregated descriptor. For our baseline method, we use a logistic regression for this final prediction layer of the model. We present a description of the baseline approach in FIG0. In experimentation, we observe that the baseline approach of the previous section works well for diffuse disease, which is evidenced in the of TAB0 for TCGA-Lung. Here, diffuse implies that the number of disease-containing tiles, pertinent to the diagnosis label, are roughly proportional to the number of tiles containing healthy tissue. However, if one applies the same approach to different WSI datasets, such as Camelyon-16, the performance significantly degrades. In the case of Camelyon-16, the diseased regions of most of the slides are highly localized, restricted to a very small area within the WSI. When presented with such imbalanced bags, simple aggregation approaches for global slide descriptors will overwhelm the features of the disease-containing tiles. Instead, we propose an adaptation and improvement of the WELDON method BID9 designed for histopathology images analysis. As in their approach, rather than creating a global slide descriptor by aggregating all tile features, instead a MIL approach is used that combines both top-instances as well as negative evidence. A visual description of approach is given in FIG1.Feature Embedding. First, a set of one-dimensional embeddings for the P = 2048 ResNet-50 features are calcualted via J one-dimensional convolutional layers strided across the tile index axis. For tile t with features k t, the embedding according to kernel j is calculated as e j,t = w j, k t. Notably, the kernels w j have dimensionality P. This one-dimensional convolution is, in essence, a shortcut for enforcing a fully-connected layer with tied weights across tiles, i.e. the same embedding for every tile BID9. In our experiments, we found that the use of a single embedding, J = 1, is an appropriate choice for WSI datasets when the number of available slides is small (< 1000). In this case, choosing J > 1 will decrease training error, but will increase generalization error. Avoiding overtraining and ensuring model generality remains a major challenge for the application of WSL to WSI datasets. Top Instances and Negative Evidence. After feature embedding, we now have a M S,i × 1 vector of local tile-level (instance) descriptors. As in BID9, these instance descriptors are sorted by value. Of these sorted embedding values, only the top and bottom R entries are retained, ing in a tensor of 2R × 1 entries to use for diagnosis classification. This can be easily accomplished through a MinMax layer on the output of the one-dimensional convolution layer. The purpose of this layer is to take not only the top instances region but also the negative evidences, that is the region which best support the absence of the class. During training, the back-propagation runs only through the selected tiles, positive and negative evidences. When applied to WSI, the MinMax serves as a powerful tile selection procedure. Multi-layer Perceptron (MLP) Classifier. In the WELDON architecture, the last layer consists of a sum applied over the 2R × 1 output from the MinMax layer. However, we find that this approach can be improved for WSI classification. We investigate the possibility of a richer interactions between the top and bottom instances by instead using an MLP as the final classifier. In our implementation of CHOWDER, we use an MLP with two fully connected layers of 200 and 100 neurons with sigmoid activations. First, for pre-processing, we fix a single tile scale for all methods and datasets. We chose a fixed zoom level of 0.5 µm/pixel, which corresponds to = 0 for slides scanned at 20x magnification, or = 1 slides scanned at 40x magnification. Next, since WSI datasets often only contain a few hundred images, far from the millions images of ImageNet dataset, strong regularization required prevent over-fitting. We applied 2 -regularization of 0.5 on the convolutional feature embedding layer and dropout on the MLP with a rate of 0.5. However, these values may not be the global optimal, as we did not apply any hyper-parameter optimization to tune these values. To optimize the model parameters, we use Adam BID18 to minimize the binary cross-entropy loss over 30 epochs with a mini-batch size of 10 and with learning rate of 0.001.To reduce variance and prevent over-fitting, we trained an ensemble of E CHOWDER networks which only differ by their initial weights. The average of the predictions made by these E networks establishes the final prediction. Although we set E = 10 for the presented in TAB0, we used a larger ensemble of E = 50 with R = 5 to obtain the best possible model and compare our method to those presented in Table 2. We also use an ensemble of E = 10 when reporting the for WELDON. As the training of one epoch requires about 30 seconds on our available hardware, the total training time for the ensemble took just over twelve hours. While the ResNet-50 features were extracted using a GPU for efficient feed-forward calculations, the CHOWDER network is trained on CPU in order to take advantage of larger system RAM sizes, compared to on-board GPU RAM. This allows us to store all the training tiles in memory to provide faster training compared to a GPU due to reduced transfer overhead. The public Cancer Genome Atlas (TCGA) provides approximately 11,000 tissue slides images of cancers of various organs 2. For our first experiment, we selected 707 lung cancer WSIs (TCGA-Lung), which were downloaded in March 2017. Subsequently, a set of new lung slides have been added to TCGA, increasing the count of lung slides to 1,009. Along with the slides themselves, TCGA also provides labels representing the type of cancer present in each WSI. However, no local segmentation annotations of cancerous tissue regions are provided. The pre-processing step extracts 1,411,043 tiles and their corresponding representations from ResNet-50. The task of these experiments is then to predict which type of cancer is contained in each WSI: adenocarcinoma or squamous cell carcinoma. We evaluate the quality of the classification according to the area under the curve (AUC) of the receiver operating characteristic (ROC) curve generated using the raw output predictions. As expected in the case of diffuse disease, the advantage provided by CHOWDER is slight as compared to the MeanPool baseline, as evidenced in TAB0. Additionally, as the full aggregation techniques work quite well in this setting, the value of R does not seem to have a strong effect on the performance of CHOWDER as it increases to R = 100. In this setting of highly homogenous tissue content, we can expect that global aggregate descriptors are able to effectively separate the two classes of carcinoma. For our second experiment, we use the Camelyon-16 challenge dataset 3, which consists of 400 WSIs taken from sentinel lymph nodes, which are either healthy or exhibit metastases of some form. In addition to the WSIs themselves, as well as their labeling (healthy, contains-metastases), a segmentation mask is provided for each WSI which represents an expert analysis on the location of metastases within the WSI. Human labeling of sentinel lymph node slides is known to be quite tedious, as noted in BID21 BID30. Teams participating in the challenge had access to, and utilized, the ground-truth masks when training their diagnosis prediction and tumor localization models. For our approach, we set aside the masks of metastasis locations and utilize only diagnosis labels. Furthermore, many participating teams developed a post-processing step, extracting handcrafted features from predicted metastasis maps to improve their segmentation. No post-processing is performed for the presented CHOWDER , the score is computed directly from the raw output of the CHOWDER model. The Camelyon-16 dataset is evaluated on two different axes. First, the accuracy of the predicted label for each WSI in the test set is evaluated according to AUC. Second, the accuracy of metastasis localization is evaluated by comparing model outputs to the ground-truth expert annotations of metastasis location. This segmentation accuracy is measured according to the free ROC metric (FROC), which is the curve of metastasis detection sensitivity to the average number of also positives. As in the Camelyon challenge, we evaluate the FROC metric as the average detection sensitivity at the average false positive rates 0.25, 0.5, 1, 2, 4, and 8.Competition Split Bias. We also conduct a set of experiments on Camelyon-16 using random train-test cross-validation (CV) splits, respecting the same training set size as in the original competition split. We note distinct difference in AUC between the competition split and those obtained via random folds. This discrepancy is especially distinct for the MeanPool baseline, as reported in TAB0. We therefore note a distinct discrepancy in the data distribution between the competition test and training splits. Notably, using the MeanPool baseline architecture, we found that the competition train-test split can be predicted with an AUC of 0.75, however one only obtains an AUC of 0.55 when using random splits. Because this distribution mismatch in the competition split could produce misleading interpretations, we report 3-fold average CV along with the obtained on the competition split. Classification Performance. In TAB0, we see the classification performance of our proposed CHOWDER method, for E = 10, as compared to both the baseline aggregation techniques, as well as the WELDON approach. In the case of WELDON, the final MLP is not used and instead a summing is applied to the MinMax layer. The value of R retains the same meaning in both cases: the number of both high and low scoring tiles to pass on to the classification layers. We test a range of values R for both WELDON and CHOWDER. We find that over all values of R, CHOWDER provides a significant advantage over both the baseline aggregation techniques as well as WELDON. We also note that the optimal performance can be obtained without using a large number of discriminative tiles, i.e. R = 5.We also present in Table 2 our performance as compared to the public Camelyon leader boards for E = 50. In this case, we are able to obtain an effective 11 th place rank, but without using any of the ground-truth disease segmentation maps. This is a remarkable , as the winning approach of BID29 required tile-level disease labels derived from expert-provided annotations in order to train a full 27-layer GoogLeNet BID28 architecture for tumor prediction. We also show the ROC curve for this in FIG3. Finally, we note that CHOWDER's performance on this task roughly is equivalent to the best-performing human pathologist, an AUC of 0.884 as reported by, and better than the average human pathologist performance, an AUC of 0.810. Notably, this human-level performance is achieved without human assistance during training, beyond the diagnosis labels themselves. Localization Performance. Obtaining high performance in terms of whole slide classification is well and good, but it is not worth much without an interpretable which can be used by pathologists to aid their diagnosis. For example, the MeanPool baseline aggregation approach provides no information during inference from which one could derive tumor locations in the WSI: all locality information is lost with the aggregation. With MaxPool, one at least retains some information via the tile locations which provide each maximum aggregate feature. For CHOWDER, we propose the use of the full set of outputs from the convolutional feature embedding layer. These are then sorted and thresholded according to value τ such that tiles with an embedded value larger than τ are classified as diseased and those with lower values are classified as healthy. We show an example of disease localization produced by CHOWDER in FIG4. Here, we see that CHOWDER is able to very accurately localize the tumorous region in the WSI even though it has only been trained using global slide-wide labels and without any local annotations. While some potential false detections occur outside of the tumor region, we see that the strongest response occurs within the tumor region itself, and follows the border regions nicely. We present further localization in Appendix A.We also present FROC scores for CHOWDER in Table 2 as compared to the leader board . Here, we obtain comparable to the 18 th rank. However, this performance is incredibly significant as all other approaches were making use of tile-level classification in order to train their segmentation techniques. We also show the FROC curve in FIG3 We have shown that using state-of-the-art techniques from MIL in computer vision, such as the top instance and negative evidence approach of BID9, one can construct an effective technique for diagnosis prediction and disease location for WSI in histopathology without the need Table 2: Final leader boards for Camelyon-16 competition. All competition methods had access to the full set of strong annotations for training their models. In contrast, our proposed approach only utilizes image-wide diagnosis levels and obtains comparable performance as top-10 methods. for expensive localized annotations produced by expert pathologists. By removing this requirement, we hope to accelerate the production of computer-assistance tools for pathologists to greatly improve the turn-around time in pathology labs and help surgeons and oncologists make rapid and effective patient care decisions. This also opens the way to tackle problems where expert pathologists may not know precisely where relevant tissue is located within the slide image, for instance for prognosis estimation or prediction of drug response tasks. The ability of our approach to discover associated regions of interest without prior localized annotations hence appears as a novel discovery approach for the field of pathology. Moreover, using the suggested localization from CHOWDER, one may considerably speed up the process of obtaining ground-truth localized annotations. A number of improvements can be made in the CHOWDER method, especially in the production of disease localization maps. As presented, we use the raw values from convolutional embedding layer, which means that the resolution of the produced disease localization map is fixed to that of the sampled tiles. However, one could also sample overlapping tiles and then use a data fusion technique to generate a final localization map. Additionally, as a variety of annotations may be available, CHOWDER could be extended to the case of heterogenous annotation, e.g. some slides with expert-produced localized annotations and those with only whole-slide annotations. A FURTHER Figure 5: Visualization of metastasis detection on test image 2 of the Camelyon-16 dataset using our proposed approach. Left: Full WSI at zoom level 6 with ground truth annotation of metastases shown via black border. Tiles with positive feature embeddings are colored from white to red according to their magnitude, with red representing the largest magnitude. Right: Detail of metastases at zoom level 2 overlaid with classification output of our proposed approach. Here, the output of all tested tiles are shown and colored according to their value, from blue to white to red, with blue representing the most negative values, and red the most positive. Tiles without color were not included when randomly selecting tiles for inference. Figure 6: Visualization of metastasis detection on test image 92 of the Camelyon-16 dataset using our proposed approach. Left: Full WSI at zoom level 6 with ground truth annotation of metastases shown via black border. Tiles with positive feature embeddings are colored from white to red according to their magnitude, with red representing the largest magnitude. Right: Detail of metastases at zoom level 2 overlaid with classification output of our proposed approach. Here, the output of all tested tiles are shown and colored according to their value, from blue to white to red, with blue representing the most negative values, and red the most positive. Tiles without color were not included when randomly selecting tiles for inference.

Represent the short summary for finding the corresponding scientific article (long)

We propose a weakly supervised learning method for the classification and localization of cancers in extremely high resolution histopathology whole slide images using only image-wide labels.

Represent the TLDR for finding the corresponding scientific article