Question: Represent the article for finding a single-sentence TLDR

Question: We present an end-to-end trainable approach for optical character recognition (OCR) on printed documents. It is based on predicting a two-dimensional character grid ('chargrid') representation of a document image as a semantic segmentation task. To identify individual character instances from the chargrid, we regard characters as objects and use object detection techniques from computer vision. We demonstrate experimentally that our method outperforms previous state-of-the-art approaches in accuracy while being easily parallelizable on GPU (thereby being significantly faster), as well as easier to train. Optical Character Recognition (OCR) on documents is an age-old problem for which numerous open-source (e.g. ) as well as proprietary solutions exist. Especially in the sub-domain of printed documents, it is often regarded as being solved. However, current state-of-the-art document-level OCR solutions (as far as the published research goes) consist of a complicated pipeline of steps, each one either a hand-optimized heuristic or requiring intermediate data and annotations to train. Deep neural networks have been proven very successful in object detection tasks. In this work, we build on top of these developments and treat OCR as a semantic segmentation and object detection task for detecting and recognizing character instances on a page. 2 We introduce a new end-toend trainable OCR pipeline for (but not limited to) printed documents that is based on deep fully convolutional neural networks. Our main contribution is to frame the OCR problem as an ultra-dense instance-segmentation task for characters over the full input document image. We do not rely on any pre-processing stages like binarization, deskewing, layout analysis. Instead, our model learns directly from the raw document pixel data. At the core of our method, we predict a chargrid representation of the input document -a 1-hot encoded grid of characters. Thus, we call our method Chargrid-OCR. Additionally, we introduce two novel post-processing steps, both of which are crucial to performing fast and accurate dense OCR. We show that our method can outperform line-based pipelines like e.g. Tesseract 4 that rely on a combination of deep convolutional and recurrent networks with CTC loss while being significantly simpler to train. * Equal contribution 2 A related task of recognizing text in natural images, referred to as Scene Text Recognition (STR), has been faster in adopting techniques from object detection in computer vision. However, compared to STR, document OCR deals with much denser text and very high accuracy requirements. 2 Chargrid-OCR: OCR as an ultra-dense object detection task Chargrid-OCR method is a lexicon-free (only character-based), end-to-end trainable approach for OCR. Given a document image, chargrid-OCR predicts character segmentation mask together with object bounding boxes for characters in one single step (see Fig 1). Both, semantic segmentation and object detection are common tasks in computer vision, e.g.. The character segmentation mask classifies each pixel into a character class and the character bounding box detects a bounding box around each character. Both, our semantic segmentation and box detection (sub-)networks are fully convolutional and consist of only a single stage (like and unlike ). Being single-stage is especially important as there may be thousands of characters (i.e. objects) on a single page which yields an ultra-dense object detection task. The chargrid representation of a document image maps each pixel that is occupied by a single character on the input document to a unique index that corresponds to that character. Given an input document, our model predicts the chargrid representation of the complete document. This is accomplished by using the chargrid as target for a semantic segmentation network. Since the chargrid does not allow one to delineate character instances, we further use class agnostic object detection to predict individual character boxes. We are thus solving an instance segmentation task. Concretely, the input to our model is an image with text, e.g. a scanned document. The output is a segmentation mask (chargrid) and a set of bounding boxes. The segmentation mask, S, classifies each pixel in the input image into characters (Fig. 1). The bounding boxes are predicted in a similar way as standard object detection methods with (i) a box mask (B c) whose confidence denotes the presence of a box at that pixel, (ii) box centers (X c, Y c), which denote the offset from the location of the predicting pixel to the center of the box and (iii) the box widths and the heights (W c, H c). Finally, for grouping characters into words, we also predict offsets to word centers (X w, Y w). The architecture of the model is based on a fully-convolutional encoder-decoder structure, with two decoders (one for semantic segmentation, one for bounding box detection) branching out of the common encoder. Fig. 1 illustrates the architecture with an example input and its corresponding outputs. The model is trained using categorical cross-entropy for the segmentation outputs (S, B c) and using Huber loss for the regression output (X c, Y c, W c, H c, X w, Y w). The character candidate boxes are those that have confidence surpassing a certain threshold (e.g. 50%) in the box mask, B c. This gives multiple boxes around the same character. In order to delete redundantly predicted box proposals of the same character instance, non-maximum suppression (NMS) is applied. However, in our scenario, the number of proposals can be of the order of 10 5. As NMS runs in quadratic time, this can become a computational bottleneck. To speed up the process, we introduce a preliminary step before NMS, which we call GraphCore. Recall that each candidate pixel predicts the offset to the center of its box. We construct a directed graph where each vertex is a candidate pixel and we add a directed edge going from pixel A to pixel B if pixel A predicts pixel B as the center of its bounding box. By taking the k-core, with k=1, of the ing graph (i.e. only the loops in the graph, which can be done efficiently in linear time) only pixels towards the center of a bounding box (typically, one or two candidate boxes per character) are selected as candidates for NMS. Another necessary post-processing step is to construct word boxes from character boxes. To do so, we cluster characters together based on their predicted word centers, which even allows us to predict rotated words. For each document input image, we require ground-truth data in the form of character bounding boxes. (WIKI dataset) We generated a dataset by synthetically rendering pages in A4 format using English Wikipedia content and applied common data augmentation. We generated 66,486 pages; the page layout and font specifications (type, height and color) were sampled randomly. This enabled to synthesize input images and perfect ground truth labels. (EDGAR dataset) We converted a vast set of publicly available scanned financial reports into images and sampled 42,918 pages with a non-repetitive layouts. We processed the images with Tesseract4 and thereby obtained noisy (i.e. including OCR errors) ground truth. We evaluated the model on both, a held-out dataset from our training data (EDGAR 77 : 77 pages and 22,521 words ; Wiki 200 : 200 pages; 76,738 words) as well as benchmark OCR dataset, i.e. Business letters (179 pages; 48,639 words; ) and UNLV. (383 pages; 133,245 words; ). We use Word Recognition Rate to measure the accuracy of OCR which can be computed by the Nm Nu+Nm, with N m and N u being the number of matched and unmatched words respectively. A predicted word matches a ground truth if and only if contents agree (i.e. identical string) and they intersect (IoU > 1%). If multiple predictions match the same ground truth, only one prediction is considered a match. The remaining unmatched predictions and unmatched ground truth words are added to obtain N u. We train various versions of our new model on the datasets described in Sec. 3.1 and report in Table 1. As baseline, we compare against Tesseract, v3 and v4, with v4 (released Oct 2018) being the publicly available state-of-the-art. 3 Tesseract v4 comes with an LSTM-based line recognition engine and achieves much higher accuracy than v3. Unfortunately, re-training Tesseract on our datasets is not possible due to needing intermediate annotations to train Tesseract. We, therefore, use off-the-shelf Tesseract without any retraining or fine tuning. However, we use test data that are domain-independent from training and/or validation data and serve as an indicator for model generalizability. Our baseline is denoted by "ChargridOCR-32", consists of 32 convolutional base channels (C = 32 in Fig. 1), and was trained on the Wiki dataset. This model outperforms Tesseract v3, but not Tesseract v4. The same model trained on Wiki+EDGAR is competitive with, and typically superior to, Tesseract4 on validation and test data. Finally, a model with twice as many convolutional channels "ChargridOCR-64" and thus with higher capacity trained on Wiki+EDGAR outperforms Tesseract on all datasets, however with significant computational overhead (rightmost column in Tab. 1). In Fig. 2 We presented a new end-to-end trainable optical character recognition pipeline that is based on state-of-the-art computer vision approaches using object detection and semantic segmentation. Our pipeline is significantly simpler compared to other sequential and line-based approaches, especially those used for document-level optical character recognition such as Tesseract 4. We empirically show that our model outperforms Tesseract 4 on a number of diverse evaluation datasets by a large margin both in terms of accuracy and run-time.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

End-to-end trainable Optical Character Recognition on printed documents; we achieve state-of-the-art results, beating Tesseract4 on benchmark datasets both in terms of accuracy and runtime, using a purely computer vision based approach.

represent the short summary for finding the corresponding scientific article (long):
For instance, <<This paper proposes a simple procedure for evaluating compositional structure in learned representations, and uses the procedure to explore the role of compositionality in four learning problems.>>  to "Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs’ learned representations. While the assessment of compositionality in languages has received significant attention in linguistics and adjacent fields, the machine learning literature lacks general-purpose tools for producing graded measurements of compositional structure in more general (e.g. vector-valued) representation spaces. We describe a procedure for evaluating compositionality by measuring how well the true representation-producing model can be approximated by a model that explicitly composes a collection of inferred representational primitives. We use the procedure to provide formal and empirical characterizations of compositional structure in a variety of settings, exploring the relationship between compositionality and learning dynamics, human judgments, representational similarity, and generalization. Figure 1: Representations arising from a communication game. In this game, an observation (b) is presented to a learned speaker model (c), which encodes it as a discrete character sequence (d) to be consumed by a listener model for some downstream task. The space of inputs has known compositional structure (a). We want to measure the extent to which this structure is reflected (perhaps imperfectly) in the structure of the learned codes. The success of modern representation learning techniques has been accompanied by an interest in understanding the structure of learned representations. One feature shared by many humandesigned representation systems is compositionality: the capacity to represent complex concepts (from objects to procedures to beliefs) by combining simple parts BID18. While many machine learning approaches make use of human-designed compositional analyses for representation and prediction BID44 BID16, it is also natural to ask whether (and how) compositionality arises in learning problems where compositional structure has not been built in from the start. Consider the example in Figure 1, which shows a hypothetical character-based encoding scheme learned for a simple communication task (similar to the one studied by). Is this encoding scheme compositional? That is, to what extent can we analyze the agents' messages as being built from smaller pieces (e.g. pieces xx meaning blue and bb meaning triangle)?A large body of work, from early experiments on language evolution to recent deep learning models BID24 ), aims to answer questions like this one. But existing solutions rely on manual (and often subjective) analysis of model outputs BID32, or at best automated procedures tailored to the specifics of individual problem domains BID7. They are difficult to compare and difficult to apply systematically. We are left with a need for a standard, formal, automatable and quantitative technique for evaluating claims about compositional structure in learned representations. The present work aims at first steps toward meeting that need. We focus on an oracle setting where the compositional structure of model inputs is known, and where the only question is whether this structure is reflected in model outputs. This oracle evaluation paradigm covers most of the existing representation learning problems in which compositionality has been studied. The first contribution of this paper is a simple formal framework for measuring how well a collection of representations (discrete-or continuous-valued) reflects an oracle compositional analysis of model inputs. We propose an evaluation metric called TRE, which provides graded judgments of compositionality for a given set of (input, representation) pairs. The core of our proposal is to treat a set of primitive meaning representations as hidden, and optimize over them to find an explicitly compositional model that approximates the true model as well as possible. For example, if the compositional structure that describes an object is a simple conjunction of attributes, we can search for a collection of "attribute vectors" that sum together to produce the observed object representations; if it is a sparse combination of (attribute, value) pairs we can additionally search for "value vectors" and parameters of a binding operation; and so on for more complex compositions. Having developed a tool for assessing the compositionality of representations, the second contribution of this paper is a survey of applications. We present experiments and analyses aimed at answering four questions about the relationship between compositionality and learning:• How does compositionality of representations evolve in relation to other measurable model properties over the course of the learning process? (Section 4) • How well does compositionality of representations track human judgments about the compositionality of model inputs? (Section 5) • How does compositionality constrain distances between representations, and how does TRE relate to other methods that analyze representations based on similarity? (Section 6) • Are compositional representations necessary for generalization to out-of-distribution inputs?(Section 7)We conclude with a discussion of possible applications and generalizations of TRE-based analysis. Arguments about whether distributed (and other non-symbolic) representations could model compositional phenomena were a staple of 1980s-era connectionist-classicist debates. BID42 provides an overview of this discussion and its relation to learnability, as well as a concrete implementation of a compositional encoding scheme with distributed representations. Since then, numerous other approaches for compositional representation learning have been proposed, with BID30 BID43 and without BID15 BID22 ) the scaffolding of explicit composition operations built into the model. The main experimental question is thus when and how compositionality arises "from scratch" in the latter class of models. In order to answer this question it is first necessary to determine whether compositional structure is present at all. Most existing proposals come from linguistics and and philosophy, and offer evaluations of compositionality targeted at analysis of formal and natural languages BID8 BID28. Techniques from this literature are specialized to the details of linguistic representations-particularly the algebraic structure of grammars BID31. It is not straightforward to apply these techniques in more general settings, particularly those featuring non-string-valued representation spaces. We are not aware of existing work that describes a procedure suitable for answering questions about compositionality in the general case. Machine learning research has responded to this absence in several ways. One class of evaluations BID32 BID11 derives judgments from ad-hoc manual analyses of representation spaces. These analyses provide insight into the organization of representations but are time-consuming and non-reproducible. Another class of evaluations BID6 BID0 BID4 ) exploits task-specific structure (e.g. the ability to elicit pairs of representations known to feature particular relationships) to give evidence of compositionality. Our work aims to provide a standard and scalable alternative to these model-and task-specific evaluations. Other authors refrain from measuring compositionality directly, and instead base analysis on measurement of related phenomena, for which more standardized evaluations exist. Examples include correlation between representation similarity and similarity of oracle compositional analyses BID7 and generalization to structurally novel inputs BID26. Our approach makes it possible to examine the circumstances under which these surrogate measures in fact track stricter notions of compositionality; similarity is discussed in Sec. 6 and generalization in Sec. 7.A long line of work in natural language processing BID13 BID2 BID12 BID19 focuses on learning composition functions to produce distributed representations of phrases and sentences-that is, for purposes of modeling rather than evaluation. We use one experiment from this literature to validate our own approach (Section 5). On the whole, we view work on compositional representation learning in NLP as complementary to the framework presented here: our approach is agnostic to the particular choice of composition function, and the aforementioned references provide well-motivated choices suitable for evaluating data from language and other sources. Indeed, one view of the present work is simply as a demonstration that we can take existing NLP techniques for compositional representation learning, fit them to representations produced by other models (even in non-linguistic settings), and view the ing training loss as a measure of the compositionality of the representation system in question. Consider again the communication task depicted in Figure 1. Here, a speaker model observes a target object described by a feature vector. The speaker sends a message to a listener model, which uses the message to complete a downstream task-for example, identifying the referent from a collection of distractors based on the content of the message BID17 ). Messages produced by the speaker model serve as representations of input objects; we want to know if these representations are compositional. Crucially, we may already know something about the structure of the inputs themselves. In this example, inputs can be identified via composition of categorical shape and color attributes. How might we determine whether this oracle analysis of input structure is reflected in the structure of representations? This section proposes an automated procedure for answering the question. Representations A representation learning problem is defined by a dataset X of observations x (Figure 1b); a space Θ of representations θ (Figure 1d); and a model f: X → Θ (Figure 1c). We assume that the representations produced by f are used in a larger system to accomplish some concrete task, the details of which are not important for our analysis. Derivations The technique we propose additionally assumes we have prior knowledge about the compositional structure of inputs. In particular, we assume that inputs can be labeled with treestructured derivations d (Figure 1a), defined by a finite set D 0 of primitives and a binary bracketing operation ·, ·, such that if DISPLAYFORM0 Compositionality In intuitive terms, the representations computed by f are compositional if each f (x) is determined by the structure of D(x). Most discussions of compositionality, following BID31, make this precise by defining a composition operation θ a * θ b → θ in the space of representations. Then the model f is compositional if it is a homomorphism from inputs to representations: we require that for any x with DISPLAYFORM1 In the linguistic contexts for which this definition was originally proposed, it is straightforward to apply. Inputs x are natural language strings. Their associated derivations D(x) are syntax trees, and composition of derivations is syntactic composition. Representations θ are logical representations of meaning (for an overview see BID47 . To argue that a particular fragment of language is compositional, it is sufficient to exhibit a lexicon D 0 mapping words to their associated meaning representations, and a grammar for composing meanings where licensed by derivations. Algorithms for learning grammars and lexicons from data are a mainstay of semantic parsing approaches to language understanding problems like question answering and instruction following BID49 BID9 BID1 .But for questions of compositionality involving more general representation spaces and more general analyses, the above definition presents two difficulties: In the absence of a clearly-defined syntax of the kind available in natural language, how do we identify lexicon entries: the primitive parts from which representations are constructed? What do we do with languages like the one in Figure 1d, which seem to exhibit some kind of regular structure, but for which the homomorphism condition given in Equation 1 cannot be made to hold exactly?Consider again the example in Figure 1. The oracle derivations tell us to identify primitive representations for dark, blue, green, square, and triangle. The derivations then suggest a process for composing these primitives (e.g. via string concatenation) to produce full representations. The speaker model is compositional (in the sense of Equation 1) as long as there is some assignment of representations to primitives such that for each model input, composing primitive representations according to the oracle derivation reproduces the speaker's prediction. In Figure 1 there is no assignment of strings to primitives that reproduces model predictions exactly. But predictions can be reproduced approximately-by taking xx to mean blue, aa to mean square, etc. The quality of the approximation serves as a measure of the compositionality of the true predictor: predictors that are mostly compositional but for a few exceptions, or compositional but for the addition of some noise, will be well-approximated on average, while arbitrary mappings from inputs to representations will not. This suggests that we should measure compositionality by searching for representations that allow an explicitly compositional model to approximate the true f as closely as possible. We define our evaluation procedure as follows:Tree Reconstruction Error (TRE)First choose: DISPLAYFORM2, a compositional approximation to f with parameters η, as: DISPLAYFORM3 f η has one parameter vector η i for every d i in D 0; these vectors are members of the representation space Θ.Given a dataset X of inputs x i with derivations d i = D(x i), compute: DISPLAYFORM4 Then we can define datum-and dataset-level evaluation metrics: DISPLAYFORM5 DISPLAYFORM6 TRE and compositionality How well does the evaluation metric TRE(X) capture the intuition behind Equation 1? The definition above uses parameters η i to witness the constructability of representations from parts, in this case by explicitly optimizing over those parts rather than taking them to be given by f. Each term in Equation 2 is analogous to an instance of Equation 1, measuring how wellf η * (x i), the best compositional prediction, matches the true model prediction f (x i). In the case of models that are homomorphisms in the sense of Equation 1, TRE reduces to the familiar case: DISPLAYFORM7 Learnable composition operators The definition of TRE leaves the choice of δ and * up to the evaluator. Indeed, if the exact form of the composition function is not known a priori, it is natural to define * with free parameters (as in e.g. BID2, treat these as another learned part off, and optimize them jointly with the η i . However, some care must be taken when choosing * (especially when learning it) to avoid trivial solutions:Remark 2. Suppose D is injective; that is, every x ∈ X is assigned a unique derivation. Then there is always some * that achieves TRE(X) = 0: DISPLAYFORM8 as in the preceding definition, and setf = f.In other words, some pre-commitment to a restricted composition function is essentially inevitable: if we allow the evaluation procedure to select an arbitrary composition function, the will be trivial. This paper features experiments with * in both a fixed functional form and a learned parametric one. Implementation details For models with continuous Θ and differentiable δ and *, TRE(X) is also differentiable. Equation 2 can be solved using gradient descent. We use this strategy in Sections 4 and 5. For discrete Θ, it may be possible to find a continuous relaxation with respect to which δ(θ, ·) and * are differentiable, and gradient descent again employed. We use this strategy in Section 7 (discussed further there). An implementation of an SGD-based TRE solver is provided in the accompanying software release. For other problems, task-specific optimizers (e.g. machine translation alignment models; BID4 or general-purpose discrete optimization toolkits can be applied to Equation 2.The remainder of the paper highlights ways of using TRE to answer questions about compositionality that arise in machine learning problems of various kinds. We begin by studying the relationship between compositionality and learning dynamics, focusing on the information bottleneck theory of representation learning proposed by BID45 . This framework proposes that learning in deep models consists of an error minimization phase followed by a compression phase, and that compression is characterized by a decrease in the mutual information between inputs and their computed representations. We investigate the hypothesis that the compression phase finds a compositional representation of the input distribution, isolating decision-relevant attributes and discarding irrelevant information. FIG1). We predict classifiers in a meta-learning framework BID39 BID37: for each sub-task, the learner is presented with two images corresponding to some compositional visual concept (e.g. "digit 8 on a black " or "green with heavy stroke") and must determine whether a held-out image is an example of the same visual concept. Given example images x 1 and x 2, a test image x *, and label y *, the model computes: DISPLAYFORM0 We use θ as the representation of a classifier for analysis. The model is trained to minimize the logistic loss between logitsŷ and ground-truth labels y *. More details are given in Appendix A.Compositional structure Visual concepts used in this task are all single attributes or conjunctions of attributes; i.e. their associated derivations are of the form attr or attr 1, attr 2. Attributes include color, digit color, digit identity and stroke type. The composition function * is addition and the distance δ(θ, θ) is cosine similarity 1 − θ θ /(θ θ).Evaluation The training dataset consists of 9000 image triplets, evenly balanced between positive and negative classes, with a validation set of 500 examples. At convergence, the model achieves validation accuracy of 75.2% on average over ten training runs. (Perfect accuracy is not possible because the true classifier is not fully determined by two training examples). We explore the relationship between the information bottleneck and compositionality by comparing TRE(X) to the mutual information I(θ; x) between representations and inputs over the course of training. Both quantities are computed on the validation set, calculating TRE(X) as described in Section 3 and I(θ; X) as described in BID41. (For discussion of limitations of this approach to computing mutual information between inputs and representations, see .) FIG2 shows the relationship between TRE(X) and I(θ; X). Recall that small TRE is indicative of a high degree of compositionality. It can be seen that both mutual information and reconstruction error are initially low (because representations initially encode little about distinctions between inputs). Both increase over the course of training, and decrease together after mutual information reaches a maximum FIG2. This pattern holds if we plot values from multiple training runs at the same time FIG2 ), or if we consider only the postulated compression phase FIG2 ). These are consistent with the hypothesis that compression in the information bottleneck framework is associated with the discovery of compositional representations. Next we investigate a more conventional representation learning task. High-dimensional embeddings of words and phrases are useful for many natural language processing applications BID46, and many techniques exist to learn them from unlabeled text BID14 BID29. The question we wish to explore is not whether phrase vectors are compositional in aggregate, but rather how compositional individual phrase representations are. Our hypothesis is that bigrams whose representations have low TRE are those whose meaning is essentially compositional, and well-explained by the constituent words, while bigrams with large reconstruction error will correspond to non-compositional multi-word expressions BID33 ).This task is already well-studied in the natural language processing literature BID36, and the analysis we present differs only in the use of TRE to search for atomic representations rather than taking them to be given by pre-trained word representations. Our goal is to validate our approach in a language processing context, and show how existing work on compositionality (and representations of natural language in particular) fit into the more general framework proposed in the current paper. We train embeddings for words and bigrams using the CBOW objective of BID29 using the implementation provided in FastText BID5 with 100-dimensional vectors and a context size of 5. Vectors are estimated from a 250M-word subset of the Gigaword dataset BID34. More details are provided in Appendix A.Compositional structure We want to know how close phrase embeddings are to the composition of their constituent word embeddings. We define derivations for words and phrases in the natural way: single words w have primitive derivations d = w; bigrams w 1 w 2 have derivations of the form w 1, w 2. The composition function is again vector addition and distance is cosine distance. (Future work might explore learned composition functions as in e.g. BID21, for future work.) We compare bigram-level judgments of compositionality computed by TRE with a dataset of human judgments about noun-noun compounds BID35. In this dataset, humans rate bigrams as compositional on a scale from 0 to 5, with highly conventionalized phrases like gravy train assigned low scores and graduate student assigned high ones. Results We reproduce the of BID36 within the tree reconstruction error framework: for a given x, TRE(x) is anticorrelated with human judgments of compositionality (ρ = −0.34, p < 0.01). Collocations rated "most compositional" by our approach (i.e. with lowest TRE) are: application form, polo shirt, research project; words rated "least compositional" are fine line, lip service, and nest egg. The next section aims at providing a formal, rather than experimental, characterization of the relationship between TRE and another perspective on the analysis of representations with help from oracle derivations. BID7 introduce a notion of topographic similarity, arguing that a learned representation captures relevant domain structure if distances between learned representations are correlated with distances between their associated derivations. This can be viewed as providing a weak form of evidence for compositionality-if the distance function rewards pairs of representations that share overlapping substructure (as might be the case with e.g. string edit distance), edit distance will be expected to correlate with some notion of derivational similarity.In this section we aim to clarify the relationship between the two evaluations. To do this we first need to equip the space of derivations described in Section 3 with a distance function. As the derivations considered in this paper are all tree-structured, it is natural to use a simple tree edit distance BID3 for this purpose. We claim the following: Proposition 1. Letf =f η * be an approximation to f estimated as in Equation 2, with all TRE(x) ≤ for some. Let ∆ be the tree edit distance (defined formally in Appendix B, Definition 2), and let δ be any distance on Θ satisfying the following properties: DISPLAYFORM0, where 0 is the identity element for *. DISPLAYFORM1 (This condition is satisfied by any translation-invariant metric.)Then ∆ is an approximate upper bound on δ: for any DISPLAYFORM2 In other words, representations cannot be much farther apart than the derivations that produce them. Proof is provided in Appendix B.We emphasize that small TRE is not a sufficient condition for topographic similarity as defined by BID7: very different derivations might be associated with the same representation (e.g. when representing arithmetic expressions by their ). But this does demonstrate that compositionality imposes some constraints on the inferences that can be drawn from similarity judgments between representations. In our final set of experiments, we investigate the relationship between compositionality and generalization. Here we focus on communication games like the one depicted in Figure 1 and in more detail in FIG3. As in the previous section, existing work argues for a relationship between compositionality and generalization, claiming that agents need compositional communication protocols to generalize to unseen referents BID26 BID11. Here we are able to evaluate this claim empirically by training a large number of agents from random initial conditions, measuring the compositional structure of the language that emerges, and seeing how this relates to their performance on both familiar and novel objects. A speaker model observes a pair of target objects, and sends a description of the objects (as a discrete code) to a listener model. The listener attempts to reconstruct the targets, receiving fractional reward for partially-correct predictions. Our experiment focuses on a reference game BID20. Two policies are trained: a speaker and a listener. The speaker observes pair of target objects represented with a feature vector. The speaker then sends a message (coded as a discrete character sequence) to the listener model. The listener observes this message and attempts to reconstruct the target objects by predicting a sequence of attribute sets. If all objects are predicted correctly, both the speaker and the listener receive a reward of 1 (partial credit is awarded for partly-correct objects; FIG3).Because the communication protocol is discrete, policies are jointly trained using a policy gradient objective BID48. The speaker and listener are implemented with RNNs; details are provided in Appendix A.Compositional structure Every target referent consists of two objects; each object has two attributes. The derivation associated with each communicative task thus has the tree structure attr 1a, attr 1b, attr 2a, attr 2b. We hold out a subset of these object pairs at training time to evaluate generalization: in each training run, 1/3 of possible reference candidates are never presented to the agent at training time. Where the previous examples involved a representation space of real embeddings, here representations are fixed-length discrete codes. Moreover, the derivations themselves have a more complicated semantics than in Sections 4 and 5: order matters, and a commutative operation like addition cannot capture the distinction between green, square, blue, triangle and green, triangle, blue, square. We thus need a different class of composition and distance operations. We represent each agent message as a sequence of one-hot vectors, and take the error function δ to be the 1 distance between vectors. The composition function has the form: DISPLAYFORM0 with free composition parameters η * = {A, B} in Equation 2. These matrices can redistribute the tokens in θ and θ across different positions of the input string, but cannot affect the choice of the tokens themselves; this makes it possible to model non-commutative aspects of string production. To (b) However, compositional languages also exhibit lower absolute performance (r = 0.57, p < 1e−9). Both facts remain true even if we restrict analysis to "successful" training runs in which agents achieve a reward > 0.5 on held-out referents (r = 0.6, p < 1e−3 and r = 0.38, p < 0.05 respectively). Figure 6: Fragment of languages ing from two multiagent training runs. In the first section, the left column shows the target referent, while the remaining columns show the message generated by speaker in the given training run after observing the referent. The two languages have substantially different TRE, but induce similar listener performance (Train and Test reward).compute TRE via gradient descent, we allow the elements of D 0 to be arbitrary vectors (intuitively assigning fractional token counts to string indices) rather than restricting them to one-hot indicators. With this change, both δ and * have subgradients and can be optimized using the same procedure as in preceding sections. Results We train 100 speaker-listener pairs with random initial parameters and measure their performance on both training and test sets. Our suggest a more nuanced view of the relationship between compositionality and generalization than has been argued in the existing literature. TRE is significantly correlated with generalization error (measured as the difference between training accuracies, FIG4). However, TRE is also significantly correlated with absolute model reward (FIG4)-"compositional" languages more often from poor communication strategies than successful ones. This is largely a consequence of the fact that many languages with low TRE correspond to trivial strategies (for example, one in which the speaker sends the same message regardless of its observation) that in poor overall performance. Moreover, despite the correlation between TRE and generalization error, low TRE is by no means a necessary condition for good generalization. We can use our technique to automatically mine a collection of training runs for languages that achieve good generalization performance at both low and high levels of compositionality. Examples of such languages are shown in Figure 6. We have introduced a new evaluation method called TRE for generating graded judgments about compositional structure in representation learning problems where the structure of the observations is understood. TRE infers a set of primitive meaning representations that, when composed, approximate the observed representations, then measures the quality of this approximation. We have applied TRE-based analysis to four different problems in representation learning, relating compositionality to learning dynamics, linguistic compositionality, similarity and generalization. Many interesting questions regarding compositionality and representation learning remain open. The most immediate is how to generalize TRE to the setting where oracle derivations are not available; in this case Equation 2 must be solved jointly with an unsupervised grammar induction problem BID25. Beyond this, it is our hope that this line of research opens up two different kinds of new work: better understanding of existing machine learning models, by providing a new set of tools for understanding their representational capacity; and better understanding of problems, by better understanding the kinds of data distributions and loss functions that give rise to compositionalor non-compositional representations of observations. Code and data for all experiments in this paper are provided at https://github.com/jacobandreas/tre. Thanks to Daniel Fried and David Gaddy for feedback on an early draft of this paper. The author was supported by a Facebook Graduate Fellowship at the time of writing. Few-shot classification The CNN has the following form:Conv(out=6, kernel=5) ReLU MaxPool(kernel=2) Conv(out=16, kernel=5) ReLU MaxPool(kernel=2) Linear(out=128) ReLU Linear(out=64) ReLUThe model is trained using ADAM BID23 ) with a learning rate of.001 and a batch size of 128. Training is ended when the model stops improving on a held-out set. Word embeddings We train FastText BID5 on the first 250 million words of the NYT section of Gigaword BID34. To acquire bigram representations, we pre-process this dataset so that each occurrence of a bigram from the BID35 dataset is treated as a single word for purposes of estimating word vectors. Communication The encoder and decoder RNNs both use gated recurrent units BID10 with embeddings and hidden states of size 256. The size of the discrete vocabulary is set to 16 and the maximum message length to 4. Training uses a policy gradient objective with a scalar baseline set to the running average reward; this is optimized using ADAM BID23 ) with a learning rate of.001 and a batch size of 256. Each model is trained for 500 steps. Models are trained by sampling from the decoder's output distribution, but greedy decoding is used to evaluate performance and produce Figure 6. First, some definitions:Definition 1. The size of a derivation is given by: DISPLAYFORM0 Definition 2. The tree edit distance between derivations is defined by: Proof. For d ∈ D 0 this follows immediately from Condition 2 in the proposition. For composed derivations it follows from Condition 3 taking θ k = θ = 0 and induction on |d|. DISPLAYFORM1"

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam/AdamW. Additionally, NovoGrad is robust to the choice of learning rate and weight initialization, works well in a large batch setting, and has two times smaller memory footprint than Adam. The most popular algorithms for training Neural Networks (NNs) are Stochastic Gradient Descent (SGD) with momentum and Adam . SGD with momentum is the preferred algorithm for computer vision, while Adam is the most commonly used for natural language processing (NLP) and speech problems. Compared to SGD, Adam is perceived as safer and more robust to weight initialization and learning rate. However, Adam has certain drawbacks. First, as noted in the original paper , the second moment can vanish or explode, especially during the initial phase of training. To alleviate this problem, a learning rate (LR) warmup is typically used. Adam often leads to solutions that generalize worse than SGD , and to improve Adam regularization, proposed AdamW with decoupled weight decay. Our motivation for this work was to find an algorithm which: performs equally well for image classification, speech recognition, machine translation, and language modeling, is robust to learning rate and weight initialization, has strong regularization properties. We start with Adam, and then replace the element-wise second moment with the layer-wise moment, compute the first moment using gradients normalized by layer-wise second moment, and decouple weight decay (similar to AdamW) from normalized gradients. The ing algorithm, NovoGrad, combines SGD's and Adam's strengths. We applied NovoGrad to a variety of large scale problems -image classification, neural machine translation, language modeling, and speech recognition -and found that in all cases, it performs as well or better than Adam/AdamW and SGD with momentum. NovoGrad belongs to the family of Stochastic Normalized Gradient Descent (SNGD) optimizers . SNGD uses only the direction of the stochastic gradient to update the weights, and the step size does not depend on the magnitude of that gradient. proved that the direction of the gradient was sufficient for convergence. Ignoring the gradient magnitude makes SNGD robust to vanishing and exploding gradients. SNGD with layer-wise gradient normalization was introduced by. The method scales up small gradients, while keeping large gradients unchanged: Similar to Adam, the weights are updated with the 1 st moment re-scaled by the 2 nd moment: Adaptive methods like Adam generalize worse than SGD with momentum as was shown in. For example, proposed to use Adam during the initial stage only and then switch to SGD. suggested to improve Adam regularization by limiting the factor 1 √ vt to a certain range: limiting from above helps to decrease the training loss while limiting from below helps to generalize better. showed that Adam's weak regularization is due to the fact that the 2 nd moment normalization effectively turns off L2-regularization. They proposed AdamW, which decouples the weight decay d · w t from the gradient and uses it directly in the weight update: Adam needs to store the 2 nd moment, and this doubles the optimizer memory compared to SGD with momentum. This affects large models like GPT-2 with 1.5 billion parameters. proposed the AdaFactor algorithm, which replaced the full 2 nd moment with moving averages of the row and column sums of the squared gradients. For a layer defined by an n × m matrix, this would reduce memory from O(n × m) to O(n + m). NovoGrad consumes the same amount of memory as SGD with momentum. NovoGrad is based on 3 ideas: layer-wise 2 nd moments instead of 2 nd moment per each parameter, gradients normalization with layer-wise 2 nd moments, decoupled weight decay. Let g l t be the stochastic gradient for layer l at step t. NovoGrad first computes the layer-wise 2 nd moment v l t using the norm ||g l t ||: where 0 ≤ β 2 ≤ 1. We use much smaller β 2 than in Adam, usually in the range [0.2, 0.5]. The moment v l t is used to normalize the gradient g l t before calculating the first moment m l t. Similarly to AdamW, we decouple weight decay d · w t from the stochastic gradient, but we add it to normalized gradient before computing moment m Algorithm 1 NovoGrad Parameters: Initial learning rate λ 0, moments β 1, β 2, weight decay d, number of steps T Weight initialization: t = 0, Initialize w 0. Moment initialization: t = 1, for each layer l set v end for end while where 0 < β 1 < 1 is the momentum, typically in the same range as in SGD or Adam [0.9 − 0.95]. The first moment can be also computed via an exponential moving average in Adam-like style: Finally, weights are updated the same way as in SGD with momentum. Similar to Adam, one can construct a counter-example for NovoGrad in the stochastic convex optimization settings . However, the "AMS-Grad" fix for Adam can also be applied in this case to guarantee NovoGrad convergence: Following (Andrew M. ; Ian J.) we will use NovoGrad to train linear model composed of two linear layers w 1, w 2 without any non-linearity. The model y = (w 1 · w 2) · x should output 1 when x = 1. This model is linear with respect to the inputs, but it is non-linear with respect to the weights, since they are factorized into the product of layers' weights. Training the model is equivalent to the minimization of the loss L(w 1, w 2) = (w 1 ·w 2 −1), 2015). The loss is not convex, and its minima are located on the hyperbola w 1 w 2 = 1 (see Figure 1). Minima close to the points (−1, −1) and are good "flat" minima which generalize well. Minima close to the axes are "sharp" minima . We trained the model with SGD with momentum, Adam, AdamW, and NovoGrad, using the same fixed learning rate, 3 weight decay, and weights initialization. The model was trained for 500 steps. Figure 2 shows the training trajectory and the zoomed-out area near the final point. All algorithms behave in a similar way: first the trajectory goes to the curve w 2 = 1/w 1, and then follows the hyperbola towards or (−1, −1). During the first phase, training loss decreases, and during the second phase, generalization improves. SGD converges nicely toward but its trajectory is still slightly off of the optimal solution. Adam oscillates wildly around hyperbola w 2 = 1/w 1, while AdamW behaves much better since weight decay decoupling significantly reduces oscillations. NovoGrad is the most stable out of four algorithms. It exhibits better generalization and closely follows the minima curve because normalized gradients prevent trajectory from going far from it. We also found that NovoGrad is more robust than other algorithms to the choice of learning rate, weight decay, and weight initialization (see for details Appendix A). 4 Each model was trained on a single DGX-1 machine with 8 NVIDIA V100 GPUs with gradient accumulation used for large batch training. In all the experiments, NovoGrad performed on par or better than other algorithms. We used ResNet-50 v2 for ImageNet classification task . We trained this model with three optimizers: SGD with momentum (SGD), AdamW, and NovoGrad. All models have been trained with the batch size of 1024 for 100 epochs. We used quadratic LR decay for SGD with momentum and cosine decay for AdamW and NovoGrad. We could not find any training recipe for ResNet-50 with AdamW, so we report the best accuracy we achieved after extensive hyper-parameter search. We used only standard data augmentation methods: re-size, flip, random crop, and did not employ any additional training tricks . The single-crop validation accuracy for each algorithm is reported in Table 1. showed that large batch size is beneficial for SNGD convergence, which motivated us to explore NovoGrad for large batch training. We trained ResNet-50 v2 with batch sizes of 8K and 32K. To compare with the previous methods, we train the model for 90 epochs using cosine LR decay. To emulate a large batch, we used a mini-batch of 128 per GPU and accumulated gradients from several mini-batches before each weight update. To establish the baseline for NovoGrad training with batch 32K we first used the method similar to proposed in: scaling the learning rate linearly with the batch size and using LR warmup. This method gives top-1=75.09% and top-5=92.27%. We found that we get much better when we increase both the learning rate λ and the weight decay d to improve the regularization (see Table 2). For comparison, we took 3 methods, which use fixed batch size during training and do not modify the original model. All 3 methods employ SGD with momentum. The first method scales LR linearly with batch size and uses the LR warmup to stabilize the initial training phase. The second method combines warmup with Layer-wise Adaptive Rate Scaling (LARS) . The last method uses warmup and dynamic weight decay (WD). NovoGrad outperformed all other methods without using any additional techniques like LR warmup , dynamic weight decay, special batch normalization initialization, etc. Using warm-up (500 steps) we slightly improved top1 accuracy to 75.99% and top5 to 92.72%. We conducted experiments with Jasper-10x5 , a very deep convolutional neural acoustic model, on the LibriSpeech speech recognition task . Jasper was trained with SGD with momentum (SGD), Adam and NovoGrad for 400 epochs with a batch of 256, polynomial LR decay, and Layerwise Adaptive Rate Clipping (LARC). 5 We found that NovoGrad yields lower Word Error Rates (WER) comparing to SGD, especially for the long runs. The model and training parameters are described in. We trained Jasper10x5 with batch sizes of 512, 4K, 8K, 16K and 32K on LibriSpeech. In all cases, we trained the model for 400 epochs. For batch size up to 8K we scaled LR linearly with the batch size and used LR warmup. To scale batch to 16K and 32K we also increased weight decay (see Table 5). The batch 16K leads to WER comparable to the baseline. Batch 32K has higher WER due to the smaller number of training steps (9 weights updates per epoch). Figure 3 shows WER on dev-clean during training for different batch sizes. We trained Transformer-XL , the state-of-the-art LM architecture on the wordlevel WikiText-103 benchmark. For all the experiments we used a 16-layer All other hyperparameters were taken from the original Transformer-XL paper, the source code was based on a publicly available implementation. 6 Each configuration was trained for 12 billion tokens which is approximately 117 epochs and 366K training iterations. Figure 4 shows that NovoGrad exhibits a much smaller gap between training and validation perplexity compared to Adam, which in better performance on the test set. Longer training for 20B tokens does not lead to overfitting as the ing validation and test perplexities improve even further. We trained Transformer on WMT 2014 English-to-German benchmark. For all the experiments, we used a 12-layer Transformer-big model with 185M parameters (d model = 1024, d ff = 4096, h = 16) with the vocabulary of 8192 tokens based on joint source-target bytepair-encodings . For Adam and AdamW we used dropout of P drop = 0.3 and for NovoGrad we used P drop = 0.2. We trained all algorithms with mixed-precision for 100K steps (approximately 150 epochs) with a 4K steps warmup on batches of up to 490K source and target tokens obtained via gradient accummulation with cosine learning rate annealing. We did not use checkpoint averaging, all the are reported for the last checkpoint in the corresponding run. w 2 ) · x should output 1 when x = 1. The model is linear function of the inputs, but it is non-linear function of the weights, since they are factorized into the product of layers' weights. Training the model is equivalent to the minimization of the loss L(w 1, w 2) = (w 1 · w 2 − 1) 2. The loss is not convex, and its minima are located on the curve: w 1 w 2 = 1. Minima close to the points (−1, −1) and are good "flat" minima which generalize well. Minima close to axes are "sharp" minima with bad generalization (see ). The 2D-contour plot of the loss function shown on Figure 5. 2 of linear model with two layers. The loss functions has many global minima located on hyperbola w 2 = 1/w 1. Solutions near (−1, −1) and are good "flat" minima, and solutions near axes are "sharp" minima. We will study how the behavior of each algorithm depends on learning rate, weight decay and initialization. We will train the model with each optimizer for 500 steps using the same learning rate, weight decay, and weights initialization. To use the same learning rate for all optimizers, we will use the "gradient averaging" for NovoGrad. We will also use the version of SGD with "gradient averaging" (similar to Adam): m t = β · m t−1 + (1 − β) · g t. For fixed learning rate this SGD version is equivalent to the regular SGD with momentum. Training trajectories for the baseline (fixed learning rate 0.2, weight decay 0.1, and β 1 = 0.95, β 2 = 0.5.) are shown on the Figure 6. All algorithms first go to the curve w 2 = 1/w 1, and then slide along hyperbola towards or (−1, −1). SGD is slightly off with respect to the optimal solution. Adam oscillates wildly around line w 2 = 1/w 1. AdamW behaves better since weight decay decoupling significantly reduces osculations. NovoGrad is the most stable out of four algorithms, it also shows much better generalization than other algorithms and converges to closely following the minima curve. Next, we increased learning rate from 0.2 to 1.0 while keeping weight decay equal to 0. Similarly, when we increased weight decay from 0.1 to 0.5 while keeping learning rate 0.2, all algorithms except NovoGrad diverge, while NovoGrad demonstrates high robustness to the weight decay choice (see Figure 8). Finally, we started training from different initial point. SGD and NovoGrad are most robust with respect to the initialization, while AdamW diverge (see Figure 9). To summarize our experiments with linear neural network: NovoGrad is more robust than other algorithms to the choice of learning rate, weight decay, and weight initialization.

Represent the TLDR for finding the corresponding scientific article

NovoGrad -  an adaptive SGD method with layer-wise gradient normalization and decoupled weight decay.

represent the short summary for finding the corresponding scientific article (long):

Examples:
Provided: "A comparison and detailed analysis of various sentence embedding models through the real-world task of automatic summarization." Match: "Vector semantics, especially sentence vectors, have recently been used successfully in many areas of natural language processing. However, relatively little work has explored the internal structure and properties of spaces of sentence vectors. In this paper, we will explore the properties of sentence vectors by studying a particular real-world application: Automatic Summarization. In particular, we show that cosine similarity between sentence vectors and document vectors is strongly correlated with sentence importance and that vector semantics can identify and correct gaps between the sentences chosen so far and the document. In addition, we identify specific dimensions which are linked to effective summaries. To our knowledge, this is the first time specific dimensions of sentence embeddings have been connected to sentence properties. We also compare the features of different methods of sentence embeddings. Many of these insights have applications in uses of sentence embeddings far beyond summarization. Vector semantics have been growing in popularity for many other natural language processing applications. Vector semantics attempt to represent words as vectors in a high-dimensional space, where vectors which are close to each other have similar meanings. Various models of vector semantics have been proposed, such as LSA BID10, word2vec BID14, and GLOVE BID17, and these models have proved to be successful in other natural language processing applications. While these models work well for individual words, producing equivalent vectors for sentences or documents has proven to be more difficult. In recent years, a number of techniques for sentence embeddings have emerged. One promising method is paragraph vectors (Also known as Doc2Vec), described by BID12. The model behind paragraph vectors resembles that behind word2vec, except that a classifier uses an additional'paragraph vector' to predict words in a Skip-Gram model. Another model, skip-thoughts, attempts to extend the word2vec model in a different way BID9. The center of the skip-thought model is an encoder-decoder neural network. The , skip-thought vectors, achieve good performance on a wide variety of natural language tasks. Simpler approaches based on linear combinations of the word vectors have managed to achieve state-of-the-art for non-domain-specific tasks BID20. Arora et al. BID1 offer one particularly promising such approach, which was found to achieve equal or greater performance in some tasks than more complicated supervised learning methods. The large volume of news articles published every day motivates effective forms of automatic summarization. The most basic and most well-studied form of automatic summarization is sentence extraction, in which entire unmodified sentences are selected from the original document. These selected sentences are concatenated to form a short summary, which ideally contains the most important information from the original document while avoiding redundancy. Any practical summarization system would likely include other steps after sentence selection, such as sentence reordering or text-simplification. However, effective algorithms for these tasks already exist, and we are primarily concerned with sentence selection in this paper. We are primarily concerned with multi-document summarization algorithms. The nature of summarization requires some specific properties from sentence vectors, making it a practical real world example for which current sentence embedding methods can be easily used. In particular, news summarization may not require some of the nuances in meaning to be represented, as we are primarily concerned with the topic, not the meaning, of a sentence. In particular, news articles will rarely contain sentences expressing contradictory views on the same subject. For instance, our algorithm will rarely need to differentiate between the sentence embeddings for sentences such as "John loves Mary" and "Mary loves John", which have divergent meanings but the same words. This is in sharp contrast to typical testing cases for sentence embeddings, such as the detection of paraphrased sentences. Then, summarization gives us an opportunity to compare the effectiveness of different sentence embeddings in practice. State-of-the-art extractive summarization techniques have been achieved with a wide variety of methods. Here, we provide a short overview of some recent successful techniques techniques. Cao et al. BID2 used a recursive neural network, which operates on a parsing tree of a sentence, to rank sentences for summarization. BID3 successfully used a neural-network-based sentence extractor, which considered the document encodings and the previously selected sentences, as well as the current sentence in making its decisions. Parveen et al. BID16 used a graph-based approach, modeling a document as "a bipartite graph consisting of sentence and entity nodes".Ren et al. Ren et al. achieved state-of-the-art through a regression-based approach. A variety of engineered features are used as inputs into a regression model. The single highest rated sentence is selected, and the process is repeated. To our knowledge, no one has explored the use of modern sentence embedding methods, such as Paragraph Vectors or Skip-Thought vectors, in summarization. However, some work has been done on summarization using word2vec representations. BID6 presented a version of text summarization based on vector semantics. However, instead of using the embeddings of a word as determined by a larger corpus, they attempted to calculate the embedding of a word based off of analysis only on the document in question. In addition they used LSA instead of newer techniques such as word2vec. Kageback et al. BID8 used cosine similarity between the sum of word2vec embeddings, as well as a recursive auto-encoder, to modify another standard summarization algorithm (sub-modular optimization) Ren et al. Ren et al. used "Average Word Embedding" as one of many independent variables in a regression model, but the degree to which word embeddings effected the final model was not clear. BID3 used word embeddings as the input to a neural network as part of summarization, but they did not directly compare embeddings. They used a single-layer convolutional neural network to encode sentences, and a LSTM neural network to encode documents. BID15 recently achieved state-of-the-art using a modified version of LexRank, using a combination of cosine similarity of weighted averages of vectors and named entity overlap. To explore potential sentence embeddings, we implement the four sentence embeddings above as vector functions, which convert sentences or documents to vectors. • SIF Average: The most basic sentence embedding is simply the weighted average of word vectors from Arora et al. BID1, without the common component removal. We use the Brown corpus (Francis & BID5 for word frequency information.• Arora: This method is simply the method described in Arora et al. It is equivalent to the one above, except with common component removal added. We use the Brown corpus both to compute the common component vector, and for word frequency information.• Paragraph Vectors: The paragraph vector approach described above. We used the 300-dimensional DBOW model pretrained by Lau et al. BID11 on the wikipedia corpus.• Skip-Thought Vectors The skip-thought vector approach described above. We used the 4800-dimensional combined-skip model BID9 BID0 .All sentence embeddings are normalized to produce unit vectors To explore the design space, we consider combinations of vector functions and selector functions, functions which, given vector representations for sentences, extracts a summary. We present a large variety of example selector functions in order to allow us to explore interaction effects between selector functions and vector functions.• Near: The most basic selector function, Near, selects the sentences whose sentence vectors have the highest cosine similarity with the document vector• Near Nonredundant: An attempt at balancing redundancy with salience, Near Nonredundant down-weights the cosine similarity scores by their average cosine similarity the sentences selected so far. Because this redundancy measure is strongly (quadratically) correlated with cosine similarity to the document, we fit a regression for redundancy for each vector function, and use the residual on this regression for the final algorithm.• LexRank: Our next selector is based off of the classical LexRank algorithm BID4. We use a modified version of this algorithm, where the weights of edges are determined by the cosine similarity between sentence embeddings.• Cluster: We use an Agglomerative Clustering algorithm (using cosine similarity as its distance metric) to find clusters in the set of sentence embeddings. We then find the sentence closest to the average of each cluster and add it to the summary. To ensure we find summaries which meet the word-length requirement, we increase the number of clusters we search for until we have selected sentences totaling 100 words.• Greedy: The greedy selector, at each step, selects the sentence such that the cosine similarity of the new summary (including previously selected sentences) is maximized. This is subtly different than the Near selector for average-based vector functions, but significantly different for Paragraph Vectors.• Brute Force: Another attempt at optimizing the cosine similarity between the summary and the document, this selector creates a pool of the 20 sentences with the highest cosine similarity. From this pool, every combination of sentences (with an appropriate word count) is tried, and the combination with the highest cosine similarity is selected as the summary.• Max Similarity: A proof-of-concept selector which computes for both the Greedy and Brute Force selectors and then selects the with the highest cosine similarity to the document vector.• Near-then-Redundancy: Starting with the pool from the Brute Force algorithm, this algorithm optimizes via brute force to minimize redundancy (defined as the average cosine similarity between pairs of sentences). Note that the size of the sentence pool, which is essentially a computational shortcut in the Brute Force selector, is now a performance-critical hyper-parameter.• PCA: This selector performs Principal Component Analysis (PCA) on the set of sentence vectors in a document. Then, the algorithm selects the one sentence closest to the first component, one sentence closest to the second component, and so on, until the length capacity is met.• Random: This selector simply selects sentences at random, until the word limit is reached. This provides a lower-bound on the performance of an effective algorithm, and is used for baseline comparisons Because evaluation of summarization is fundamentally a subjective task, human evaluations are ideal. However, human evaluations are often expensive and time-consuming to obtain. Luckily, some metrics of automatically evaluating summaries, by comparison to a human-written summary, have been developed. Traditionally, various forms of the ROUGE metric, which compare shared n-grams, have been used. BID13. ROUGE has been shown to correlate strongly with human judgments BID18, and is our primary metric for evaluating summaries 1. We report ROUGE-1 and ROUGE-2 statistics, which correspond to unigrams and bigrams, respectively. We split the document clusters in the DUC 2004 dataset into a testing set and a validation set of approximately equal sizes. The pre-defined training set of the DUC 2001 dataset was used as a training set for some of the graphs and data analysis presented here. We present for Multi-Document Summarization on the DUC 2004 dataset TAB1. A few notes on the :• The best performing selector, Greedy, is both very simple and based on fundamental principles of vector semantics. • Paragraph Vectors work much worse with the Clustering and Greedy algorithms, and work much better with Near and SVMs.• Many combinations of selector function and vector function do not work above the level of random chance.• In general, despite their sophistication, Paragraph Vectors and Skip-Thought vectors perform worse than much more basic approaches. Despite the poor performance of our models compared to the baselines, analyses of the underlying data provide many useful insights into the behavior of vector semantics in real-world tasks. The cosine scores between all sentence vectors and the corresponding document vectors follow a normal distribution for all vector functions FIG0 ), but this effect is most pronounced for paragraph vectors (r 2 = .996). In addition, the sentence embeddings for paragraph vectors and skip-thought vectors are far closer to the document embedding than would be expected from a random distribution, with mean cosine similarities of.65 and.84, respectively (Unsurprisingly, this also holds for Average and Arora vectors). By identifying the sentences present in an optimal summarization, we show that optimal sentences have higher cosine scores, and that this effect is increased after adjusting cosine scores for word length (Fig. 2). However, there is a lot of overlap, implying that, although this method has some power to discern good summaries from bad summaries, the power of this method alone is not high enough to product good summaries. We calculated the isolated ROUGE score of each individual sentence in the training set, and the sentence embeddings for these sentences on all four vector functions. To partially eliminate the effects of the sentence's context, we subtract the corresponding document vector from all sentence vectors before regression. Due to the large number of predictor variables, we use a Bonferroni correction, considering values significant only if they have p-values of α n, which, for α =.05, corresponds approximately to p <.00001 for the skip-thought vectors, and p <.00016 for all other vectors. Three dimensions are significant at this level for SIF Average vectors. No dimensions are significant at this level for Arora vectors, though the three values significant for SIF Average achieve p values of.0021,.0485 and.0006. 29 dimensions are significant for Paragraph Vectors. 5 dimensions are significant, at the much higher threshold, for Skip-Thought Vectors. It appears that these specific dimensions correspond to aspects of a sentence that make it somehow more suited for a summary. Despite the theoretical implications of this , the regression models do not have enough predictive power to create good summaries by themselves. The Greedy algorithm is the most successful algorithm among those we have explored. As its name implies, the Greedy algorithm appears to be simply an attempt at maximizing the following objective function:f cos (summary) = vector(summary) · vector(document)Of course, any simple objective function can only be an approximation to the informally-defined criteria for good summaries. Even so, TAB1 suggests that the performance of the greedy algorithm is not based on the accuracy of the corresponding objective function. In particular, the two other strategies which try to maximize the same objective functio (Brute force, and Maximum Similarity) consistently and significantly create summaries with higher cosine similarity to the document, outperforming the Greedy selector on its objective function, but both of these algorithms perform much worse than the Greedy algorithm. Deeper analysis into the decisions of the Greedy algorithm reveals some reasons for this discrepancy. It appears that the good performance of the Greedy algorithm not from the associated objective function, but by the way in which it maximizes this objective function. In particular, the Greedy algorithm selects sentences with low cosine similarity scores in a vacuum, but which increase the cosine similarity of the overall sentence. To understand why this is true, it we consider the step-by-step behavior of the Greedy algorithm, which can be understood as maximizing the following equation (derived in Appendix A): DISPLAYFORM0 wheres p = i j=0s j i j=0s jNote that this equation consists of three parts: id ·s 1 (a constant wrt.s),d ·s, which is simply the salience of a sentence measured by cosine similarity, and the denominator, which is essentially a measure of redundancy. Not only does this simple metric lead to a'natural' penalty for redundancy, it performs better than our handcrafted redundancy penalties. The way this expression scales when picking the i th sentence is particularity noteworthy: the behavior of this function changes as i increases. In particular, the function becomes more sensitive to redundancy, and less sensitive to salience, as the algorithm selects more sentences. In other words, the algorithm will first try to select important sentences, and then select sentences to fill in the gaps. This , and the success of the ing algorithm, has implications for balancing salience and redundancy in future summarization algorithms. In general, there are two ways to compute a document vector. The most obvious is to pass the entire text of the document into the vector function. This has two theoretical problems. The first is that the'documents' in our algorithms are really clusters of documents, and are therefore non-coherent. The second is that Skip-thought vectors are not designed to handle text longer than a sentence. However, an alternative document vector, docvec-avg, is defined as the mean of the (normalized) sentence vectors. This corresponds to treating the document as a collection of sentences, instead of a collection of words. We compare the two here, and present full in Appendix B.As expected, Skipthought vectors, which are not designed for text larger than a sentence, perform significantly better with the docvec-avg strategy. More notable is the poor performance of the docvec-avg strategy with Paragraph Vectors. The size of the performance gap here implies that Paragraph Vectors can combine information from multiple sentences in a manner more sophisticated than simple averaging. More interesting is the performance for SIF Average and Arora vectors. For these vector functions, which are based on taking the average of words, docvec-avg very closely resembles the simple strategy. And yet there is a small but significant performance gap. The difference between the two document vectors is the weighting. Docvec-avg, which normalizes vectors before adding them together, removes some weighting information present in the simple strategy. In particular, the simple strategy assigns more weight to sentences with a lot of highly-weighted words. Presumably, docvec-avg, by ignoring this weighting, leaves out useful information. This hypothesis is supported by the greater performance gap for Arora vectors, which effectively downweights certain common words and therefore could be expected to carry more information in word weightings. Similar, but much smaller, gaps exist when computing the vectors for summaries at each step in the greedy algorithm. We present a broad comparison of the properties of different sentence embedding schemes.6.6.1 SIF AVERAGE/ARORA VECTORS Three selector functions perform better with both SIF Average and Arora vectors: Near Nonredundant, Greedy, and PCA. These functions seem to be unified by their comparisons between the vectors of sentence embeddings (implicitly, in the case of the greedy algorithm). These selector functions correspond to the most basic test for sentence embeddings: Judging the similarity of two sentences. The exact difference the common component removal makes is less clear. Arora vectors hold a slight performance edge for all selectors except for Near and Greedy (the Greedy algorithm loses a full 2 points). Two selector functions perform better with Paragraph Vectors: Near and Brute Force. Both of these are very similar: They require comparing sentence vectors to the document vector. The poor performance on algorithms such as Near Nonredundant suggests that Paragraph Vectors are especially poor at comparing sentence vectors to each other. These suggest that Paragraph Vectors are especially good at computing document vectors, a hypothesis also implied by the of Section 6.5. The other distinguishing property of Paragraph Vectors is their very high correlation when regressing on the individual features. It is hard to disentangle the properties of Skipthought vectors from the high dimensionality of the pretrained vectors we used. In general, Skipthought vectors performed poorly. They only performed better than other vector functions with one selector, Clustering, although their performance with this selector was significant. We have identified differences in different forms of sentence vectors when applied to real-world tasks. In particular, each sentence vector form seems to be more successful when used in a particular way. Roughly speaking, Arora's vectors excel at judging the similarity of two sentences while Paragraph Vectors excel at representing document vectors, and at representing features as dimensions of vectors. While we do not have enough data to pinpoint the strengths of Skipthought vectors, they seem to work well in specific contexts that our work did not fully explore. These differences are extremely significant, and will likely make or break real-world applications. Therefore, special care should be taken when selecting the sentence vector method for a real-world task. This material is based upon work supported by the National Science Foundation under Grant No. 1659788 and 1359275 A APPENDIX A The first choice of the greedy algorithm is simple: it chooses the sentence with maximum cosine similarity to the document vector:s 1 = argmax s∈Ss ·d (Recall that all vectors have unit-length, so cosine similarity is equivalent to the dot product).To select the second vector, the greedy algorithm is maximizing the following equation: DISPLAYFORM0"

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.