Question: Represent the article for finding a single-sentence TLDR

Question: Contextualized word representations such as ELMo and BERT have become the de facto starting point for incorporating pretrained representations for downstream NLP tasks. In these settings, contextual representations have largely made obsolete their static embedding predecessors such as Word2Vec and GloVe. However, static embeddings do have their advantages in that they are straightforward to understand and faster to use. Additionally, embedding analysis methods for static embeddings are far more diverse and mature than those available for their dynamic counterparts. In this work, we introduce simple methods for generating static lookup table embeddings from existing pretrained contextual representations and demonstrate they outperform Word2Vec and GloVe embeddings on a variety of word similarity and word relatedness tasks. In doing so, our also reveal insights that may be useful for subsequent downstream tasks using our embeddings or the original contextual models. Further, we demonstrate the increased potential for analysis by applying existing approaches for estimating social bias in word embeddings. Our analysis constitutes the most comprehensive study of social bias in contextual word representations (via the proxy of our distilled embeddings) and reveals a number of inconsistencies in current techniques for quantifying social bias in word embeddings. We publicly release our code and distilled word embeddings to support reproducible research and the broader NLP community. Word embeddings (; ;) have been a hallmark of modern natural language processing (NLP) for several years. Pretrained embeddings in particular have seen widespread use and have experienced parallel and complementary innovations alongside neural networks for NLP. Advances in embedding quality in part have come from integrating additional information such as syntax (b;), morphology (Cotterell & Schütze, 2015), subwords , subcharacters and, most recently, context . As a consequence of their representational potential, pretrained word representations have seen widespread adoption across almost every task in NLP and reflect one of the greatest successes of both representation learning and transfer learning for NLP (b). The space of pretrained word representations can be partitioned into static vs. dynamic embeddings methods. Static methods such as Word2Vec , GloVe , and FastText yield representations that are fixed after training and generally associate a single vector with a given word in the style of a lookup table. While subsequent work addressed the fact that words may have multiple senses and should have different representations for different senses (; ; ; ;), fundamentally these methods cannot easily adapt to the inference time context in which they are applied. This contrasts with contextual, or dynamic, methods such as CoVe , ELMo , and BERT , which produce vector representations for a word conditional on the inference time context in which it appears. Given that dynamic representations are arguably more linguistically valid, more expressive (static embeddings are a special-case of dynamic embeddings that are optimally ineffective at being dynamic), and have yielded significant empirical improvements (b; a; a), it would seem that static embeddings are outdated. Static embeddings, however, have significant advantages over dynamic embeddings with regard to speed, computational resources, and ease of use. These benefits have important implications for time-sensitive systems, resource-constrained settings or environmental concerns , and broader accessibility of NLP technologies 1. As a consequence of this dichotomy between static and dynamic representations and their disparate benefits, we propose in this work a simple yet effective mechanism for converting from dynamic representations to static representations. We begin by demonstrating that our method when applied to pretrained contextual models (BERT, GPT-2, RoBERTa, XLNet, DistilBERT) yields higher quality static embeddings than Word2Vec and GloVe when evaluated intrinsically on four word similarity and word relatedness datasets. Further, since our procedure does not rely on specific properties of the pretrained contextual model, it can be applied as needed to generate ever-improving static embeddings that will track advances in pretrained contextual word representations. Our approach offers the hope that high-quality embeddings can be maintained in both settings given their unique advantages and appropriateness in different settings. At the same time, we show that by distilling static embeddings from their dynamic counterparts, we can then employ the more comprehensive arsenal of embedding analysis tools that have been developed in the static embedding setting to better understand the original contextual embeddings. As an example, we employ methods for identifying gender, racial, and religious bias (; ;) to our distilled representations and find that these experiments not only shed light on the properties of our distilled embeddings for downstream use but can also serve as a proxy for understanding existing biases in the original pretrained contextual representations. Our large-scale and exhaustive evaluation of bias further reveals dramatic inconsistencies in existing measures of social bias and highlights sizeable discrepancies in the bias estimates obtained for distilled embeddings drawn from different pretrained models and individual model layers. In this work, we study pretrained word embeddings, primarily of the static variety. As such, we focus on comparing our embeddings against existing pretrained static embeddings that have seen widespread adoption. We identify Word2Vec and GloVe as being the most prominent static embeddings currently in use and posit that these embeddings have been frequently chosen not only because of their high quality representations but also because lookup tables pretrained on large corpora are publicly accessible and easy to use. Similarly, in considering contextual models to distill from, we begin with BERT as it has been the most prominent in downstream use among the growing number of alternatives (e.g. ELMo , GPT , BERT , Transformer-XL, GPT-2 , XLNet, RoBERTa, and DistilBERT ) though we provide similar analyses for several of the other models (GPT-2, XLNet, RoBERTa, DistilBERT) and more comprehensively address them in the appendices. We primarily report for the bert-base-uncased model and include complete for the bert-large-uncased model in the appendices as well. In order to use a contextual model like BERT to compute a single context-agnostic representation for a given word w, we define two operations. The first is subword pooling: the application of a pooling mechanism over the subword representations generated for w in context c to compute a single representation for w in c, i.e. {w 1 c, . . ., w k c} → w c. Beyond this, we define context combination to be the mapping from representations w c1,..., w cn of w in different contexts c 1,..., c n to a single static embedding w that is agnostic of context. The tokenization procedure for BERT can be decomposed into two steps: performing a simple word-level tokenization and then potentially deconstructing a word into multiple subwords, yielding w 1,..., w k such that cat(w 1, . . ., w k) = w where cat(·) indicates concatenation. In English, the subword tokenization algorithm is WordPiece . As a consequence, the decomposition of a word into subwords is the same across contexts and the subwords can be unambiguously associated with their source word. Therefore, any given layer of the model outputs vectors w 1 c,..., w k c. We consider four potential pooling mechanisms to compute w c given these vectors: min(·) and max(·) are element-wise min and max pooling, mean(·) indicates mean pooling, i.e. |X | and last(·) indicates selecting the last vector, w k c. In order to convert contextual representations into static ones, we describe two methods of specifying contexts c 1,..., c n and then combining the ing representations w c1,..., w cn. Decontextualized -For a word w, we use a single context where c 1 = w. That is, we feed the single word w by itself into the pretrained contextual model and consider the ing vector to be the representation (applying subword pooling if the word is split into multiple subwords). Aggregated -Observing that the Decontextualized strategy may be presenting an unnatural input to the pretrained encoder which may have never encountered w by itself without a surrounding phrase or sentence, we instead consider ways of combining the representations for w in multiple contexts. In particular, we sample n sentences from a large corpus D, each of which contains the word w, and compute the vectors w c1,..., w cn. Then, we apply a pooling strategy to yield a single representation that aggregates the representations across the n contexts as is shown in Equation 2. To assess the representational quality of our static embeddings, we evaluate on several word similarity and word relatedness datasets (see §A.2 for additional commentary). We consider 4 such datasets: RG65 , WS353 , SIMLEX999 and SIMVERB3500 . Taken together, these datasets contain 4917 examples and contain a vocabulary V of 2005 unique words. Each example is a pair of words (w 1, w 2) with a gold-standard annotation (provided by one or more humans depending on the dataset) of how semantically similar or how semantically related w 1 and w 2 are. A word embedding is evaluated by the relative correctness of its ranking of the similarity/relatedness of all examples in a dataset with respect to the gold-standard ranking using the Spearman ρ coefficient. Embedding predictions are computed using cosine similarity as in Equation 3: We begin by studying how the choices of f and g 2 impact the performance of embeddings distilled from bert-base-uncased. In Figure 1, we show the performance on all four datasets of the ing static embeddings where embeddings computed using the Aggregated strategy are pooled over N = 100000 sentences. Here, N is the number of total contexts for all words (see §A.4). Across all four datasets, we see that g = mean is the best performing pooling mechanism within the Aggregated strategy and also outperforms the Decontexualized strategy by a substantial margin. Fixing g = mean, we further observe that mean pooling at the subword level also performs best. We further find that this trend that f = mean, g = mean is optimal among the 16 possible pairs consistently holds for almost all pretrained contextual models we considered. If we further consider the impacts of N as shown in Table 1, we see that performance for both bert-base-uncased and bert-large-uncased tends to steadily increase for all datasets with increasing N (and this trend holds for the 7 other pretrained models). In particular, in the largest setting with N = 1000000, the bert-large-uncased embeddings distilled from the best performing layer for each dataset dramatically outperform both Word2Vec and GloVe. However, this can be seen as an unfair comparison given that we are selecting the layer for specific datasets. As the middle band of table shows, we can fix a layer and still outperform both Word2Vec and Glove. Beyond the benefits of using a larger N, Table 1 reveals an interesting relationship between N and the best-performing layer. In Figure 1, there is a clear preference towards the first quarter of the model's layers (layers 0-3) with a sharp drop-off in performance immediately thereafter (we see a similar preference for the first quarter in models with a different number of layers, e.g. Figure 3, Figure 10). Given that our intrinsic evaluation is centered on lexical semantic understanding, this appears to be largely consistent with the findings of Liu et al. (2019a);. However, as we pool over a larger number of contexts, we see that the best-performing layer monotonically (with a single exception) shifts to be later and later within the pretrained model. What this indicates is that since the later layers did not perform better for smaller values of N, these layers demonstrate greater variance with respect to the layer-wise distributional mean and reducing this variance helps in our evaluation 3. This may have implications for downstream use, given that later layers of the model are generally preferred by downstream practitioners and it is precisely these layers where we see the greatest variance. Accordingly, combining our stable static embeddings from layer with the contextual example-specific embeddings also from layer of the pretrained model as was suggested in may be a potent strategy in downstream settings. In general, we find these suggest there may be merits towards further work studying the unification of static and dynamic methods. Along with a trend towards later layers for larger values of N, we see a similar preference towards later layers as we consider each column of from left to right. In particular, while the datasets are ordered chronologically 4, each dataset was explicitly introduced as an improvement over its predecessors (perhaps transitively, see §A.3). While it is unclear from our evaluation as to what differences in the examples in each dataset may cause this behavior, we find this correlation with dataset difficulty and layer-wise optimality to be intriguing. In particular, we see that SIMVERB3500 which contains verbs primarily (as opposed to nouns or adjectives which dominate the other datasets) tends to yield the best performance for embeddings distilled from the intermediary layers of the model (most clear for bert-large-uncased). Remarkably, we find that most tendencies we observe generalize well to all other pretrained models we study (specifically the optimality of f = mean, g = mean, the improved performance for larger N, and the layer-wise tendencies with respect to N and dataset). In Table 2, we summarize the of all models employing the Aggregated strategy with f = mean, g = mean and N = 100000 contexts. Surprisingly, despite the fact that many of these models perform approximately equally on many downstream evaluations, we observe that their corresponding distilled embeddings perform radically differently even when the same distillation procedure is applied. These can be interpreted as suggesting that some models learn better lexical semantic representations whereas others learn other behaviors such as context representation and semantic composition more accurately. More generally, we argue that these warrant reconsideration of analyses performed on only one pretrained model as they may not generalize to other pretrained models even when the models considered have (nearly) identical Transformer architectures. A noteworthy in Table 2 is that of DistilBert-6 which outperforms BERT-12 on three out of the four datasets despite being distilled using knowledge distillation from BERT-12. Analogously, RoBERTa, which was introduced as a direct improvement over BERT, does not reliably outperform the corresponding BERT models when comparing the derived static embeddings. Table 2: Performance of static embeddings from different pretrained models on word similarity and word relatedness tasks. f and g are set to mean for all models, N = 100000, and (#) indicates the layer the embeddings are distilled from. Bold indicates best performing embeddings for a given dataset of those depicted. Bias is a complex and highly relevant topic in developing representations and models in machine learning and natural language processing. In this context, we study the social bias encoded within static word representations. As Kate Crawford argued for in her NIPS 2017 keynote, while studying individual models is important given that specific models may propagate, accentuate, or diminish biases in different ways, studying the representations that serve as the starting point and that are shared across models (which are used for possibly different tasks) allows for more generalizable understanding of bias . In this work, we simultaneously consider multiple axes of social bias (i.e. gender, race, and religion) and multiple proposed methods for computationally quantifying these biases. We do so precisely because we find that existing NLP literature has primarily prioritized gender (which may be a technically easier setting) and because we find that different computational specifications of bias that evaluate the same social phenomena yield different . As a direct consequence, we strongly caution that the should be taken with respect to the definitions of bias being applied. Further, we note that an embedding which receives low bias scores cannot be assumed to be (nearly) unbiased, rather that under existing definitions the embedding exhibits low bias and perhaps additional more nuanced definitions are needed. introduced a definition for computing gender bias which assumes access to a set P = {(m 1, f 1),..., (m n, f n)} of (male, female) word pairs where m i and f i only differ in gender (e.g. 'men' and 'women'). They compute a gender direction g: where E(·) is the embedding function, ";" indicates horizontal concatenation/stacking and indicates taking the first principal component. Then, given a set N of target words that we are interested in evaluating the bias with respect to, specifies the bias as: This definition is only inherently applicable to binary bias settings, i.e. where there are exactly two protected classes, but still is difficult to apply to binary settings beyond gender as constructing a set P can be challenging. Similarly, multi-class generalizations of this bias definition are also difficult to propose due to the issue of constructing k-tuples that only differ in the underlying social attribute. This definition also assumes the first principal component is capable of explaining a large fraction of the variance. introduced a different definition for computing binary bias that is not restricted to gender, which assumes access to sets A 1 = {m 1, · · ·, m n} and A 2 = {f 1, · · ·, f n} of representative words for each of the two protected classes. For each class, µ i = mean w∈Ai E(w) is computed. computes the bias in the following ways: Compared to the definition of , these definitions may be more general as constructing P is strictly more difficult than constructing A 1, A 2 (as P can always be split into two such sets but the reverse is not generally true) and's definition does not rely on the first principal component explaining a large fraction of the variance. However, unlike the first definition, computes the bias in favor of/against a specific class (meaning if N = {'programmer', 'homemaker'} and'programmer' was equally male-biased as'homemaker' was female-biased, then under the definition of , there would be no bias in aggregate). For the purposes of comparison, we adjust their definition by taking the absolute value of each term in the mean over N. introduced a definition for quantifying multi-class bias which assumes access to sets A 1,..., A k of representative words as in. They quantify the bias as 5: Similar to the adjustment made for the definition, we again take the absolute value of each term in the mean over N. Figure 2: Layer-wise bias of distilled BERT-12 embeddings for f = mean, g = mean, N = 100000 Left: Gender, Center: Race, Right: Religion Table 3: Social bias within static embeddings from different pretrained models with respect to a set of professions N prof. Parameters are set as f = mean, g = mean, N = 100000 and the layer of the pretrained model used in distillation is X 4. Lowest bias in a particular column is denoted in bold. Inspired by the of , in this work we transparently report social bias in existing static embeddings as well as the embeddings we compute. In particular, we exhaustively report the bias for all 3542 valid (pretrained model, layer, social attribute, bias definition) 4-tuples which describe all combinations of static embeddings and bias measures referenced in this work. We specifically report for binary gender (male, female), two-class religion (Christianity, Islam) and three-class race (white, Hispanic, and Asian), directly following. These are by no means intended to be comprehensive with regards to the breadth of bias socially and only address a restricted class of social biases which notably does not include the important class of intersectional biases. The types of biases being evaluated for are taken with respect to specific word lists (which are sometimes subjective albeit being peer-reviewed) that serve as exemplars and with respect to definitions of bias grounded in the norms of the United States. Beginning with bert-base-uncased, we report the layer-wise bias across all (attribute, definition) pairs in Figure 2. What we immediately observe is that for any given social attribute, there is a great deal of variation across the layers in the quantified amount of bias. Further, while we are unsurprised that different bias measures for the same social attribute assign different absolute scores, we observe that they also do not agree in relative judgments. For gender, we observe that the bias estimated by the definition of steadily increases before peaking at the penultimate layer and slightly decreasing thereafter. In contrast, under bias GARG-EUC we see a distribution with two peaks corresponding to layers at the start or end of the pretrained contextual model with lower bias observed in the intermediary layers. For estimating the same quantity, bias GARG-COS is mostly uniform across the layers (though the scale of the axes visually lessens the variation displayed). Similarly, in looking at the religious bias, we see similar inconsistencies with the bias increasing monotonically from layers 2 through 8 under bias MANZINI, decreasing monotonically under bias GARG-EUC, and remaining roughly constant under bias GARG-COS. In general, while the choice of N (and the choice of A i in the gender bias case) does affect the absolute bias estimates under any given definition, we find that the general trends in the bias across layers are approximately invariant under these choices for a specific definition. Taken together, our analysis suggests a concerning state of affairs regarding bias quantification measures for (static) word embeddings. In particular, while estimates are seemingly stable to some types of choices regarding word lists, bias scores for a particular word embedding are tightly related to the definition being used and existing bias measures are markedly inconsistent with each other. We find this has important consequences beyond understanding the social biases in our representations. Concretely, we argue that without certainty regarding the extent to which embeddings are biased, it is impossible to properly interpret the meaningfulness of debiasing procedures (; a; b;) as we cannot reliably estimate the bias in the embeddings both before and after the procedure. This is further compounded with the existing evidence that current intrinsic measures of social bias may not handle geometric behavior such as clustering . In light of the above, next we compare bias estimates across different pretrained models in Table 3. Given the conflicting scores assigned by different definitions, we retain all definitions along with all social attributes in this comparison. However, we only consider target words given by N prof for visual clarity as well as due to the aforementioned stability to the choice of N, with the for adjectives provided in Table 8. We begin by noting that since we do not perform preprocessing to normalize embeddings, the scores using bias GARG-EUC are not comparable (and may not have been proper to compare in the layer-wise case either) as they are sensitive to the absolute norms of the embeddings which cannot be expected to be similar across models 6. Further, we note that bias BOLUKBASI may not be a reliable indicator as similar to Zhao et al. (2019a), we find that the first principal component explains less than 35% of the variance in the majority of the static embeddings distilled from contextual models. Of the two bias definitions not mentioned thus far, we find that all distilled static embeddings have substantially higher scores under bias MANZINI but generally lower scores under bias GARG-COS when compared to Word2Vec and GloVe. Interestingly, we see that under bias MANZINI both GPT-2 and RoBERTa embedding consistently get high scores across social attributes when compared to other distilled embeddings but under bias GARG-COS they receive the lowest scores among distilled embeddings. Ultimately, given the aforementioned issues regarding the reliability of bias measures, it is difficult to arrive at a clear consensus of the comparative bias between our distilled embeddings and prior static embeddings. What our analysis does resolutely reveal is a pronounced and likely problematic effect of existing bias definitions on the ing bias scores. Distilled Static Representations. introduced an approach similar to our Aggregated strategy where representations are gradually aggregated across instances in a dataset during training to model global information. Between epochs, the memory of past instances is reset and during testing, inference-time instances are added into the memory. In that work, the computed static embeddings are an additional feature that is used to achieve the state-of-the-art on several NER datasets. Based on our , we believe their approach could be further improved by different decisions in pretrained model and layer choice. Their may be explained by the (desirable) variance reduction we observe in pooling over many contexts. Additionally, since they only pool over instances in an online fashion within an epoch, the number of contexts is relatively small in their approach as compared to ours which may help to explain why they find that min or max pooling perform slightly better than mean pooling as the choice for g. proposes a different approach to convert representations from sentence encoders into static embeddings as a means for applying the WEAT implicit bias tests to a sentence encoder. In their method, a single semantically-bleached sentence is synthetically constructed from a template and then fed into the encoder to compute a static embedding for the word of interest. We argue that this approach may inherently not be appropriate for quantifying bias in sentence encoders 7 in the general case as sentence encoders are trained on semantically-meaningful sentences and semantically-bleached constructions are not representative of this distribution. Moreover, the types of templated constructions presented heavily rely on deictic expressions and therefore are difficult to adapt for certain syntactic categories such as verbs (as would be required for the SimVerb3500 dataset especially) without providing arguments for the verb. These concerns are further exacerbated by our findings given the poor representational behavior seen in our Decontextualized embeddings which have similar deficiencies with their static embeddings and the poor representational behavior when we pool over relatively few semantically-meaningful contexts using the Aggregated strategy (e.g. our for N = 10000 which is still 50 instances per word on average and is much more than the single instance they consider). We believe our quantification of bias as a can be taken as a more faithful estimator of bias in sentence encoders. considers a similar approach towards diachronic sense modelling. In particular, given a word, they find its senses and example sentences of each sense in the Oxford English Dictionary and use these to compute static embeddings using the Aggregated strategy with the last layer of bert-base-uncased and n i upper-bounded at 10. Given our , their performance could likely be improved by pooling over more sentences, using bert-large-uncased, and considering layer choice as their task heavily relies on lexical understanding which seems to be better captured in earlier layers of the model than the last one. Since they require sense annotations for their setting (and the number of example sentences in a dictionary for a sense is inherently constrained), our findings also suggest that additional sense-annotated or weakly sense-annotated sentences would be beneficial. Lightweight Pretrained Representations. Taken differently, our approach can be seen as a method for integrating pretraining in a more lightweight fashion. Model compression and knowledge distillation are well-studied techniques in machine learning that have been recently applied for similar purposes. In particular, several concurrent approaches have been proposed to yield lighter pretrained sentence encoders and contextual word representations (; ; ;). Our approach along with these recent approaches yield representations that are more appropriate for resource-constrained settings such as on-device models for mobile phones, for real-time settings where we require low-latency and short inference times, and for users that may not have access to GPU or TPU computational resources . Additionally, this line of work is particularly timely given the emergent concerns of the environmental impact/harm of training and using increasingly large models in NLP , machine learning , and the broader AI community . Bias. Social bias in NLP has been primarily evaluated in three ways: (a) using geometric similarity between embeddings (; ;), (b) adapting psychological association tests , and (c) considering down-stream behavior 2018a; 2019a; ) 8. In relation to this body of work, our bias evaluation is in the style of (a) as we are interested in intrinsic bias in embeddings and considers (potentially) multi-class social bias in the lens of gender, race, and religion whereas prior work has primarily focused on gender. Additionally, while most of the work on bias in embeddings has considered the static embedding setting, recent work has considered sentence encoders and contextual models. Zhao et al. (2019a) considers gender bias in ELMo when applied to extends these by considering not only NER but also bias using WEAT by leveraging the masked language modeling objective of BERT. considers intrinsic gender bias using ELMo by studying gender-swapped sentences. When compared to these approaches, we study a broader class of biases under more than one bias definition and consider more than one model. Further, while these approaches generally neglect reporting bias values for different layers of the model, we show this is crucial as bias is not uniformly distributed throughout model layers and downstream practitioners often do not use the last layer of deep Transformer models (a; ; b) 9. Pretrained contextual word representations have quickly gained traction in the NLP community, largely because of the flurry of empirical successes that have followed since their introduction. For downstream practitioners, our work suggests several simple (e.g. subword pooling mechanism choice) and more sophisticated (e.g. layer choice, benefits of variance reduction by using multiple contexts) strategies that may yield better downstream performance. Additionally, some recent models have combined static and dynamic embeddings (; ;) and our representations may support drop-in improvements in these settings. Beyond furthering efforts in representation learning, this work introduces a new approach towards the understanding of contextual word representations via proxy analysis. In particular, while in this work we choose to study social bias, similar analyses toward other forms of interpretability and understanding would be valuable. Additionally, post-processing approaches that go beyond analysis such as dimensionality reduction may be particularly intriguing given that this is often challenging to do within large multi-layered networks like BERT but has been successfully done for static embeddings (; ;). Future work may also consider the choice of the corpus D from which contexts are drawn. In particular, we believe choosing D to be drawn from the target domain for some downstream task may serve as an extremely lightweight domain adaptation strategy. Additionally, in this work we choose to provide contexts of sentence length in order to facilitate regularity in the comparison across models. But for some models, such as Transformer-XL or XLNet which are trained with memories to handle larger contexts, better performance may be achieved by using larger contexts. In this work, we propose simple but effective procedures for converting contextual word representations into static word embeddings. When applied to pretrained models like BERT, we find the ing embeddings outperform Word2Vec and GloVe substantially under intrinsic evaluation and provide insights into the pretrained model. We further demonstrate the ing embeddings are more amenable to (existing) embedding analysis methods and report the extent of various social biases (gender, race, religion) across a number of measures. Our large-scale analysis furnishes several findings with respect to social bias encoded in popular pretrained contextual representations via the proxy of our embeddings and has implications towards the reliability of existing protocols for quantifying bias in word embeddings. All data, code, visualizations (and code to produce to them), and distilled word embeddings will be publicly released. Additional reproducibility details are provided in Appendix A. URL http://papers.nips.cc/paper/ 6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddi pdf. Rishi Bommasani, Arzoo Katiyar, and Claire Cardie. SPARSE: Structured prediction using argument-relative structured encoding. In this work, we chose to conduct intrinsic evaluation experiments that focused on word similarity and word relatedness. We did not consider the related evaluation of lexical understanding via word analogies as they have been shown to decompose into word similarity subtasks (a) and there are significant concerns about the validity of these analogies tests . We acknowledge that word similarity and word relatedness tasks have also been heavily scrutinized . A primary concern is that are highly sensitive to (hyper)parameter selection . In our setting, where the parameters of the embeddings are largely fixed based on which pretrained models are publicly released and where we exhaustively report the impact of most remaining parameters, we find these concerns to still be valid but less relevant. To this end, prior work has considered various preprocessing operations on static embeddings such as clipping embeddings on an elementwise basis when performing intrinsic evaluation. We chose not to study these preprocessing choices as they create discrepancies between the embeddings used in intrinsic evaluation and those used in downstream tasks (where this form of preprocessing is generally not considered) and would have added additional parameters implicitly. Instead, we directly used the computed embeddings from the pretrained model with no changes throughout this work. A.3 introduced a set of 65 noun-pairs and demonstrated strong correlation (exceeding 95%) between the scores in their dataset and additional human validation. introduced a larger collection of pairs which they argued was an improvement over RG65 as it more faithfully addressed semantic similarity. followed this work by introducing a even more pairs that included those of as a subset and again demonstrated correlations with human scores exceeding 95%. argued that SIMLEX999 was an improvement in coverage over RG65 and more correctly quantified semantic similarity as opposed to semantic relatedness or association when compared to WS353. Beyond this, SIMVERB3500 was introduced by to further increase coverage over all predecessors. Specifically, it shifted the focus towards verbs which had been heavily neglected in the prior datasets which centered on nouns and adjectives. We used PyTorch throughout this work with the pretrained contextual word representations taken from the HuggingFace pytorch-transformers repository 13. Tokenization for each model was conducted using its corresponding tokenizer, i.e. for GPT2 use the GPT2Tokenizer in pytorch-transformers. For simplicity, throughout this work, we introduce N as the total number of contexts used in distilling with the Aggregated strategy. Concretely, N = wi∈V n i where V is the vocabulary used (generally the 2005 words in the four datasets considered). As a , in finding contexts, we filter for sentences in D that contain at least one word in V. We choose to do this as this requires a number of candidate sentences upper bounded with respect to the most frequent word in V as opposed to filtering for a specific value for n which requires a number of sentences scaling in the frequency of the least frequent word in V. The N samples from D for the Aggregated strategy were sampled uniformly at random. Accordingly, as the aforementioned discussion suggests, for word w i, the number of examples n i which contain w i scales in the frequency of w i in the vocabulary being used. As a consequence, for small values of N, it is possible that rare words would have no examples and computing a representation w using the Aggregated strategy would be impossible. In this case, we back-offed to using the Decontextualized representation for w i. Given this concern, in the bias evaluation, we fix n i = 20 for every w i. In initial experiments, we found the bias to be fairly stable when choosing values n i ∈ {20, 50, 100}. The choice of n i would correspond to N = 40100 (as the vocabulary size was 2005) in the representation quality section in some sense (however this assumes a uniform distribution of word frequency as opposed to a Zipf distribution). The embeddings in the bias evaluation are drawn from layer X 4 using f = mean, g = mean as we found these to be the best performing embeddings generally across pretrained models and datasets in the representational quality evaluation. The set of gender-paired tuples P were taken from. In the gender bias section, P for definitions involving sets A i indicates that P was split into equal-sized sets of male and female work. For the remaining gender , the sets described in §G.3 were used. The various attribute sets A i and target sets N j were taken from which can be further sourced to a number of prior works in studying social bias. We remove any multi-word terms from these lists. B BERT-LARGE Table 8: Social bias within static embeddings from different pretrained models with respect to a set of adjectives, N adj. Parameters are set as f = mean, g = mean, N = 100000 and the layer of the pretrained model used in distillation is X 4.

A procedure for distilling contextual models into static embeddings; we apply our method to 9 popular models and demonstrate clear gains in representation quality wrt Word2Vec/GloVe and improved analysis potential by thoroughly studying social bias.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: The brain performs unsupervised learning and (perhaps) simultaneous supervised learning. This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning. Inspired by the rich space of Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal. We present the Hebbian-augmented training algorithm (HAT) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights. We test HAT's effect on a simple problem (Fashion-MNIST) and find consistently higher performance than supervised learning alone. This finding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods. We further find that the meta-learned update rule is a time-varying function; thus, it is difficult to pinpoint an interpretable Hebbian update rule that aids in training. We do find that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner's convergence. Backpropagation achieves great performance in neural net optimization, but might not be biologically plausible because most problems are not explicitly phrased as classification with true labels, because neurons only know local signals (e.g. synaptic density, ACh levels, current), and because backpropagation uses the computational graph, a separate data structure with no known biological basis. Although some supervised training schemes are more biologically plausible (e.g. contrastive Hebbian learning and equilibrium propagation ), it's currently unknown whether the behavior of all neurons is accurately encapsulated by these models. We speculate that some local, unsupervised learning occurs in the brain and demonstrate that the addition of local, unsupervised rules to standard backpropagation actually improves the speed and robustness of learning. We begin by defining a local learning rule. Consider two adjacent neurons i, j with weight w ij: given an impulse traversing i, j with activations v i, v j, a local learning rule computes updates ∆w ij using local data v i, w ij, v j. Note that by this definition, a local learning rule is unsupervised at face value. Many neuroscientists have hypothesized specific functions that describe the brain's true (unsupervised) local learning rule. Most such rules involve using the correlation of activations as part of the update rule. Examples include Hebb's Rule, Oja's Rule, the Generalized Hebbian Algorithm, and nonlineear Hebbian rules. It is not obvious which of these rules (if any) describe the true behavior of neurons. We employ meta-learning (learning how to learn) as an investigative tool. Optimization functions are algorithms too; it stands to reason that we can learn the best optimization function. In the meta-learning framework, one model A learns a task (e.g. Fashion-MNIST) while another model B learns how to optimize A. Meta-learning has achieved great in finding robust optimization schemes. Andrychowicz et. al. used meta-learning to find the best gradient-based optimization function (B learns to update A using A's gradients), and Chen et. al. used meta-learning to find the best gradient-free optimization function (B learns to update A using only the sequence of A's losses). Finally, Metz et al. demonstrated a fully differentiable architecture for learning to learn unsupervised local rules and demonstrate better-than-random performance on a few-shot basis. If B consistently converges to some stable rule, we take it as strong evidence that this rule may occur in biological brains as well. We therefore wish to extend Metz's approach to learning semisupervised local rules not only to improve performance but also to investigate the functional form of the meta-learned update rule. The Hebbian-Augmented Training algorithm (HAT) is an algorithm that trains the neural net L twice per sample: using local, unsupervised rules on the forward pass and using backpropagationbased gradient descent on the backward pass. Formally, we create 2 multilayer perceptrons: a learner L(· | φ L) with parameters φ L and a metalearner M (v i, v j, w ij | φ M) with parameters φ M, which takes inputs v i, w ij, v j and returns ∆w ij. For a single sample (x, y), we train L without supervision using M and x; we simultaneously train L and M with supervision using A and y. On the forward pass, we compute activations for each layer. For a given layer, we now have the inputs, outputs, and current weights -all of the inputs of local learning rule. We can then apply the outputs of meta-learner M to update the weights of layer. We then recompute the activations of layer using the new weights. This process is done efficiently by convolution (for details, see Appendix A). We compute the activations of the first layer 1, update 1, compute the activations of the second layer 2, update 2, and so on until we compute the predicted Weightsˆ y and update |L|. On the backward pass, we backpropagate. Since we recomputed the activations of each layer using weights updated by M, the weights of M are upstream of the weights of L in the computational graph; thus, a single iteration of the backpropagation algorithm will compute gradients for both M and L. Given a gradient ∇ p for each parameter p ∈ φ L ∪ φ M, we then perform a supervised update p ← p + A(p, ∇ p). The key insight is that the convolution of the meta-learner over the weights of the learner forms a fully differentiable framework M L y. Algorithm 1 Hebbian-Augmented Training Algorithm for weights W, v +1 is a placeholder output as input to M 5: Updates weight using local rule M 6: Backpropagate loss H(v |L|, y). for layer weight W in L and M do Backward pass 9: W ← A Apply gradient update using optimizer A 10: return L, M Return updated learner and updated meta-learner We hypothesize that the HAT algorithm will have three positive effects. • HAT will train the learner L faster since there are twice as many updates. In ordinary backpropagation the metadata generated from the forward pass is computed and wasted; in HAT, the metadata is computed and used to generate a (potentially) useful update. • HAT will improve the convergence of L. The second update should introduce some stochasticity in the loss landscape since it is not directly tied to gradient descent, which may lead L into better local optima. • HAT will improve the performance of L when some examples are not labeled. Backpropagation has no ability to learn from just the input x, while HAT is able to perform the unsupervised update. We generate two learning curves to test these hypotheses: one with respect to time and one with respect to the proportion of labeled examples. The charts below represent the aggregated learning curves of 100 pairs (L i, M i). We find that the effects of HAT on training are clearly positive. The median accuracy of the neural nets trained by HAT is clearly increased along the learning curve, and the HAT-group neural nets reach a higher asymptotic value than the control group. We do note that the two learning curves seem to inflect around the same point -HAT does not seem to cause a faster convergence, just a better one. We attribute this to the meta-learner's convergence; it may take the meta-learner up to 0.5 epochs to start to have positive effects. One potential concern with adding unsupervised meta-learner updates is that after the convergence of the base learner L, the meta-learner's continued output of non-zero updates might "bounce" the base learner out of an optimum. Remarkably, we see in the above plot that the performance of the HAT-trained neural nets is quite stable for the entire 18 epochs of post-convergence duration. To our surprise, we find that HAT is more effective when there are more labels, even though the self-supervised component of the algorithm is designed to take advantage of scarce labels. We attribute this to slow convergence of the meta-learner M -when labels are scarce, the meta-learner may actually converge slower than the learner and thus provide bad update suggestions. We would like insight into why HAT improves the training of neural nets over vanilla gradient descent. Thus, we will analyze the functional form of the learned update rule M after it has fully converged. Recall the setting from experiments 1 and 2: we generate 100 pairs of learners and meta-learners: (L i, M i) for i ∈ {1, ..., 100}. We then investigate the pointwise mean M of these meta-learners. We first visualize the dependence of the function M on its inputs (v i, v j, w ij). We find that a remarkably linear dependence on v j explains almost all of the variance in the outputs of the meta-learned update rule. This indicates that the rule is a "rich-get-richer" scheme: neurons that already fired with high magnitude will experience larger incoming weights and thus be encouraged to fire with high activation in the future. This linear dependence is surprising since all of the hypothesized rules in neuroscience have a dependence on v i ·v j. As a sanity check, we attempted to directly apply this update rule (∆w ij ≈ 2·v j) without meta-learning to see if we can replicate HAT's performance improvement. However, the were decisively negative -HAT improves performance, but the a priori application of HAT's update rule decreases it. We present three hypotheses: • Perhaps M learns a good update rule while L is training, then learns a degenerate rule once L has converged. The sole purpose of this degenerate rule would be to not un-learn the important weights that have already converged (thus explaining the rich-gets-richer behavior of the rule f (·) = 2v j ). Thus, analyzing the black-box function at epoch 20 is merely the wrong time -perhaps observing the meta-learned rule at epoch 1 would be more insightful and useful. • Perhaps M learns a good update rule in each run, and these update rules are all complex functions with no good low-order polynomial approximations; however, their pointwise mean (which is itself not a good local update rule) happens to be linear. Thus, M is the wrong object to analyze and presents behaviors that are not indicative of the of experiments 1 and 2. • Perhaps the learning of M is extremely transient. For any given point in time, there is a different optimal learning rule, and our exercise in finding a fixed local, unsupervised update rule that is universal across training is futile. The HAT algorithm demonstrates that local, unsupervised signals can provide performance-improving weight updates. Neural nets under HAT converge to better asymptotic losses as long as there is sufficient time (> 0.5 epochs) and a sufficient number of labels (> 20% of the data is labeled). The latter finding is surprising since the addition of an unsupervised learning algorithm depends on the presence of labels in order to deliver marginal benefits over gradient descent. The underlying form of the learned rule that makes HAT successful is still a mystery; we find that while the meta-learner may learn a useful update rule during training, the meta-learner does not converge to this useful rule in the long run and instead devolves into a linear function ConvergedRule. This converged function preserves fully-converged weights by reinforcing incoming weights for neurons with high activations. The discovery that HAT does not stably converge to a function makes analysis quite difficult. However, there is potential for future work to do more subtle analyses. Imagine a time t during training in which the meta-learner M has converged to a useful function, but the learner L has not yet finished training. A follow-up to this thesis might be to discover whether there such a time t exists, what the structure of M at time t is, and how M changes the weights of L at time t. One potential methodology might be to observe the function f not as a 3-dimensional function in (v i, w ij, v j) but rather as a 4-dimensional function in (v i, w ij, v j, t). Observing the function along the t-axis and checking for phase changes would shed light on whether a single useful update rule is learned during training or whether HAT's learning is truly transient and continuous. If this follow-up were to succeed, then we could have an a priori rule to apply without having to metalearn update rules. Extracting the local rules from multiple domains could either find that HAT learns a universal rule or that functional distance between two rules describes the "difference" between their originating domains. • Suppose we always metalearn the same rule, regardless of problem domain. Optimal-Hebb is then a universal learning rule. • Suppose Optimal-Hebb is not universal for all problems. For local rules R A, R B on problems A, B, integrating gives an explicit measure for how similar A and B are. This provides a systematic way to identify pairs of learning problems that are good candidates for transfer learning. One implementation detail is notably not covered in the HAT pseudocode; this implementation detail patches an inadequacy in modern deep learning frameworks. Given two neural net layers i and i+1 and minibatches of size B, we have B instances of | i | × i+1 neuron pairs, each of which has 3 salient properties (v i, w ij, v j). Therefore, we would like to apply the function M over the zeroth dimension of a tensor of size 3 × B × | i | × | i+1 | in order to compute the unsupervised weight updates. However, as of this writing date, it is not possible to apply an arbitrary function M to slices of a tensor in parallel in any modern deep learning framework (e.g. Tensorflow, PyTorch, Keras); the reason is that this plays poorly with optimization of the computational graph. We thus implement the application of M's updates to the weights by convoluting M over a state tensor. This is best clarified with an example. Suppose we have a neural net with consecutive layers 1, 2 of size 784 and 183, respectively. Suppose further that we have batches of size 50. Finally, suppose that we require a meta-learner that is a neural net of architecture 3 × 100 × 1. We then copy the tensors along the boxed dimensions to stack them. We instantiate M as a sequence of 3 composed functions: 1. a convolutional layer of kernel size 1 × 1 with 3 in-channels and 100 out-channels, 2. a ReLU activation, and 3. a convolutional layer of kernel size 1 × 1 with 100 in-channels and 1 out-channels. Applying this series of functions to a 1× image with 3 channels is equivalent to passing the 3 channels into a neural net with architecture 3 × 100 × 1. PyTorch (the framework used for this research) does not support the vectorization of arbitrary functions along torch tensors. However, it does support (and heavily optimize for) convolutions. Thus, we implement our neural net function M as a series of convolutions, and we convolve the function over the input tensor of size 3 × 50 × 183 × 784. The output of M is of size 50 × 183 × 784; we average over the zeroth dimension to finally get a weight update of dimension 183 × 784, which is the same size as the original weight tensor.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Metalearning unsupervised update rules for neural networks improves performance and potentially demonstrates how neurons in the brain learn without access to global labels.

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep convolutional networks often append additive constant ("bias") terms to their convolution operations, enabling a richer repertoire of functional mappings. Biases are also used to facilitate training, by subtracting mean response over batches of training images (a component of "batch normalization"). Recent state-of-the-art blind denoising methods seem to require these terms for their success. Here, however, we show that bias terms used in most CNNs (additive constants, including those used for batch normalization) interfere with the interpretability of these networks, do not help performance, and in fact prevent generalization of performance to noise levels not including in the training data. In particular, bias-free CNNs (BF-CNNs) are locally linear, and hence amenable to direct analysis with linear-algebraic tools. These analyses provide interpretations of network functionality in terms of projection onto a union of low-dimensional subspaces, connecting the learning-based method to more traditional denoising methodology. Additionally, BF-CNNs generalize robustly, achieving near-state-of-the-art performance at noise levels well beyond the range over which they have been trained. Denoising -recovering a signal from measurements corrupted by noise -is a canonical application of statistical estimation that has been studied since the 1950's. Achieving high-quality denoising requires (at least implicitly) quantifying and exploiting the differences between signals and noise. In the case of natural photographic images, the denoising problem is both an important application, as well as a useful test-bed for our understanding of natural images. The classical solution to the denoising problem is the Wiener filter, which assumes a translation-invariant Gaussian signal model. Under this prior, the Wiener filter is the optimal estimator (in terms of mean squared error). It operates by mapping the noisy image to the frequency domain, shrinking the amplitude of all components, and mapping back to the signal domain. In the case of natural images, the high-frequency components are shrunk more aggressively than the lower-frequency components because they tend to contain less energy in natural images. This is equivalent to convolution with a lowpass filter, implying that each pixel is replaced with a weighted average over a local neighborhood. In the 1990's, more powerful solutions were developed based on multi-scale ("wavelet") transforms. These transforms map natural images to a domain where they have sparser representations. This makes it possible to perform denoising by applying nonlinear thresholding operations in order to reduce or discard components that are small relative to the noise level (4; 12; 1). From a linear-algebraic perspective, these algorithms operate by projecting the noisy input onto a lower-dimensional subspace that contains plausible signal content. The projection eliminates the orthogonal complement of the subspace, which mostly contains noise. This general methodology laid the foundations for the state-of-the-art models in the 2000's (e.g.), some of which added a data-driven perspective, learning sparsifying transforms, or more general nonlinear shrinkage functions directly from natural images (6; 10). In the past decade, purely data-driven models based on convolutional neural networks have come to dominate all previous methods in terms of performance. These models consist of cascades of convolutional filters, and rectifying nonlinearities, which are capable of representing a diverse and powerful set of functions. Training such architectures to minimize mean square error over large databases of noisy natural-image patches achieves current state-of-the-art (see also for a related approach). Neural networks have achieved particularly impressive on the blind denoising problem, in which the noise amplitude is unknown (14; 15; 9). Despite their success, We lack intuition about the denoising mechanisms these solutions implement. Network architecture and functional units are often borrowed from the image-recognition literature, and it is unclear which of these aspects contribute positively, or limit, the denoising performance. Many authors claim critical importance of specific aspects of architecture (e.g., skip connections, batch normalization, recurrence), but the benefits of these attributes are difficult to isolate and evaluate in the context of the many other elements of the system. In this work, we show that bias terms used in most CNNs (additive constants, including those used for batch normalization) interfere with the interpretability of these networks, do not help performance, and in fact prevent generalization of performance to noise levels not including in the training data. In particular, bias-free CNNs (BF-CNNs) are locally linear, and hence amenable to direct analysis with linear-algebraic tools. And BF-CNNs generalize robustly, achieving near-state-of-the-art performance at noise levels well beyond the range over which they have been trained. We assume a measurement model in which images are corrupted by additive noise: y = x + n, where x ∈ R N is the original image, containing N pixels, n is an image of i.i.d. samples of Gaussian noise with variance σ 2, and y ∈ R N is the observed noisy image. The denoising problem consists of finding a function f: R N → R N that provides a good estimate of the original image, x. Commonly, one minimizes the mean squared error: f (y) = arg min g E||x − g(y)|| 2, where the expectation is taken over some distribution over images, x, as well as over the distribution of noise realizations. Finally, if the noise standard deviation, σ, is unknown, the expectation should also be taken over a distribution of this variable. This problem is often called blind denoising in the literature. Feedforward neural networks with rectified linear units (ReLUs) are piecewise affine: for a given input signal, the effect of the network on the input is a cascade of linear transformations (convolutional or fully connected layers, each represented by a matrix, (W), additive constants (b), and pointwise multiplication by a binary mask representing the sign of the affine responses (R). Since each stage is affine, the entire cascade implements a single affine transformation. The function computed by a denoising neural network with L layers may be written where A y ∈ R N ×N is the Jacobian of f (·) evaluated at input y, and b y ∈ R N represents the net additive bias. The subscripts on A y and b y serve as a reminder that the corresponding matrix and vector, respectively, depend on the ReLU activation patterns, which in turn depend on the input vector y. If we remove all the additive ("bias") terms from every stage of a CNN, the ing bias-free CNN (BF-CNN) is strictly linear, and its net action may be expressed as where A y is again the Jacobian of f BF (·) evaluated at y. We analyze this local representation to reveal and visualize the noise-removal mechanisms implemented by BF-CNNs. We illustrate our analysis using a BF-CNN based on the architecture of the Denoising CNN (DnCNN,), although our observations also hold for other architectures (7; 11; 15). The linear representation of the denoising map given by equation 2 implies that the ith pixel of the output image is computed as an inner product between the ith row of A y and the input image. The rows of A y can be interpreted as adaptive filters that produce an estimate of the denoised pixel via a weighted average of noisy pixels. Examination of these filters reveals their diversity, and their relationship to the underlying image content: they are adapted to the local features of the noisy image, averaging over homogeneous regions of the image without blurring across edges (Figure 2). We observe that the equivalent filters of all architectures adapt to image structure. The local linear structure of a BF-CNN allows analysis of its functional capabilities via the singular value decomposition (SVD). For a given input y, we compute the SVD of the Jacobian matrix: The output is a linear combination of the left singular vectors, each weighted by the projection of the input onto the corresponding right singular vector, and scaled by the corresponding singular value. Analyzing the SVD of a BF-CNN on natural images reveals that most singular values are close to zero (Figure 1a). The network is thus discarding all but a very low-dimensional portion of the input image. We can measure an "effective dimensionality", d, of The three rightmost images show the weighting functions used to compute each of the indicated pixels (red squares). All weighting functions sum to one, and thus compute a local average (note that some weights are negative, indicated in red). Their shapes vary substantially, and are adapted to the underlying image content. this preserved subspace by computing the total noise variance remaining in the denoised image, f BF (y), which corresponds to the sum of the squares of singular values. where We also observe that the left and right singular vectors corresponding to the singular values with non-negligible amplitudes are approximately the same (Figure 1c). This means that the Jacobian is (approximately) symmetric, and we can interpret the action of the network as projecting the noisy signal onto a low-dimensional subspace, as is done in wavelet thresholding schemes. For inputs of the form y:= x + n, the subspace spanned by the singular vectors corresponding to the non-negligible singular values contains x almost entirely, in the sense that projecting x onto the subspace preserves most of its norm. The low-dimensional subspace encoded by the Jacobian is therefore tailored to the input image. This is confirmed by visualizing the singular vectors as images. The singular vectors corresponding to non-negligible singular values capture features of the input image; the ones corresponding to near-zero singular values are unstructured (Figure 3). BF-CNN therefore implements an approximate projection onto an adaptive signal subspace that preserves image structure, while suppressing much of the noise. The signal subspace depends on the noise level. We find that for a given clean image corrupted by noise, the effective dimensionality of the signal subspace decreases as the noise level increases (Figure 1b). At lower noise levels the network detects a richer set of image features, that lie in a larger signal subspace. In addition, these signal subspaces are nested: subspaces corresponding to lower noise levels contain at least 95% of the subspaces corresponding to higher noise levels. The empirical that dimensionality is equal to α σ, combined with the observation that the signal subspace contains the clean image, explains the observed denoising performance across different noise levels (Figure 4). Specifically, if we assume A y x ≈ x, the mean squared error is proportional to σ: The scaling of MSE with the square root of the noise variance implies that the PSNR of the denoised image should be a linear function of the input PSNR, with a slope of 1/2. This provides an empirical target for generalization beyond training range. We investigate generalization across noise levels, comparing networks with and without net bias. We implement BF-CNNs based on several Denoising CNNs (14; 7; 11; 15). These architectures include popular features of existing neural-network techniques in image processing: recurrence, multiscale filters, and skip connections. To construct BF-CNNs, we remove all sources of additive bias, including the mean parameter of the batch-normalization in every layer (note however that the rescaling parameters are preserved). We train the networks, following the training scheme described in, using images corrupted by i.i.d. Gaussian noise with a range of standard deviations. This range is the training range of the network. We then evaluate the networks for noise levels that are both within and beyond the training range. Figure 4 compares DnCNN from and its equivalent BF-CNN for different noise levels, inside and outside of the training range. In all cases, DnCNN generalizes very poorly to noise levels outside the training range. In contrast, BF-CNN generalizes robustly, as predicted with a slope of 1/2, even when trained only on modest levels of noise (σ =). Figure 5 shows an example that demonstrates visually the striking difference in generalization performance. We found that the same holds for the other architectures.. The CNN performs poorly at high noise levels (σ = 90, far beyond the training range), whereas BF-CNN performs at state-of-the-art levels. The CNN used for this example is DnCNN; using alternative architectures yields similar .

Represent the short summary for finding the corresponding scientific article (long):

We show that removing constant terms from CNN architectures provides interpretability of the denoising method via linear-algebra techniques and also boosts generalization performance across noise levels.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.


Question: Represent the text to find a very short summary (1-2 sentences max):

Question: The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry. Through their myriad successful applications across a wide range of disciplines, it is now well established that deep neural networks possess an unprecedented ability to model complex real-world datasets, and in many cases they can do so with minimal overfitting. Indeed, the list of practical achievements of deep learning has grown at an astonishing rate, and includes models capable of human-level performance in tasks such as image recognition , speech recognition, and machine translation . Yet to each of these deep learning triumphs corresponds a large engineering effort to produce such a high-performing model. Part of the practical difficulty in designing good models stems from a proliferation of hyperparameters and a poor understanding of the general guidelines for their selection. Given a candidate network architecture, some of the most impactful hyperparameters are those governing the choice of the model's initial weights. Although considerable study has been devoted to the selection of initial weights, relatively little has been proved about how these choices affect important quantities such as rate of convergence of gradient descent. In this work, we examine the effect of initialization on the rate of convergence of gradient descent in deep linear networks. We provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. In particular, we show that for deep networks, the width needed for efficient convergence for orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence of Gaussian networks scales linearly in the depth. Orthogonal weight initializations have been the subject of a significant amount of prior theoretical and empirical investigation. For example, in a line of work focusing on dynamical isometry, it was found that orthogonal weights can speed up convergence for deep linear networks and for deep non-linear networks;;;;; ) when they operate in the linear regime. In the context of recurrent neural networks, orthogonality can help improve the system's stability. A main limitation of prior work is that it has focused almost exclusively on model's properties at initialization. In contrast, our analysis focuses on the benefit of orthogonal initialization on the entire training process, thereby establishing a provable benefit for optimization. The paper is organized as follows. After reviewing related work in Section 2 and establishing some preliminaries in Section 3, we present our main positive on efficient convergence from orthogonal initialization in Section 4. In Section 5, we show that Gaussian initialization leads to exponentially long convergence time if the width is too small compared with the depth. In Section 6, we perform experiments to support our theoretical . Deep linear networks. Despite the simplicity of their input-output maps, deep linear networks define high-dimensional non-convex optimization landscapes whose properties closely reflect those of their non-linear counterparts. For this reason, deep linear networks have been the subject of extensive theoretical analysis. A line of work (; ; ; ; ; Laurent & von) studied the landscape properties of deep linear networks. Although it was established that all local minima are global under certain assumptions, these properties alone are still not sufficient to guarantee global convergence or to provide a concrete rate of convergence for gradient-based optimization algorithms. Another line of work directly analyzed the trajectory taken by gradient descent and established conditions that guarantee convergence to global minimum (; ;). Most relevant to our work is the of , which shows that if the width of hidden layers is larger than the depth, gradient descent with Gaussian initialization can efficiently converge to a global minimum. Our establishes that for Gaussian initialization, this linear dependence between width and depth is necessary, while for orthogonal initialization, the width can be independent of depth. Our negative for Gaussian initialization also significantly generalizes the of , who proved a similar negative for 1-dimensional linear networks. Orthogonal weight initializations. Orthogonal weight initializations have also found significant success in non-linear networks. In the context of feedforward models, the spectral properties of a network's input-output Jacobian have been empirically linked to convergence speed (; ; 2018;). It was found that when this spectrum concentrates around 1 at initialization, a property dubbed dynamical isometry, convergence times improved by orders of magnitude. The conditions for attaining dynamical isometry in the infinitewidth limit were established by Pennington et al. (2017; 2018) and basically require that input-output map to be approximately linear and for the weight matrices to be orthogonal. Therefore the training time benefits of dynamical isometry are likely rooted in the benefits of orthogonality for deep linear networks, which we establish in this work. Orthogonal matrices are also frequently used in the context of recurrent neural networks, for which the stability of the state-to-state transition operator is determined by the spectrum of its Jacobian (; Laurent & von). Orthogonal matrices can improve the conditioning, leading to an ability to learn over long time horizons (; ; ;). While the benefits of orthogonality can be quite large at initialization, little is known about whether or in what contexts these benefits persist during training, a scenario that has lead to the development of efficient methods of constraining the optimization to the orthogonal group (; ;). Although we do not study the recurrent setting in this work, an extension of our analysis might help determine when orthogonality is beneficial in that setting. Denote by · the 2 norm of a vector or the spectral norm of a matrix. Denote by · F the Frobenius norm of a matrix. For a symmetric matrix A, let λ max (A) and λ min (A) be its maximum and minimum eigenvalues, and let λ i (A) be its i-th largest eigenvalue. For a matrix B ∈ R m×n, let σ i (B) be its i-th largest singular value (i = 1, 2, . . ., min{m, n}), and let σ max (B) = σ 1 (B), σ min (B) = σ min{m,n} (B). Denote by vec (A) be the vectorization of a matrix A in column-first order. The Kronecker product between two matrices A ∈ R m1×n1 and B ∈ R m2×n2 is defined as where a i,j is the element in the (i, j)-th entry of A. We use the standard O(·), Ω(·) and Θ(·) notation to hide universal constant factors. We also use C to represent a sufficiently large universal constant whose specific value can differ from line to line. Suppose that there are n training examples dx×n the input data matrix and by Y = (y 1, . . ., y n) ∈ R dy×n the target matrix. Consider an L-layer linear neural network with weight matrices W 1,..., W L, which given an input x ∈ R dx computes where and α is a normalization constant which will be specified later according to the initialization scheme. We study the problem of training the deep linear network by minimizing the 2 loss over training data: The algorithm we consider to minimize the objective is gradient descent with random initialization, which first randomly samples the initial weight matrices from a certain distribution, and then updates the weights using gradient descent: for time t = 0, 1, 2,..., where η > 0 is the learning rate. For convenience, we denote The time index t is used on any variable that depends on W 1,..., W L to represent its value at time t, e.g., W j: In this section we present our main positive for orthogonal initialization. We show that orthogonal initialization enables efficient convergence of gradient descent to a global minimum provided that the hidden width is not too small. In order to properly define orthogonal weights, we let the widths of all hidden layers be equal:.., W L−1 are m × m square matrices, and We sample each initial weight matrix W i independently from a uniform distribution over scaled orthogonal matrices satisfying The same scaling factor was adopted in , which preserves the expectation of the squared 2 norm of any input. F. Then * is the minimum value for the objective. Denote r = rank(X), κ = λmax(X X) λr(X X) 2 Our main theorem in this section is the following: for some δ ∈ and a sufficiently large universal constant C > 0. Set the learning rate η ≤ dy 2L X 2. Then with probability at least 1 − δ over the random initialization, we have where (t) is the objective value at iteration t. Notably, in Theorem 4.1, the width m need not depend on the depth L. This is in sharp contrast with the of for Gaussian initialization, which requires m ≥Ω(Lrκ 3 d y). It turns out that a near-linear dependence between m and L is necessary for Gaussian initialization to have efficient convergence, as we will show in Section 5. Therefore the requirement in is nearly tight in terms of the dependence on L. These together rigorously establish the benefit of orthogonal initialization in optimizing very deep linear networks. If we set the learning rate optimally according to Theorem 4.1 to η = Θ(dy L X 2), we obtain that (t) − * decreases by a ratio of 1 − Θ(κ −1) after every iteration. This matches the convergence rate of gradient descent on the (1-layer) linear regression problem min The proof uses the high-level framework from , which tracks the evolution of the network's output during optimization. This evolution is closely related to a time-varying positive semidefinite (PSD) matrix (defined in), and the proof relies on carefully upper and lower bounding the eigenvalues of this matrix throughout training, which in turn implies the desired convergence . First, we can make the following simplifying assumption without loss of generality. See Appendix B in for justification. Assumption 4.1. (Without loss of generality) X ∈ R dx×r, rank(X) = r, Y = W * X, and * = 0.'s framework. The key idea is to look at the network's output, defined as We also write U (t) = αW L:1 (t)X as the output at time t. Note that F. According to the gradient descent update rule, we write. 2r is known as the stable rank of X, which is always no more than the rank. where E(t) contains all the high-order terms (i.e., those with η 2 or higher). With this definition, the evolution of U (t) can be written as the following equation: where Notice that P (t) is always PSD since it is the sum of L PSD matrices. Therefore, in order to establish convergence, we only need to (i) show that the higher-order term E(t) is small and (ii) prove upper and lower bounds on P (t)'s eigenvalues. For the second task, it suffices to control the singular values of 3 Under orthogonal initialization, these matrices are perfectly isometric at initialization, and we will show that they stay close to isometry during training, thus enabling efficient convergence. The following lemma summarizes some properties at initialization. Lemma 4.2. At initialization, we have Furthermore, with probability at least 1 − δ, the loss at initialization satisfies Proof sketch. The spectral property follows directly from. To prove, we essentially need to upper bound the magnitude of the network's initial output. This turns out to be equivalent to studying the magnitude of the projection of a vector onto a random lowdimensional subspace, which we can bound using standard concentration inequalities. The details are given in Appendix A.1. Now we proceed to prove Theorem 4.1. We define F which is the upper bound on from. Conditioned on being satisfied, we will use induction on t to prove the following three properties A(t), B(t) and C(t) for all t = 0, 1,...: • B(t): • C(t): A and B are true according to Lemma 4.2, and C is trivially true. In order to prove A(t), B(t) and C(t) for all t, we will prove the following claims for all t ≥ 0: Claim 4.4. C(t) =⇒ B(t). The proofs of these claims are given in Appendix A. Notice that we finish the proof of Theorem 4.1 once we prove A(t) for all t ≥ 0. In this section, we show that gradient descent with Gaussian random initialization necessarily suffers from a running time that scales exponentially with the depth of the network, unless the width becomes nearly linear in the depth. Since we mostly focus on the dependence of width and running time on depth, we will assume the depth L to be very large. Recall that we want to minimize the objective F by gradient descent. We assume Y = W * X for some W * ∈ R dy×dx, so that the optimal objective value is 0. For convenience, we assume X F = Θ and Y F = Θ., and all weights in the network are independent. We set the scaling factor α such that the initial output of the network does not blow up exponentially (in expectation): Note that E f (x; We also assume that the magnitude of initialization at each layer cannot vanish with depth: Note that the assumptions and are just sanity checks to rule out the obvious pathological cases -they are easily satisfied by all the commonly used initialization schemes in practice. Now we formally state our main theorem in this section. Theorem 5.1. for some universal constant 0 < γ ≤ 1. Then there exists a universal constant c > 0 such that, if gradient descent is run with learning rate η ≤ e cL γ, then with probability at least 0.9 over the random initialization, for the first e Theorem 5.1 establishes that efficient convergence from Gaussian initialization is impossible for large depth unless the width becomes nearly linear in depth. This nearly linear dependence is the best we can hope for, since proved a positive when the width is larger than linear in depth. Therefore, a phase transition from untrainable to trainable happens at the point when the width and depth has a nearly linear relation. Furthermore, Theorem 5.1 generalizes the of , which only treats the special case of For convenience, we define a scaled version of We first give a simple upper bound on A j: Lemma 5.2. With probability at least 1 − δ, we have A j: The proof of Lemma 5.2 is given in Appendix B.1. It simply uses Markov inequality and union bound. Furthermore, a key property at initialization is that if j − i is large enough, A j:i will become exponentially small. Lemma 5.3. With probability at least Proof. We first consider a fixed pair (i, j) such that j − i ≥ L 10. In order to bound A j:i, we first take an arbitrary unit vector v ∈ R di−1 and bound A j:iv. We can write A j:.., j). Recall the expression for the moments of chi-squared random variables: (∀λ > 0). Taking λ = 1 2 and using the bound Choose a sufficiently small constant c > 0. By Markov inequality we have. Therefore we have shown that for any fixed unit vector v ∈ R di−1, with probability at least 1 − e −Ω(L γ) we have Next, we use this to bound A j:i via an -net argument. We partition the index set Now, for any u ∈ R di−1, we write it as u = q l=1 a l u l where a l is a scalar and u l is a unit vector supported on S l. By the definition of The above inequality is valid for any u ∈ R di−1. Thus we can take the unit vector u that maximizes A j:iu. This gives us A j: Finally, we take a union bound over all possible (i, j). The failure probaility is at most The following lemma shows that the properties in Lemmas 5.2 and 5.3 are still to some extent preserved after applying small perturbations on all the weight matrices. Lemma 5.4. Suppose that the initial weights satisfy A j:, where c 1 > 0 is a universal constant. Then for another set of matrices, we must have Proof. It suffices to show that the difference Expanding this product, except for the one term corresponding to A j:i, every other term has the form A j: By assumption, each ∆ k has spectral norm e −0.6c1L γ, and each A j:i has spectral norm O(L 3), so we have Therefore we have The proof of the second part of the lemma is postponed to Appendix B.2. As a consequence of Lemma 5.4, we can control the objective value and the gradient at any point sufficiently close to the random initialization. Lemma 5.5. For a set of weight matrices, the objective and the gradient satisfy The proof of Lemma 5.5 is given in Appendix B.3. Finally, we can finish the proof of Theorem 5.1 using the above lemmas. Proof of Theorem 5.1. From Lemmas 5.2 and 5.3, we know that with probability at least 0.9, we have (i) A j: Here c 1 > 0 is a universal constant. From now on we are conditioned on these properties being satisfied. We suppose that the learning rate η is at most e 0.2c1L We say that a set of weight matrices W 1,..., W L are in the "initial neighborhood" if. From Lemmas 5.4 and 5.5 we know that in the "initial neighborhood" the objective value is always between 0.4 Y 2 F and 0.6 Y 2 F. Therefore we have to escape the "initial neighborhood" in order to get the objective value out of this interval. Now we calculate how many iterations are necessary to escape the "initial neighborhood." According to Lemma 5.5, inside the "initial neighborhood" each W i can move at most η(in one iteration by definition of the gradient descent algorithm. In order to leave the "initial neighborhood," some W i must satisfy In order to move this amount, the number of iterations has to be at least This finishes the proof. In this section, we provide empirical evidence to support the in Sections 4 and 5. To study how depth and width affect convergence speed of gradient descent under orthogonal and Gaussian initialization schemes, we train a family of linear networks with their widths ranging from 10 to at t = 1258 and t = 10000, for different depth-width configurations and different initialization schemes. Darker color means smaller loss. 1000 and depths from 1 to 700, on a fixed synthetic dataset (X, Y). 4 Each network is trained using gradient descent staring from both Gaussian and orthogonal initializations. In Figure 1, We lay out the logarithm of the relative training loss (t), using heap-maps, at steps t = 1258 and t = 10000. In each heat-map, each point represents the relative training loss of one experiment; the darker the color, the smaller the loss. Figure 1 clearly demonstrates a sharp transition from untrainable to trainable (i.e., from red to black) when we increase the width of the network: • for Gaussian initialization, this transition occurs across a contour characterized by a linear relation between width and depth; • for orthogonal initialization, the transition occurs at a width that is approximately independent of the depth. These observations excellently verify our theory developed in Sections 4 and 5. To have a closer look into the training dynamics, we also plot "relative loss v.s. training time" for a variety of depth-width configurations. See Figure 2. There again we can clearly see that orthogonal initialization enables fast training at small width (independent of depth), and that the required width for Gaussian initialization depends on depth. In this work, we studied the effect of the initialization parameter values of deep linear neural networks on the convergence time of gradient descent. We found that when the initial weights are iid Gaussian, the convergence time grows exponentially in the depth unless the width is at least as large 4 We choose X ∈ R 1024×16 and W * ∈ R 10×1024, and set Y = W * X. Entries in X and W * are drawn i.i.d. from N. as the depth. In contrast, when the initial weight matrices are drawn from the orthogonal group, the width needed to guarantee efficient convergence is in fact independent of the depth. These establish for the first time a concrete proof that orthogonal initialization is superior to Gaussian initialization in terms of convergence time. A.1 PROOF OF LEMMA 4.2 Proof of Lemma 4.2. We only need to prove. We first upper bound the magnitude of the network's initial output on any given input / z 2 has the same distribution as. Note that m > C · log(r/δ). We know that with probability at least 1 − δ r we have which implies Finally, taking a union bound, we know that with probability at least 1 − δ, the inequality holds for every x ∈ {x 1, . . ., x r}, which implies A.2 PROOF OF CLAIM 4.3 Proof of Claim 4.3.. Thus we can bound the gradient norm as follows for all 0 ≤ s ≤ t and all i ∈ [L]: where we have used B(s). Then for all i ∈ [L] we have:. A.3 PROOF OF CLAIM 4.4 Proof of Claim 4.4. Expanding this product, each term except W j:i has the form: where i ≤ k 1 < · · · < k s ≤ j are locations where terms like ∆ k l are taken out. Note that every factor in of the form according to. Thus, we can bound the sum of all terms of the form as Here the last step uses m > C(LR) 2 which is implied by. Combined with, this proves B(t). Proof of Claim 4.5. Recall that we have the dynamics for U (t). In order to establish convergence from we need to prove upper and lower bounds on the eigenvalues of P (t), as well as show that the high-order term E(t) is small. We will prove these using B(t). Using the definition and property B(t), we have In the lower bound above, we make use of the following relation on dimensions: m ≥ d x ≥ r, which enables the inequality λ min (W i−1:1 (t)X) (Next, we will prove the following bound on the high-order term E(t): Recall that E(t) is the sum of all high-order terms in the product Same as, we have It suffices to show that the above bound is at most 1 6 ηλ min (P t) U (t) − Y F = 1 6 ηλ min (P t) 2 (t). Since λ min (P t) ≥ Using, and noting that either L − i − 1 or i − 1 is greater than L 4, we have

We provide for the first time a rigorous proof that orthogonal initialization speeds up convergence relative to Gaussian initialization, for deep linear networks.

Represent the short summary for finding the corresponding scientific article (long).

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.