Question: Represent the scientific article to find a short TLDR

Question: We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion. In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution. While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate. Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss. Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation. Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes. Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art. Text sequence transduction systems convert a given text sequence from one domain to another. These techniques can be applied to a wide range of natural language processing applications such as machine translation , summarization , and dialogue response generation . In many cases, however, parallel corpora for the task at hand are scarce. Therefore, unsupervised sequence transduction methods that require only non-parallel data are appealing and have been receiving growing attention (; ; ; ; ; . This trend is most pronounced in the space of text style transfer tasks where parallel data is particularly challenging to obtain (; ;). Style transfer has historically referred to sequence transduction problems that modify superficial properties of texti.e. style rather than content. We focus on a standard suite of style transfer tasks, including formality transfer , author imitation , word decipherment , sentiment transfer , and related language translation . General unsupervised translation has not typically been considered style transfer, but for the purpose of comparison we also conduct evaluation on this task . Recent work on unsupervised text style transfer mostly employs non-generative or non-probabilistic modeling approaches. For example, and design adversarial discriminators to shape their unsupervised objective -an approach that can be effective, but often introduces training instability. Other work focuses on directly designing unsupervised training objectives by incorporating intuitive loss terms (e.g. backtranslation loss), and demonstrates state-ofthe-art performance on unsupervised machine translation and style transfer . However, the space of possible unsupervised objectives is extremely large and the underlying modeling assumptions defined by each objective can only be reasoned about indirectly. As a , the process of designing such systems is often heuristic. In contrast, probabilistic models (e.g. the noisy channel model ) define assumptions about data more explicitly and allow us to reason about these assumptions during system design. Further, the corresponding objectives are determined naturally by principles of probabilistic inference, reducing the need for empirical search directly in the space of possible objectives. That said, classical probabilistic models for unsupervised sequence transduction (e.g. the HMM or semi-HMM) typically enforce overly strong independence assumptions about data to make exact inference tractable (; ;). This has restricted their development and caused their performance to lag behind unsupervised neural objectives on complex tasks. Luckily, in recent years, powerful variational approximation techniques have made it more practical to train probabilistic models without strong independence assumptions . Inspired by this, we take a new approach to unsupervised style transfer. We directly define a generative probabilistic model that treats a non-parallel corpus in two domains as a partially observed parallel corpus. Our model makes few independence assumptions and its true posterior is intractable. However, we show that by using amortized variational inference , a principled probabilistic technique, a natural unsupervised objective falls out of our modeling approach that has many connections with past work, yet is different from all past work in specific ways. In experiments across a suite of unsupervised text style transfer tasks, we find that the natural objective of our model actually outperforms all manually defined unsupervised objectives from past work, supporting the notion that probabilistic principles can be a useful guide even in deep neural systems. Further, in the case of unsupervised machine translation, our model matches the current state-of-the-art non-generative approach. We first overview text style transfer, which aims to transfer a text (typically a single sentence or a short paragraph -for simplicity we refer to simply "sentences" below) from one domain to another while preserving underlying content. For example, formality transfer is the task of transforming the tone of text from informal to formal without changing its content. Other examples include sentiment transfer , word decipherment , and author imitation . If parallel examples were available from each domain (i.e. the training data is a bitext consisting of pairs of sentences from each domain) supervised techniques could be used to perform style transfer (e.g. attentional Seq2Seq and Transformer ). However, for most style transfer problems, only non-parallel corpora (one corpus from each domain) can be easily collected. Thus, work on style transfer typically focuses on the more difficult unsupervised setting where systems must learn from non-parallel data alone. The model we propose treats an observed non-parallel text corpus as a partially observed parallel corpus. Thus, we introduce notation for both observed text inputs and those that we will treat as latent variables. Specifically, we let X = {x Corresponding indices represent parallel sentences. Thus, none of the observed sentences share indices. In our model, we introduce latent sentences to complete the parallel corpus. Specifically,X = {x (m+1),x (m+2), · · ·,x (n) } represents the set of latent parallel sentences in D 1, whilē } represents the set of latent parallel sentences in D 2. Then the goal of unsupervised text transduction is to infer these latent variables conditioned the observed non-parallel corpora; that is, to learn p(ȳ|x) and p(x|y). First we present our generative model of bitext, which we refer to as a deep latent sequence model. We then describe unsupervised learning and inference techniques for this model class. Directly modeling p(ȳ|x) and p(x|y) in the unsupervised setting is difficult because we never directly observe parallel data. Instead, we propose a generative model of the complete data that defines a joint likelihood, p(X,X, Y,Ȳ). In order to perform text transduction, the unobserved halves can be treated as latent variables: they will be marginalized out during learning and inferred via posterior inference at test time. Our model assumes that each observed sentence is generated from an unobserved parallel sentence in the opposite domain, as depicted in Figure 1. Specifically, each sentence, and prior, p D1 (x (j) ). We let θ x|ȳ and θ y|x represent the parameters of the two transduction distributions respectively. We assume the prior distributions are pretrained on the observed data in their respective domains and therefore omit their parameters for simplicity of notation. Together, this gives the following joint likelihood: The log marginal likelihood of the data, which we will approximate during training, is: Note that if the two transduction models share no parameters, the training problems for each observed domain are independent. Critically, we introduce parameter sharing through our variational inference procedure, which we describe in more detail in Section 3.2. Architecture: Since we would like to be able to model a variety of transfer tasks, we choose a parameterization for our transduction distributions that makes no independence assumptions. Specifically, we employ an encoder-decoder architecture based on the standard attentional Seq2Seq model which has been shown to be successful across various tasks . Similarly, our prior distributions for each domain are parameterized as recurrent language models which, again, make no independence assumptions. In contrast, traditional unsupervised generative sequence models typically make strong independence assumptions to enable exact inference (e.g. the HMM makes a Markov assumption on the latent sequence and emissions are one-to-one). Our model is more flexible, but exact inference via dynamic programming will be intractable. We address this problem in the next section. Ideally, learning should directly optimize the log data likelihood, which is the marginal of our model shown in Eq. 2. However, due to our model's neural parameterization which does not factorize, computing the data likelihood cannot be accomplished using dynamic programming as can be done with simpler models like the HMM. To overcome the intractability of computing the true data likelihood, we adopt amortized variational inference in order to derive a surrogate objective for learning, the evidence lower bound (ELBO) on log marginal likelihood 2: The surrogate objective introduces q(ȳ|x (i); φȳ |x ) and q(x|y (j); φx |y ), which represent two separate inference network distributions that approximate the model's true posteriors, p(ȳ|x (i); θ x|ȳ ) and p(x|y (j); θ y|x ), respectively. Learning operates by jointly optimizing the lower bound over both variational and model parameters. Once trained, the variational posterior distributions can be used directly for style transfer. The KL terms in Eq. 3, that appear naturally in the ELBO objective, can be intuitively viewed as regularizers that use the language model priors to bias the induced sentences towards the desired domains. Amortized variational techniques have been most commonly applied to continuous latent variables, as in the case of the variational autoencoder (VAE) . Here, we use this approach for inference over discrete sequences, which has been shown to be effective in related work on a semi-supervised task . Inference Network and Parameter Sharing: Note that the approximate posterior on one domain aims to learn the reverse style transfer distribution, which is exactly the goal of the generative distribution in the opposite domain. For example, the inference network q(ȳ|x (i); φȳ |x ) and the generative distribution p(y|x (i); θ y|x ) both aim to transform D 1 to D 2. Therefore, we use the same architecture for each inference network as used in the transduction models, and tie their parameters: φx |y = θ x|ȳ, φȳ |x = θ y|x. This means we learn only two encoder-decoders overall -which are parameterized by θ x|ȳ and θ y|x respectively -to represent two directions of transfer. In addition to reducing number of learnable parameters, this parameter tying couples the learning problems for both domains and allows us to jointly learn from the full data. Moreover, inspired by recent work that builds a universal Seq2Seq model to translate between different language pairs , we introduce further parameter tying between the two directions of transduction: the same encoder is employed for both x and y, and a domain embedding c is provided to the same decoder to specify the transfer direction, as shown in Figure 2. Ablation analysis in Section 5.3 suggests that parameter sharing is important to achieve good performance. The reconstruction terms and the KL terms in Eq. 3 still involve intractable expectations due to the marginalization of the latent sequence, thus we need to approximate their gradients. We can approximate the expectations by sampling latent sequences, but gradients are difficult to back-propagate directly through discrete samples. Gumbel-softmax and REINFORCE are often used as stochastic gradient estimators in the discrete case. Since the latent text variables have an extremely large domain, we find that REINFORCE-based gradient estimates in high variance. Thus, we use the Gumbel-softmax straight-through estimator to backpropagate gradients from the KL terms. However, we find that approximating gradients of the reconstruction loss is much more challenging -both the Gumbelsoftmax estimator and REINFORCE are unable to outperform a simple stop-gradient baseline, which confirms a similar observation in previous work on unsupervised machine translation . Therefore, we simply stop computing the gradients for the inference network that would from the reconstruction term, and perform greedy sampling of the latent sequences during training. Note that the inference networks still receive gradients from the prior through the KL term, and their parameters are shared with the decoders which do receive gradients from reconstruction. We consider this to be the best empirical compromise at the moment. Initialization. Good initialization is often necessary for successful optimization of unsupervised learning objectives. In preliminary experiments, we find that the encoder-decoder structure has difficulty generating realistic sentences during the initial stages of training, which usually in a disastrous local optimum. This is mainly because the encoder-decoder is initialized randomly and there is no direct training signal to specify the desired latent sequence in the unsupervised setting. Therefore, we apply a self-reconstruction loss L rec at the initial epochs of training. We denote the output the encoder as e(·) and the decoder distribution as p dec, then α decays from 1.0 to 0.0 linearly in the first k epochs. k is a tunable parameter and usually less than 3 in all our experiments. Our probabilistic formulation can be connected with recent advances in unsupervised text transduction methods. For example, back translation loss plays an important role in recent unsupervised machine translation (; ;) and unsupervised style transfer systems. In back translation loss, the source language x is translated to the target language y to form a pseudo-parallel corpus, then a translation model from y to x can be learned on this pseudo bitext just as in supervised setting. While back translation was often explained as a data augmentation technique, in our probabilistic formulation it appears naturally with the ELBO objective as the reconstruction loss term. Some previous work has incorporated a pretrained language models into neural semi-supervised or unsupervised objectives. uses the log likelihood of a pretrained language model as the reward to update a supervised machine translation system with policy gradient. utilize a similar idea for unsupervised machine translation. employed a similar approach, but interpret the LM as an adversary, with the generator is trained to fool the LM. We show how our ELBO objective is connected with these more heuristic LM regularizers by expanding the KL loss term (assume x is observed): Note that the loss used in previous work does not include the negative entropy term, −H q. Our objective in this additional "regularizer", the negative entropy of the transduction distribution, −H q. Intuitively, −H q is helps avoid a peaked transduction distribution, preventing the transduction from constantly generating similar sentences to satisfy the language model. In experiments we will show that this additional regularization is important and helps bypass bad local optimum and improve performance. These important differences with past work suggest that a probabilistic view of the unsupervised sequence transduction may provide helpful guidance in determining effective training objectives. We test our model on five style transfer tasks: sentiment transfer, word substitution decipherment, formality transfer, author imitation, and related language translation. For completeness, we also evaluate on the task of general unsupervised machine translation using standard benchmarks. We compare with the unsupervised machine translation model (UNMT) which recently demonstrated state-of-the-art performance on transfer tasks such as sentiment and gender transfer . 4 To validate the effect of the negative entropy term in the KL loss term Eq. 5, we remove it and train the model with a back-translation loss plus a language model negative log likelihood loss (which we denote as BT+NLL) as an ablation baseline. For each task, we also include strong baseline numbers from related work if available. For our method we select the model with the best validation ELBO, and for UNMT or BT+NLL we select the model with the best back-translation loss. Complete model configurations and hyperparameters can be found in Appendix A.1. Word Substitution Decipherment. Word decipherment aims to uncover the plain text behind a corpus that was enciphered via word substitution where word in the vocabulary is mapped to a unique type in a cipher dictionary (; ;). In our formulation, the model is presented with a non-parallel corpus of English plaintext and the ciphertext. We use the data in which provides 200K sentences from each domain. While previous work controls the difficulty of this task by varying the percentage of words that are ciphered, we directly evaluate on the most difficult version of this task -100% of the words are enciphered (i.e. no vocabulary sharing in the two domains). We select the model with the best unsupervised reconstruction loss, and evaluate with BLEU score on the test set which contains 100K parallel sentences. Sentiment Transfer. Sentiment transfer is a task of paraphrasing a sentence with a different sentiment while preserving the original content. Evaluation of sentiment transfer is difficult and is still an open research problem . Evaluation focuses on three aspects: attribute control, content preservation, and fluency. A successful system needs to perform well with respect to all three aspects. We follow prior work by using three automatic metrics : classification accuracy, self-BLEU (BLEU of the output with the original sentence as the reference), and the perplexity (PPL) of each system's output under an external language model. We pretrain a convolutional classifier to assess classification accuracy, and use an LSTM language model pretrained on each domain to compute the PPL of system outputs. We use the Yelp reviews dataset collected by which contains 250K negative sentences and 380K positive sentences. We also use a small test set that has 1000 human-annotated parallel sentences introduced in. We denote the positive sentiment as domain D 1 and the negative sentiment as domain D 2. We denote self-BLEU score as BLEU s and reference BLEU score on the 1000 sentences as BLEU r. Formality Transfer. Next, we consider a harder task of modifying the formality of a sequence. We use the GYAFC dataset , which contains formal and informal sentences from two different domains. In this paper, we use the Entertainment and Music domain, which has about 52K training sentences, 5K development sentences, and 2.5K test sentences. This dataset actually contains parallel data between formal and informal sentences, which we use only for evaluation. We follow the evaluation of sentiment transfer task and test models on three axes. Since the test set is a parallel corpus, we only compute reference BLEU and ignore self-BLEU. We use D 1 to denote formal text, and D 2 to denote informal text. Author Imitation. Author imitation is the task of paraphrasing a sentence to match another author's style. The dataset we use is a collection of Shakespeare's plays translated line by line into modern English. It was collected by 5 and used in prior work on supervised style trans-4 The model they used is slightly different from the original model of in certain details -e.g. the addition of a pooling layer after attention. We re-implement their model in our codebase for fair comparison and verify that our re-implementation achieves performance competitive with the original paper. 5 https://github.com/tokestermw/tensorflow-shakespeare fer . This is a parallel corpus and thus we follow the setting in the formality transfer task. We use D 1 to denote modern English, and D 2 to denote Shakespeare-style English. Related Language Translation. Next, we test our method on a challenging related language translation task . This task is a natural test bed for unsupervised sequence transduction, because the goal is to preserve the meaning of the source sentence while rewriting it into the target language. For our experiments, we choose Bosnian (bs) and Serbian (sr) as the related language pairs. We follow to report BLEU-1 score on this task since BLEU-4 score is close to zero. Unsupervised MT. In order to draw connections with a related work on general unsupervised machine translation, we also evaluate on the WMT'16 German English translation task. This task is substantially more difficult than the style transfer tasks considered so far. We compare with the state-of-the-art UNMT system using the existing implementation from the XLM codebase, 6 and implement our approach in the same framework with XLM initialization for fair comparison. We train both systems on 5M non-parallel sentences from each language. Results for the full suite of tasks are collected in Tables 1 and 2. We list the PPL of the test set under the external LM for both the source and target domain in Table 1. PPL of system outputs should be compared to PPL of the test set itself because extremely low PPL often indicates that the generated sentences are short or trivial. Tables 1 and 2 demonstrate some general trends. First, UNMT is able to outperform other prior methods in unsupervised text style transfer, such as (; ;). The performance improvements of UNMT indicate that flexible and powerful architectures are crucial (prior methods generally do not have attention mechanism). Second, our model achieves comparable classification accuracy to UNMT but outperforms it in all style transfer tasks in terms of the reference-BLEU, which is probably the most important metric since it directly measures the quality of the final generations against gold parallel data. This indicates that our method is both effective and consistent across many different tasks. Finally, the BT+NLL baseline is sometimes quite competitive, which indicates that the addition of a language model alone can be beneficial. However, our method consistently outperforms the simple BT+NLL method, which indicates the effectiveness of the additional entropy regularizer in Eq. 5 that is the byproduct of our probabilistic formulation. Next, we examine the PPL of the system outputs under pretrained domain LMs, which should be evaluated in comparison with the PPL of the test set itself. For both the sentiment transfer and the formality transfer tasks in Table 1, BT+NLL achieves extremely low PPL, lower than the PPL of the test corpus in the target domain. After a close examination of the output, we find that it contains many repeated and overly simple outputs. For example, the system generates many examples of "I love this place" when transferring negative to positive sentiment (see Appendix A.3 for examples). It is not surprising that such a trivial output has low perplexity, high accuracy, and low BLEU score. On the other hand, our system obtains reasonably competitive PPL, and our approach achieves the highest accuracy and higher BLEU score than the UNMT baseline. Parameter Sharing. We also conducted an experiment on the word substitution decipherment task, where we remove parameter sharing (as explained in Section 3.2) between two directions of transduction distributions, and optimize two encoder-decoder instead. We found that the model only obtained an extremely low BLEU score and failed to generate any meaningful outputs. Performance vs. Domain Divergence. Figure 3 plots the relative improvement of our method over UNMT with respect to accuracy of a naive Bayes' classifier trained to predict the domain of test sentences. Tasks with high classification accuracy likely have more divergent domains. We can see that for decipherment and en-de translation, where the domains have different vocabularies and thus are easily distinguished, our method yields a smaller gain over UNMT. This likely indicates that the (discrimination) regularization effect of the LM priors is less important or necessary when the two domains are very different. Why do we do better than UNMT? Finally, we examine in detail the output of our model and UNMT for the author imitation task. We pick this task because the reference outputs for the test set are provided, aiding analysis. Examples shown in Table 3 demonstrate that UNMT tends to make overly large changes to the source so that the original meaning is lost, while our method is better at preserving the content of the source sentence. Next, we quantitatively examine the outputs from UNMT and our method by comparing the F1 measure of words bucketed by their syntactic tags. We use the open-sourced compare-mt tool , and the are shown in Figure 4. Our system has an advantage over UNMT in all word categories. In particular, our system is much better at generating nouns, which contribute to preserving the content of the sentences. Greedy vs. Sample-based Gradient Approximation. In our experiments, we use greedy decoding from the inference network to approximate the expectation required by ELBO, which is a biased estimator. The main purpose of this approach is to reduce the variance of the gradient estimator during training, especially in the early stages when the variance of sample-based approaches is quite high. As an ablation experiment on the sentiment transfer task we compare greedy and sample-based gradient approximations in terms of both train and test ELBO, as well as task performance corresponding to best test ELBO. After the model is fully trained, we find that the sample-based approximation has Table 4, where the sampled-based training underperforms on both ELBO and task evaluations. As noted above, to stabilize the training process, we stop gradients from propagating to the inference network from the reconstruction loss. Does this approach indeed better optimize the actual probabilistic objective (i.e. ELBO) or only indirectly lead to improved task evaluations? In this section we use sentiment transfer as an example task to compare different methods for propagating gradients and evaluate both ELBO and task evaluations. Specifically, we compare three different methods: • Stop Gradient: The gradients from reconstruction loss are not propagated to the inference network. This is the method we use in all previous experiments. • Gumbel Softmax : Gradients from the reconstruction loss are propagated to the inference network with the straight-through Gumbel estimator. • REINFORCE : Gradients from reconstruction loss are propagated to the inference network with ELBO as a reward function. This method has been used in previous work for semi-supervised sequence generation , but often suffers from instability issues. We report the train and test ELBO along with task evaluations in Table 5, and plot the learning curves on validation set in Figure 5. 7 While being much simpler, we show that the stop-gradient trick produces superior ELBO over Gumbel Softmax and REINFORCE. This suggests that stopping gradient helps better optimize the likelihood objective under our probabilistic formulation in comparison with other optimization techniques that propagate gradients, which is counter-intuitive. A likely explanation is that as a gradient estimator, while clearly biased, stop-gradient has substantially reduced variance. In comparison with other techniques that offer reduced bias but extremely high variance when applied to our model class (which involves discrete sequences as latent variables), stop-gradient actually leads to better optimization of our objective because it achieves better balance of bias and variance overall. A.1 MODEL CONFIGURATIONS. We adopt the following attentional encoder-decoder architecture for UNMT, BT+NLL, and our method across all the experiments: • We use word embeddings of size 128. • We use 1 layer LSTM with hidden size of 512 as both the encoder and decoder. • We apply dropout to the readout states before softmax with a rate of 0.3. • , we add a max pooling operation over the encoder hidden states before feeding it to the decoder. Intuitively the pooling window size would control how much information is preserved during transduction. A window size of 1 is equivalent to standard attention mechanism, and a large window size corresponds to no attention. See Appendix A.2 for how to select the window size. • There is a noise function for UNMT baseline in its denoising autoencoder loss (; . We use the default noise function and noise hyperparameters in . For BT+NLL and our method we found that adding the extra noise into the self-reconstruction loss (Eq. 4) often hurts the performance, because we already have a language model to avoid the local optimum of direct-copy generation. A.2 HYPERPARAMETER TUNING. We vary pooling windows size as {1, 5}, the decaying patience hyperparameter k for selfreconstruction loss (Eq. 4) as {1, 2, 3}. For the baseliens UNMT and BT+NLL, we also try the option of not annealing the self-reconstruction loss at all as in the unsupervised machine translation task . We vary the weight λ for the NLL term (BT+NLL) or the KL term (our method) as {0.001, 0.01, 0.03, 0.05, 0.1}. We list some examples of the sentiment transfer task in Table 6. Notably, the BT+NLL method tends to produce extremely short and simple sentences. In Section 5 we mentioned that the baseline BT+NLL has a low perplexity for some tasks because it tends to generate overly simple and repetitive sentences. From Table 1 we see that two representative tasks are sentiment transfer and formatliy transfer. In Appendix A.3 we have demonstrated some examples for sentiment transfer, next we show some repetitive samples of BT+NLL in Table 7.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We formulate a probabilistic latent sequence model to tackle unsupervised text style transfer, and show its effectiveness across a suite of unsupervised text style transfer tasks.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Current practice in machine learning is to employ deep nets in an overparametrized limit, with the nominal number of parameters typically exceeding the number of measurements. This resembles the situation in compressed sensing, or in sparse regression with $l_1$ penalty terms, and provides a theoretical avenue for understanding phenomena that arise in the context of deep nets. One such phenonemon is the success of deep nets in providing good generalization in an interpolating regime with zero training error. Traditional statistical practice calls for regularization or smoothing to prevent "overfitting" (poor generalization performance). However, recent work shows that there exist data interpolation procedures which are statistically consistent and provide good generalization performance\cite{belkin2018overfitting} ("perfect fitting"). In this context, it has been suggested that "classical" and "modern" regimes for machine learning are separated by a peak in the generalization error ("risk") curve, a phenomenon dubbed "double descent"\cite{belkin2019reconciling}. While such overfitting peaks do exist and arise from ill-conditioned design matrices, here we challenge the interpretation of the overfitting peak as demarcating the regime where good generalization occurs under overparametrization. We propose a model of Misparamatrized Sparse Regression (MiSpaR) and analytically compute the GE curves for $l_2$ and $l_1$ penalties. We show that the overfitting peak arising in the interpolation limit is dissociated from the regime of good generalization. The analytical expressions are obtained in the so called "thermodynamic" limit. We find an additional interesting phenomenon: increasing overparametrization in the fitting model increases sparsity, which should intuitively improve performance of $l_1$ penalized regression. However, at the same time, the relative number of measurements decrease compared to the number of fitting parameters, and eventually overparametrization does lead to poor generalization. Nevertheless, $l_1$ penalized regression can show good generalization performance under conditions of data interpolation even with a large amount of overparametrization. These provide a theoretical avenue into studying inverse problems in the interpolating regime using overparametrized fitting functions such as deep nets. Modern machine learning has two salient characteristics: large numbers of measurements m, and non-linear parametric models with very many fitting parameters p, with both m and p in the range of 10 6 9 for many applications. Fitting data with such large numbers of parameters stands in contrast to the inductive scientific process where models with small numbers of parameters are normative. Nevertheless, these large-parameter models are successful in dealing with real life complexity, raising interesting theoretical questions about the generalization ability of models with large numbers of parameters, particularly in the overparametrized regime µ = p/m > 1. Classical statistical procedures trade training (TE) and generalization error (GE) by controlling the model complexity. Sending TE to zero (for noisy data) is expected to increase GE. However deep nets seem to over-parametrize and drive TE to zero (data interpolation) while maintaining good GE. Over-parametrization has the benefit that global minima of the empirical loss function Note that for µ↵ > 1, the GE values for the l 1 case are close to zero, whereas the values for the l 2 penalized case can be much larger. Note also that the overfitting peak is much larger for ↵ < 1 than for ↵ > 1, and that the region of good generalization starts at µ = 1/↵, which can be to the left or right of the overfitting peak depending on the value of the undersampling parameter ↵. For the simulations with! 0, in the l 2 case a pseudoinverse was used. For the l 1 case a numerically small value = 10 5 was used, and it was checked that the do not change on decreasing. proliferate and become easier to find. These observations have led to recent theoretical activity. Regression and classification algorithms have been shown that interpolate data but also generalize optimally. An interesting related phenomenon has been noted: the existence of a peak in GE with increasing fitting model complexity. In it was suggested that this peak separates a classical regime from a modern (interpolating) regime where over-parametrization improves performance. While the presence of a peak in the GE curve is in stark contrast with the classical statistical folk wisdom where the GE curve is thought to be U-shaped, understanding the significance of such peaks is an open question, and motivates the current paper. Parenthetically, similar over-fitting peaks were reported almost twenty years ago (cf. statistical physics approach to learning) and attributed to increased fitting model entropy near the peak (see in particular Figs 4.3 and 5.2 in ). 1. We introduce a model, Misparametrized (or Misspecified) Sparse Regression (MiSpaR), which separates the number of measurements m, the number of model parameters n (which can be controlled for sparsity by a parameter ⇢), and the number of fitting degrees of freedom p. 2. We obtain analytical expressions for the GE and TE curves for l 2 penalized regression in the "high-dimensional" or "thermodynamic" asymptotic regime m, p, n! 1 keeping the ratios µ = p/m and ↵ = m/n fixed. We are also able to analytically compute GE for l 1 penalized regression, and exhibit explicit expressions for µ < 1 and µ >> 1 as! 0. 3. We show that for! 0 and for > 0, the overfitting peak appears at the data interpolation point µ = 1 (p = m) for both l 2 and l 1 penalized interpolation (GE ⇠ |1 µ| 1 near µ = 1), but does not demarcate the point at which "good generalization" first occurs, which for small corresponds to the point p = n (µ↵ = 1) (Figure 1). The region of good generalization can start before or after the overfitting peak. The overfitting peak is suppressed for finite. 4. For infinitely large overparametrization, generalization does not occur: GE(µ ! 1) = 1 for both l 2 and l 1 penalized interpolation. However, for small values of the sparsity parameter ⇢ and measurement noise variance 2, there is a large range of values of µ where l 1 regularized interpolation generalizes well, but l 2 penalized interpolation does not (Fig. 1). This range is given by 1 << log(µ) << 2, ⇢/↵ << 1. In this regime the sparsity penalty is effective, and suppresses noise-driven mis-estimation of parameters for the l 1 penalty. This shows how generalization properties of penalized interpolation depend strongly on the inductive bias, and are not properties of data interpolation per se. This has important implications for the usage of deep nets for solving inverse problems. 2 for small µ µ c ) and GE 1 (µ ! 1) = 1. 6. For = 0 and ↵ > ↵ c (⇢), GE 1 goes to zero linearly at µ↵ = 1 (GE 1 / ( . In this case GE 1 goes to zero with a nontrivial on the left, but rises quadratically on the right Usually in linear regression the same (known) design matrix x ij is used both for data generation and for parameter inference. In MiSpaR the generative model has a fixed number n of parameters j, which generate m measurements y i, but the number of parameters p in the inference model is allowed to vary freely, with p < n corresponding the under-parametrized and p > n the over-parametrized case. For the under-parametrized case, a truncated version of the design matrix is used for inference, whereas for the over-parametrized case, the design matrix is augmented with extra rows. In addition, we assume that the parameters in the generative model are sparse, and consider the effect of sparsity-inducing regularization in the interpolation limit. Combining misparametrization with sparsity is important to our study for two reasons • Dissociating data interpolation (which happens when µ = 1, ! 0) from the regime where good generalization can occur (this is controlled by the undersampling ↵ as well as by the model sparsity ⇢). • We are able to study the effect of different regularization procedures on data interpolation in an analytically tractable manner and obtain analytical expressions for the generalization error. Generative Model ("Teacher") We assume that the (known/measured) design variables are i.i.d. Gaussian distributed 2 from one realization of the generative model to another with variance 1/n. This choice of variance is important to fix normalization. Other choices have also been employed in the literature (notably x ij ⇠ N (0, 1/m)) -this is important to keep in mind when comparing with literature formulae where factors of ↵ may need to be inserted appropriately to obtain a match. Undersampling: ↵ = m/n Sparsity: Here ⇡ is the distribution of the non-zero model parameters. We assume this distribution to be Gaussian as this permits closed form evaluation of integrals appearing in the l 1 case. Note that we term µ = p/m as overpametrization (referring to the case where µ > 1) and we term ↵ = m/n as undersampling (referring to the case where ↵ < 1). Inference Model ("Student") The design matrix used for inference is mis-parametrized or mis-specified: under-specified (or partially observed) when µ↵ < 1 ⌘ p < n; over-specified, with extra, effect-free rows in the design matrix when µ↵ > 1 ⌘ p > n Parameter inference is carried out by minimizing a penalized mean squared error Note that for p > n, the model parameters are augmented by p n zero entries. We consider l 2 and l 1 penalties (correspondingly V = Note that the expectation E is taken simultaneously over the parameter values, the design matrix and measurement noise. We obtain exact analytical expressions for the risk (generalization error) in the (thermodynamic) limit where n, p, m all tend to infinity, but the ratios ↵ = m/n, µ = p/m are held finite. Similar "thermodynamic" or "high-dimensional" limiting procedures are used in statistical physics, eg in the study of random matrices and spin-glass models in large spatial dimensions. Such limits are also well-studied in modern statistics (for example to understand phase-transition phenomena in the LASSO algorithm ). While there is a large literature on the LASSO phase transition, we were unable to find any computations of the GE curves that span across the underparametrized and overparametrized regimes in a systematic model as presented here. We derive analytical formulae for TE and GE with l 2 or ridge regularization. For l 1 regularization, explicit formulae are given in some parameter regimes. More generally for the l 1 case we obtain a pair of simultaneous nonlinear equations in two variables which implicitly define the MSE. These can be solved numerically to obtain the GE. The nonlinear equations are given in closed form without hidden parameters and do not require integration. Analytical Formulae: T E 2, GE 2 are the training and generalization errors for the l 2 penalized case, and GE 1 the generalization error for the l 1 penalized case. Due to lack of space we do not present the analytical formulae for > 0 as these expressions are complex, but the corresponding analytical expressions were used to generated the theory curves in Fig.1 for the case > 0. The derivations employ the cavity mean field theory approach. Here 2 ef f = 2 + ⇢(1 µ↵). Note that the formulae for GE agree for the l 2 and l! cases in the underparametrized regime µ < 1, but diverge in the overparametrized regime: infinitesimal l 1 regularization provides no better generalization than the pseudoinverse based procedure unless there is overparametrization. Further note that "good generalization" (GE small) begins when µ↵ > 1 not at the overfitting peak (µ = 1).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Proposes an analytically tractable model and inference procedure (misparametrized sparse regression, inferred using L_1 penalty and studied in the data-interpolation limit) to study deep-net related phenomena in the context of inverse problems.

represent the tldr for finding the corresponding scientific article.

------

For example, We show that several claims of the information bottleneck theory of deep learning are not true in the general case. should be similar to The practical successes of deep neural networks have not been matched by theoretical progress that satisfyingly explains their behavior. In this work, we study the information bottleneck (IB) theory of deep learning, which makes three specific claims: first, that deep networks undergo two distinct phases consisting of an initial fitting phase and a subsequent compression phase; second, that the compression phase is causally related to the excellent generalization performance of deep networks; and third, that the compression phase occurs due to the diffusion-like behavior of stochastic gradient descent. Here we show that none of these claims hold true in the general case. Through a combination of analytical and simulation, we demonstrate that the information plane trajectory is predominantly a function of the neural nonlinearity employed: double-sided saturating nonlinearities like tanh yield a compression phase as neural activations enter the saturation regime, but linear activation functions and single-sided saturating nonlinearities like the widely used ReLU in fact do not. Moreover, we find that there is no evident causal connection between compression and generalization: networks that do not compress are still capable of generalization, and vice versa. Next, we show that the compression phase, when it exists, does not arise from stochasticity in training by demonstrating that we can replicate the IB findings using full batch gradient descent rather than stochastic gradient descent. Finally, we show that when an input domain consists of a subset of task-relevant and task-irrelevant information, hidden representations do compress the task-irrelevant information, although the overall information about the input may monotonically increase with training time, and that this compression happens concurrently with the fitting process rather than during a subsequent compression period. Deep neural networks (; are the tool of choice for real-world tasks ranging from visual object recognition BID16, to unsupervised learning BID11 BID19 and reinforcement learning . These practical successes have spawned many attempts to explain the performance of deep learning systems BID12, mostly in terms of the properties and dynamics of the optimization problem in the space of weights (; BID8 BID1, or the classes of functions that can be efficiently represented by deep networks BID20). This paper analyzes a recent inventive proposal to study the dynamics of learning through the lens of information theory . In this view, deep learning is a question of representation learning: each layer of a deep neural network can be seen as a set of summary statistics which contain some but not all of the information present in the input, while retaining as much information about the target output as possible. The amount of information in a hidden layer regarding the input and output can then be measured over the course of learning, yielding a picture of the optimization process in the information plane. Crucially, this method holds the promise to serve as a general analysis that can be used to compare different architectures, using the common currency of mutual information. Moreover, the elegant information bottleneck (IB) theory provides a fundamental bound on the amount of input compression and target output information that any representation can achieve . The IB bound thus serves as a method-agnostic ideal to which different architectures and algorithms may be compared. A preliminary empirical exploration of these ideas in deep neural networks has yielded striking findings . Most saliently, trajectories in the information plane appear to consist of two distinct phases: an initial "fitting" phase where mutual information between the hidden layers and both the input and output increases, and a subsequent "compression" phase where mutual information between the hidden layers and the input decreases. It has been hypothesized that this compression phase is responsible for the excellent generalization performance of deep networks, and further, that this compression phase occurs due to the random diffusion-like behavior of stochastic gradient descent. Here we study these phenomena using a combination of analytical methods and simulation. In Section 2, we show that the compression observed by arises primarily due to the double-saturating tanh activation function used. Using simple models, we elucidate the effect of neural nonlinearity on the compression phase. Importantly, we demonstrate that the ReLU activation function, often the nonlinearity of choice in practice, does not exhibit a compression phase. We discuss how this compression via nonlinearity is related to the assumption of binning or noise in the hidden layer representation. To better understand the dynamics of learning in the information plane, in Section 3 we study deep linear networks in a tractable setting where the mutual information can be calculated exactly. We find that deep linear networks do not compress over the course of training for the setting we examine. Further, we show a dissociation between generalization and compression. In Section 4, we investigate whether stochasticity in the training process causes compression in the information plane. We train networks with full batch gradient descent, and compare the to those obtained with stochastic gradient descent. We find comparable compression in both cases, indicating that the stochasticity of SGD is not a primary factor in the observed compression phase. Moreover, we show that the two phases of SGD occur even in networks that do not compress, demonstrating that the phases are not causally related to compression. These may seem difficult to reconcile with the intuition that compression can be necessary to attain good performance: if some input channels primarily convey noise, good generalization requires excluding them. Therefore, in Section 5 we study a situation with explicitly task-relevant and task-irrelevant input dimensions. We show that the hidden-layer mutual information with the task-irrelevant subspace does indeed drop during training, though the overall information with the input increases. However, instead of a secondary compression phase, this task-irrelevant information is compressed at the same time that the taskrelevant information is boosted. Our highlight the importance of noise assumptions in applying information theoretic analyses to deep learning systems, and put in doubt the generality of the IB theory of deep learning as an explanation of generalization performance in deep architectures. The starting point for our analysis is the observation that changing the activation function can markedly change the trajectory of a network in the information plane. In FIG0, we show our replication of the reported by for networks with the tanh nonlinearity.1 This replication was performed with the code supplied by the authors of , and closely follows the experimental setup described therein. Briefly, a neural network with 7 fully connected hidden layers of width 12-10-7-5-4-3-2 is trained with stochastic gradient descent to produce a binary classification from a 12-dimensional input. In our replication we used 256 randomly selected samples per batch. The mutual information of the network layers with respect to the input and output variables is calculated by binning the neuron's tanh output activations into 30 equal intervals between -1 and 1. Discretized values for each neuron in each layer are then used to directly calculate the joint distributions, over the 4096 equally likely input patterns and true output labels. In line with prior work , the dynamics in FIG0 show a;, no compression is observed except in the final classification layer with sigmoidal neurons. See Appendix B for the KDE MI method applied to the original Tishby dataset; additional using a second popular nonparametric k-NN-based method BID15; and for other neural nonlinearities.transition between an initial fitting phase, during which information about the input increases, and a subsequent compression phase, during which information about the input decreases. We then modified the code to train deep networks using rectified linear activation functions (f (x) = max(0, x)). While the activities of tanh networks are bounded in the range [−1, 1], ReLU networks have potentially unbounded positive activities. To calculate mutual information, we first trained the ReLU networks, next identified their largest activity value over the course of training, and finally chose 100 evenly spaced bins between the minimum and maximum activity values to discretize the hidden layer activity. The ing information plane dynamics are shown in FIG0. The mutual information with the input monotonically increases in all ReLU layers, with no apparent compression phase. To see whether our were an artifact of the small network size, toy dataset, or simple binning-based mutual information estimator we employed, we also trained larger networks on the MNIST dataset and computed mutual information using a state-of-the-art nonparametric kernel density estimator which assumes hidden activity is distributed as a mixture of Gaussians (see Appendix B for details). Fig. C-D show that, again, tanh networks compressed but ReLU networks did not. Appendix B shows that similar also obtain with the popular nonparametric k-nearest-neighbor estimator of BID15, and for other neural nonlinearities. Thus, the choice of nonlinearity substantively affects the dynamics in the information plane. To understand the impact of neural nonlinearity on the mutual information dynamics, we develop a minimal model that exhibits this phenomenon. In particular, consider the simple three neuron network shown in FIG1. We assume a scalar Gaussian input distribution X ∼ N, which is fed through the scalar first layer weight w 1, and passed through a neural nonlinearity f (·), yielding the hidden unit activity h = f (w 1 X). To calculate the mutual information with the input, this hidden unit activity is then binned yielding the new discrete variable T = bin(h) (for instance, into 30 evenly spaced bins from -1 to 1 for the tanh nonlinearity). This binning process is depicted in FIG1. In this simple setting, the mutual information I(T ; X) between the binned hidden layer activity T and the input X can be calculated exactly. In particular, DISPLAYFORM0 where H(·) denotes entropy, and we have used the fact that H(T |X) = 0 since T is a deterministic function of X. Here the probabilities p i = P (h ≥ b i and h < b i+1) are simply the probability that an input X produces a hidden unit activity that lands in bin i, defined by lower and upper bin limits b i and b i+1 respectively. This probability can be calculated exactly for monotonic nonlinearities f (·) using the cumulative density of X, DISPLAYFORM1 where f −1 (·) is the inverse function of f (·).As shown in FIG1 -D, as a function of the weight w 1, mutual information with the input first increases and then decreases for the tanh nonlinearity, but always increases for the ReLU nonlinearity. Intuitively, for small weights w 1 ≈ 0, neural activities lie near zero on the approximately linear part of the tanh function. Therefore f (w 1 X) ≈ w 1 X, yielding a rescaled Gaussian with information that grows with the size of the weights. However for very large weights w 1 → ∞, the tanh hidden unit nearly always saturates, yielding a discrete variable that concentrates in just two bins. This is more or less a coin flip, containing mutual information with the input of approximately 1 bit. Hence the distribution of T collapses to a much lower entropy distribution, yielding compression for large weight values. With the ReLU nonlinearity, half of the inputs are negative and land in the bin containing a hidden activity of zero. The other half are Gaussian distributed, and thus have entropy that increases with the size of the weight. Hence double-saturating nonlinearities can lead to compression of information about the input, as hidden units enter their saturation regime, due to the binning procedure used to calculate mutual information. The crux of the issue is that the actual I(h; X) is infinite, unless the network itself adds noise to the hidden layers. In particular, without added noise, the transformation from X to the continuous hidden activity h is deterministic and the mutual information I(h; X) would generally be infinite (see Appendix C for extended discussion). Networks that include noise in their processing (e.g.,) can have finite I(T ; X). Otherwise, to obtain a finite MI, one must compute mutual information as though there were binning or added noise in the activations. But this binning/noise is not actually a part of the operation of the network, and is therefore somewhat arbitrary (different binning schemes can in different mutual information with the input, as shown in Fig. 14 of Appendix C).We note that the binning procedure can be viewed as implicitly adding noise to the hidden layer activity: a range of X values map to a single bin, such that the mapping between X and T is no longer perfectly invertible BID17. The binning procedure is therefore crucial to obtaining a finite MI value, and corresponds approximately to a model where noise enters the system after the calculation of h, that is, T = h +, where is noise of fixed variance independent from h and X. This approach is common in information theoretic analyses of deterministic systems, and can serve as a measure of the complexity of a system's representation (see Sec 2.4 of). However, neither binning nor noise is present in the networks that considered, nor the ones in FIG1, either during training or testing. It therefore remains unclear whether robustness of a representation to this sort of noise in fact influences generalization performance in deep learning systems. Furthermore, the addition of noise means that different architectures may no longer be compared in a common currency of mutual information: the binning/noise structure is arbitrary, and architectures that implement an identical input-output map can nevertheless have different robustness to noise added in their internal representation. For instance, Appendix C describes a family of linear networks that compute exactly the same input-output map and therefore generalize identically, but yield different mutual information with respect to the input. Finally, we note that approaches which view the weights obtained from the training process as the random variables of interest may sidestep this issue BID0. Hence when a tanh network is initialized with small weights and over the course of training comes to saturate its nonlinear units (as it must to compute most functions of practical interest, see discussion in Appendix D), it will enter a compression period where mutual information decreases. FIG0 of Appendix E show histograms of neural activity over the course of training, demonstrating that activities in the tanh network enter the saturation regime during training. This nonlinearity-based compression furnishes another explanation for the observation that training slows down as tanh networks enter their compression phase : some fraction of inputs have saturated the nonlinearities, reducing backpropagated error gradients. The preceding section investigates the role of nonlinearity in the observed compression behavior, tracing the source to double-saturating nonlinearities and the binning methodology used to calculate mutual information. However, other mechanisms could lead to compression as well. Even without nonlinearity, neurons could converge to highly correlated activations, or project out irrelevant directions of the input. These phenomena are not possible to observe in our simple three neuron minimal model, as they require multiple inputs and hidden layer activities. To search for these mechanisms, we turn to a tractable model system: deep linear neural networks BID3;; Saxe et al. FORMULA1 ). In particular, we exploit recent on the generalization dynamics in simple linear networks trained in a student-teacher setup (; BID1 . In a student-teacher setting, one "student" neural network learns to approximate the output of another "teacher" neural network. This setting is a way of generating a dataset with interesting structure that nevertheless allows exact calculation of the generalization performance of the network, exact calculation of the mutual information of the representation (without any binning procedure), and, though we do not do so here, direct comparison to the IB bound which is already known for linear Gaussian problems BID6.We consider a scenario where a linear teacher neural network generates input and output examples which are then fed to a deep linear student network to learn FIG2. Following the formulation of BID1, we assume multivariate Gaussian inputs X ∼ N (0, 1 Ni I Ni) and a scalar output Y. The output is generated by the teacher network according to DISPLAYFORM0 represents aspects of the target function which cannot be represented by a neural network (that is, the approximation error or bias in statistical learning theory), and the teacher weights W o are drawn independently from N (0, σ 2 w). Here, the weights of the teacher define the rule to be learned. The signal to noise ratio SNR = σ 2 w /σ 2 o determines the strength of the rule linking inputs to outputs relative to the inevitable approximation error. We emphasize that the "noise" added to the teacher's output is fundamentally different from the noise added for the purpose of calculating mutual information: o models the approximation error for the task-even the best possible neural network may still make errors because the target function is not representable exactly as a neural network-and is part of the construction of the dataset, not part of the analysis of the student network. To train the student network, a dataset of P examples is generated using the teacher. The student network is then trained to minimize the mean squared error between its output and the target output using standard (batch or stochastic) gradient descent on this dataset. Here the student is a deep linear neural network consisting of potentially many layers, but where the the activation function of each neuron is simply f (u) = u. That is, a depth D deep linear network computes the output DISPLAYFORM1 While linear activation functions stop the network from computing complex nonlinear functions of the input, deep linear networks nevertheless show complicated nonlinear learning trajectories , the optimization problem remains nonconvex BID3, and the generalization dynamics can exhibit substantial overtraining BID10 BID1.Importantly, because of the simplified setting considered here, the true generalization error is easily shown to be DISPLAYFORM2 where W tot (t) is the overall linear map implemented by the network at training epoch t (that is, DISPLAYFORM3 Furthermore, the mutual information with the input and output may be calculated exactly, because the distribution of the activity of any hidden layer is Gaussian. Let T be the activity of a specific hidden layer, and letW be the linear map from the input to this activity (that is, for layer l,W = W l · · · W 2 W 1). Since T =W X, the mutual information of X and T calculated using differential entropy is infinite. For the purpose of calculating the mutual information, therefore, we assume that Gaussian noise is added to the hidden layer activity, T =W X + M I, with mean 0 and variance σ 2 M I = 1.0. This allows the analysis to apply to networks of any size, including overcomplete layers, but as before we emphasize that we do not add this noise either during training or testing. With these assumptions, T and X are jointly Gaussian and we have DISPLAYFORM4 where |·| denotes the determinant of a matrix. Finally the mutual information with the output Y, also jointly Gaussian, can be calculated similarly (see Eqns. FORMULA22 - FORMULA25 of Appendix G). Here the network has an input layer of 100 units, 1 hidden layer of 100 units each and one output unit. The network was trained with batch gradient descent on a dataset of 100 examples drawn from the teacher with signal to noise ratio of 1.0. The linear network behaves qualitatively like the ReLU network, and does not exhibit compression. Nevertheless, it learns a map that generalizes well on this task and shows minimal overtraining. Hence, in the setting we study here, generalization performance can be acceptable without any compression phase. The in BID1 ) show that, for the case of linear networks, overtraining is worst when the number of inputs matches the number of training samples, and is reduced by making the number of samples smaller or larger. FIG4 shows learning dynamics with the number of samples matched to the size of the network. Here overfitting is substantial, and again no compression is seen in the information plane. Comparing to the in FIG2, both networks exhibit similar information dynamics with respect to the input (no compression), but yield different generalization performance. Hence, in this linear analysis of a generic setting, there do not appear to be additional mechanisms that cause compression over the course of learning; and generalization behavior can be widely different for networks with the same dynamics of information compression regarding the input. We note that, in the setting considered here, all input dimensions have the same variance, and the weights of the teacher are drawn independently. Because of this, there are no special directions in the input, and each subspace of the input contains as much information as any other. It is possible that, in real world tasks, higher variance inputs are also the most likely to be relevant to the task (here, have large weights in the teacher). We have not investigated this possibility here. To see whether similar behavior arises in nonlinear networks, we trained tanh networks in the same setting as Section 2, but with 30% of the data, which we found to lead to modest overtraining. Next, we test a core theoretical claim of the information bottleneck theory of deep learning, namely that randomness in stochastic gradient descent is responsible for the compression phase. In particular, because the choice of input samples in SGD is random, the weights evolve in a stochastic way during training. distinguish two phases of SGD optimization: in the first "drift" phase, the mean of the gradients over training samples is large relative to the standard deviation of the gradients; in the second "diffusion" phase, the mean becomes smaller than the standard deviation of the gradients. The authors propose that compression should commence following the transition from a high to a low gradient signal-to-noise ratio (SNR), i.e., the onset of the diffusion phase. The proposed mechanism behind this diffusion-driven compression is as follows. The authors state that during the diffusion phase, the stochastic evolution of the weights can be described as a Fokker-Planck equation under the constraint of small training error. Then, the stationary distribution over weights for this process will have maximum entropy, again subject to the training error constraint. Finally, the authors claim that weights drawn from this stationary distribution will maximize the entropy of inputs given hidden layer activity, H(X|T), subject to a training error constraint, and that this training error constraint is equivalent to a constraint on the mutual information I(T ; Y) for small training error. Since the entropy of the input, H(X), is fixed, the of the diffusion dynamics will be to minimize I(X; T):= H(X) − H(X|T) for a given value of I(T ; Y) reached at the end of the drift phase. However, this explanation does not hold up to either theoretical or empirical investigation. Let us assume that the diffusion phase does drive the distribution of weights to a maximum entropy distribution subject to a training error constraint. Note that this distribution reflects stochasticity of weights across different training runs. There is no general reason that a given set of weights sampled from this distribution (i.e., the weight parameters found in one particular training run) will maximize H(X|T), the entropy of inputs given hidden layer activity. In particular, H(X|T) reflects (conditional) uncertainty about inputs drawn from the data-generating distribution, rather than uncertainty about any kind of distribution across different training runs. We also show empirically that the stochasticity of the SGD is not necessary for compression. To do so, we consider two distinct training procedures: offline stochastic gradient descent (SGD), which learns from a fixed-size dataset, and updates weights by repeatedly sampling a single example from the dataset and calculating the gradient of the error with respect to that single sample (the typical procedure used in practice); and batch gradient descent (BGD), which learns from a fixed-size dataset, and updates weights using the gradient of the total error across all examples. Batch gradient descent uses the full training dataset and, crucially, therefore has no randomness or diffusion-like behavior in its updates. We trained tanh and ReLU networks with SGD and BGD and compare their information plane dynamics in FIG5 (see Appendix H for a linear network). We find largely consistent information dynamics in both instances, with robust compression in tanh networks for both methods. Thus randomness in the training process does not appear to contribute substantially to compression of information about the input. This finding is consistent with the view presented in Section 2 that compression arises predominantly from the double saturating nonlinearity. Finally, we look at the gradient signal-to-noise ratio (SNR) to analyze the relationship between compression and the transition from high to low gradient SNR. FIG1 of Appendix I shows the gradient SNR over training, which in all cases shows a phase transition during learning. Hence the gradient SNR transition is a general phenomenon, but is not causally related to compression. Appendix I offers an extended discussion and shows gradient SNR transitions without compression on the MNIST dataset and for linear networks. Our finding that generalization can occur without compression may seem difficult to reconcile with the intuition that certain tasks involve suppressing irrelevant directions of the input. In the extreme, if certain inputs contribute nothing but noise, then good generalization requires ignoring them. To study this, we consider a variant on the linear student-teacher setup of Section 3: we partition the input X into a set of task-relevant inputs X rel and a set of task-irrelevant inputs X irrel, and alter the teacher network so that the teacher's weights to the task-irrelevant inputs are all zero. Hence the inputs X irrel contribute only noise, while the X rel contain signal. We then calculate the information plane dynamics for the whole layer, and for the task-relevant and task-irrelevant inputs separately. FIG7 shows information plane dynamics for a deep linear neural network trained using SGD (5 samples/batch) on a task with 30 task-relevant inputs and 70 task-irrelevant inputs. While the overall dynamics show no compression phase, the information specifically about the task-irrelevant subspace does compress over the course of training. This compression process occurs at the same time as the fitting to the task-relevant information. Thus, when a task requires ignoring some inputs, the The information with the task-relevant subspace increases robustly over training. (C) However, the information specifically about the task-irrelevant subspace does compress after initially growing as the network is trained.information with these inputs specifically will indeed be reduced; but overall mutual information with the input in general may still increase. Our suggest that compression dynamics in the information plane are not a general feature of deep networks, but are critically influenced by the nonlinearities employed by the network. Doublesaturating nonlinearities lead to compression, if mutual information is estimated by binning activations or by adding homoscedastic noise, while single-sided saturating nonlinearities like ReLUs do not compress in general. Consistent with this view, we find that stochasticity in the training process does not contribute to compression in the cases we investigate. Furthermore, we have found instances where generalization performance does not clearly track information plane behavior, questioning the causal link between compression and generalization. Hence information compression may parallel the situation with sharp minima: although empirical evidence has shown a correlation with generalization error in certain settings and architectures, further theoretical analysis has shown that sharp minima can in fact generalize well BID9. We emphasize that compression still may occur within a subset of the input dimensions if the task demands it. This compression, however, is interleaved rather than in a secondary phase and may not be visible by information metrics that track the overall information between a hidden layer and the input. Finally, we note that our address the specific claims of one scheme to link the information bottleneck principle with current practice in deep networks. The information bottleneck principle itself is more general and may yet offer important insights into deep networks BID0. Moreover, the information bottleneck principle could yield fundamentally new training algorithms for networks that are inherently stochastic and where compression is explicitly encouraged with appropriate regularization terms BID5 BID2 This Appendix investigates the generality of the finding that compression is not observed in neural network layers with certain activation functions. FIG0 of the main text shows example using a binning-based MI estimator and a nonparametric KDE estimator, for both the tanh and ReLU activation functions. Here we describe the KDE MI estimator in detail, and present extended on other datasets. We also show for other activation functions. Finally, we provide entropy estimates based on another nonparametric estimator, the popular k-nearest neighbor approach of BID15. Our findings consistently show that double-saturating nonlinearities can yield compression, while single-sided nonlinearities do not. The KDE approach of; estimates the mutual information between the input and the hidden layer activity by assuming that the hidden activity is distributed as a mixture of Gaussians. This assumption is well-suited to the present setting under the following interpretation: we take the input activity to be distributed as delta functions at each example in the dataset, corresponding to a uniform distribution over these specific samples. In other words, we assume that the empirical distribution of input samples is the true distribution. Next, the hidden layer activity h is a deterministic function of the input. As mentioned in the main text and discussed in more detail in Appendix C, without the assumption of noise, this would have infinite mutual information with the input. We therefore assume for the purposes of analysis that Gaussian noise of variance σ 2 is added, that is, T = h + where ∼ N (0, σ 2 I). Under these assumptions, the distribution of T is genuinely a mixture of Gaussians, with a Gaussian centered on the hidden activity corresponding to each input sample. We emphasize again that the noise is added solely for the purposes of analysis, and is not present during training or testing the network. In this setting, an upper bound for the mutual information with the input is ) DISPLAYFORM0 where P is the number of training samples and h i denotes the hidden activity vector in response to input sample i. Similarly, the mutual information with respect to the output can be calculated as DISPLAYFORM1 DISPLAYFORM2 where L is the number of output labels, P l denotes the number of data samples with output label l, p l = P l /P denotes the probability of output label l, and the sums over i, Y i = l indicate a sum over all examples with output label l. FIG10 -B shows the of applying this MI estimation method on the dataset and network architecture of , with MI estimated on the full dataset and averaged over 50 repetitions. Mutual information was estimated using data samples from the test set, and we took the noise variance σ 2 = 0.1. These look similar to the estimate derived from binning, with compression in tanh networks but no compression in ReLU networks. Relative to the binning estimate, it appears that compression is less pronounced in the KDE method. The network was trained using SGD with minibatches of size 128. As before, mutual information was estimated using data samples from the test set, and we took the noise variance σ 2 = 0.1. The smaller layer sizes in the top three hidden layers were selected to ensure the quality of the kernel density estimator given the amount of data in the test set, since the estimates are more accurate for smaller-dimensional data. Because of computational expense, the MNIST are from a single training run. More detailed for the MNIST dataset are provided in FIG12 for the tanh activation function, and in FIG0 for the ReLU activation function. In these figures, the first row shows the evolution of the cross entropy loss (on both training and testing data sets) during training. The second row shows the mutual information between input and the activity of different hidden layers, using the nonparametric KDE estimator described above. The blue region in the second row shows the range of possible MI values, ranging from the upper bound described above (Eq. 10) to the following lower bound, DISPLAYFORM3 The third row shows the mutual information between input and activity of different hidden layers, estimated using the binning method (here, the activity of each neuron was discretized into bins of size 0.5). For both the second and third rows, we also plot the entropy of the inputs, H(X), as a dashed line. H(X) is an upper bound on the mutual information I(X; T), and is computed using the assumption of a uniform distribution over the 10,000 testing points in the MNIST dataset, giving H(X) = log 2 10000.Finally, the fourth row visualizes the dynamics of the SGD updates during training. For each layer and epoch, the green line shows the 2 norm of the weights. We also compute the vector of mean updates across SGD minibatches (this vector has one dimension for each weight parameter), as well as the vector of the standard deviation of the updates across SGD minibatches. The 2 norm of the mean update vector is shown in blue, and the 2 norm of the standard deviation vector is shown in orange. The gradient SNR, computed as the ratio of the norm of the mean vector to the norm of the standard deviation vector, is shown in red. For both the tanh and ReLU networks, the gradient SNR shows a phase transition during training, and the norm of the weights in each layer increases. Importantly, this phase transition occurs despite a lack of compression in the ReLU network, indicating that noise in SGD updates does not yield compression in this setting. Upper and lower bounds for the mutual information I(X; T) between the input (X) and each layer's activity (T), using the nonparametric KDE estimator. Dotted line indicates H(X) = log 2 10000, the entropy of a uniform distribution over 10,000 testing samples. Row 3: Binning-based estimate of the mutual information I(X; T), with each neuron's activity discretized using a bin size of 0.5. Row 4: Gradient SNR and weight norm dynamics. The gradient SNR shows a phase transition during training, and the norm of the weights in each layer increases. Upper and lower bounds for the mutual information I(X; T) between the input (X) and each layer's activity (T), using the nonparametric KDE estimator. Dotted line indicates H(X) = log 2 10000, the entropy of a uniform distribution over 10,000 testing samples. Row 3: Binning-based estimate of the mutual information I(X; T), with each neuron's activity discretized using a bin size of 0.5. Row 4: Gradient SNR and weight norm dynamics. The gradient SNR shows a phase transition during training, and the norm of the weights in each layer increases. Importantly, this phase transition occurs despite a lack of compression in the ReLU network, indicating that noise in SGD updates does not yield compression in this setting. Next, in FIG10 -D, we show from the kernel MI estimator from two additional nonlinear activation functions, the softsign function DISPLAYFORM0 and the softplus function DISPLAYFORM1 These functions are plotted next to tanh and ReLU in FIG0. The softsign function is similar to tanh but saturates more slowly, and yields less compression than tanh. The softplus function is a smoothed version of the ReLU, and yields similar dynamics with no compression. Because softplus never saturates fully to zero, it retains more information with respect to the input than ReLUs in general. We additionally investigated the widely-used nonparametric MI estimator of BID15. This estimator uses nearest neighbor distances between samples to compute an estimate of the entropy of a continuous random variable. Here we focused for simplicity only on the compression phenomenon in the mutual information between the input and hidden layer activity, leaving aside the information with respect to the output (as this is not relevant to the compression phenomenon). Again, without additional noise assumptions, the MI between the hidden representation and the input would be infinite because the mapping is deterministic. Rather than make specific noise assumptions, we instead use the Kraskov method to estimate the entropy of the hidden representations T. Note that the entropy of T is the mutual information up to an unknown constant so long as the noise assumption is homoscedastic, that is, T = h + Z where the random variable Z is independent of X. To see this, note that DISPLAYFORM0 where the constant c = H(Z). Hence observing compression in the layer entropy H(T) is enough to establish that compression occurs in the mutual information. The Kraskov estimate is given by DISPLAYFORM1 where d is the dimension of the hidden representation, P is the number of samples, r i is the distance to the k-th nearest neighbor of sample i, is a small constant for numerical stability, Γ(·) is the Gamma function, and ψ(·) is the digamma function. Here the parameter prevents infinite terms when the nearest neighbor distance r i = 0 for some sample. We took = 10 −16. FIG0 shows the entropy over training for tanh and ReLU networks trained on the dataset of and with the network architecture in , averaged over 50 repeats. In these experiments, we used k = 2. Compression would correspond to decreasing entropy over the course of training, while a lack of compression would correspond to increasing entropy. Several tanh layers exhibit compression, while the ReLU layers do not. Hence qualitatively, the Kraskov estimator returns similar to the binning and KDE strategies. A B FIG0: Entropy dynamics over training for the network architecture and training dataset of Shwartz-Ziv & Tishby FORMULA7, estimated with the nonparametric k-nearest-neighbor-based method of BID15. Here the x-axis is epochs of training time, and the y-axis plots the entropy of the hidden representation, as calculated using nearest-neighbor distances. Note that in this setting, if T is considered to be the hidden activity plus independent noise, the entropy is equal to the mutual information up to a constant (see derivation in text). Layers 0-4 correspond to the hidden layers of size 10-7-5-4-3. (A) tanh neural network layers can show compression over the course of training.(B) ReLU neural network layers show no compression. A recurring theme in the reported in this paper is the necessity of noise assumptions to yield a nontrivial information theoretic analysis. Here we give an extended discussion of this phenomenon, and of issues relating to discrete entropy as opposed to continuous (differential) entropy. The activity of a neural network is often a continuous deterministic function of its input. That is, in response to an input X, a specific hidden layer might produce activity h = f (X) for some function f. The mutual information between h and X is given by DISPLAYFORM0 If h were a discrete variable, then the entropy would be given by DISPLAYFORM1 where p i is the probability of the discrete symbol i, as mentioned in the main text. Then H(h|X) = 0 because the mapping is deterministic and we have I(h; X) = H(h).However h is typically continuous. The continuous entropy, defined for a continuous random variable Z with density p Z by analogy to Eqn. FORMULA8 as DISPLAYFORM2 can be negative and possibly infinite. In particular, note that if p Z is a delta function, then H(Z) = −∞. The mutual information between hidden layer activity h and the input X for continuous h, X is DISPLAYFORM3 Now H(h|X) = −∞ since given the input X, the hidden activity h is distributed as a delta function at f (X). The mutual information is thus generally infinite, so long as the hidden layer activity has finite entropy (H(h) is finite).Figure 13: Effect of binning strategy on minimal three neuron model. Mutual information for the simple three neuron model shown in FIG1 with bin edges b i ∈ tanh(linspace(−50, 50, N)). In contrast to linear binning, the mutual information continues to increase as weights grow. To yield a finite mutual information, some noise in the mapping is required such that H(h|X) remains finite. A common choice (and one adopted here for the linear network, the nonparametric kernel density estimator, and the k-nearest neighbor estimator) is to analyze a new variable with additive noise, T = h + Z, where Z is a random variable independent of X. Then H(T |X) = H(Z) which allows the overall information I(T ; X) = H(T) − H(Z) to remain finite. This noise assumption is not present in the actual neural networks either during training or testing, and is made solely for the purpose of calculating the mutual information. Another strategy is to partition the continuous variable h into a discrete variable T, for instance by binning the values (the approach taken in). This allows use of the discrete entropy, which remains finite. Again, however, in practice the network does not operate on the binned variables T but on the continuous variables h, and the binning is solely for the purpose of calculating the mutual information. Moreover, there are many possible binning strategies, which yield different discrete random variables, and different mutual information with respect to the input. The choice of binning strategy is an assumption analogous to choosing a type of noise to add to the representation in the continuous case: because there is in fact no binning in the operation of the network, there is no clear choice for binning methodology. The strategy we use in binning-based experiments reported here is the following: for bounded activations like the tanh activation, we use evenly spaced bins between the minimum and maximum limits of the function. For unbounded activations like ReLU, we first train the network completely; next identify the minimum and maximum hidden activation over all units and all training epochs; and finally bin into equally spaced bins between these minimum and maximum values. We note that this procedure places no restriction on the magnitude that the unbounded activation function can take during training, and yields the same MI estimate as using infinite equally spaced bins (because bins for activities larger than the maximum are never seen during training).As an example of another binning strategy that can yield markedly different , we consider evenly spaced bins in a neuron's net input, rather than its activity. That is, instead of evenly spaced bins in the neural activity, we determine the bin edges by mapping a set of evenly spaced values through the neural nonlinearity. For tanh, for instance, this spaces bins more tightly in the saturation region as compared to the linear region. FIG0 shows the of applying this binning strategy FIG0: Effect of binning strategy on information plane dynamics. Results for the same tanh network and training regime as 1A, but with bin edges b i ∈ tanh(linspace(−50, 50, N)). Measured with this binning structure, there is no compression in most layers.to the minimal three neuron model with tanh activations. This binning scheme captures more information as the weights of the network grow larger. FIG0 shows information plane dynamics for this binning structure. The tanh network no longer exhibits compression. (We note that the broken DPI in this example is an artifact of performing binning only for analysis, as discussed below).Any implementation of a neural network on digital hardware is ultimately of finite precision, and hence is a binned, discrete representation. However, it is a very high resolution binning compared to that used here or by: single precision would correspond to using roughly 2 32 bins to discretize each hidden unit's activity, as compared to the 30-100 used here. If the binning is fine-grained enough that each input X yields a different binned activity pattern h, then H(h) = log(P) where P is the number of examples in the dataset, and there will be little to no change in information during training. As an example, we show in FIG0 the of binning at full machine precision. Finally, we note two consequences of the assumption of noise/binning for the purposes of analysis. First, this means that the data processing inequality (DPI) does not apply to the noisy/binned mutual information estimates. The DPI states that information can only be destroyed through successive transformations, that is, if X → h 1 → h 2 form a Markov chain, then I(X; h 1) ≥ I(X; h 2) (see, eg,). Because noise is added only for the purpose of analysis, however, this does not apply here. In particular, for the DPI to apply, the noise added at lower layers would have to propagate through the network to higher layers. That is, if the transformation from hidden layer 1 to hidden layer 2 is h 2 = f (h 1) and T 1 = h 1 + Z 1 is the hidden layer activity after adding noise, then the DPI would hold for the variableT 2 = f (T 1) + Z 2 = f (h 1 + Z 1) + Z 2, not the quantity Information in most layers stays pinned to log 2 (P) = 12. Compression is only observed in the highest and smallest layers near the very end of training, when the saturation of tanh is strong enough to saturate machine precision. DISPLAYFORM4 used in the analysis. Said another way, the Markov chain for DISPLAYFORM5, so the DPI states only that I(X; h 1) ≥ I(X; T 2).A second consequence of the noise assumption is the fact that the mutual information is no longer invariant to invertible transformations of the hidden activity h. A potentially attractive feature of a theory based on mutual information is that it can allow for comparisons between different architectures: mutual information is invariant to any invertible transformation of the variables, so two hidden representations could be very different in detail but yield identical mutual information with respect to the input. However, once noise is added to a hidden representation, this is no longer the case: the variable T = h + Z is not invariant to reparametrizations of h. As a simple example, consider a minimal linear network with scalar weights w 1 and w 2 that computes the outputŷ = w 2 w 1 X. The hidden activity is h = w 1 X. Now consider the family of networks in which we scale down w 1 and scale up w 2 by a factor c = 0, that is, these networks have weightsw 1 = w 1 /c andw 2 = cw 2, yielding the exact same input-output mapŷ =w 2w1 X = cw 2 (w 1 /c)X = w 2 w 1 X. Because they compute the same function, they necessarily generalize identically. However after introducing the noise assumption the mutual information is DISPLAYFORM6 where we have taken the setting in Section 3 in which X is normal Gaussian, and independent Gaussian noise of variance σ 2 M I is added for the purpose of MI computation. Clearly, the mutual information is now dependent on the scaling c of the internal layer, even though this is an invertible linear transformation of the representation. Moreover, this shows that networks which generalize identically can nevertheless have very different mutual information with respect to the input when it is measured in this way. Our argument relating neural saturation to compression in mutual information relies on the notion that in typical training regimes, weights begin small and increase in size over the course of training. We note that this is a virtual necessity for a nonlinearity like tanh, which is linear around the origin: when initialized with small weights, the activity of a tanh network will be in this linear regime and the network can only compute a linear function of its input. Hence a real world nonlinear task can only be learned by increasing the norm of the weights so as to engage the tanh nonlinearity on some examples. This point can also be appreciated from norm-based capacity bounds on neural networks, which show that, for instance, the Rademacher complexity of a neural network with small weights must be low BID4 ). Finally, as an empirical matter, the networks trained in this paper do in fact increase the norm of their weights over the course of For the linear setting considered here, the mutual information between a hidden representation T and the output Y may be calculated using the relations DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 H STOCHASTIC VS BATCH TRAINING FIG0 shows information plane dynamics for stochastic and batch gradient descent learning in a linear network. Randomness in the training process does not dramatically alter the information plane dynamics. The proposed mechanism of compression in is noise arising from stochastic gradient descent training. The in Section 4 of the main text show that compression still occurs under batch gradient descent learning, suggesting that in fact noise in the gradient updates is not the cause of compression. Here we investigate a related claim, namely that during training, networks switch between two phases. These phases are defined by the ratio of the mean of the gradient to the standard deviation of the gradient across training examples, called the gradient signal-to-noise ratio. In the first "drift" phase, the SNR is high, while in the second "diffusion" phase the SNR is low. hypothesize that the drift phase corresponds to movement toward the minimum with no compression, while the diffusion phase corresponds to a constrained diffusion in weight configurations that attain the optimal loss, during which representations compress. However, two phases of gradient descent have been described more generally, sometimes known as the transient and stochastic phases or search and convergence phases BID21 BID7, suggesting that these phases might not be related specifically to compression behavior. In FIG1 we plot the gradient SNR over the course of training for the tanh and ReLU networks in the standard setting of. In particular, for each layer l we calculate the mean and standard deviation as DISPLAYFORM0 DISPLAYFORM1 where · denotes the mean and ST D(·) denotes the element-wise standard deviation across all training samples, and · F denotes the Frobenius norm. The gradient SNR is then the ratio m l /s l. We additionally plot the norm of the weights W l F over the course of training. Both tanh and ReLU networks yield a similar qualitative pattern, with SNR undergoing a step-like transition to a lower value during training. Figures 9 and 10, fourth row, show similar plots for MNIST-trained networks. Again, SNR undergoes a transition from high to low over training. Hence the two phase nature of gradient descent appears to hold across the settings that we examine here. Crucially, this finding shows that the SNR transition is not related to the compression phenomenon because ReLU networks, which show the gradient SNR phase transition, do not compress. Finally, to show the generality of the two-phase gradient SNR behavior and its independence from compression, we develop a minimal model of this phenomenon in a three neuron linear network. We consider the student-teacher setting of FIG2 but with N i = N h = 1, such that the input and hidden layers have just a single neuron (as in the setting of FIG1). Here, with just a single hidden neuron, clearly there can be no compression so long as the first layer weight increases over the course of training. FIG0 shows that even in this simple setting, the SNR shows the phase transition but the weight norm increases over training. Hence again, the two phases of the gradient are present even though there is no compression. To intuitively understand the source of this behavior, note that the weights are initialized to be small and hence early in learning all must be increased in magnitude, yielding a consistent mean gradient. Once the network reaches the vicinity of the minimum, the mean weight change across all samples by definition goes to zero. The standard deviation remains finite, however, because on some specific examples error could be improved by increasing or decreasing the weights-even though across the whole dataset the mean error has been minimized. Hence overall, our show that a two-phase structure in the gradient SNR occurs in all settings we consider, even though compression occurs only in a subset. The gradient SNR behavior is therefore not causally related to compression dynamics, consistent with the view that saturating nonlinearities are the primary source of compression.

We establish that the scaling laws derived in (Bora et al., 2017) are optimal or near-optimal in the absence of further assumptions.