Question: Represent the scientific article to find a short TLDR

Question: In the Information Bottleneck (IB), when tuning the relative strength between compression and prediction terms, how do the two terms behave, and what's their relationship with the dataset and the learned representation? In this paper, we set out to answer these questions by studying multiple phase transitions in the IB objective: IB_β[p(z|x)] = I(X; Z) − βI(Y; Z) defined on the encoding distribution p(z|x) for input X, target Y and representation Z, where sudden jumps of dI(Y; Z)/dβ and prediction accuracy are observed with increasing β. We introduce a definition for IB phase transitions as a qualitative change of the IB loss landscape, and show that the transitions correspond to the onset of learning new classes. Using second-order calculus of variations, we derive a formula that provides a practical condition for IB phase transitions, and draw its connection with the Fisher information matrix for parameterized models. We provide two perspectives to understand the formula, revealing that each IB phase transition is finding a component of maximum (nonlinear) correlation between X and Y orthogonal to the learned representation, in close analogy with canonical-correlation analysis (CCA) in linear settings. Based on the theory, we present an algorithm for discovering phase transition points. Finally, we verify that our theory and algorithm accurately predict phase transitions in categorical datasets, predict the onset of learning new classes and class difficulty in MNIST, and predict prominent phase transitions in CIFAR10. The Information Bottleneck (IB) objective : explicitly trades off model compression (I(X; Z), I(·; ·) denoting mutual information) with predictive performance (I(Y ; Z)) using the Lagrange multiplier β, where X, Y are observed random variables, and Z is a learned representation of X. The IB method has proved effective in a variety of scenarios, including improving the robustness against adversarial attacks , learning invariant and disentangled representations (a; b), underlying information-based geometric clustering (b), improving the training and performance in adversarial learning , and facilitating skill discovery and learning goal-conditioned policy in reinforcement learning. From Eq. we see that when β → 0 it will encourage I(X; Z) = 0 which leads to a trivial representation Z that is independent of X, while when β → +∞, it reduces to a maximum likelihood objective 1 that does not constrain the information flow. Between these two extremes, how will the IB objective behave? Will prediction and compression performance change smoothly, or do there exist interesting transitions in between? , the authors observe and study the learnability transition, i.e. the β value such that the IB objective transitions from a trivial global minimum to learning a nontrivial representation. They also show how this first phase transition relates to the structure of the dataset. However, to answer the full question, we need to consider the full range of β. Motivation. To get a sense of how I(Y ; Z) and I(X; Z) vary with β, we train Variational Information Bottleneck (VIB) models on the CIFAR10 dataset , where each experiment is at a different β and random initialization of the model. Fig. 1 shows the I(X; Z), I(Y ; Z) and accuracy vs. β, as well as I(Y ; Z) vs. I(X; Z) for CIFAR10 with 20% label noise (see Appendix I for details). are discontinuous and the accuracy has discrete jumps. The observation lets us refine our question: When do the phase transitions occur, and how do they depend on the structure of the dataset? These questions are important, since answering them will help us gain a better understanding of the IB objective and its close interplay with the dataset and the learned representation. Moreover, the IB objective belongs to a general form of two-term trade-offs in many machine learning objectives: L = Prediction-loss + β · Complexity, where the complexity term generally takes the form of regularization. Usually, learning is set at a specific β. Many more insights can be gained if we understand the behavior of the prediction loss and model complexity with varying β, and how they depend on the dataset. The techniques developed to address the question in the IB setting may also help us understand the two-term tradeoff in other learning objectives. Contributions. In this work, we begin to address the above question in IB settings. Specifically: • We identify a qualitative change of the IB loss landscape w.r.t. p(z|x) for varying β as IB phase transitions (Section 3). • Based on the definition, we introduce a quantity G[p(z|x)] and use it to prove a theorem giving a practical condition for IB phase transitions. We further reveal the connection between G[p(z|x)] and the Fisher information matrix when p(z|x) is parameterized by θ (Section 3). • We reveal the close interplay between the IB objective, the dataset and the learned representation, by showing that in IB, each phase transition corresponds to learning a new nonlinear component of maximum correlation between X and Y, orthogonal to the previously-learned Z, and each with decreasing strength (Section 4). To the best of our knowledge, our work provides the first theoretical formula to address IB phase transitions in the most general setting. In addition, we present an algorithm for iteratively finding the IB phase transition points (Section 5). We show that our theory and algorithm give tight matches with the observed phase transitions in categorical datasets, predict the onset of learning new classes and class difficulty in MNIST, and predict prominent transitions in CIFAR10 experiments (Section 6). The Information Bottleneck Method provides a tabular method based on the Blahut-Arimoto (BA) Algorithm to numerically solve the IB functional for the optimal encoder distribution P (Z|X), given the trade-off parameter β and the cardinality of the representation variable Z. This work has been extended in a variety of directions, including to the case where all three variables X, Y, Z are multivariate Gaussians , cases of variational bounds on the IB and related functionals for amortized learning (; a;), and a more generalized interpretation of the constraint on model complexity as a Kolmogorov Structure Function . Previous theoretical analyses of IB include , which looks at IB through the lens of copula functions, and , which starts to tackle the question of how to bound generalization with IB. We will make practical use of the original IB algorithm, as well as the amortized bounds of the Variational Informormation Bottleneck and the Conditional Entropy Bottleneck . Phase transitions, where key quantities change discontinuously with varying relative strength in the two-term trade-off, have been observed in many different learning domains, for multiple learning objectives. , the authors observe phase transitions in the latent representation of β-VAE for varying β. Strouse & Schwab (2017b) utilize the kink angle of the phase transitions in the Deterministic Information Bottleneck (DIB) (a) to determine the optimal number of clusters for geometric clustering. explicitly considers critical points in binary classification tasks using a discrete information bottleneck with a non-convex Pareto-optimal frontier. In Achille & Soatto (2018a) x, and Σ x is the covariance matrix. This work provides valuable insights for IB, but is limited to the special case that X, Y and Z are jointly Gaussian. Phase transitions in the general IB setting have also been observed, which describes as "information bifurcation". , the authors study the first phase transition, i.e. the learnability phase transition, and provide insights on how the learnability depends on the dataset. Our work is the first work that addresses all the IB phase transitions in the most general setting, and provides theoretical insights on the interplay between the IB objective, its phase transitions, the dataset, and the learned representation. 3 FORMULA FOR IB PHASE TRANSITIONS 3.1 DEFINITIONS Let X ∈ X, Y ∈ Y, Z ∈ Z be random variables denoting the input, target and representation, respectively, having a joint probability distribution p(X, Y, Z), with X × Y × Z its support. X, Y and Z satisfy the Markov chain Z − X − Y, i.e. Y and Z are conditionally independent given X. We assume that the integral (or summing if X, Y or Z are discrete random variables) is on X × Y × Z. We use x, y and z to denote the instances of the respective random variables. The above settings are used throughout the paper. We can view the IB objective IB β [p(z|x)] (Eq. 1) as a functional of the encoding distribution p(z|x). To prepare for the introduction of IB phase transitions, we first define relative perturbation function and second variation, as follows. Definition 1. Relative perturbation function: For p(z|x), its relative perturbation function r(z|x) is a bounded function that maps X × Z to R and satisfies E z∼p(z|x) [r(z|x)] = 0. Formally, define We have that r(z|x) ∈ Q Z|X iff r(z|x) is a relative perturbation function of p(z|x). The perturbed probability (density) is p (z|x) = p(z|x) (1 + · r(z|x)) for some > 0. Definition 2. Second variation: Let functional F [f (x)] be defined on some normed linear space R. Let us add a perturbative function · h(x) to f (x), and now the functional F [f (x) + · h(x)] can be expanded as is a linear functional of ·h(x), and is called the first variation, denoted as δF is a quadratic functional of · h(x), and is called the second variation, denoted as δ We can think of the perturbation function · h(x) as an infinite-dimensional "vector" (x being the indices), with being its amplitude and h(x) its direction. Here β + and 0 − denote one-sided limits. We can understand the δ 2 IB β [p(z|x)] as a local "curvature" of the IB objective IB β (Eq. 1) w.r.t. p(z|x), along some relative perturbation r(z|x). A phase transition occurs when the convexity of IB β [p(z|x)] w.r.t. p(z|x) changes from a minimum to a saddle point in the neighborhood of its optimal solution p * β (z|x) as β increases from β c to β c + 0 +. This means that there exists a perturbation to go downhill and find a better minimum. We validate this definition empirically below. The definition for IB phase transition (Definition 3) indicates the important role δ 2 IB β [p(z|x)] plays on the optimal solution in providing the condition for phase transitions. To concretize it and prepare for a more practical condition for IB phase transitions, we expand IB β [p(z|x)(1 + · r(z|x))] to the second order of, giving: The proof is given in Appendix B, in which we also give Eq. for empirical estimation. Note that Lemma 0.1 is very general and can be applied to any p(z|x), not only at the optimal solution p * β (z|x). The Fisher Information matrix. In practice, the encoder p θ (z|x) is usually parameterized by some parameter vector θ = (θ 1, θ 2, ...θ k) T ∈ Θ, e.g. weights and biases in a neural net, where Θ is the parameter field. An infinitesimal change of θ ← θ + ∆θ induces a relative perturbation · r(z|x) ∆θ, from which we can compute the threshold function where are the conditional Fisher information matrix of θ for Z conditioned on X and Y, respectively. λ max is the largest eigenvalue of C −1 I Z|Y (θ) − I Z (θ) (C T) −1 with v max the corresponding eigenvector, where CC T is the Cholesky decomposition of the matrix I Z|X (θ) − I Z (θ), and v max is the eigenvector for λ max. The infimum is attained at ∆θ = (The proof is in appendix C. We see that for parameterized encoders p θ (z|x), each term of G[p(z|x)] in Eq. can be replaced by a bilinear form with the Fisher information matrix of the respective variables. Although this lemma is not required to understand the more general setting of Lemma 0.1, where the model is described in a functional space, Lemma 0.2 helps understand G[p(z|x)] for parameterized models, which permits directly linking the phase transitions to the model's parameters. Phase Transitions. Now we introduce Theorem 1 that gives a concrete and practical condition for IB phase transitions, which is the core of the paper: Theorem 1. The IB phase transition points {β c i} as defined in Definition 3 are given by the roots of the following equation: where We can understand Eq. as the condition when δ 2 IB β [p(z|x)] is about to be able to be negative at the optimal solution p * β (z|x) for a given β. The proof for Theorem 1 is given in Appendix D. In Section 4, we will analyze Theorem 1 in detail. In this section we set out to understand G[p(z|x)] as given by Eq. and the phase transition condition as given by Theorem 1, from the perspectives of Jensen's inequality and representational maximum correlation. The condition for IB phase transitions given by Theorem 1 involves which is in itself an optimization problem. We can understand using Jensen's inequality: The equality between A and B holds when the perturbation r(z|x) is constant w.r.t. x for any z; the equality between B and C holds when E x∼p(x|y,z) [r(z|x)] is constant w.r.t. y for any z. Therefore, the minimization of A−C B−C encourages the relative perturbation function r(z|x) to be as constant w.r.t. x as possible (minimizing intra-class difference), but as different w.r.t. different y as possible (maximizing inter-class difference), ing in a clustering of the values of r(z|x) for different examples x according to their class y. Because of this clustering property in classification problems, we conjecture that there are at most |Y| − 1 phase transitions, where |Y| is the number of classes, with each phase transition differentiating one or more classes. Under certain conditions we can further simplify G[p(z|x)] and gain a deeper understanding of it. Firstly, inspired by maximum correlation , we introduce two new concepts, representational maximum correlation and conditional maximum correlation, as follows. Definition 4. Given a joint distribution p(X, Y), and a representation Z satisfying the Markov chain Z − X − Y, the representational maximum correlation ρ r (X, Y ; Z) is defined as where The conditional maximum correlation ρ m (X, Y |Z) is defined as: where We prove the following Theorem 2, which expresses G[p(z|x)] in terms of representational maximum correlation and related quantities, with proof given in Appendix F. Z|X and Q Z|X satisfy:, then we have: (i) The representation maximum correlation and G: (ii) The representational maximum correlation and conditional maximum correlation: where z * = arg max z∈Z ρ m (X, Y |Z = z), and h * (x) is the optimal solution for the learn- (iv) For discrete X, Y and Z, we have where σ 2 (Z) is the second largest singular value of the matrix Q X,Y |Z:= Theorem 2 furthers our understanding of G[p(z|x)] and the phase transition condition (Theorem 1), which we elaborate as follows. Discovering maximum correlation in the orthogonal space of a learned representation: Intuitively, the representational maximum correlation measures the maximum linear correlation between f (X, Z) and g(Y, Z) among all real-valued functions f, g, under the constraint that f (X, Z) is "orthogonal" to p(X|Z) and is the inverse square of this representational maximum correlation. Theorem 2 (ii) further shows that G[p(z|x)] is finding a specific z * on which maximum (nonlinear) correlation between X and Y 2 For discrete X, Z such that the cardinality |Z| ≥ |X |, this is generally true since in this scenario, h(x, z) and s(z) have |X ||Z| + |Z| unknown variables, but the condition has only |X ||Z| + |X | linear equations. The difference between Q Z|X and Q conditioned on Z can be found. Combined with Theorem 1, we have that when we continuously increase β, for the optimal representation Z * β given by p * β (z|x) at β, ρ r (X, Y ; Z * β) shall monotonically decrease due to that X and Y has to find their maximum correlation on the orthogonal space of an increasingly better representation Z * β that captures more information about X. A phase transition occurs when ρ r (X, Y ; Z * β) reduces to, after which as β continues to increase, ρ r (X, Y ; Z * β) will try to find maximum correlation between X and Y orthogonal to the full previously learned representation. This is reminiscent of canonical-correlation analysis (CCA) in linear settings, where components with decreasing linear maximum correlation that are orthogonal to previous components are found one by one. In comparison, we show that in IB, each phase transition corresponds to learning a new nonlinear component of maximum correlation between X and Y in Z, orthogonal to the previously-learned Z. In the case of classification where different classes may have different difficulty (e.g. due to label noise or support overlap), we should expect that classes that are less difficult as measured by a larger maximum correlation between X and Y are learned earlier. Conspicuous subset conditioned on a single z: Furthermore, we show in (iii) that an optimal relative perturbation function r(z|x) can be decomposed into a product of two factors, a factor that only focus on perturbing a specific point z * in the representation space, and an h * (x) factor that is finding the "conspicuous subset" , i.e. the most confident, large, typical, and imbalanced subset in the X space for the distribution Singular values In categorical settings, (iv) reveals a connection between G[p(z|x)] and the singular value of the Q X,Y |Z matrix. Due to the property of SVD, we know that the square of the singular values of Q X,Y |Z equals the non-negative eigenvalue of the matrix Q T X,Y |Z Q X,Y |Z. Then the phase transition condition in Theorem 1 is equivalent to a (nonlinear) eigenvalue problem. This is resonant with previous analogy with CCA in linear settings, and is also reminiscent of the linear eigenvalue problem in Gaussian IB . As a consequence of the theoretical analysis above, we are able to derive an algorithm to efficiently estimate the phase transitions for a given model architecture and dataset. This algorithm also permits us to empirically confirm some of our theoretical in Section 6. Typically, classification involves high-dimensional inputs X. Without sweeping the full range of β where at each β it is a full learning problem, it is in general a difficult task to estimate the phase transitions. In Algorithm 1, we present a two-stage approach. In the first stage, we train a single maximum likelihood neural network f θ with the same encoder architecture as in the (variational) IB to estimate p(y|x), and obtain an N × C matrix p(y|x), where N is the number of examples in the dataset and C is the number of classes. In the second stage, we perform an iterative algorithm w.r.t. G and β, alternatively, to converge to a phase transition point. Specifically, for a given β, we use a Blahut-Arimoto type IB algorithm to efficiently reach IB optimal p * β (z|x) at β, then use SVD (with the formula given in Theorem 2 (iv)) to efficiently estimate G[p * β (z|x)] at β (step 8). We then use the G[p * β (z|x)] value as the new β and do it again (step 7 in the next iteration). At convergence, we will reach the phase transition point given by G[p * β (z|x)] = β (Theorem 1). After convergence as measured by patience parameter K, we slightly increase β by δ (step 13), so that the algorithm can discover the subsequent phase transitions. We quantitatively and qualitatively test the ability of our theory and Algorithm 1 to provide good predictions for IB phase transitions. We first verify them in fully categorical settings, where X, Y, Z are all discrete, and we show that the phase transitions can correspond to learning new classes as we increase β. We then test our algorithm on versions of the MNIST and CIFAR10 datasets with added label noise. 8: 6.1 CATEGORICAL DATASET For categorical datasets, X and Y are discrete, and p(X) and p(Y |X) are given. To test Theorem 1, we use the Blahut-Arimoto IB algorithm to compute the optimal p * β (z|x) for each β. I(Y ; Z *) vs. β is plotted in Fig. 2 (a). There are two phase transitions at β Moreover, starting at β = 1, Alg. 1 converges to each phase transition points within few iterations. Our other experiments with random categorical datasets show similarly tight matches. Furthermore, in Appendix G we show that the phase transitions correspond to the onset of separation of p(z|x) for subsets of X that correspond to different classes. This supports our conjecture from Section 4.1 that there are at most |Y| − 1 phase transitions in classification problems. For continuous X, how does our algorithm perform, and will it reveal aspects of the dataset? We first test our algorithm in a 4-class MNIST with noisy labels 3, whose confusion matrix and experimental settings are given in Appendix H. Fig. 3 (a) shows the path Alg. 1 takes. We see again that in each Figure 3: (a) Path of Alg. 1 starting with β = 1, where the maximum likelihood model f θ is using the same encoder architecture as in the CEB model. This stairstep path shows that Alg. 1 is able to ignore very large regions of β, while quickly and precisely finding the phase transition points. Also plotted is an accumulation of G[p * β (z|x)] vs. β by running Alg. 1 with varying starting β (blue dots). (b) Per-class accuracy vs. β, where the accuracy at each β is from training an independent CEB model on the dataset. The per-class accuracy denotes the fraction of correctly predicted labels by the CEB model for the observed labelỹ. phase Alg. 1 converges to the phase transition points within a few iterations, and it discovers in total 3 phase transition points. Similar to the categorical case, we expect that each phase transition corresponds to the onset of learning a new class, and that the last class is much harder to learn due to a larger separation of β. Therefore, this class should have a much larger label noise so that it is hard to capture this component of maximum correlation between X and Y, as analyzed in representational maximum correlation (Section 4.2). Fig. 3 (b) plots the per-class accuracy with increasing β for running the Conditional Entropy Bottleneck (another variational bound on IB). We see that the first two predicted phase transition points β c 0, β c 1 closely match the observed onset of learning class 3 and class 0. Class 1 is observed to learn earlier than expected, possibly due to the gap between the variational IB objective and the true IB objective in continuous settings. By looking at the confusion matrix for the label noise (Fig. 7), we see that the ordering of onset of learning: class 2, 3, 0, 1, corresponds exactly to the decreasing diagonal element p(ỹ = 1|y = 1) (increasing noise) of the classes, and as predicted, class 1 has a much smaller diagonal element p(ỹ = 1|y = 1) than the other three classes, which makes it much more difficult to learn. This ordering of classes by difficulty is what our representational maximum correlation predicts. The most interesting region is right before β = 2, where accuracy decreases with β. Alg. 1 identifies both sides of that region, as well as points at or near all of the early obvious phase transitions. It also seems to miss later transitions, possibly due to the gap between the variational IB objective and the true IB objective in continuous settings. Finally, we investigate the CIFAR10 experiment from Section 1. The details of the experimental setup are described in Appendix I. This experiment stretches the current limits of our discrete approximation to the underlying continuous representation being learned by the models. Nevertheless, we can see in Fig. 4 that many of the visible empirical phase transitions are tightly identified by Alg. 1. Particularly, the onset of learning is predicted quite accurately; the large interval between the predicted β 3 = 1.21 and β 4 = 1.61 corresponds well to the continuous increase of I(X; Z) and I(Y ; Z) at the same interval. And Alg. 1 is able to identify many dense transitions not obviously seen by just looking at I(Y ; Z) vs. β curve alone. Alg. 1 predicts 9 phase transitions, exactly equal to |Y| − 1 for CIFAR10. In this work, we observe and study the phase transitions in IB as we vary β. We introduce the definition for IB phase transitions, and based on it derive a formula that gives a practical condition for IB phase transitions. We further understand the formula via Jensen's inequality and representational maximum correlation. We reveal the close interplay between the IB objective, the dataset and the learned representation, as each phase transition is learning a nonlinear maximum correlation component in the orthogonal space of the learned representation. We present an algorithm for finding the phase transitions, and show that it gives tight matches with observed phase transitions in categorical datasets, predicts onset of learning new classes and class difficulty in MNIST, and predicts prominent transitions in CIFAR10 experiments. This work is a first theoretical step towards a deeper understanding of the phenomenon of phase transitions in the Information Bottleneck. We believe our approach will be applicable to other "trade-off" objectives, like β-VAE and InfoDropout (a), where the model's ability to predict is balanced against a measure of complexity. Here we prove the Lemma 2.1, which will be crucial in the lemmas and theorems in this paper that follows. Lemma 2.1. For a relative perturbation function r(z|x) ∈ Q Z|X for a p(z|x), where r(z|x) satisfies E z∼p(z|x) [r(z|x)] = 0, we have that the IB objective can be expanded as Proof. Suppose that we perform a relative perturbation r(z|x) on p(z|x) such that the perturbed conditional probability is p (z|x) = p(z|x) (1 + · r(z|x)), then we have Therefore, we can denote the corresponding relative perturbation r(z) on p(z) as Similarly, we have And we can denote the corresponding relative perturbation r(z|y) on p(z|y) as We have The 0 th -order term is simply IB β [p(z|x)]. The first order term is The n th -order term for n ≥ 2 is In the last equality we have used Combining the terms with all orders, we have As a side note, the KL-divergence between p (z|x) = p(z|x)(1 + · r(z|x)) and p(z|x) is Therefore, to the second order, we have Similarly, we have up to the second order. Using similar procedure, we have up to the second-order, B PROOF OF LEMMA 0.1 Proof. From Lemma 2.1, we have The condition of is equivalent to Using Jensen's inequality and the convexity of the square function, we have The equality holds iff r(z|y) = E x∼p(x|y,z) [r(z|x)] is constant w.r.t. y, for any z., where the equality holds iff r(z|x) is constant w.r.t. x for any z. where r(z|y) = E x∼p(x|y,z) [r(z|x)] and r(z) = E x∼p(x|z) [r(z|x)]. which is always true due to that E[r 2 (z|x)] ≥ E[r 2 (z)], and will be a looser condition than Eq. above. Above all, we have Eq.. To empirically estimate G[p(z|x)] from a minibatch of {(x i, y i)}, i = 1, 2,...N and the encoder p(z|x), we can make the following Monte Carlo importance sampling estimation, where we use the samples {x j} ∼ p(x) and also get samples of {z i} ∼ p(z) = p(x)p(z|x), and have: Here Ω x (y i) denotes the set of x examples that has label of y i, and 1[·] is an indicator function that takes value 1 if its argument is true, 0 otherwise. for any x j. Combining all terms, we have that the empiricalĜ[p(z|x)] is given bŷ where {z i} ∼ p(z) and {x i} ∼ p(x). It is also possible to use different distributions for importance sampling, which will in different formulas for empirical estimation of Proof. For the parameterized 4 p θ (z|x) with θ ∈ Θ, after θ ← θ + ∆θ, where 5 ∆θ ∈ Θ is an infinitesimal perturbation on θ, we have that the distribution changes from p θ (z|x) to p θ+∆θ (z|x), 4 In this paper, θ = (θ1, θ2, ...θ k) T and,... is a k × k matrix with (i, j) element of ∂θ i ∂θ j. 5 Note that since Θ is a field, it is closed under subtraction, we have ∆θ ∈ Θ. ∂θ 2 1 = 0, and similarly E y,z∼p θ (y,z) [In other words, the ∆θ 2 terms in the first-order variation δIB β [p θ (z|x)] vanish, and the remaining ∆θ 2 are all Similarly, we have Combining the continuity of T β (β) at β = β, and Eq. and Proof. When we r(z|x) is shifted by a global transformation r (z|x) ← r(z|x) + s(z), we have, and similarly r (z|y) ← r(z|y) + s(z). The numerator of G[r(z|x); p(z|x)] is then Proof. Using the condition of the theorem, we have that ∀r(z|x) ∈ Q 0 Z|X, there exists r 1 (z|x) ∈ Q Z|X and s(z) ∈ {s : Z → R|s bounded} s.t. r(z|x) = r 1 (z|x) + s(z). Note that the only difference between Q Z|X and Q Z|X is that Q Z|X requires E p(z|x) [r 1 (z|x)] = 0. Using Lemma 2.2, we have where r(z|x) doesn't have the constraint of E p(z|x) [·] = 0. After dropping the constraint of E z∼p(z|x) [r(z|x)] = 0, again using Lemma 2.2, we can let r(z) = E x∼p(x|z) [r(z|x)] = 0 (since we can perform the transformation r (z|x) ← r(z|x) − r(z), so that the new r (z) ≡ 0). Now we get a simpler formula for G[p(z|x)], as follows: where Q From Eq., we can further require that We have where F (y, z):= dxp(x|y, z)f (x, z). We have used Cauchy-Schwarz inequality, where the equality holds when g(y, z) = αF (y, z) for some α. Comparing Eq. and the supremum: we see that the only difference is that in the latter Therefore, where in the last equality we have let c(z) have "mass" only on the place where ρ 2 m (X, Y |Z = z) attains supremum w.r.t. z. Z|X, satisfying the requirement for ρ s (X, Y ; Z) (which equals ρ r (X, Y ; Z) by Eq. 28). In this section we study the behavior of p(z|x) on the phase transitions. We use the same categorical dataset (where |X| = |Y | = |Z| = 3 and p(x) is uniform, and p(y|x) is given in Fig. 5). In Fig. 6 we show the p(z|x) on the simplex before and after each phase transition. We see that the first phase transition corresponds to the separation of x = 2 (belonging to y = 2) w.r.t. x ∈ {0, 1} (belonging to classes y ∈ {0, 1}), on the p(z|x) simplex. The second phase transition corresponds to the separation of x = 0 with x = 1. Therefore, each phase transition corresponds to the ability to distinguish subset of examples, and learning of new classes. We use the MNIST training examples with class 0, 1, 2, 3, with a hidden label-noise matrix as given in Fig. 7, based on which at each minibatch we dynamically sample the observed label. We use conditional entropy bottleneck (CEB) as the variational IB objective, and run multiple independent instances with different the target β. We jump start learning by started training at β = 100 for 100 epochs, annealing β from 100 down to the target β over 600 epochs, and continue to train at the target epoch for another 800 epochs. The encoder is a three-layer neural net, where each hidden layer has 512 neurons and leakyReLU activation, and the last layer has linear activation. The classifier p(y|z) is a 2-layer neural net with a 128-neuron ReLU hidden layer. The backward encoder p(z|y) is also a 2-layer neural net with a 128-neuron ReLU hidden layer. We trained with Adam at learning rate of 10 −3, and anneal down with factor 1/(1 + 0.01 · epoch). For Alg. 1, for the f θ we use the same architecture as the encoder of CEB, and use |Z| = 50 in Alg. 1. We use the same CIFAR10 class confusion matrix provided in to generate noisy labels with about 20% label noise on average (reproduced in Table 1). We trained 28 × 1 Wide ResNet models using the open source implementation from as encoders for the Variational Information Bottleneck (VIB) . The 10 dimensional output of the encoder parameterized a mean-field Gaussian with unit covariance. Samples from the encoder were passed to the classifier, a 2 layer MLP. The marginal distributions were mixtures of 500 fully covariate 10-dimensional Gaussians, all parameters of which are trained. With this standard model, we trained 251 different models at β from 1.0 to 6.0 with step size of 0.02. As in , we jump-start learning by annealing β from 100 down to the target β. We do this over the first 4000 steps of training. The models continued to train for another 56,000 gradient steps after that, a total of 600 epochs. We trained with Adam at a base learning rate of 10 −3, and reduced the learning rate by a factor of 0.5 at 300, 400, and 500 epochs. The models converged to essentially their final accuracy within 40,000 gradient steps, and then remained stable. Figure 5: p(y|x) for the categorical dataset in Fig. 2 and Fig. 6. The value in i th row and j th column denotes p(y = j|x = i). p(x) is uniform. The accuracies reported in Figure 4 are averaged across five passes over the training set. We use |Z| = 50 in Alg. 1.

Represent the TLDR for finding the corresponding scientific article

We give a theoretical analysis of the Information Bottleneck objective to understand and predict observed phase transitions in the prediction vs. compression tradeoff.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing scalable hyper-parameters in each layer (layer-wise) and for every neuron separately (neuron-wise), and then optimizing it using the stochastic gradient descent algorithm. Introduction of neuron-wise activation function acts like a vector activation function as opposed to the traditional scalar activation function given by fixed, global and layer-wise activations. In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which further accelerate convergence, thereby reducing the training cost. For numerical experiments, a nonlinear discontinuous function is approximated using a deep neural network with layer-wise and neuron-wise locally adaptive activation functions with and without the slope recovery term and compared with its global counterpart. Moreover, solution of the nonlinear Burgers equation, which exhibits steep gradients, is also obtained using the proposed methods. On the theoretical side, we prove that in the proposed method the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate. Furthermore, the proposed adaptive activation functions with the slope recovery are shown to accelerate the training process in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets with and without data augmentation. In recent years, research on neural networks (NNs) has intensified around the world due to their successful applications in many diverse fields such as speech recognition, computer vision , natural language translation , etc. Training of NN is performed on data sets before using it in the actual applications. Various data sets are available for applications like image classification, which is a subset of computer vision. MNIST and their variants like, Fashion-MNIST , and KMNIST are the data sets for handwritten digits, images of clothing and accessories, and Japanese letters, respectively. Apart from MNIST, Semeion is a handwritten digit data set that contains 1593 digits collected from 80 persons. SVHN is another data set for street view house numbers obtained from house numbers in Google Street View images. CI-FAR is the popular data set containing color images commonly used to train machine learning algorithms. In particular, the CIFAR-10 data set contains 50000 training and 10000 testing images in 10 classes with image resolution of 32x32. CIFAR-100 is similar to the CIFAR-10, except it has 100 classes with 600 images in each class, which is more challenging than the CIFAR-10 data set. problems, where the approximate solutions of governing equations are obtained, as well as inverse problems, where parameters involved in the governing equation are inferred from the training data. Highly efficient and adaptable algorithms are important to design the most effective NN which not only increases the accuracy of the solution but also reduces the training cost. Various architectures of NN like Dropout NN are proposed in the literature, which can improve the efficiency of the algorithm for specific applications. Activation function plays an important role in the training process of NN. In this work, we are particularly focusing on adaptive activation functions, which adapt automatically such that the network can be trained faster. Various methods are proposed in the literature for adaptive activation function, like the adaptive sigmoidal activation function proposed by for multilayer feedforward NNs, while focuses on learning activation functions in convolutional NNs by combining basic activation functions in a data-driven way. Multiple activation functions per neuron are proposed , where individual neurons select between a multitude of activation functions. proposed a tunable activation function, where only a single hidden layer is used and the activation function is tuned. , used a similar idea of tunable activation function but with multiple outputs. Recently, Kunc and Kléma proposed a transformative adaptive activation functions for gene expression inference, see (Kunc & Kléma, 2019). One such adaptive activation function is proposed by introducing scalable hyper-parameter in the activation function, which can be optimized. Mathematically, it changes the slope of activation function thereby increasing the learning process, especially during the initial training period. Due to single scalar hyper-parameter, we call such adaptive activation functions globally adaptive activations, meaning that it gives an optimized slope for the entire network. One can think of doing such optimization at the local level, where the scalable hyper-parameter are introduced hidden layer-wise or even for each neuron in the network. Such local adaptation can further improve the performance of the network. Figure 1 shows a sketch of a neuron-wise locally adaptive activation function based physics-informed neural network (LAAF-PINN), where both the NN part along with the physicsinformed part can be seen. In this architecture, along with the output of NN and the residual term from the governing equation, the activation slopes from every neuron are also contributing to the loss function in the form of slope recovery term. The rest of the paper is organized as follows. Section 2 presents the methodology of the proposed layer-wise and neuron-wise locally adaptive activations in detail. This also includes a discussion on the slope recovery term, expansion of parametric space due to layer-wise and neuron-wise introduction of hyper-parameters, its effect on the overall training cost, and a theoretical for gradient decent algorithms. Section 3 gives numerical experiments, where we approximate a nonlinear discontinuous function using deep NN by the proposed approaches. We also solve the Burgers equation using the proposed algorithm and present various comparisons in appendix B. Section 4 presents numerical with various standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets. Finally, in section 5, we summarize the of our work. We use a NN of depth D corresponding to a network with an input layer, D − 1 hidden-layers and an output layer. In the k th hidden-layer, N k number of neurons are present. Each hidden-layer of the network receives an output z k−1 ∈ R N k−1 from the previous layer where an affine transformation of the form is performed. The network weights w k ∈ R N k ×N k−1 and bias term b k ∈ R N k associated with the k th layer are chosen from independent and identically distributed sampling. The nonlinearactivation function σ(·) is applied to each component of the transformed vector before sending it as an input to the next layer. The activation function is an identity function after an output layer. Thus, the final neural network representation is given by the composition where the operator • is the composition operator, represents the trainable parameters in the network, u is the output and z 0 = z is the input. In supervised learning of solution of PDEs, the training data is important to train the neural network, which can be obtained from the exact solution (if available) or from high-resolution numerical solution given by efficient numerical schemes and it can be even obtained from carefully performed experiments, which may yield both high-and low-fidelity data sets. We aim to find the optimal weights for which the suitably defined loss function is minimized. In PINN the loss function is defined as where the mean squared error (MSE) is given by Here {x represents the residual training points in space-time domain, while {x represents the boundary/initial training data. The neural network solution must satisfy the governing equation at randomly chosen points in the domain, which constitutes the physicsinformed part of neural network given by first term, whereas the second term includes the known boundary/initial conditions, which must be satisfied by the neural network solution. The ing optimization problem leads to finding the minimum of a loss function by optimizing the parameters like, weights and biases, i.e., we seek to find w *, b * = arg min w,b∈Θ (J(w, b)). One can approximate the solutions to this minimization problem iteratively by one of the forms of gradient descent algorithm. The stochastic gradient descent (SGD) algorithm is widely used in machine learning community see, for a complete survey. In SGD the weights are updated as, where η l > 0 is the learning rate. SGD methods can be initialized with some starting value w 0. In this work, the ADAM optimizer , which is a variant of the SGD method is used. A deep network is required to solve complex problems, which on the other hand is difficult to train. In most cases, a suitable architecture is selected based on the researcher's experience. One can also think of tuning the network to get the best performance out of it. In this regard, we propose the following two approaches to optimize the adaptive activation function. Instead of globally defining the hyper-parameter a for the adaptive activation function, let us define this parameter hidden layer-wise as This gives additional D − 1 hyper-parameters to be optimized along with weights and biases. Here, every hidden-layer has its own slope for the activation function. One can also define such activation function at the neuron level as This gives additional k=1 N k hyper-parameters to be optimized. Neuron-wise activation function acts as a vector activation function as opposed to scalar activation function given by L-LAAF and global adaptive activation function (GAAF) approaches, where every neuron has its own slope for the activation function. The ing optimization problem leads to finding the minimum of a loss function by optimizing a k i along with weights and biases, i.e., we seek to find (a The process of training NN can be further accelerated by multiplying a with scaling factor n > 1. The final form of the activation function is given by σ(na It is important to note that the introduction of the scalable hyper-parameter does not change the structure of the loss function defined previously. Then, the final adaptive activation function based neural network representation of the solution is given by In this case, the set of trainable parametersΘ consists of {w and {a In all the proposed methods, the initialization of scalable hyper-parameters is done such that na k i = 1, ∀n. The main motivation of adaptive activation function is to increase the slope of activation function, ing in non-vanishing gradients and fast training of the network. It is clear that one should quickly increase the slope of activation in order to improve the performance of NN. Thus, instead of only depending on the optimization methods, another way to achieve this is to include the slope recovery term based on the activation slope in the loss function as where the slope recovery term S(a) is given by where N is a linear/nonlinear operator. Although, there can be several choices of this operator, including the linear identity operator, in this work we use the exponential operator. The main reason behind this is that such term contributes to the gradient of the loss function without vanishing. The overall effect of inclusion of this term is that it forces the network to increase the value of activation slope quickly thereby increasing the training speed. We now provide a theoretical regarding the proposed methods. The following theorem states that a gradient descent algorithm minimizing our objective functionJ(Θ) in equation 3 does not converge to a sub-optimal critical point or a sub-optimal local minimum, for neither L-LAAF nor N-LAAF, given appropriate initialization and learning rates. In the following theorem, we treatΘ as a real-valued vector. Let Jc = M SE F + M SE u with the constant network u Θ (z) = u Θ (z) = c ∈ R N D for all z, z where c is a constant. Theorem 2.1. Let (Θ m) m∈N be a sequence generated by a gradient descent algorithm asΘ m+1 = Θ m − η m ∇J(Θ). Assume that J(Θ 0) < Jc + S for any c ∈ R N D, J is differentiable, and that for each i ∈ {1, . . ., N f}, there exist differentiable function ϕ i and input. Assume that at least one of the following three conditions holds. (i) (constant learning rate) ∇J is Lipschitz continuous with Lipschitz constant C (i.e., ∇J(Θ) − ∇J(Θ) 2 ≤ C Θ −Θ 2 for allΘ,Θ in its domain), and ≤ η m ≤ (2 −)/C, where is a fixed positive number. (ii) (diminishing learning rate) ∇J is Lipschitz continuous, η m → 0 and ∞ m=0 η m = ∞. (iii) (adaptive learning rate) the learning rate η m is chosen by the minimization rule, the limited minimization rule, the Armjio rule, or the Goldstein rule . Then, for both L-LAAF and N-LAAF, no limit point of (Θ m) m∈N is a sub-optimal critical point or a sub-optimal local minimum. The initial condition J(Θ 0) < Jc + S means that the initial value J(Θ 0) needs to be less than that of a constant network plus the highest value of the slope recovery term. Here, note that S < S. The proof of Theorem 2.1 is included in appendix A. In this section, we shall solve a regression problem of a nonlinear function approximation using deep neural network. The Burgers equation using physics-informed neural network is solved in appendix B. In this test case, a standard neural network (without physics-informed part) is used to approximate a discontinuous function. In this case, the loss function consists of the data mismatch and the slope recovery term as The following discontinuous function with discontinuity at x = 0 location is approximated by a deep neural network. Here, the domain is [−3, 3] and the number of training points used is 300. The activation function is tanh, learning rate is 2.0e-4 and the number of hidden layers are four with 50 neurons in each layer. Figure 2 shows the solution (first column), solution in frequency domain (second column) and pointwise absolute error in log scale (third column). The solution by standard fixed activation function is given in the first row, GAAF solution is given in second row, whereas the third row shows the solution given by L-LAAF without and with (fourth row) slope recovery term. The solution given by N-LAAF without slope recovery term is shown in the fifth row and with slope recovery term in the sixth row. We see that the NN training speed increases for the locally adaptive activation functions compared to fixed and globally adaptive activations. Moreover, both L-LAAF and N-LAAF with slope recovery term accelerate training and yield the least error as compared to other methods. Figure 3 (top) shows the variation of na for GAAF, whereas the second row, left and right shows the layer-wise variation of na k for L-LAAF without and with the slope recovery term respectively. The third row, left and right shows the variation of scaled hyper-parameters for N-LAAF without and with the slope recovery term respectively, where the mean value of na k i along with its standard deviation (Std) are plotted for each hidden-layer. We see that the value of na is quite large with the slope recovery term which shows the rapid increase in the activation slopes. Finally, the comparison of the loss function is shown in figure 4 for standard fixed activation, GAAF, L-LAAF and N-LAAF without the slope recovery (left) and for L-LAAF and N-LAAF with the slope recovery (right) using a scaling factor of 10. The Loss function for both L-LAAF and N-LAAF without the slope recovery term decreases faster, especially during the initial training period compared to the fixed and global activation function based algorithms. The previous sections demonstrated the advantages of adaptive activation functions with PINN for physics related problems. One of the remaining questions is whether or not the advantage of adaptive activations remains with standard deep neural networks for other types of deep learning applications. To explore the question, this section presents numerical with various standard benchmark problems in deep learning. Figures 5 and 6 shows the mean values and the uncertainty intervals Figure 2: Discontinuous function: Neural network solution using standard fixed activation (first row), GAAF (second row), L-LAAF without (third row) and with (fourth row) slope recovery term, and N-LAAF without (fifth row) and with (sixth row) slope recovery term using the tanh activation. First column shows the solution which is also plotted in frequency domain (zoomed-view) as shown by the corresponding second column. Third column gives the point-wise absolute error in the log scale for all the cases. accelerates the minimization process of the training loss values. Here, all of GAAF, L-LAAF and N-LAAF use the slope recovery term, which improved the methods without the recovery term. Accordingly, the of GAAF are also new contributions of this paper. In general, L-LAAF improved against GAAF as expected. The standard cross entropy loss was used for training and plots. We used pre-activation ResNet with 18 layers for CIFAR-10, CIFAR-100, and SVHN data sets, whereas we used a standard variant of LeNet with ReLU for other data sets; i.e., the architecture of the variant of LeNet consists of the following five layers (with the three hidden layers): input layer, convolutional layer with 64 5 × 5 filters, followed by max pooling of size of 2 by 2 and ReLU, convolutional layer with 64 5 × 5 filters, followed by max pooling of size of 2 by 2 and ReLU, fully connected layer with 1014 output units, followed by ReLU, and Fully connected layer with the number of output units being equal to the number of target classes. All hyper-parameters were fixed a priori across all different data sets and models. We fixed the mini-batch size s to be 64, the initial learning rate to be 0.01, the momentum coefficient to be 0.9 and we use scaling factor n = 1 and 2. The learning rate was divided by 10 at the beginning of 10th epoch for all experiments (with and without data augmentation), and of 100th epoch for those with data augmentation. In this paper, we present two versions of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions. Such local activation functions further improve the training speed of the neural network compared to its global predecessor. To further accelerate the training process, an activation slope based slope recovery term is added in the loss function for both layer-wise and neuron-wise activation functions, which is shown to enhance the performance of the neural network. Various NN and PINN test cases like nonlinear discontinuous function approximation and Burgers equation respectively, and benchmark deep learning problems like MNIST, CIFAR, SVHN etc are solved to verify our claim. Moreover, we theoretically prove that no sub-optimal critical point or local minimum attracts gradient descent algorithms in the proposed methods (L-LAAF and N-LAAF) with the slope recovery term under only mild assumptions. k=1 is a limit point of (Θ m) m∈N and a sub-optimal critical point or a sub-optimal local minimum. and h Following the proofs in (, Propositions 1.2.1-1.2.4), we have that ∇J(Θ) = 0 and J(Θ) < Jc + S, for all three cases of the conditions corresponding the different rules of the learning rate. Therefore, we have that for all k ∈ {1, . . ., D − 1}, Furthermore, we have that for all k ∈ {1, . . ., D − 1} and all j ∈ {1, . . ., N k}, By combining equation 5-equation 7, for all k ∈ {1, . . ., D − 1}, which implies that for all a k = 0 since (D − 1) exp(a k) = 0. This implies that J(Θ) = Jc + S, which contradicts with J(Θ) < Jc + S. This proves the desired statement for L-LAAF. For N-LAAF, we prove the statement by contradiction. Suppose that the parameter vectorΘ consisting of {w k=1 ∀j = 1, 2, · · ·, N k is a limit point of (Θ m) m∈N and a suboptimal critical point or a sub-optimal local minimum. Redefine and h for all j ∈ {1, . . ., N k}, where w k,j ∈ R 1×N k−1 and b k,j ∈ R. Then, by the same proof steps, we have that ∇J(Θ) = 0 and J(Θ) < Jc + S, for all three cases of the conditions corresponding the different rules of the learning rate. Therefore, we have that for all k ∈ {1, . . ., D − 1} and all j ∈ {1, . . ., N k}, By combining equation 6-equation 8, for all k ∈ {1, . . ., D − 1} and all j ∈ {1, . . ., N k},, which implies that for all a This implies that J(Θ) = Jc + S, which contradicts with J(Θ) < Jc + S. This proves the desired statement for N-LAAF. The Burgers equation is one of the fundamental partial differential equation arising in various fields such as nonlinear acoustics, gas dynamics, fluid mechanics etc, see for more details. The Burgers equation was first introduced by H. and later studied by J.M. in the context of theory of turbulence. Here, we consider the viscous Burgers equation given by equation equation 9 along with its initial and boundary conditions. The non-linearity in the convection term develops very steep solution due to small˜ value. We consider the Burgers equation given by with initial condition u(x, 0) = − sin(πx), boundary conditions u(−1, t) = u(1, t) = 0 and˜ = 0.01/π. The analytical solution can be obtained using the Hopf-Cole transformation, see for more details. The number of boundary and initial training points is 400, whereas the number of residual training points is 10000. The activation function is tanh, learning rate is 0.0008 and the number of hidden layers are 6 with 20 neurons in each layer. Figure 7 shows the evolution of frequency plots of the solution at three different times using the standard fixed activation function (first row), global adaptive activation function (second row), L-LAAF without (third row) and with (fourth row) slope recovery term, N-LAAF without (fifth row) and with (sixth row) slope recovery term using scaling factor n = 10. Again, for both L-LAAF and N-LAAF the frequencies are converging faster towards the exact solution (shown by black line) with and without slope recovery term as compared to the fixed and global activation function based algorithms. decreases faster for all adaptive activations, in particular GAAF. Even though it is difficult to see from the actual solution plots given by figure 8, one can see from the Figure 10: Burgers equation: comparison of na k for L-LAAF for all six hidden-layers. First three columns represent for L-LAAF without slope recovery term whereas the last three columns are with slope recovery term. In all simulations, the scaling factor n is 10. Figure 10 shows the comparison of layer-wise variation of na k for L-LAAF with and without slope recovery term. It can be seen that, the presence of slope recovery term further increases the slope of activation function thereby increasing the training speed. Similarly, figure 11 shows the mean and standard deviation of na k i for N-LAAF with and without slope recovery term, which again validates that with slope recovery term network training speed increases. for N-LAAF for all six hidden-layers. First three columns represent resuls for N-LAAF without the slope recovery term whereas the last three columns are with slope recovery term. In all simulations, the scaling factor n is 10.

Proposing locally adaptive activation functions in deep and physics-informed neural networks for faster convergence

Represent the TLDR for finding the corresponding scientific article

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the article for finding a single-sentence TLDR

Question: Learning in Gaussian Process models occurs through the adaptation of hyperparameters of the mean and the covariance function. The classical approach entails maximizing the marginal likelihood yielding fixed point estimates (an approach called Type II maximum likelihood or ML-II). An alternative learning procedure is to infer the posterior over hyperparameters in a hierarchical specification of GPs we call Fully Bayesian Gaussian Process Regression (GPR). This work considers two approximations to the intractable hyperparameter posterior, 1) Hamiltonian Monte Carlo (HMC) yielding a sampling based approximation and 2) Variational Inference (VI) where the posterior over hyperparameters is approximated by a factorized Gaussian (mean-field) or a full rank Gaussian accounting for correlations between hyperparameters. We analyse the predictive performance for fully Bayesian GPR on a range of benchmark data sets. The Gaussian process (GP) posterior is heavily influenced by the choice of the covariance function which needs to be set a priori. Specification of a covariance function and setting the hyperparameters of the chosen covariance family are jointly referred to as the model selection problem . A preponderance of literature on GPs address model selection through maximization of the marginal likelihood, ML-II . This is an attractive approach as the marginal likelihood is tractable in the case of a Gaussian noise model. Once the point estimate hyperparameters have been selected typically using conjugate gradient methods the posterior distribution over latent function values and hence predictions can be derived in closed form; a compelling property of GP models. While straightforward to implement the non-convexity of the marginal likelihood surface can pose significant challenges for ML-II. The presence of multiple modes can make the process prone to overfitting especially when there are many hyperparameters. Further, weakly identified hyperparameters can manifest in flat ridges in the marginal likelihood surface (where different combinations of hyperparameters give similar marginal likelihood value) making gradient based optimisation extremely sensitive to starting values. Overall, the ML-II point estimates for the hyperparameters are subject to high variability and underestimate prediction uncertainty. The central challenge in extending the Bayesian treatment to hyperparameters in a hierarchical framework is that their posterior is highly intractable; this also renders the predictive posterior intractable. The latter is typically handled numerically by Monte Carlo integration yielding a non-Gaussian predictive posterior; it yields in fact a mixture of GPs. The key question about quantifying uncertainty around covariance hyperparameters is examining how this effect propagates to the posterior predictive distribution under different approximation schemes. Given observations (X, y) where y i are noisy realizations of some latent function values f corrupted with Gaussian noise, j ) denote a positive definite covariance function parameterized with hyperparameters θ and the corresponding covariance matrix K θ. The hierarchical GP framework is given by, Prior over hyperparameters θ ∼ p(θ) The generative model in implies the joint posterior over unknowns given as, where Z is the unknown normalization constant. The predictive distribution for unknown test inputs X integrates over the joint posterior, (where we have suppressed the conditioning over inputs X, X for brevity). The inner integral p(f |f, y, θ)p(f |θ, y)df reduces to the standard GP predictive posterior with fixed hyperparameters, where, where K θ denotes the covariance matrix evaluated between the test inputs X and K * θ denotes the covariance matrix evaluated between the test inputs X and training inputs X. Under a Gaussian noise setting the hierarchical predictive posterior is reduced to, where f is integrated out analytically and θ j are draws from the hyperparameter posterior. The only intractable integral we need to deal with is p(θ|y) ∝ p(y|θ)p(θ) and predictive posterior follows as per eq.. Hence, the hierarchical predictive posterior is a multivariate mixture of Gaussians (Appendix section 6.2). The distinct advantage of HMC over other MCMC methods is the suppression of the random walk behaviour typical of Metropolis and variants. Refer to for a detailed tutorial. In the experiments we use a self-tuning variant of HMC called the No-U-TurnSampler (NUTS) proposed in in which the path length is deterministically adjusted for every iteration. Empirically, NUTS is shown to work as well as a hand-tuned HMC. By using NUTS we avoid the overhead in determining good values for the step-size and path length (L). We use an identity mass matrix with 500 warm-up iterations and run 4 chains to detect mode switching which can sometimes adversely affect predictions. Further, the primary variables are declared as the log of the hyperparameters log(θ) as this eliminates the positivity constraints that we otherwise we need to account for. The computational cost of the HMC scheme is dominated by the need to invert the covariance matrix K θ which is O(N 3). We largely follow the approach in. We transform the support of hyperparameters θ such that they live in the real space R J where J is the number of hyperparameters. Let η = g(θ) = log(θ) and we proceed by setting the variational family to, in the mean-field approximation where λ mf = (µ 1, . . ., µ J, ν 1, . . ., ν J) is the vector of unconstrained variational parameters (log(σ 2 j) = ν j ) which live in R 2J. In the full rank approximation the variational family takes the form, where we use the Cholesky factorization of the covariance matrix Σ so that the variational parameters λ f r = (µ, L) are unconstrained in R J+J(J+1)/2. The variational objective, ELBO is maximised in the transformed η space using stochastic gradient ascent and any intractable expectations are approximated using monte carlo integration. where the term |J g −1 (η)| denotes the Jacobian of the inverse transformation hinges on automatic differentiation and the re-parametrization trick . The computational cost per iteration is O(N M J) where J is the number of hyperparameters and M is the number of MC samples used in computing stochastic gradients. We evaluate 4 UCI benchmark regression data sets under fully Bayesian GPR (see Table 1). For VI we evaluate the mean-field and full-rank approximations. The top line shows the baseline ML-II method. The two metrics shown are: 1) RMSE -square root mean squared error and 2) NLPD -negative log of the predictive density averaged across test data. Except for'wine' which is a near linear dataset, HMC and full-rank variational schemes exceed the performance of ML-II. By looking at Fig.1 one can notice how the prediction intervals under the full Bayesian schemes capture the true data points. HMC generates a wider span of functions relative to VI (indicated by the uncertainty interval 1). The mean-field (MF) performance although inferior to HMC and full-rank (FR) VI still dominates the ML-II method. Further, while HMC is the gold standard and gives a more exact approximation, the VI schemes provide a remarkably close approximation to HMC in terms of error. The higher RMSE of the MF scheme compared to FR and HMC indicates that taking into account correlations between the hyperparameters improves prediction quality. Data set CO 2 Wine Concrete Airline We demonstrate the feasibility of fully Bayesian GPR in the Gaussian likelihood setting for moderate sized high-dimensional data sets with composite kernels. We present a concise comparative analysis across different approximation schemes and find that VI schemes based on the Gaussian variational family are only marginally inferior in terms of predictive performance to the gold standard HMC. While sampling with HMC can be tuned to generate samples from multi-modal posteriors using tempered transitions , the predictions can remain invariant to samples from different hyperparameter modes. Fully Bayesian bottom: Airline). In the CO 2 data where we undertake long-range extrapolation, the uncertainty intervals under the full Bayesian schemes capture the true observations while ML-II underestimates predictive uncertainty. For the Airline dataset, red in each twoway plot denotes ML-II, the uncertainty intervals under the full Bayesian schemes capture the upward trend better than ML-II. The latter also misses on structure that the other schemes capture. inference in GPs is highly intractable and one has to consider the trade-off between computational cost, accuracy and robustness of uncertainty intervals. Most interesting real-world applications of GPs entail hand-crafted kernels involving many hyperparameters where there risk of overfitting is not only higher but also hard to detect. A more robust solution is to integrate over the hyperparameters and compute predictive intervals that reflect these uncertainties. An interesting question is whether conducting inference over hierarchies in GPs increases expressivity and representational power by accounting for a more diverse range of models consistent with the data. More specifically, how does it compare to the expressivity of deep GPs with point estimate hyperparameters. Further, these general approximation schemes can be considered in conjunction with different incarnations of GP models where transformations are used to warp the observation space yielding warped GPs or warp the input space either using parametric transformations like neural nets yielding deep kernel learning or non-parametric ones yielding deep GPs (6. Appendix In early accounts, , and explore the integration over covariance hyperparameters using HMC in the regression and classification setting. More recently, use a slice sampling scheme for covariance hyperparameters in a general likelihood setting specifically addressing the coupling between latent function values f and hyperparameters θ. conduct a comparative evaluation of MCMC schemes for the full Bayesian treatment of GP models. Other works like explore the MCMC approach to variationally sparse GPs by using a scheme that jointly samples inducing points and hyperparameters. explore a full Bayesian inference framework for regression using HMC but only applies to separable covariance structures together with grid-structured inputs for scalability. On the variational learning side, ; jointly select inducing points and hyperparameters, hence the posterior over hyperparameters is obtained as a side-effect where the inducing points are the main goal. In more recent work, propose a novel variational scheme for sparse GPR which extends the Bayesian treatment to hyperparameters. Extract the 2.5 th percentile ⇒ f i(r l) where r l = 2.5 100 × T Extract the 97.5 th percentile ⇒ f i(ru) where r u = 97.5 All the four data sets use composite kernels constructed from base kernels. Table 2 summarizes the base kernels used and the set of hyperparameters for each kernel. All hyperparameters are given vague N priors in log space. Due to the sparsity of Airline data, several of the hyperparameters were weakly identified and in order to constrain inference to a reasonable range we resorted to a tighter normal prior around the ML-II estimates and Gamma(2, 0.1) priors for the noise hyperparameters. All the experiments were done in python using pymc3 . In the case of HMC, 4 chains were run to convergence and one chain was selected to compute predictions. For mean-field and full rank VI, a convergence threshold of 1e-4 was set for the variational parameters, optimisation terminated when all the variational parameters (means and standard deviations) concurrently changed by less than 1e-4. For'wine' and'concrete' data sets we use a random 50/50 training/test split. For'CO 2' we use the first 545 observations as training and for'Airline' we use the first 100 observations as training. Table 2: Base kernels used in the UCI experiments. k SE denotes the squared exponential kernel, k ARD denotes the automatic relevance determination kernel (squared exponential over dimensions), k P er denotes the periodic kernel, k RQ denotes the rational quadratic kernel and k N oise denotes the white kernel for stationary noise. Data set Composite Kernel In the figures and tables below, a prefix's' denotes signal std. deviation, a prefix'ls' denotes lengthscale and a prefix'n' denotes noise std. deviation. The figure below shows marginal posteriors of the hyperparamters used in the Airline kernel. We can make the following remarks: 1. It is evident that sampling and variational optimisation do not converge to the same region of the hyperparameter space as ML-II. 2. Given that the predictions are better under the full Bayesian schemes, this indicates that ML-II is in an inferior local optimum. 3. The mean-field marginal posteriors are narrower than the full rank and HMC posteriors as is expected. Full rank marginal posteriors closely approximate the HMC marginals. 4. The noise std. deviation distribution learnt under the full Bayesian schemes is higher than ML-II point estimate indicating overfitting in this particular example.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Analysis of Bayesian Hyperparameter Inference in Gaussian Process Regression

Represent the short summary for finding the corresponding scientific article (long)


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We consider the problem of using variational latent-variable models for data compression. For such models to produce a compressed binary sequence, which is the universal data representation in a digital world, the latent representation needs to be subjected to entropy coding. Range coding as an entropy coding technique is optimal, but it can fail catastrophically if the computation of the prior differs even slightly between the sending and the receiving side. Unfortunately, this is a common scenario when floating point math is used and the sender and receiver operate on different hardware or software platforms, as numerical round-off is often platform dependent. We propose using integer networks as a universal solution to this problem, and demonstrate that they enable reliable cross-platform encoding and decoding of images using variational models. The task of information transmission in today's world is largely divided into two separate endeavors: source coding, or the representation of data (such as audio or images) as sequences of bits, and channel coding, representing sequences of bits as analog signals on imperfect, physical channels such as radio waves BID7. This decoupling has substantial benefits, as the binary representations of arbitrary data can be seamlessly transmitted over arbitrary physical channels by only changing the underlying channel code, rather than having to design a new code for every possible combination of data source and physical channel. Hence, the universal representation of any compressed data today is the binary channel, a representation which consists of a variable number of binary symbols, each with probability 1 2, and no noise (i.e. uncertainty). As a latent representation, the binary channel unfortunately is a severe restriction compared to the richness of latent representations defined by many variational latent-variable models in the literature (e.g., BID13 BID22 BID18, and in particular models targeted at data compression BID23 BID0 . Variational latent-variable models such as VAEs BID13 consist of an encoder model distribution e(y | x) bringing the data x into a latent representation y, and a decoder model distribution d(x | y), which represents the data likelihood conditioned on the latents. Given an encoder e, we observe the marginal distribution of latents m(y) = E x [e(y | x)], where the expectation runs over the (unknown) data distribution. The prior p(y) is a variational estimate of the marginal BID1.By choosing the parametric forms of these distributions and the training objective appropriately, many such models succeed in representing relevant information in the data they are trained for quite compactly (i.e., with a small expected Kullback-Leibler (KL) divergence between the encoder and the prior, E x D KL [e p]), and so may be called compressive in a sense. However, not all of them can be directly used for practical data compression, as the representation needs to be further converted into binary (entropy encoded). This conversion is typically performed by range coding, or arithmetic coding BID20. Range coding is asymptotically optimal: the length of the binary sequence quickly converges to the expected KL divergence in bits, for reasonably large sequences (such as, for one image). For this to hold, the following requirements must be satisfied: Figure 1: The same image, decoded with a model computing the prior using integer arithmetic (left), and the same model using floating point arithmetic (right). The image was decoded correctly, beginning in the top-left corner, until floating point round-off error caused a small discrepancy between the sender's and the receiver's copy of the prior, at which point the error propagated catastrophically.• The representation must be discrete-valued, i.e. have a finite number of states, and be noiseless -i.e. the conditional entropy of the encoder must be zero: DISPLAYFORM0 • All scalar elements of the representation y must be brought into a total ordering, and the prior needs to be written using the chain rule of calculus (as a product of conditionals), as the algorithm can only encode or decode one scalar random variable at a time.• Both sides of the binary channel (i.e. sender and receiver) must be able to evaluate the prior, and they must have identical instances of it. The latter point is crucial, as range coding is extremely sensitive to differences in p between sender and receiver -so sensitive, in fact, that even small perturbations due to floating point round-off error can lead to catastrophic error propagation. Unfortunately, numerical round-off is highly platform dependent, and in typical data compression applications, sender and receiver may well employ different hardware or software platforms. Round-off error may even be non-deterministic on one and the same computer. Figure 1 illustrates a decoding failure in a model which computes p using floating point math, caused by such computational non-determinism in sender vs. receiver. Recently, latent-variable models have been explored that employ artificial neural networks (ANNs) to compute hierarchical or autoregressive priors BID22 BID18, including some of the best-performing learned image compression models BID17 BID14. Because ANNs are typically based on floating point math, these methods are vulnerable to catastrophic failures when deployed on heterogeneous platforms. To address this problem, and enable use of powerful learned variational models for real-world data compression, we propose to use integer arithmetic in these ANNs, as floating-point arithmetic cannot presently be made deterministic across arbitrary platforms. We formulate a type of quantized neural network we call integer networks, which are specifically targeted at generative and compression models, and at preventing computational non-determinism in computation of the prior. Because full determinism is a feature of many existing, widely used image and video compression methods, we also consider using integer networks end to end for computing the representation itself. ANNs are typically composite functions that alternate between linear and elementwise nonlinear operations. One linear operation followed by a nonlinearity is considered one layer of the network. To ensure that such a network can be implemented deterministically on a wide variety of hardware platforms, we restrict all the data types to be integral, and all operations to be implemented either with basic arithmetic or lookup tables. Because integer multiplications (including matrix multiplications or convolutions) increase the dynamic range of the output compared to their inputs, we introduce an additional step after each linear operator, where we divide each of its output by a learned parameter.. This nonlinearity can be implemented deterministically either using a lookup table or simply using a clipping operation. The corresponding scaled cumulative of a generalized Gaussian with β = 4 used for computing gradients is plotted in cyan, and other choices of β in gray. Right: Example nonlinearity approximating hyperbolic tangent for 4-bit signed integer outputs, given by g Qtanh (v) = Q(7 tanh( v 15)). This nonlinearity can be implemented deterministically using a lookup table. The corresponding scaled hyperbolic tangent used for computing gradients is plotted in cyan. Concretely, we define the relationship between inputs u and outputs w of one layer as: DISPLAYFORM0 In order, the inputs u are subjected to a linear transform H (a matrix multiplication, or a convolution); a bias vector b is added; the is divided elementwise by a vector c, yielding an intermediate vector v; and finally, an elementwise nonlinearity g is applied to v. The activations w and all intermediate , as well as the parameters H, b, and c are all defined as integers. However, they may use differing number formats. For v to be integral, we define here to perform rounding division (equivalent to division followed by rounding to the nearest integer). In programming languages such as C, this can be implemented with integer operands m, n as DISPLAYFORM1 where Q rounds to the nearest integer and / / is floor division; here, the addition can be folded into the bias b as an optimization. We constrain the linear filter coefficients H and the bias vector b to generally use signed integers, and the scaling vector c to use unsigned integers. We implement the accumulators of the linear transform with larger bit width than the activations and filter coefficients, in order to reflect the potentially increased dynamic range of multiplicative operations. We assume here that the bias and scaling vectors, as well as the intermediate vector v, have the same bit width as the accumulators. The elementwise nonlinearity g must be saturating on both ends of its domain, because integers can only represent finite number ranges. In order to maximize utility of the dynamic range, we scale nonlinearities such that their range matches the bit width of w, while their domain can be scaled somewhat arbitrarily. Depending on the range of the nonlinearity, the activations w may use a signed or unsigned number format. For instance, a reasonable choice of number formats and nonlinearity would be:H: 8-bit signed b, v: 32-bit signed (same as accumulator)c: 32-bit unsigned w: 8-bit unsigned g QReLU (v) = max (min(v, 255), 0) In this example, the nonlinearity can be implemented with a simple clipping operation. Refer to figure 2, left, for a visualization (for visualization purposes, the figure shows a smaller bit width). H: 4-bit signed b, v: 16-bit signed (same as accumulator) c: 16-bit unsigned w: 4-bit signed DISPLAYFORM0 Here, the nonlinearity approximates the hyperbolic tangent, a widely used nonlinearity. It may be best implemented using a lookup table (see figure 2, right, for a visualization). We scale its range to fill the 4-bit signed integer number format of w by multiplying its output with 7. The domain can be scaled somewhat arbitrarily, since v has a larger bit width than w. When it is chosen too small, w may not utilize all integer values, leading to a large quantization error. When it is chosen too large, overflow may occur in v, or the size of the lookup table may grow too large for practical purposes. Therefore, it is best to determine the input scaling based on the shape of the nonlinearity and the available dynamic range. Here, we simply chose the value of 15 "by eye", so that the nonlinearity is reasonably well represented with the lookup table (i.e., we made sure that at least two or three input values are mapped to each output value, in order to preserve the approximate shape of the nonlinearity). To effectively accumulate small gradient signals, we train the networks entirely using floating point computations, rounded to integers after every computational operation, while the backpropagation is done with full floating point precision. More concretely, we define the integer parameters H, b, and c as functions of their floating point equivalents H, b, and c, respectively: DISPLAYFORM0... DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 Here, we simply rescale each element of b using a constant K, which is the bit-width of the kernel H (e.g. 8-bits in the QReLu networks), and round it to the nearest integer. The reparameterization mapping r is borrowed from: DISPLAYFORM4 When c is small, perturbations in c can lead to excessively large fluctuations of the quotient (i.e., the input to the nonlinearity). This leads to instabilities in training. r ensures that values of c are always positive, while gracefully scaling down gradient magnitudes on c near zero. Effectively, the step size on c is multiplied with a factor that is approximately linear in c.Before rounding the linear filter coefficients in H = [h 1, . . ., h N], we apply a special rescaling function s to each of its filters h: DISPLAYFORM5 s rescales each filter such that at least one of its minimum and maximum coefficients hits one of the dynamic range bounds (−2 K−1 and 2 K−1 −1), while keeping zero at zero. This represents the finest possible quantization of the filter given its integer representation, and thus maximizes accuracy. To prevent division by zero, we ensure the divisor is larger than or equal to a small constant (for example, = 10 −20).In order to backpropagate gradient signals into the parameters, one cannot simply take gradients of the loss function with respect to H, b, or c, since the rounding function Q has zero gradients almost everywhere, except for the half-integer positions where the gradient is positive infinity. A simple remedy is to replace the derivative of Q with the identity function, since this is the smoothed gradient across all rounded values. Further, we treat the rescaling divisor s as if it were a constant. That is, we compute the derivatives of the loss function with respect to H, b, and c as with the chain rule of calculus, but overriding: DISPLAYFORM6 where r is the replacement gradient function for r as proposed by. After training is completed, we compute the integer parameters H, b and c one more time, and from then on use them for evaluation. Note that further reparameterization of the kernels H, such as Sadam, or of the biases b or scaling parameters c, is possible by simply chaining reparameterizations. In addition to rounding the parameters, it is necessary to round the activations. To obtain gradients for the rounding division, we simply substitute the gradient of floating point division. To estimate gradients for the rounded activation functions, we replace their gradient with the corresponding nonrounded activation function, plotted in cyan in figure 2. In the case of QReLU, the gradient of the clipping operation is a box function, which can lead to training getting stuck, since if activations consistently hit one of the bounds, no gradients are propagated back (this is sometimes called the "dead unit" problem). As a remedy, we replace the gradient instead with DISPLAYFORM7 where DISPLAYFORM8 β, and L is the bit width of w. This function corresponds to a scaled generalized Gaussian probability density with shape parameter β. In this context, we can think of β as a temperature parameter that makes the function converge to the gradient of the clipping operation as β goes to infinity. Although this setting permits an annealing schedule, we simply chose β = 4 and obtained good . The integral of this function is plotted in figure 2 (left) in cyan, along with other choices of β in gray. Suppose our prior on the latent representation is p(y | z), where z summarizes other latent variables of the representation (it may be empty). To apply range coding, we need to impose a total ordering on the elements of y and write it as a chain of conditionals: DISPLAYFORM0 where y:i denotes the vector of all elements of y preceding the ith. A common assumption is that p is a known distribution, with parameters θ i computed by an ANN g: DISPLAYFORM1 We simply propose here to compute g deterministically using an integer network, discretizing the parameters θ to a reasonable accuracy. If p(y i | θ i) itself cannot be computed deterministically, we can precompute all possible values and express it as a lookup table over y i and θ i.As an example, consider the prior used in the image compression model proposed by, which is a modified Gaussian with scale parameters conditioned on another latent variable: DISPLAYFORM2 We reformulate the scale parameters σ as: DISPLAYFORM3 where θ = g(z) is computed using an integer network. The last activation function in g is chosen to have integer outputs of L levels in the range [0, L − 1]. Constants σ min, σ max, and L determine the discretized selection of scale parameters used in the model. The discretization is chosen to be logarithmic, as this choice minimizes E x D KL [e p] for a given number of levels. During training, we can simply backpropagate through this reformulation, and through g as described in the previous section. After training, we precompute all possible values of p as a function of y i and θ i and form a lookup table, while g is implemented with integer arithmetic. For certain applications, it can be useful not only to be able to deploy a compression model across heterogenous platforms, but to go even further in also ensuring identical reconstructions of the data across platforms. To this end, it can be attractive to make the entire model robust to non-determinism. To use integer networks in the encoder or decoder, one can use the equivalent construction as in FORMULA0 Jang et al. FORMULA0 and Ágústsson et al. are concerned with producing gradients for categorical distributions and vector quantization (VQ), respectively. In both methods, the representation is found by evaluating an ANN followed by an arg max function, while useful gradients are obtained by substituting the arg max with a softmax function. Since arg max can be evaluated deterministically in a platform-independent way, and evaluating a softmax function with rounded inputs is feasible, integer networks can be combined with these models without additional modifications. Theis et al. FORMULA0 and differ mostly in the details of interaction between the encoder and the prior. These two approaches are particularly interesting for image compression, as they scale well: Image compression models are often trained with a rate-distortion objective with a Lagrange parameter λ, equivalent to β in the β-VAE objective BID10 BID1. Depending on the parameter, the latent representation carries vastly different amounts of information, and the optimal number of latent states in turn varies with that. While the number of latent states is a hyperparameter that needs to be chosen ahead of time in the categorical/VQ case, the latter two approaches can extend it as needed during training, because the latent states are organized along the real line. Further, for categorical distributions as well as VQ, the required dimensionality of the function computing the parameters grows linearly with the number of latent states due to their use of the arg max function. In the latter two models, the number of states can grow arbitrarily without increasing the dimensionality of g. Both BID23 and use deterministic encoder distributions (i.e. degenerating to delta distributions) during evaluation, but replace them with probabilistic versions for purposes of estimating E x D KL [e p] during training. BID23 propose to use the following encoder distribution: DISPLAYFORM0 where U is the uniform distribution and g is an ANN. They replace the gradient of the quantizer with the identity. During evaluation, y = Q(g(x)) is used as the representation. use the following distribution during training: DISPLAYFORM1 which makes y shift-invariant. During evaluation, they determine the representation as y = Q(g(x) − o), where o is a sub-integer offset chosen such that the mode (or, if it cannot be estimated easily, the median) of the distribution is centered on one of the quantization bins. If g is implemented with integer networks, the latter approach becomes equivalent to the former, because g then inherently computes integer outputs, and this is effectively equivalent to the quantization in. However, we've found that training with this construction leads to instabilities, such that the prior distribution never converges to a stable set of parameters. The reason may be that with quantization in e, the marginal m(y) = E x e(y | x) resembles a piecewise constant function, while the prior p must be forced to be smooth, or E x D KL [e p] would not yield any useful gradients. Because the prior is a variational approximation of the marginal, this means that the prior must be regularized (which we did not attempt here -we used the nonparametric density model described in). On the other hand, when using without quantization, the marginal is typically a smooth density, and the prior can approximate it closely without the need for regularization. As a remedy for the instabilities, we propose the following trick: We simply use during training, but define the last layer of g without a nonlinearity and with floating point division, such that the representation is compressed on CPU 1 CPU 1 CPU 1 CPU 1 GPU 1 GPU 1 GPU 1 GPU 1 decompressed on CPU 1 GPU 1 CPU 2 GPU 2 CPU 1 GPU 1 CPU 2 GPU 2 Tecnick dataset: 100 RGB images of 1200 × 1200 pixels 0% 71% 54% 66% 63% 41% 59% 34% ditto, integer prior 0% 0% 0% 0% 0% 0% 0% 0% CLIC dataset: 2021 RGB images of various pixel sizes 0% 78% 68% 78% 77% 52% 78% 54% ditto, integer prior 0% 0% 0% 0% 0% 0% 0% 0% DISPLAYFORM2 CPU 1: Intel Xeon E5-1650 GPU 1: NVIDIA Titan X (Pascal) CPU 2: Intel Xeon E5-2690 GPU 2: NVIDIA Titan X (Maxwell) Table 1: Decompression failure rates due to floating point round-off error on Tecnick and CLIC image datasets. When compressing and decompressing on the same CPU platform (first column), the model decompresses all images correctly. However, when compressing on a GPU or decompressing on a different platform, a large percentage of the images fail to be decoded correctly. Implementing the prior of the same model using integer networks ensures correct decompression across all tested platforms.during training, where u is the input to the last layer and / represents elementwise floating point division, and DISPLAYFORM3 during evaluation. This can be rewritten strictly using integer arithmetic as: DISPLAYFORM4 where represents elementwise multiplication, and the rounded product can be folded into the bias b as an optimization. This way, the representation is computed deterministically during evaluation, while during training, the marginal still resembles a smooth function, such that no regularization of the prior is necessary. In order to assess the efficacy of integer networks to enable platform-independent compression and decompression, we re-implemented the image compression model described in, FORMULA0 model, evaluated on BID15, corresponding to the rate point at approximately 0.7 bits per pixel in FIG2, right panel. Generally, training of integer models takes somewhat longer and is somewhat noisier than training of floating point models. When matching floating point and integer networks for asymptotic performance (128 vs. 256 filters, respectively), integer networks take longer to converge (likely due to their larger number of filters). When matching by number of filters FORMULA0, it appears that the training time to convergence is about the same, but the performance ends up worse.which is defined with a hyperprior. We compare the original model with a version in which the network h s computing the prior is replaced with an integer network. We used the same network architectures in terms of number of layers, filters, etc., and the same training parameters as in the original paper. The rate-distortion performance of the model was assessed on BID15 and is shown in FIG2 (left). The modified model performs identically to the original model, as it maps out the same rate-distortion frontier. However, it is much more robust to cross-platform compression and decompression (table 1). We tested compression and decompression on four different platforms (two CPU platforms and two GPU platforms) and two different datasets, Tecnick BID2 BID2. The original model fails to correctly decompress more than half of the images on average when compression and decompression occurs on different platforms. The modified model brings the failure rate down to 0% in all cases. It should be noted that the decreased accuracy of integer arithmetic generally leads to a lower approximation capacity than with floating point networks. We found that when implementing the models described in Ballé using integer networks throughout, the rate-distortion performance decreased (figure 3, right). The loss in approximation capacity can be compensated for by increasing the number of filters per layer. Note that this appears to increase the training time necessary for convergence (figure 4). However, note that increasing the number of parameters may not necessarily increase the size of the model parameters or the runtime, as the storage requirements for integer parameters (kernels, biases, etc.) are lower than for floating point parameters, and integer arithmetic is computationally less complex than floating point arithmetic in general. There is a large body of recent research considering quantization of ANNs mostly targeted at image recognition applications. BID6 train classification networks on lower precision multiplication. BID11 and BID19 perform quantization down to bilevel (i.e., 1-bit integers) at inference time to reduce computation in classification networks. More recently, BID24 and others have used quantization during training as well as inference, to reduce computation on gradients as well as activations, and BID5 use non-uniform quantization to remove floating point computation, replacing it completely with integer offsets into an integer lookup table. While the quantization of neural networks is not a new topic, the from the above techniques focus almost exclusively on classification networks. BID8, BID9, and others have demonstrated that these types of networks are particularly robust to capacity reduction. Models used for image compression, like many generative models, are much more sensitive to capacity constraints since they tend to underfit. As illustrated in and in figure 3 (right), this class of models is much more sensitive to reductions of capacity, both in terms of network size and the expressive power of the activation function. This may explain why our experiments with post-hoc quantization of network activations have never yielded competitive for this class of model (not shown).As illustrated in figure 1 and table 1, small floating point inconsistencies in variational latent-variable models can have disastrous effects when we use range coding to employ the models for data compression across different hardware or software platforms. The reader may wonder whether there exists other entropy coding algorithms that can convert discrete latent-variable representations into a binary representation, and which do not suffer from a sensitivity to perturbations in the probability model. Unfortunately, such an algorithm would always produce suboptimal for the following reason. The source coding theorem BID21 ) establishes a lower bound on the average length of the ing bit sequences, which range coding achieves asymptotically (i.e. for long bit sequences). The lower bound is given by the cross entropy between the marginal and the prior: DISPLAYFORM0 where |b(y)| is the length of the binary representation of y. If an entropy coding algorithm tolerates error in the values of p(y | θ), this means it must operate under the assumption of identical probability values for a range of values of θ -in other words, discretize the probability values. Since the cross entropy is minimal only for p(y | θ) = m(y) (for all y), this would impose a new lower bound on |b(y)| given by the cross entropy with the discretized probabilities, which is greater or equal to the cross entropy given above. Thus, the more tolerant the entropy coding method is to errors in p, the further it deviates from optimal performance. Moreover, it is hard to establish tolerance intervals for probability values computed with floating point arithmetic, in particular when ANNs are used, due to error propagation. Hence, it is generally difficult to provide guarantees that a given tolerance will not be exceeded. For similar reasons, current commercial compression methods model probabilities exclusively in the discrete domain (e.g., using lookup tables; BID16.Our approach to neural network quantization is the first we are aware of which specifically addresses non-deterministic computation, as opposed to computational complexity. It enables a variety of possible variational model architectures and distributions to be effectively used for platformindependent data compression. While we aren't assessing its effects on computational complexity here, it is conceivable that complexity reductions can also be achieved with the same approach; this is a topic for future work.

Represent the short summary for finding the corresponding scientific article (long):

We train variational models with quantized networks for computational determinism. This enables using them for cross-platform data compression.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.