Question: Represent the text to find a very short summary (1-2 sentences max).

Question: The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima in the training loss -- minima where the output of the network is resilient to small random noise added to its parameters. So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either \textit{stochastic} or \textit{compressed}. In this work, we present a general PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned -- a network that is deterministic and uncompressed. What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves {\em generalize} to the interactions between the matrices on test data, thereby implying a wide test loss minimum. We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data). In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices -- a guarantee that would not have been possible with prior approaches. Modern deep neural networks contain millions of parameters and are trained on relatively few samples. Conventional wisdom in machine learning suggests that such models should massively overfit on the training data, as these models have the capacity to memorize even a randomly labeled dataset of similar size . Yet these models have achieved state-ofthe-art generalization error on many real-world tasks. This observation has spurred an active line of research (; BID2 BID11) that has tried to understand what properties are possessed by stochastic gradient descent (SGD) training of deep networks that allows these networks to generalize well. One particularly promising line of work in this area (; BID0 has been bounds that utilize the noise-resilience of deep networks on training data i.e., how much the training loss of the network changes with noise injected into the parameters, or roughly, how wide is the training loss minimum. While these have yielded generalization bounds that do not have a severe exponential dependence on depth (unlike other bounds that grow with the product of spectral norms of the weight matrices), these bounds are quite limited: they either apply to a stochastic version of the classifier (where the parameters are drawn from a distribution) or a compressed version of the classifier (where the parameters are modified and represented using fewer bits).In this paper, we revisit the PAC-Bayesian analysis of deep networks in Neyshabur et al. (2017; and provide a general framework that allows one to use noise-resilience of the deep network on training data to provide a bound on the original deterministic and uncompressed network. We achieve this by arguing that if on the training data, the interaction between the 'activated weight matrices' (weight matrices where the weights incoming from/outgoing to inactive units are zeroed out) satisfy certain conditions which in a wide training loss minimum, these conditions themselves generalize to the weight matrix interactions on the test data. After presenting this general PAC-Bayesian framework, we specialize it to the case of deep ReLU networks, showing that we can provide a generalization bound that accomplishes two goals simultaneously: i) it applies to the original network and ii) it does not scale exponentially with depth in terms of the products of the spectral norms of the weight matrices; instead our bound scales with more meaningful terms that capture the interactions between the weight matrices and do not have such a severe dependence on depth in practice. We note that all but one of these terms are indeed quite small on networks in practice. However, one particularly (empirically) large term that we use is the reciprocal of the magnitude of the network pre-activations on the training data (and so our bound would be small only in the scenario where the pre-activations are not too small). We emphasize that this drawback is more of a limitation in how we characterize noise-resilience through the specific conditions we chose for the ReLU network, rather than a drawback in our PAC-Bayesian framework itself. Our hope is that, since our technique is quite general and flexible, by carefully identifying the right set of conditions, in the future, one might be able to derive a similar generalization guarantee that is smaller in practice. To the best of our knowledge, our approach of generalizing noise-resilience of deep networks from training data to test data in order to derive a bound on the original network that does not scale with products of spectral norms, has neither been considered nor accomplished so far, even in limited situations. One of the most important aspects of the generalization puzzle that has been studied is that of the flatness/width of the training loss at the minimum found by SGD. The general understanding is that flatter minima are correlated with better generalization behavior, and this should somehow help explain the generalization behavior BID7 BID6 BID8. Flatness of the training loss minimum is also correlated with the observation that on training data, adding noise to the parameters of the network only in little change in the output of the network -or in other words, the network is noise-resilient. Deep networks are known to be similarly resilient to noise injected into the inputs ; but note that our theoretical analysis relies on resilience to parameter perturbations. While some progress has been made in understanding the convergence and generalization behavior of SGD training of simple models like two-layered hidden neural networks under simple data distributions (; ; BID2 BID11, all known generalization guarantees for SGD on deeper networks -through analyses that do not use noise-resilience properties of the networks -have strong exponential dependence on depth. In particular, these bounds scale either with the product of the spectral norms of the weight matrices BID0 BID1 or their Frobenius norms BID4 . In practice, the weight matrices have a spectral norm that is as large as 2 or 3, and an even larger Frobenius norm that scales with √ H where H is the width of the network i.e., maximum number of hidden units per layer.1 Thus, the generalization bound scales as say, 2 D or H D 2, where D is the depth of the network. At a high level, the reason these bounds suffer from such an exponential dependence on depth is that they effectively perform a worst case approximation of how the weight matrices interact with each other. For example, the product of the spectral norms arises from a naive approximation of the Lipschitz constant of the neural network, which would hold only when the singular values of the 1 To understand why these values are of this order in magnitude, consider the initial matrix that is randomly initialized with independent entries with variance 1 √ H. It can be shown that the spectral norm of this matrix, with high probability, lies near its expected value, near 2 and the Frobenius norm near its expected value which is √ H. Since SGD is observed not to move too far away from the initialization regardless of H , these values are more or less preserved for the final weight matrices.weight matrices all align with each other. However, in practice, for most inputs to the network, the interactions between the activated weight matrices are not as adverse. By using noise-resilience of the networks, prior approaches BID0 ) have been able to derive bounds that replace the above worst-case approximation with smaller terms that realistically capture these interactions. However, these works are limited in critical ways. BID0 use noise-resilience of the network to modify and "compress" the parameter representation of the network, and derive a generalization bound on the compressed network. While this bound enjoys a better dependence on depth because its applies to a compressed network, the main drawback of this bound is that it does not apply on the original network. On the other hand, take advantage of noise-resilience on training data by incorporating it within a PAC-Bayesian generalization bound BID14. However, their final guarantee is only a bound on the expected test loss of a stochastic network. In this work, we revisit the idea in , by pursuing the PAC-Bayesian framework BID14 to answer this question. The standard PAC-Bayesian framework provides generalization bounds for the expected loss of a stochastic classifier, where the stochasticity typically corresponds to Gaussian noise injected into the parameters output by the learning algorithm. However, if the classifier is noise-resilient on both training and test data, one could extend the PAC-Bayesian bound to a standard generalization guarantee on the deterministic classifier. Other works have used PAC-Bayesian bounds in different ways in the context of neural networks. BID9; BID3 optimize the stochasticity and/or the weights of the network in order to numerically compute good (i.e., non-vacuous) generalization bounds on the stochastic network. BID0 derive generalization bounds on the original, deterministic network by working from the PAC-Bayesian bound on the stochastic network. However, as stated earlier, their work does not make use of noise resilience in the networks learned by SGD.OUR CONTRIBUTIONS The key contribution in our work is a general PAC-Bayesian framework for deriving generalization bounds while leveraging the noise resilience of a deep network. While our approach is applied to deep networks, we note that it is general enough to be applied to other classifiers. In our framework, we consider a set of conditions that when satisfied by the network, makes the output of the network noise-resilient at a particular input datapoint. For example, these conditions could characterize the interactions between the activated weight matrices at a particular input. To provide a generalization guarantee, we assume that the learning algorithm has found weights such that these conditions hold for the weight interactions in the network on training data (which effectively implies a wide training loss minimum). Then, as a key step, we generalize these conditions over to the weight interactions on test data (which effectively implies a wide test loss minimum) 2. Thus, with the guarantee that the classifier is noise-resilient both on training and test data, we derive a generalization bound on the test loss of the original network. Finally, we apply our framework to a specific set up of ReLU based feedforward networks. In particular, we first instantiate the above abstract framework with a set of specific conditions, and then use the above framework to derive a bound on the original network. While very similar conditions have already been identified in prior work BID0 ) (see Appendix G for an extensive discussion of this), our contribution here is in showing how these conditions generalize from training to test data. Crucially, like these works, our bound does not have severe exponential dependence on depth in terms of products of spectral norms. We note that in reality, all but one of our conditions on the network do hold on training data as necessitated by the framework. The strong, non-realistic condition we make is that the pre-activation values of the network are sufficiently large, although only on training data; however, in practice a small proportion of the pre-activation values can be arbitrarily small. Our generalization bound scales inversely with the smallest absolute value of the pre-activations on the training data, and hence in practice, our bound would be large. Intuitively, we make this assumption to ensure that under sufficiently small parameter perturbations, the activation states of the units are guaranteed not to flip. It is worth noting that BID0 too require similar, but more realistic assumptions about pre-activation values that effectively assume only a small proportion of units flip under noise. However, even under our stronger condition that no such units exist, it is not apparent how these approaches would yield a similar bound on the deterministic, uncompressed network without generalizing their conditions to test data. We hope that in the future our work could be developed further to accommodate the more realistic conditions from BID0. In this section, we present our general PAC-Bayesian framework that uses noise-resilience of the network to convert a PAC-Bayesian generalization bound on the stochastic classifier to a generalization bound on the deterministic classifier. NOTATION. Let KL(⋅ ⋅) denote the KL-divergence. Let ⋅, ⋅ ∞ denote the 2 norm and the ∞ norms of a vector, respectively. Let ⋅ 2, ⋅ F, ⋅ 2,∞ denote the spectral norm, Frobenius norm and maximum row 2 norm of a matrix, respectively. Consider a K-class learning task where the labeled datapoints (x, y) are drawn from an underlying distribution D over X × {1, 2, ⋯, K} where X ∈ R N. We consider a classifier parametrized by weights W. For a given input x and class k, we denote the output of the classifier by f (x; W) [k]. In our PAC-Bayesian analysis, we will use U ∼ N (0, σ 2) to denote parameters whose entries are sampled independently from a Gaussian, and W + U to denote the entrywise addition of the two sets of parameters. We use DISPLAYFORM0 Given a training set S of m samples, we let (x, y) ∼ S to denote uniform sampling from the set. Finally, for any γ > 0, let L γ (f (x; W), y) denote a margin-based loss such that the loss is 0 only when f (x; W) [y] ≥ max j≠y f (x; W) [j] + γ, and 1 otherwise. Note that L 0 corresponds to 0-1 error. See Appendix A for more notations. TRADITIONAL PAC-BAYESIAN BOUNDS. The PAC-Bayesian framework BID14 b) allows us to derive generalization bounds for a stochastic classifier. Specifically, letW be a random variable in the parameter space whose distribution is learned based on training data S. Let P be a prior distribution in the parameter space chosen independent of the training data. The PAC-Bayesian framework yields the following generalization bound on the 0-1 error of the stochastic classifier that holds with probability 1 − δ over the draw of the training set S of m samples 3: DISPLAYFORM1 Typically, and in the rest of this discussion,W is a Gaussian with covariance σ 2 I for some σ > 0 centered at the weights W learned based on the training data. Furthermore, we will set P to be a Gaussian with covariance σ 2 I centered at the random initialization of the network like in BID3, instead of at the origin, like in BID0. This is because the ing KL-divergence -which depends on the distance between the means of the prior and the posterior -is known to be smaller, and to save a √ H factor in the bound . To extend the above PAC-Bayesian bound to a standard generalization bound on a deterministic classifier W, we need to replace the training and the test loss of the stochastic classifier with that of the original, deterministic classifier. However, in doing so, we will have to introduce extra terms in the upper bound to account for the perturbation suffered by the train and test loss under the Gaussian perturbation of the parameters. To tightly bound these two terms, we need that the network is noise-resilient on training and test data respectively. Our hope is that if the learning algorithm has found weights such that the network is noise-resilient on the training data, we can then generalize this noise-resilience over to test data as well, allowing us to better bound the excess terms. DISPLAYFORM0 For convenience, we also define an additional R + 1th set to be the singleton set containing the margin of the classifier on the input: f (x; W) [y] − max j≠y f (x; W) [j]. Note that if this term is positive (negative) then the classification is (in)correct. We will also denote the constant ∆ ⋆ R+1,1 as γ class.ORDERING OF THE SETS OF PROPERTIES We now impose a crucial constraint on how these sets of properties depend on each other. Roughly speaking, we want that for a given input, if the first r − 1 sets of properties approximately satisfy the condition in Equation 1, then the properties in the rth set are noise-resilient i.e., under random parameter perturbations, these properties do not suffer much perturbation. This kind of constraint would naturally hold for deep networks if we have chosen the properties carefully e.g., we will show that, for any given input, the perturbation in the pre-activation values of the dth layer is small as long as the absolute pre-activation values in the layers below d − 1 are large, and a few other norm-bounds on the lower layer weights are satisfied. We formalize the above requirement by defining expressions ∆ r,l (σ) that bound the perturbation in the properties ρ r,l, in terms of the variance σ 2 of the parameter perturbations. For any r ≤ R + 1 and for any (x, y), our framework requires the following to hold: DISPLAYFORM1 Let us unpack the above constraint. First, although the above constraint must hold for all inputs (x, y), it effectively applies only to those inputs that satisfy the pre-condition of the if-then statement: namely, it applies only to inputs (x, y) that approximately satisfy the first r − 1 conditions in DISPLAYFORM2. Next, we discuss the second part of the above if-then statement which specifies a probability term that is required to be small for all such inputs. In words, the first event within the probability term above is the event that for a given random perturbation U, the properties involved in the rth condition suffer a large perturbation. The second is the event that the properties involved in the first r − 1 conditions do not suffer much perturbation; but, given that these r − 1 conditions already hold approximately, this second event implies that these conditions are still preserved approximately under perturbation. In summary, our constraint requires the following: for any input on which the first r − 1 conditions hold, there should be very few parameter perturbations that significantly perturb the rth set of properties while preserving the first r − 1 conditions. When we instantiate the framework, we have to derive closed form expressions for the perturbation bounds ∆ r,l (σ) (in terms of only σ and the constants ∆ ⋆ r,l). As we will see, for ReLU networks, we will choose the properties in a way that this constraint naturally falls into place in a way that the perturbation bounds ∆ r,l (σ) do not grow with the product of spectral norms (Lemma E.1).THEOREM STATEMENT In this setup, we have the following'margin-based' generalization guarantee on the original network. That is, we bound the 0-1 test error of the network by a margin-based error on the training data. Our generalization guarantee, which scales linearly with the number of conditions R, holds under the setting that the training algorithm always finds weights such that on the training data, the conditions in Equation 1 is satisfied for all r = 1, ⋯, R. Theorem 3.1. Let σ * be the maximum standard deviation of the Gaussian parameter perturbation such that the constraint in Equation 2 holds with ∆ r,l (σ ⋆) ≤ ∆ ⋆ r,l ∀r ≤ R + 1 and ∀l. Then, for any δ > 0, with probability 1 − δ over the draw of samples S from D m, for any W we have that, if W satisfies the conditions in Equation 1 for all r ≤ R and for all training examples (x, y) ∈ S, then DISPLAYFORM3 The crux of our proof (in Appendix D) lies in generalizing the conditions of Equation 1 satisfied on the training data to test data one after the other, by proving that they are noise-resilient on both training and test data. Crucially, after we generalize the first r − 1 conditions from training data to test data (i.e., on most test and training data, the r − 1 conditions are satisfied), we will have from Equation 2 that the rth set of properties are noise-resilient on both training and test data. Using the noise-resilience of the rth set of properties on test/train data, we can generalize even the rth condition to test data. We emphasize a key, fundamental tool that we present in Theorem C.1 to convert a generic PACBayesian bound on a stochastic classifier, to a generalization bound on the deterministic classifier. Our technique is at a high level similar to approaches in BID12 BID13. In Section C.1, we argue how this technique is more powerful than other approaches in Neyshabur et al. FORMULA2; BID10; BID5 in leveraging the noiseresilience of a classifier. The high level argument is that, to convert the PAC-Bayesian bound, these latter works relied on a looser output perturbation bound, one that holds on all possible inputs, with high probability over all perturbations i.e., a bound on max x f (x; W) − f (x; W + U) ∞ w.h.p over draws of U. In contrast, our technique relies on a subtly different but significantly tighter bound: a bound on the output perturbation that holds with high probability given an input i.e., a bound on f (x; W) − f (x; W + U) ∞ w.h.p over draws of U for each x. When we do instantiate our framework as in the next section, this subtle difference is critical in being able to bound the output perturbation without suffering from a factor proportional to the product of the spectral norms of the weight matrices (which is the case in Neyshabur et al. FORMULA2). NOTATION. In this section, we apply our framework to feedforward fully connected ReLU networks of depth D (we care about D > 2) and width H (which we will assume is larger than the input dimensionality N, to simplify our proofs) and derive a generalization bound on the original network that does not scale with the product of spectral norms of the weight matrices. Let φ (⋅) denote the ReLU activation. We consider a network parameterized by DISPLAYFORM0 We denote the value of the hth hidden unit on the dth layer before and after the activation by DISPLAYFORM1 to be the Jacobian of the pre-activations of layer d with respect to the pre-activations of layer d ′ for d ′ ≤ d (each row in this Jacobian corresponds to a unit in layer d). In short, we will call this, Jacobian d d ′. Let Z denote the random initialization of the network. Informally, we consider a setting where the learning algorithm satisfies the following conditions on the training data that make it noise-resilient on training data: a) the 2 norm of the hidden layers are all small, b) the pre-activation values are all sufficiently large in magnitude, c) the Jacobian of any layer with respect to a lower layer, has rows with a small 2 norm, and has a small spectral norm. We cast these conditions in the form of Equation 1 by appropriately defining the properties ρ's and the margins ∆ ⋆'s in the general framework. We note that these properties are quite similar to those already explored in BID0; Neyshabur et al. FORMULA2; we provide more intuition about these properties, and how we cast them in our framework in Appendix E.1.Having defined these properties, we first prove in Lemma E.1 in Appendix E a guarantee equivalent to the abstract inequality in Equation 2. Essentially, we show that under random perturbations of the parameters, the perturbation in the output of the network and the perturbation in the input-dependent properties involved in (a), (b), (c) themselves can all be bounded in terms of each other. Crucially, these perturbation bounds do not grow with the spectral norms of the network. Having instantiated the framework as above, we then instantiate the bound provided by the framework. Our generalization bound scales with the bounds on the properties in (a) and (c) above as satisfied on the training data, and with the reciprocal of the property in (b) i.e., the smallest absolute value of the pre-activations on the training data. Additionally, our bound has an explicit dependence on the depth of the network, which arises from the fact that we generalize R = O(D) conditions. Most importantly, our bound does not have a dependence on the product of the spectral norms of the weight matrices. Theorem 4.1. (shorter version; see Appendix F for the complete statement) For any margin γ class > 0, and any δ > 0, with probability 1 − δ over the draw of samples from D m, for any W, we have that: DISPLAYFORM2 (an upper bound on the spectral norm of the Jacobian for each layer).In FIG0, we show how the terms in the bound vary for networks of varying depth with a small width of H = 40 on the MNIST dataset. We observe that B layer-2, B output, B jac-row-2, B jac-spec typically lie in the range of and scale with depth as ∝ 1.57 D. In contrast, the equivalent term from Neyshabur et al. FORMULA2 consisting of the product of spectral norms can be as large as 10 3 or 10 5 and scale with D more severely as 2.15 D.The bottleneck in our bound is B preact, which scales inversely with the magnitude of the smallest absolute pre-activation value of the network. In practice, this term can be arbitrarily large, even though it does not depend on the product of spectral norms/depth. This is because some hidden units can have arbitrarily small absolute pre-activation values -although this is true only for a small proportion of these units. To give an idea of the typical, non-pathological magnitude of the pre-activation values, we plot two other variations of B preact: a) 5%-B preact which is calculated by ignoring 5% of the training datapoints with the smallest absolute pre-activation values and b) median-B preact which is calculated by ignoring half the hidden units in each layer with the smallest absolute pre-activation values for each input. We observe that median-B preact is quite small (of the order of 10 2), while 5%-B preact, while large (of the order of 10 4), is still orders of magnitude smaller than B preact. In Figure 2 we show how our overall bound and existing product-of-spectral-norm-based bounds BID1 BID0 vary with depth. While our bound is orders of magnitude larger than prior bounds, the key point here is that our bound grows with depth as 1.57 D while prior bounds grow with depth as 2.15D indicating that our bound should perform asymptotically better with respect to depth. Indeed, we verify that our bound obtains better values than the other existing bounds when D = 28 (see Figure 2 b). We also plot hypothetical variations of our bound replacing B preact with 5%-B preact (see "Ours-5%") and median-B preact (see "Ours-Median") both of which perform orders of magnitude better than our actual bound (note that these two hypothetical bounds do not actually hold good). In fact for larger depth, the bound with 5%-B preact performs better than all other bounds (including existing bounds). This indicates that the only bottleneck in our bound comes from the dependence on the smallest pre-activation magnitudes, and if this particular dependence is addressed, our bound has the potential to achieve tighter guarantees for even smaller D such as D = 8. In the left, we vary the depth of the network (fixing H = 40) and plot the logarithm of various generalization bounds ignoring the dependence on the training dataset size and a log(DH) factor in all of the considered bounds. Specifically, we consider our bound, the hypothetical versions of our bound involving 5%-B preact and median-B preact respectively, and the bounds from Neyshabur et al. DISPLAYFORM3 and Bartlett et al. FORMULA2 maxx DISPLAYFORM4 both of which have been modified to include distance from initialization instead of distance from origin for a fair comparison. Observe the last two bounds have a plot with a larger slope than the other bounds indicating that they might potentially do worse for a sufficiently large D. Indeed, this can be observed from the plots on the right where we report the distribution of the logarithm of these bounds for D = 28 across 12 runs (although under training settings different from the experiments on the left; see Appendix F.3 for the exact details).We refer the reader to Appendix F.3 for added discussion where we demonstrate how all the quantities in our bound vary with depth for H = 1280 (Finally, as noted before, we emphasize that the dependence of our bound on the pre-activation values is a limitation in how we characterize noise-resilience through our conditions rather than a drawback in our general PAC-Bayesian framework itself. Specifically, using the assumed lower bound on the pre-activation magnitudes we can ensure that, under noise, the activation states of the units do not flip; then the noise propagates through the network in a tractable, "linear" manner. Improving this analysis is an important direction for future work. For example, one could modify our analysis to allow perturbations large enough to flip a small proportion of the activation states; one could potentially formulate such realistic conditions by drawing inspiration from the conditions in Neyshabur et al. FORMULA2 ; BID0 .However, we note that even though these prior approaches made more realistic assumptions about the magnitudes of the pre-activation values, the key limitation in these approaches is that even under our non-realistic assumption, their approaches would yield bounds only on stochastic/compressed networks. Generalizing noise-resilience from training data to test data is crucial to extending these bounds to the original network, which we accomplish. In this work, we introduced a novel PAC-Bayesian framework for leveraging the noise-resilience of deep neural networks on training data, to derive a generalization bound on the original uncompressed, deterministic network. The main philosophy of our approach is to first generalize the noise-resilience from training data to test data using which we convert a PAC-Bayesian bound on a stochastic network to a standard margin-based generalization bound. We apply our approach to ReLU based networks and derive a bound that scales with terms that capture the interactions between the weight matrices better than the product of spectral norms. For future work, the most important direction is that of removing the dependence on our strong assumption that the magnitude of the pre-activation values of the network are not too small on training data. More generally, a better understanding of the source of noise-resilience in deep ReLU networks would help in applying our framework more carefully in these settings, leading to tighter guarantees on the original network. We will use upper-case symbols to denote matrices, and lower-case bold-face symbols to denote vectors. In order to make the mathematical statements/derivations easier to read, if we want to emphasize a term, say x, we write, x. Recall that we consider a neural netork of depth D (i.e., D − 1 hidden layers and one output layer) mapping from R N → R K, where K is the number of class labels in the learning task. The layers are fully connected with H units in each hidden layer, and with ReLU activations φ (⋅) on all the hidden units and linear activations on the output units. We denote the parameters of the network using the symbol W, which in turn denotes a set of weight matrices DISPLAYFORM0 to be the Jacobian corresponding to the pre-activation values of layer d with respect to the pre-activation values of layer d ′ on an input x. That is, DISPLAYFORM1 In other words, this corresponds to the product of the'activated' portion of the matrices DISPLAYFORM2, where the weights corresponding to inactive inputs are zeroed out. In short, we will call this' Jacobian d d ′'. Note that each row in this Jacobian corresponds to a unit on the dth layer, and each column corresponds to a unit on the d ′ th layer. We will denote the parameters of a random initialization of the network by Z = (Z 1, Z 2, ⋯, Z d). Let D be an underlying distribution over R N × {1, 2, ⋯, K} from which the data is drawn. In our PAC-Bayesian analysis, we will use U to denote a set of D weight matrices U 1, U 2, ⋯, U D whose entries are sampled independently from a Gaussian. Furthermore, we will use U d to denote only the first d of the randomly sampled weight matrices, and W + U d to denote a network where the d random matrices are added to the first d weight matrices in W. Note that W + U 0 = W. Thus, f (x; W + U d) is the output of a network where the first d weight matrices have been perturbed. In our analysis, we will also need to study a perturbed network where the hidden units are frozen to be at the activation state they were at before the perturbation; we will use the notation W[+U d] to denote the weights of such a network. For our statements regarding probability of events, we will use ∧, ∨, and ¬ to denote the intersection, union and complement of events (to disambiguate from the set operators). In this section, we present some standard . The first two below will be useful for our noise resilience analysis. HOEFFDING BOUND Lemma B.1. For i = 1, 2, ⋯, n, let X i be independent random variables sampled from a Gaussian with mean µ i and variance σ 2 i. Then for all t ≥ 0, we have: DISPLAYFORM0 Or alternatively, for δ ∈ DISPLAYFORM1 Note that an identical inequality holds good symmetrically for the event ∑ n i=1 X i − µ i ≤ −t, and so the probability that the event ∑ n i=1 X i − µ i > t holds, is at most twice the failure probability in the above inequalities. Lemma B.2. Let U be a H 1 × H 2 matrix where each entry is sampled from N (0, σ 2). Let x be an arbitrary vector in DISPLAYFORM0 Proof. U x is a random vector sampled from a multivariate Gaussian with mean E[U x] = 0 and co-variance E[U xx T U T]. The (i, j)th entry in this covariance matrix is E[(u DISPLAYFORM1 2 . When i ≠ j, since u i and u j are independent random variables, we will have E[(u DISPLAYFORM2 SPECTRAL NORM OF ENTRY-WISE GAUSSIAN MATRIX The following bounds the spectral norm of a matrix with Gaussian entries, with high probability: DISPLAYFORM3 2 ) or alternatively, for any δ > 0, DISPLAYFORM4 2 ) KL DIVERGENCE OF GAUSSIANS. We will use the following KL divergence equality to bound the generalization error in our PAC-Bayesian analyses. Lemma B.4. Let P be the spherical Gaussian N (µ 1, σ 2 I) and Q be the spherical Gaussian N (µ 2, σ 2 I). Then, the KL-divergence between Q and P is: DISPLAYFORM5 In this section, we will present our main PAC-Bayesian theorem that will guide our analysis of generalization in our framework. Concretely, our extends the generalization bound provided by conventional PAC-Bayesian analysis BID13 -which is a generalization bound on the expected loss of a distribution of classifiers i.e., a stochastic classifier -to a generalization bound on a deterministic classifier. The way we reduce the PAC-Bayesian bound to a standard generalization bound, is different from the one pursued in previous works BID0 BID10.The generalization bound that we state below is a bit more general than standard generalization bounds on deterministic networks. Typically, generalization bounds are on the classification error; however, as discussed in the main paper we will be dealing with generalizing multiple different conditions on the interactions between the weights of the network from the training data to test data. So to state a bound that is general enough, we consider a set of generic functions ρ r (W, x, y) for r = 1, 2, ⋯R ′ (we use R ′ to distinguish it from R, the number of conditions in the abstract classifier of Section 3.1). Each of these functions compute a scalar value that corresponds to some input-dependent property of the network with parameters W for the datapoint (x, y). As an example, this property could simply be the margin of the function on the yth class i.e., f (x; W) [y] − max j≠y f (x; W) [j]. Theorem C.1. Let P be a prior distribution over the parameter space that is chosen independent of the training dataset. Let U be a random variable sampled entrywise from N (0, σ 2). Let ρ r (⋅, ⋅, ⋅) and ∆ r > 0 for r = 1, 2, ⋯R ′, be a set of input-dependent properties and their corresponding margins. We define the network W to be noise-resilient with respect to all these functions, at a given data point (x, y) if: DISPLAYFORM6, W) denote the probability over the random draw of a point (x, y) drawn from D, that the network with weights W is not noise-resilient at (x, y) according to Equation 3. DISPLAYFORM7, W) denote the fraction of data points (x, y) in a dataset S for which the network is not noise-resilient according to Equation 3. Then for any δ, with probability 1 − δ over the draws of a sample set S = {(x i, y i) ∼ D i = 1, 2, ⋯, m}, for any W we have: DISPLAYFORM8 The reader maybe curious about how one would bound the term µ D in the above bound, as this term corresponds to noise-resilience with respect to test data. This is precisely what we bound later when we generalize the noise-resilience-related conditions satisfied on train data over to test data. The above approach differs from previous approaches used by BID0; BID10 in how strong a noise-resilience we require of the classifier to provide the generalization guarantee. The stronger the noise-resilience requirement, the more price we have to pay when we jump from the PAC-Bayesian guarantee on the stochastic classifier to a guarantee on the deterministic classifier. We argue that our noise-resilience requirement is a much milder condition and therefore promises tighter guarantees. Our requirement is in fact philosophically similar to BID12 BID13, although technically different. More concretely, to arrive at a reasonable generalization guarantee in our setup, we would need that µ D andμ S are both only as large as O(1 √ m). In other words, we would want the following for (x, y) ∼ D and for (x, y) ∼ S: DISPLAYFORM0 Previous works require a noise resilience condition of the form that with high probability a particular perturbation does not perturb the classifier output on any input. For example, the noise-resilience condition used in BID0 written in terms of our notations, would be: DISPLAYFORM1 The main difference between the above two formulations is in what makes a particular perturbation (un)favorable for the classifier. In our case, we deem a perturbation unfavorable only after fixing the datapoint. However, in the earlier works, a perturbation is deemed unfavorable if it perturbs the classifier output sufficiently on some datapoint from the domain of the distribution. While this difference is subtle, the earlier approach would lead to a much more pessimistic analysis of these perturbations. In our analysis, this weakened noise resilience condition will be critical in analyzing the Gaussian perturbations more carefully than in Neyshabur et al. FORMULA2 i.e., we can bound the perturbation in the classifier output more tightly by analyzing the Gaussian perturbation for a fixed input point. Note that one way our noise resilience condition would seem stronger in that on a given datapoint we want less than 1 √ m mass of the perturbations to be unfavorable for us, while in previous bounds, there can be as much as 1 2 probability mass of perturbations that are unfavorable. In our analysis, this will only weaken our generalization bound by a ln √ m factor in comparison to previous bounds (while we save other significant factors). Proof. The starting point of our proof is a standard PAC-Bayesian theorem BID13 which bounds the generalization error of a stochastic classifier. Let P be a data-independent prior over the parameter space. Let L(W, x, y) be any loss function that takes as input the network parameter, and a datapoint x and its true label y and outputs a value in. Then, we have that, with probability 1 − δ over the draw of S ∼ D m, for every distribution Q over the parameter space, the following holds: DISPLAYFORM0 In other words, the statement tells us that except for a δ proportion of bad draws of m samples, the test loss of the stochastic classifierW ∼ Q would be close to its train loss. This holds for every possible distribution Q, which allows us to cleverly choose Q based on S. As is the convention, we choose Q to be the distribution of the stochastic classifier picked from N (W, σ 2 I) i.e., a Gaussian perturbation of the deterministic classifier W.RELATING TEST LOSS OF STOCHASTIC CLASSIFIER TO DETERMINISTIC CLASSIFIER. Now our task is to bound the loss for the deterministic classifier W, Pr (x,y)∼D [∃r ρ r (W, x, y) < 0]. To this end, let us define the following margin-based variation of this loss for some c ≥ 0: DISPLAYFORM1 and so we have Pr (x,y)∼D [∃r ρ r (W, x, y) DISPLAYFORM2 First, we will bound the expected L 0 of a deterministic classifier by the expected L 1 2 of the stochastic classifier; then we will bound the test L 1 2 of the stochastic classifier using the PAC-Bayesian bound. We will split the expected loss of the deterministic classifier into an expectation over datapoints for which it is noise-resilient with respect to Gaussian noise and an expectation over the rest. To write this out, we define, for a datapoint (x, y), N(W, x, y) to be the event that W is noise-resilient at (x, y) as defined in Equation 3 in the theorem statement: DISPLAYFORM3 To further continue the upper bound on the left hand side, we turn our attention to the stochastic classifier's loss on the noise-resilient part of the distribution D (we will lower bound this term in terms of the first term on the right hand side above). For simplicity of notations, we will write D ′ to denote the distribution D conditioned on N(W, x, y). Also, let U(W, x, y) be the favorable event that for a given data point (x, y) and a draw of the stochastic classifier,W, it is the case that for every r, ρ r (W, x, y) − ρ r (W, x, y) ≤ ∆ r 2. Then, the stochastic classifier's loss L 1 2 on D ′ is: DISPLAYFORM4 splitting the inner expectation over the favorable and unfavorable perturbations, and using linearity of expectations, DISPLAYFORM5 to lower bound this, we simply ignore the second term (which is positive) DISPLAYFORM6 Next, we use the following fact: if L 1 2 (W, x, y) = 0, then for all r, ρ r (W, x, y) ≥ ∆ r 2 and ifW is a favorable perturbation of W, then for all r, ρ r (W, DISPLAYFORM7 Hence ifW is a favorable perturbation then, DISPLAYFORM8 . Therefore, we can lower bound the above expression by replacing the stochastic classifier with the deterministic classifier (and thus ridding ourselves of the expectation over Q): DISPLAYFORM9 Since the favorable perturbations for a fixed datapoint drawn from D ′ have sufficiently high probability (that is, PrW ∼Q U(W, x, y) ≥ 1 − 1 √ m), we have: DISPLAYFORM10 Thus, we have a lower bound on the stochastic classifier's loss that is in terms of the deterministic classifier's loss on the noise-resilient datapoints. Rearranging it, we get an upper bound on the latter: DISPLAYFORM11 Thus, we have an upper bound on the expected loss of the deterministic classifier W on the noiseresilient part of the distribution. Plugging this back in the first term of the upper bound on the deterministic classifier's loss on the whole distribution D in Equation 5 we get: DISPLAYFORM12, W) rearranging, we get: DISPLAYFORM13 rewriting the expectation over D ′ explicitly as an expectation over D conditioned on N(W, x, y), we get: DISPLAYFORM14 the first term above is essentially an expectation of a loss over the distribution D with the loss set to be zero over the non-noise-resilient datapoints and set to be L 1 2 over the noise-resilient datapoints; thus we can upper bound it with the expectation of the L 1 2 loss over the whole distribution D: DISPLAYFORM15 Now observe that we can upper bound the first term here using the PAC-Bayesian bound by plugging in L 1 2 for the generic L in FIG5; however, the bound would still be in terms of the stochastic classifier's train error. To get the generalization bound we seek, which involves the deterministic classifier's train error, we need to take one final step mirroring these tricks on the train loss. LOSS. Our analysis here is almost identical to the previous analysis. Instead of working with the distribution D and D ′ we will work with the training data set S and a subset of it S ′ for which noise resilience property is satisfied by W. Below, to make the presentation neater, we use (x, y) ∼ S to denote uniform sampling from S. First, we upper bound the stochastic classifier's train loss (L 1 2) as follows: DISPLAYFORM0 splitting over the noise-resilient points S ′ ((x, y) ∈ S for which N(W, x, y) holds) like in Equation 5, we can upper bound as: DISPLAYFORM1 We can upper bound the first term by first splitting it over the favorable and unfavorable perturbations like we did before: DISPLAYFORM2 To upper bound this, we apply a similar argument. First, if L 1 2 (W, x, y) = 1, then ∃r such that ρ r (W, x, y) < ∆ r 2 and ifW is a favorable perturbation then for that value of r, ρ r (W, x, y) < ρ r (W, x, y) + ∆ r 2 < ∆ r. Thus ifW is a favorable perturbation then, L 1 (W, x, y) = 1 whenever L 1 2 (W, x, y) = 1 i.e., L 1 2 (W, x, y) ≤ L 1 (W, x, y). Next, we use the fact that the unfavorable perturbations for a fixed datapoint drawn from S ′ have sufficiently low probability i.e., PrW ∼Q ¬U(W, x, y) ≤ 1 √ m. Then, we get the following upper bound on the above equations, by replacing the stochastic classifier with the deterministic classifier (and thus ignoring the expectation over Q): DISPLAYFORM3 Plugging this back in the first term of Equation 7, we get: DISPLAYFORM4, W) since the first term is effectively the expectation of a loss over the whole distribution with the loss set to be zero on the non-noise-resilient points and set to L 1 over the rest, we can upper bound it by setting the loss to be L 1 over the whole distribution: DISPLAYFORM5 Applying the above upper bound and the bound in Equation 6 into the PAC-Bayesian of Equation 4 yields our (Note that combining these equations would produce the term DISPLAYFORM6 which is at most DISPLAYFORM7, which we reflect in the final bound.). In this section, we present the proof for the abstract generalization guarantee presented in Section 3. Our proof is based on the following recursive inequality that we demonstrate for all r ≤ R (we will prove a similar, but slightly different inequality for r = R + 1): DISPLAYFORM0 Recall that the rth condition in Equation 1 is that ∀l, ρ r,l > ∆ ⋆ r,l. Above, we bound the probability mass of test points such that any one of the first r conditions in Equation 1 is not even approximately satisfied, in terms of the probability mass of points where one of the first r − 1 conditions is not even approximately satisfied, and a term that corresponds to how much error there can be in generalizing the rth condition from the training data. Our proof crucially relies on Theorem C.1. This theorem provides an upper bound on the proportion of test data that fail to satisfy a set of conditions, in terms of four quantities. The first quantity is the proportion of training data that do not satisfy the conditions; the second and third quantities, which we will in short refer to asμ S andμ D, correspond to the proportion of training and test data on which the properties involved in the conditions are not noise-resilient. The fourth quantity is the generalization error. First, we consider the base case when r = 1, and apply the PAC-Bayes-based guarantee from Theorem C.1 on the first set of properties {ρ 1,1, ρ 1,2, ⋯} and their corresponding constants DISPLAYFORM1 Effectively this establishes that the noise-resilience requirement of Equation 3 in Theorem C.1 holds on all possible inputs, thus proving our claim that the termsμ S andμ D would be zero. Thus, we will get that DISPLAYFORM2 which proves the recursion statement for the base case. To prove the recursion for some arbitrary r ≤ R, we again apply the PAC-Bayes-based guarantee from Theorem C.1, but on the union of the first r sets of properties. Again, we will have that the first term in the guarantee would be zero, since the corresponding conditions are satisfied on the training data. Now, to bound the proportion of bad pointsμ S andμ D, we make the following claim:the network is noise-resilient as per Equation 3 in Theorem C.1 for any input that satisfies the r−1 conditions approximately i.e., ∀q ≤ r−1 and ∀l, ρ q,l (W, x, y) > 0.The above claim can be used to prove Equation 8 as follows. Since all the conditions are assumed to be satisfied by a margin on the training data, this claim immediately implies that µ S is zero. Similarly, this claim implies that for the test data, we can bound µ D in terms of Pr (x,y)∼D [∃q<r ∃l ρ q,l (W, x, y) < 0], thus giving rise to the recursion in Equation 8. Now, to prove our claim, consider an input (x, y) such that ρ q,l (W, x, y) > 0 for q = 1, 2, ⋯, r − 1 and for all possible l. First from the assumption in our theorem statement that ∆ q,l (σ ⋆) ≤ ∆ ⋆ q,l, we have the following upper bound on the proportion of parameter perturbations under which any of the properties in the first r sets suffer a large perturbation: DISPLAYFORM3 Now, this can be expanded as a summation over q = 1, 2, ⋯, r as: DISPLAYFORM4 and because (x, y) satisfies ρ q,l (W, x, y) > 0 for q = 1, 2, ⋯, r − 1 and for all possible l, by the constraint assumed in Equation 2, we have: DISPLAYFORM5 Thus, we have that (x, y) satisfies the noise-resilience condition from Equation 3 in Theorem C.1 if it also satisfies ρ q,l (W, x, y) > 0 for q = 1, 2, ⋯, r − 1 and for all possible l. This proves our claim, and hence in turn proves the recursion in Equation 8.Finally, we can apply a similar argument for the R + 1th set of input-dependent properties (which is a singleton set consisting of the margin of the network) with a small change since the first term in the guarantee from Theorem C.1 is not explicitly assumed to be zero; we will get an inequality in terms of the number of training points that are not classified correctly by a margin, giving rise to the margin-based bound: DISPLAYFORM6 Note that in the first term on the right hand side, ρ R+1,1 (W, x, y) corresponds to the margin of the classifier on (x, y). Now, by using the fact that the test error is upper bounded by the left hand side in the above equation, applying the recursion on the right hand side R + 1 times, we get our final . In this section, we will quantify the noise resilience of a network in different aspects. Each of our bounds has the following structure: we fix an input point (x, y), and then say that with high probability over a Gaussian perturbation of the network's parameters, a particular input-dependent property of the network (say the output of the network, or the pre-activation value of a particular unit h at a particular layer d, or say the Frobenius norm sof its active weight matrices), changes only by a small magnitude proportional to the variance σ 2 of the Gaussian perturbation of the parameters. A key feature of our bounds is that they do not involve the product of the spectral norm of the weight matrices and hence save us an exponential factor in the final generalization bound. Instead, the bound in the perturbation of a particular property will be in terms of i) the magnitude of the some'preceding' properties (typically, these are properties of the lower layers) of the network, and ii) how those preceding properties themselves respond to perturbations. For example, an upper bound in the perturbation of the dth layer's output would involve the 2 norm of the lower layers d ′ < d, and how much they would blow up under these perturbations. E.2 SOME NOTATIONS.To formulate our lemma statement succinctly, we design a notation wherein we define a set of'tolerance parameters' which we will use to denote the extent of perturbation suffered by a particular property of the network. LetĈ denote a'set' (more on what we mean by a set below) of positive tolerance values, consisting of the following elements: • We callĈ a'set' to denote a group of related constants into a single symbol. Each element in this set has a particular semantic associated with it, unlike the standard notation of a set, and so when we refer to, sayζ d d ′ ∈Ĉ, we are indexing into the set to pick a particular element.• We will use the subscriptĈ d to index into a subset of only those tolerance values corresponding to layers from 1 until d. Next we define two events. The first event formulates the scenario that for a given input, a particular perturbation of the weights until layer d brought about very little change in the properties of these layers (within some tolerance levels). The second event formulates the scenario that the perturbation did not flip the activation states of the network. Definition E.1. Given an input x, and an arbitrary set of constantsĈ ′, for any perturbation U of W, we denote by PERT-BOUND(W+U,Ĉ ′, x) the event that:• for eachα DISPLAYFORM0 • for eachγ ′ d ∈Ĉ ′, the maximum perturbation in the preactivation of hidden units on layer d DISPLAYFORM1 • for eachζ DISPLAYFORM2 • for eachψ DISPLAYFORM3 NOTE: If we supply only a subset ofĈ (sayĈ d instead of the whole ofĈ) to the above event, PERT-BOUND(W + U, ⋅, x), then it would denote the event that the perturbations suffered by only that subset of properties is within the respective tolerance values. Next, we define the event that the perturbations do not affect the activation states of the network. Definition E.2. For any perturbation U of the matrices W, let UNCHANGED-ACTS d (W + U, x) denote the event that none of the activation states of the first d layers change on perturbation. E.3 MAIN LEMMA.Our here are styled similar to the equations required by Equation 2 presented in the main paper. For a given input point and for a particular property of the network, roughly, we bound the the probability that a perturbation affects the value of the property while none of the preceding preceding properties themselves are perturbed beyond a certain tolerance level. Lemma E.1. LetĈ be a set of constants (that denote the amount of perturbation in the properties preceding a considered property). For anyδ > 0, defineĈ ′ (which is a bound on the perturbation of a considered property) in terms ofĈ and the perturbation parameter σ, for all d = 1, 2, ⋯, D and for all d ′ = d − 1, ⋯, 1 as follows: DISPLAYFORM4 2 ) for any d. Then, the following statements hold good:1. Bound on perturbation of of 2 norm of the output of layer d. DISPLAYFORM5 3. Bound on perturbation of 2 norm on the rows of the Jacobians d d ′.P r U ¬PERT-BOUND(W + U, {ζ P r U ¬PERT-BOUND(W + U, {ψ DISPLAYFORM6 DISPLAYFORM7 Proof. For the most part of this discussion, we will consider a perturbed network where all the hidden units are frozen to be at the same activation state as they were at, before the perturbation. We will denote the weights of such a network by W [+U] and its output at the dth layer by f d (x; W[+U]). By having the activations states frozen, the Gaussian perturbations propagate linearly through the activations, effectively remaining as Gaussian perturbations; then, we can enjoy the well-established properties of the Gaussian even after they propagate. DISPLAYFORM8 Now, the spectral norm of Y d ′′ is at most the products of the spectral norms of each of these three matrices. Using Lemma B.3, the spectral norm of the middle term U d ′′ can be bounded by σ 2H ln 2DĤ δ with high probability 1 −δ D over the draws of U d ′′. We will also decompose the spectral norm of the first term so that our final bound does not involve any Jacobian of the dth layer. When d ′′ = d, this term has spectral norm 1 because the Jacobian d d is essentially the identity matrix. When DISPLAYFORM0. Furthermore, since, for a ReLU network, DISPLAYFORM1 effectively W d with some columns zerod out, the spectral norm of the Jacobian is upper bounded by the spectral norm W d.Putting all these together, we have that with probability 1 −δ D over the draws of U d ′′, the following holds good: DISPLAYFORM2 By a union bound, we then get that with probability 1 −δ over the draws of U d, we can upper bound Equation 14 as: DISPLAYFORM3 δ Note that the above bound simultaneously holds over all d ′ (without the application of a union bound). Finally we get the of the lemma by a similar argument as in the case of the perturbation bound on the output of each layer. 10 Again, note that the below succinct formula works even for corner cases like d Below, we present our main for this section, a generalization bound on a class of networks that is based on certain norm bounds on the training data. We provide a more intuitive presentation of these bounds after the proof in Appendix F.3. Theorem. 4.1 For any δ > 0, with probability 1 − δ over the draw of samples S ∼ D m, for any W, we have that: DISPLAYFORM0 DISPLAYFORM1 GROUPING AND ORDERING THE PROPERTIES. Now to apply the abstract generalization bound in Theorem 3.1, recall that we need to come up with an ordered grouping of the functions above such that we can realize the constraint given in Equation 2. Specifically, this constraint effectively required that, for a given input, the perturbation in the properties grouped in a particular set be small, given that all the properties in the preceding sets satisfy the corresponding conditions on them. To this end, we make use of Lemma E.1 where we have proven perturbation bounds relevant to the properties we have defined above. Our lemma also naturally induces dependencies between these properties in a way that they can be ordered as required by our framework. The order in which we traverse the properties is as follows, as dictated by Lemma E.1. We will go from layer 0 uptil D. For a particular layer d, we will first group the properties corresponding to the spectral norms of the Jacobians of that layer whose corresponding margins are {ψ DISPLAYFORM2 Next, we will group the row 2 norms of the Jacobians of layer d, whose corresponding margins are {ζ DISPLAYFORM3 Followed by this, we will have a singleton set of the layer output's 2 norm whose corresponding margin is α ▵ d . We then will group the pre-activations of layer d, each of which has the corresponding margin γ ▵ d . For the output layer, instead of the pre-activations or the output 2 norm, we will consider the margin-based property we have defined above.12 13 Observe that the number of sets R that we have created in this manner, is at most 4D since there are at most 4 sets of properties in each layer. that is required by our framework. For any r, the rth set of properties need to satisfy the following statement: DISPLAYFORM0 Furthermore, we want the perturbation bounds ∆ r,l (σ) to satisfy ∆ r,l (σ ⋆) ≤ ∆ ⋆ r,l, where σ ⋆ is the standard deviation of the parameter perturbation chosen in the PAC-Bayesian analysis. The next step in our proof is to show that our choice of σ ⋆, and the input-dependent properties, all satisfy the above requirements. To do this, we instantiate Lemma E.1 with σ = σ ⋆ as in Theorem 4.1 DISPLAYFORM1 Then, it can be verified that the values of the perturbation bounds inĈ ′ in Lemma E.1 can be upper bounded by the corresponding value in C ▵ 2. In other words, we have that for our chosen value of σ, the perturbations in all the properties and the output of the network can be bounded by the constants specified in C ▵ 2. Succinctly, let us say: DISPLAYFORM2 Given that these perturbation bounds hold for our chosen value of σ, we will focus on showing that a constraint of the form Equation 2 holds for the row 2 norms of the Jacobians d d ′ for all d ′ < d. A similar approach would apply for the other properties. First, we note that the sets of properties preceding the ones corresponding to the row 2 norms of Jacobian d d ′, consists of all the properties upto layer d − 1. Therefore, the precondition for Equation 2 which is of the form ρ(W, x, y) > 0 for all the previous properties ρ, translates to norm bound on these properties involving the constants C † d−1 as discussed in Fact F.1. Succinctly, these norm bounds can be expressed as NORM-BOUND(W + U, DISPLAYFORM3 Given that these norm bounds hold for a particular x, our goal is to argue that the rest of the constraint in Equation 2 holds. To do this, we first argue that given these norm bounds, if PERT-BOUND(W + U, C ▵ d−1 2, x) holds, then so does UNCHANGED-ACTS d−1 (W + U, x). This is because, the event PERT-BOUND(W + U, C ▵ d−1 2, x) implies that the pre-activation values of layer d − 1 suffer a perturbation of at most γ DISPLAYFORM4 holds, we have that the preactivation values of this layer have a magnitude of at least γ † DISPLAYFORM5 From these two equations, we have that the hidden units even at layer d−1 of the network do not change their activation state (i.e., the sign of the pre-activation does not change) under this perturbation. We can similarly argue for the layers below d − 1, thus proving that UNCHANGED- DISPLAYFORM6 Then, from the above discussion on the activation states, and from Equation 15, we have that Lemma E.1 boils down to the following inequality, when we plug σ = σ ⋆:12 For layer 0, the only property that we have defined is the 2 norm of the input. 13 Note that the Jacobian for d d is nothing but an identity matrix regardless of the input datapoint; thus we do not need any generalization analysis to bound its value on a test datapoint. Hence, we ignore it in our analysis, as can be seen from the list of properties that we have defined. Note, that the ing bound would have a log term that does not affect our bound in an asymptotic sense. In this section, we provide more detailed demonstration of the dependence of the terms in our bound on the depth/width of the network. In all the experiments, including the ones in the main paper (except the one in Figure 2 (b)) we use SGD with learning rate 0.1 and mini-batch size 64. We train the network on a subset of 4096 random training examples from the MNIST dataset to minimize cross entropy loss. We stop training when we classify at least 0.99 of the data perfectly, with a margin of γ class = 10. In Figure 2 (b) where we train networks of depth D = 28, the above training algorithm is quite unstable. Instead, we use Adam with a learning rate of 10 −5 until the network achieves an accuracy of 0.95 on the training dataset. Finally, we note that all logarithmic transformations in our plots are to the base 10.In FIG2 we show how the norm-bounds on the input-dependent properties of the network do not scale as large as the product of spectral norms. For the remaining experiments in this section, we will present a slightly looser bound than the one presented in our main , motivated by the fact that computing our actual bound is expensive as it involves computing spectral norms of Θ(D 2) Jacobians on m training datapoints. We note that even this looser bound does not have a dependence on the product of spectral norms, and has similar overall dependence on the depth. Specifically, we will consider a bound that is based on a slightly modified noise-resilience analysis. Recall that in Lemma E.1, when we considered the perturbation in the row 2 norm Jacobian d d ′, we bounded Equation 13 in terms of the spectral norms of the Jacobians. Instead of taking this route, if we retained the bound in Equation 13, we will get a slightly different upper bound on the perturbation of the Jacobian row 2 norm as: DISPLAYFORM0 By using this bound in our analysis, we can ignore the spectral norm terms ψ Thus, the row 2 norms of these Jacobians must be split into separate sets of properties, and the bound on them generalized one after the other (instead of grouped into one set and generalized all at one go as before). This would Recall from the discussion in the introduction in the main paper that, prior works (; BID0 have also characterized noise resilience in terms of conditions on the interactions between the activated weight matrices. Below, we discuss the conditions assumed by these works, which parallel the conditions we have studied in our paper (such as the bounded 2 norm in each layer). There are two main high level similarities between the conditions studied across these works. First, these conditions -all of which characterize the interactions between the activated weights matrices in the network -are assumed only for the training inputs; such an assumption implies noise-resilience of the network on training inputs. Second, there are two kinds of conditions assumed. The first kind allows one to bound the propagation of noise through the network under the assumption that the activation states do not flip; the second kind allows one to bound the extent to which the activation states do flip.

represent the tldr for finding the corresponding scientific article

E.g.:
We provably recover the span of a deep multi-layered neural network with latent structure and empirically apply efficient span recovery algorithms to attack networks by obfuscating inputs. == The tremendous success of deep neural networks has motivated the need to better understand the fundamental properties of these networks, but many of the theoretical proposed have only been for shallow networks. In this paper, we study an important primitive for understanding the meaningful input space of a deep network: span recovery. For $k<n$, let $\mathbf{A} \in \mathbb{R}^{k \times n}$ be the innermost weight matrix of an arbitrary feed forward neural network $M: \mathbb{R}^n \to \mathbb{R}$, so $M(x)$ can be written as $M(x) = \sigma(\mathbf{A} x)$, for some network $\sigma: \mathbb{R}^k \to \mathbb{R}$. The goal is then to recover the row span of $\mathbf{A}$ given only oracle access to the value of $M(x)$. We show that if $M$ is a multi-layered network with ReLU activation functions, then partial recovery is possible: namely, we can provably recover $k/2$ linearly independent vectors in the row span of $\mathbf{A}$ using poly$(n)$ non-adaptive queries to $M(x)$. Furthermore, if $M$ has differentiable activation functions, we demonstrate that \textit{full} span recovery is possible even when the output is first passed through a sign or $0/1$ thresholding function; in this case our algorithm is adaptive. Empirically, we confirm that full span recovery is not always possible, but only for unrealistically thin layers. For reasonably wide networks, we obtain full span recovery on both random networks and networks trained on MNIST data. Furthermore, we demonstrate the utility of span recovery as an attack by inducing neural networks to misclassify data obfuscated by controlled random noise as sensical inputs. Consider the general framework in which we are given an unknown function f: R n → R, and we want to learn properties about this function given only access to the value f (x) for different inputs x. There are many contexts where this framework is applicable, such as blackbox optimization in which we are learning to optimize f (x) , PAC learning in which we are learning to approximate f (x) , adversarial attacks in which we are trying to find adversarial inputs to f (x) , or structure recovery in which we are learning the structure of f (x). For example in the case when f (x) is a neural network, one might want to recover the underlying weights or architecture (; . In this work, we consider the setting when f (x) = M (x) is a neural network that admits a latent low-dimensional structure, namely M (x) = σ(Ax) where A ∈ R k×n is a rank k matrix for some k < n, and σ: R k → R is some neural network. In this setting, we focus primarily on the goal of recovering the row-span of the weight matrix A. We remark that we can assume that A is full-rank as our extend to the case when A is not full-rank. Span recovery of general functions f (x) = g(Ax), where g is arbitrary, has been studied in some contexts, and is used to gain important information about the underlying function f. By learning Span(A), we in essence are capturing the relevant subspace of the input to f; namely, f behaves identically on x as it does on the projection of x onto the row-span of A. In statistics, this is known as effective dimension reduction or the multi-index model;. Another important motivation for span recovery is for designing adversarial attacks. Given the span of A, we compute the kernel of A, which can be used to fool the function into behaving incorrectly on inputs which are perturbed by vectors in the kernel. Specifically, if x is a legitimate input correctly classified by f and y is a large random vector in the kernel of A, then x + y will be indistinguishable from noise but we will have f (x) = f (x + y). Several works have considered the problem from an approximation-theoretic standpoint, where the goal is to output a hypothesis function f which approximates f well on a bounded domain. For instance, in the case that A ∈ R n is a rank 1 matrix and g(Ax) is a smooth function with bounded derivatives, gives an adaptive algorithm to approximate f. Their also give an approximation A to A, under the assumption that A is a stochastic vector (A i ≥ 0 for each i and i A i = 1). Extending this to more general rank k matrices A ∈ R k×n, and give algorithms with polynomial sample complexity to find approximations f to twice differentiable functions f. However, their do not provide any guarantee that the original matrix A In this paper, we provably show that span recovery for deep neural networks with high precision can be efficiently accomplished with poly(n) function evaluations, even when the networks have poly(n) layers and the output of the network is a scalar in some finite set. Specifically, for deep networks M (x): R n → R with ReLU activation functions, we prove that we can recover a subspace V ⊂ Span(A) of dimension at least k/2 with polynomially many non-adaptive queries. 1 First, we use a volume bounding technique to show that a ReLU network has sufficiently large piece-wise linear sections and that gradient information can be derived from function evaluations. Next, by using a novel combinatorial analysis of the sign patterns of the ReLU network along with facts in polynomial algebra, we show that the gradient matrix has sufficient rank to allow for partial span recovery. Theorem 3.4 (informal) Suppose we have the network M (x) = w T φ(W 1 φ(W 2 φ(. . . W d φ(Ax))... ), where φ is the ReLU and W i ∈ R ki×ki+1 are weight matrices, with k i possibly much smaller than k. Then, under mild assumptions, there is a non-adaptive algorithm that makes O(kn log k) queries to M (x) and returns in poly(n, k)-time a subspace V ⊆ span(A) of dimension at least k 2 with probability 1 − δ. We remark that span recovery of the first weight layer is provably feasible even in the surprising case when the neural network has many "bottleneck" layers with small O(log(n)) width. Because this does not hold in the linear case, this implies that the non-linearities introduced by activations in deep learning allow for much more information to be captured by the model. Moreover, our algorithm is non-adaptive, which means that the points x i at which M (x i) needs to be evaluated can be chosen in advance and span recovery will succeed with high probability. This has the benefit of being parallelizable, and possibly more difficult to detect when being used for an adversarial attack. In contrast with previous papers, we do not assume that the gradient matrix has large rank; rather our main focus and novelty is to prove this statement under minimal assumptions. We require only two mild assumptions on the weight matrices. The first assumption is on the orthant probabilities of the matrix A, namely that the distribution of sign patterns of a vector Ag, where g ∼ N (0, I n), is not too far from uniform. Two examples of matrices which satisfy this property are random matrices and matrices with nearly orthogonal rows. The second assumption is a non-degeneracy condition on the matrices W i, which enforces that products of rows of the matrices W i in vectors with non-zero coordinates. Our next is to show that full span recovery is possible for thresholded networks M (x) with twice differentiable activation functions in the inner layers, when the network has a 0/1 threshold function in the last layer and becomes therefore non-differentiable, i.e., M (x) ∈ {0, 1}. Since the activation functions can be arbitrarily non-linear, our algorithm only provides an approximation of the true subspace Span(A), although the distance between the subspace we output and Span(A) can be made exponentially small. We need only assume bounds on the first and second derivatives of the activation functions, as well as the fact that we can find inputs x ∈ R n such that M (x) = 0 with good probability, and that the gradients of the network near certain points where the threshold evaluates to one are not arbitrarily small. We refer the reader to Section 4 for further details on these assumptions. Under these assumptions, we can apply a novel gradient-estimation scheme to approximately recover the gradient of M (x) and the span of A. Theorem 4.3 (informal) Suppose we have the network M (x) = τ (σ(Ax)), where τ: R → {0, 1} is a threshold function and σ: R k → R is a neural network with twice differentiable activation functions, and such that M satisfies the conditions sketched above (formally defined in Section 4). Then there is an algorithm that runs in poly(N) time, making at most poly(N) queries to M (x), where N = poly(n, k, log, log(1 δ)), and returns with probability 1 − δ a subspace V ⊂ R n of dimension k such that for any x ∈ V, we have where P Span(A) is the orthogonal projection onto the span of A. Empirically, we verify our theoretical findings by running our span recovery algorithms on randomly generated networks and trained networks. First, we confirm that full recovery is not possible for all architectures when the network layer sizes are small. This implies that the standard assumption that the gradient matrix is full rank does not always hold. However, we see that realistic network architectures lend themselves easily to full span recovery on both random and trained instances. We emphasize that this holds even when the network has many small layers, for example a ReLU network that has 6 hidden layers with nodes, in that order, can still admit full span recovery of the rank 80 weight matrix. Furthermore, we observe that we can effortlessly apply input obfuscation attacks after a successful span recovery and cause misclassifications by tricking the network into classifying noise as normal inputs with high confidence. Specifically, we can inject large amounts of noise in the null space of A to arbitrarily obfuscate the input without changing the output of the network. We demonstrate the utility of this attack on MNIST data, where we use span recovery to generate noisy images that are classified by the network as normal digits with high confidence. We note that this veers away from traditional adversarial attacks, which aim to drastically change the network output with humanly-undetectable changes in the input. In our case, we attempt the arguably more challenging problem of drastically changing the input without affecting the output of the network. Notation For a vector x ∈ R k, the sign pattern of x, denoted sign(x) ∈ {0, 1} k, is the indicator vector for the nonzero coordinates of x. Namely, sign(x) i = 1 if x i = 0 and sign(x) i = 0 otherwise. Given a matrix A ∈ R n×m, we denote its singular values as σ min (A) = σ min{n,m},..., σ 1 (A) = σ max (A). The condition number of A is denoted κ(A) = σ max (A)/σ min (A). We let I n ∈ R n×n denote the n × n identity matrix. For a subspace V ⊂ R n, we write P V ∈ R n×n to denote the orthogonal projection matrix onto V. If µ ∈ R n and Σ ∈ R n×n is a PSD matrix, we write N (µ, Σ) to denote the multi-variate Gaussian distribution with mean µ and covariance Σ. Gradient Information For any function f (x) = g(Ax), note that ∇f (x) = A g(Ax) must be a vector in the row span of A. Therefore, span recovery boils down to understanding the span of the gradient matrix as x varies. Specifically, note that if we can find points x 1,.., x k such that {∇f (x i)} are linearly independent, then the full span of A can be recovered using the span of the gradients. To our knowledge, previous span recovery algorithms heavily rely on the assumption that the gradient matrix is full rank and in fact well-conditioned. Specifically, for some distribution D, it is assumed that H f = x∼D ∇f (x)∇f (x) dx is a rank k matrix with a minimum non-zero singular value bounded below by α and the number of gradient or function evaluations needed depends inverse polynomially in α. In contrast, in this paper, when f (x) is a neural network, we provably show that H f is a matrix of sufficiently high rank or large minimum non-zero singular value under mild assumptions, using tools in polynomial algebra. In this section, we demonstrate that partial span recovery is possible for deep ReLU networks. Specifically, we consider neural networks M (x): R n → R of the form where φ(x) i = max{x i, 0} is the RELU (applied coordinate-wise to each of its inputs), and W i ∈ R ki×ki+1, and w ∈ R k d, and A has rank k. We note that k i can be much smaller than k. In order to obtain partial span recovery, we make the following assumptions parameterized by a value γ > 0 (our algorithms will by polynomial in 1/γ): • Assumption 2: Si is the matrix with the rows j / ∈ S i set equal to 0. Moreover, we assume Our first assumption is an assumption on the orthant probabilities of the distribution Ag. Specifically, observe that Ag ∈ R k follows a multi-variate Gaussian distribution with covariance matrix AA T. Assumption 1 then states that the probability that a random vector x ∼ N (0, AA T) lies in a certain orthant of R k is not too far from uniform. We remark that orthant probabilities of multivariate Gaussian distributions are well-studied (see e.g., ; ;), and thus may allow for the application of this assumption to a larger class of matrices. In particular, we show it is satisfied by both random matrices and orthogonal matrices. Our second assumption is a non-degeneracy condition on the weight matrices W i -namely, that products of w T with non-empty sets of rows of the W i in entry-wise non-zero vectors. In addition, Assumption 2 requires that the network is non-zero with probability that is not arbitrarily small, otherwise we cannot hope to find even a single x with M (x) = 0. In the following lemma, we demonstrate that these conditions are satisfied by randomly initialized networks, even when the entries of the W i are not identically distributed. Lemma 3.1. If A ∈ R k×n is an arbitrary matrix with orthogonal rows, or if n > Ω(k 3) and A has entries that are drawn i.i.d. from some sub-Gaussian distribution D with expectation 0, unit variance, and constant sub-Gaussian norm 1/p then with probability at least 1 − e −k 2, A satisfies Assumption 1 with have entries that are drawn independently (and possibly non-identically) from continuous symmetric distributions, and if, then Assumption 2 holds with probability 1 − δ. The algorithm for recovery is given in Algorithm 1. Our algorithm computes the gradient ∇M (g i) for different Gaussian vectors g i ∼ N (0, I k), and returns the subspace spanned by these gradients. To implement this procedure, we must show that it is possible to compute gradients via the perturbational method (i.e. finite differences), given only oracle queries to the network M. Namely, we firstly must show that if g ∼ N (0, I n) then ∇M (g) exists, and moreover, that ∇M (x) exists for all x ∈ B (g), where B (g) is a ball of radius centered at g, and is some value with polynomial bit complexity which we can bound. To demonstrate this, we show that for any fixing of the sign patterns of the network, we can write the region of R n which satisfies this sign pattern and is -close to one of the O(dk) ReLU thresholds of the network as a linear program. We then show that the feasible polytope of this linear program is contained inside a Euclidean box in R n, which has one side of length. Using this containment, we upper bound the volume of the polytope in R n which is close to each ReLU, and union bound over all sign patterns and ReLUs to show that the probability that a Gaussian lands in one of these polytopes is exponentially small. Compute Gradient: There is an algorithm which, given g ∼ N (0, I k), with probability 1 − exp(−n c) for any constant c > 1 (over the randomness in g), computes ∇M (g) ∈ R n with O(n) queries to the network, and in poly(n) runtime. Now observe that the gradients of the network lie in the row-span of A. To see this, for a given input x ∈ R n, let S 0 (x) ∈ R k be the sign pattern of φ(Ax) ∈ R k, and more generally define, which demonstrates the claim that the gradients lie in the row-span of, and let Z be the matrix where the i-th row is equal to z i. We will prove that Z has rank at least k/2. To see this, first note that we can write Z = VA, where V is some matrix such that the non-zero entries in the i-th row are precisely the coordinates in the set S i 0, where S i j = S j (g i) for any j = 0, 1, 2,..., d and i = 1, 2,..., r. We first show that V has rank at least ck for a constant c > 0. To see this, suppose we have computed r gradients so far, and the rank of V is less than ck for some 0 < c < 1/2. Now V ∈ R r×k is a fixed rank-ck matrix, so the span of the matrix can be expressed as a linear combination of some fixed subset of ck of its rows. We use this fact to show in the following lemma that the set of all possible sign patterns obtainable in the row span of V is much smaller than 2 k. Thus a gradient z r+1 with a uniform (or nearly uniform) sign pattern will land outside this set with good probability, and thus will increase the rank of Z when appended. Lemma 3.3. Let V ∈ R r×k be a fixed at most rank ck matrix for c ≤ 1/2. Then the number of sign patterns S ⊂ [k] with at most k/2 non-zeros spanned by the rows of V is at most. In other words, the set, where φ is the ReLU, satisfies Assumptions 1 and 2. Then the algorithm given in Figure 1 makes O(kn log(k/δ)/γ) queries to M (x) and returns in poly(n, k, 1/γ, log(1/δ)) time a subspace V ⊆ span(A) of dimension at least k 2 with probability 1 − δ. In this section, we consider networks that have a threshold function at the output node, as is done often for classification. Specifically, let τ: R → {0, 1} be the threshold function: τ (x) = 1 if x ≥ 1, and τ (x) = 0 otherwise. Again, we let A ∈ R k×n where k < n, be the innermost weight matrix. The networks M: R n → R we consider are then of the form: and each φ i is a continuous, differentiable activation function applied entrywise to its input. We will demonstrate that even for such functions with a binary threshold placed at the end, giving us minimal information about the network, we can still achieve full span recovery of the weight matrix A, albeit with the cost of anapproximation to the subspace. Note that the latter fact is inherent, since the gradient of any function that is not linear in some ball around each point cannot be obtained exactly without infinitely small perturbations of the input, which we do not allow in our model. We can simplify the above notation, and write σ(x) = W 1 φ 1 (W 2 φ 2 (. . . φ d Ax))... ), and thus M (x) = τ (σ(x)). Our algorithm will involve building a subspace V ⊂ R n which is a good approximation to the span of A. At each step, we attempt to recover a new vector which is very close to a vector in Span(A), but which is nearly orthogonal to the vectors in V. Specifically, after building V, on an input x ∈ R n, we will query M for inputs M ((I n − P V)x). Recall that P V is the projection matrix onto V, and P V ⊥ is the projection matrix onto the subspace orthogonal to V. Thus, it will help here to think of the functions M, σ as being functions of x and not (I n − P V)x, and so we define For the of this section, we make the following assumptions on the activation functions. 1. The function φ i: R → R is continuous and twice differentiable, and φ i = 0. 2. φ i and φ i are L i -Lipschitz, meaning: The network is non-zero with bounded probability: for every subspace V ⊂ R n of dimension dim(V) < k, we have that Pr g∼N (0,In) [σ V (g) ≥ 1] ≥ γ for some value γ > 0. 4. Gradients are not arbitrarily small near the boundary: for every subspace V ⊂ R n of dimension dim(V) < k for some values η, γ > 0, where ∇ g σ V (cg) is the directional derivative of σ V in the direction g. The first two conditions are standard and straightforward, namely φ i is differentiable, and has bounded first and second derivatives (note that for our purposes, they need only be bounded in a ball of radius poly(n)). Since M (x) is a threshold applied to σ(x), the third condition states that it is possible to find inputs x with non-zero network evaluation M (x). Our condition is slightly stronger, in that we would like this to be possible even when x is projected away from any k < k dimensional subspace (note that this ensures that Ax is non-zero, since A has rank k). The last condition simply states that if we pick a random direction g where the network is non-zero, then the gradients of the network are not arbitrarily small along that direction at the threshold points where σ(c · g) = 1. Observe that if the gradients at such points are vanishingly small, then we cannot hope to recover them. Moreover, since M only changes value at these points, these points are the only points where information about σ can be learned. Thus, the gradients at these points are the only gradients which could possibly be learned. We note that the running time of our algorithms will be polynomial in log(1/η), and thus we can even allow the gradient size η to be exponentially small. We now formally describe and analyze our span recovery algorithm for networks with differentiable activation functions and 0/1 thresholding. Let κ i be the condition number of the i-th weight matrix W i, and let δ > 0 be a failure probability, and let > 0 be a precision parameter which will affect the how well the subspace we output will approximate Span(A). Now fix N = poly(n, k, The running time and query complexity of our algorithm will be polynomial in N . Our algorithm for approximate span recovery is given formally in Algorithm 2. Proposition 4.1. Let V ⊂ R n be a subspace of dimension k < k, and fix any 0 > 0. Then we can find a vector x with 0 ≤ σ V (x) − 1 ≤ 2 0 in expected O(1/γ + N log(1/ 0)) time. Moreover, with probability γ/2 we have that ∇ x σ V (x) > η/4 and the tighter bound of 0 η2 We will apply the above proposition as input to the following Lemma 4.2, which is the main technical of this section. Our approach involves first taking the point x from Proposition 4.1 such that σ V (x) is close but bounded away from the boundary, and generating n perturbations at this point M V (x + u i) for carefully chosen u i. While we do not know the value of σ V (x + u i), we can tell for a given scaling c > 0 if σ V (x + cu i) has crossed the boundary, since we will then have M V (x + cu i) = 0. Thus, we can estimate the directional derivative ∇ ui σ(x) by finding a value c i via a binary search such that σ V (x + c i u i) is exponentially closer to the boundary than σ V (x). In order for our estimate to be accurate, we must carefully upper and lower bound the gradients and Hessian of σ v near x, and demonstrate that the linear approximation of σ v at x is still accurate at the point x + c i u i where the boundary is crossed. Since each value of 1/c i is precisely proportional to ∇ ui σ(x) = ∇σ(x), u i, we can then set up a linear system to approximately solve for the gradient ∇σ(x) (lines 8 and 9 of Algorithm 2). Lemma 4.2. Fix any, δ > 0, and let N be defined as above. Then given any subspace V ⊂ R n with dimension dim(V) < k, and given x ∈ R n, such that 0 η2, and such that ∇ x σ V (x) > η/2, then with probability 1 − 2 −N/n 2, we can find a vector v ∈ R n in expected poly(N) time, such that P Span(A) v 2 ≥ (1 −) v 2, and such that P V v 2 ≤ v 2. Theorem 4.3. Suppose the network M (x) = τ (σ(Ax)) satisfies the conditions described at the beginning of this section. Then Algorithm 2 runs in poly(N) time, making at most poly(N) queries to M (x), where Find a scaling α > 0 via binary search on values τ (σ V (αg)) such that x = αg satisfies Generate g 1,..., g n ∼ N (0, I n), and set u i = g i 2 −N − x/ x 2. For each i ∈ [n], binary search over values c to find c i such that If any c i satisfies |c i | ≥ (10 · 2 −N 0 /η), restart from line 5 (regenerate the Gaussian g). Otherwise, define B ∈ R n×n via B *,i = u i, where B *,i is the i-th column of B. Define b ∈ R n by b i = 1/c i. Let y * be the solution to: min Set v i = y * and V ← Span(V, v i). 10 end 11 return V poly(n, k,, and returns with probability 1 − δ a subspace V ⊂ R n of dimension k such that for any x ∈ V, we have P Span(A) x 2 ≥ (1 −) x 2. Figure 1: Partial span recovery of small networks with layer sizes specified in the legend. Note that 784->80-> indicates a 4 layer neural network with hidden layer sizes 784, 80, 6, and 3, in that order. Full span recovery is not always possible and recovery deteriorates as width decreases and depth increases. When applying span recovery for a given network, we first calculate the gradients analytically via auto-differentiation at a fixed number of sample points distributed according to a standard Gaussian. Our networks are feedforward, fully-connected with ReLU units; therefore, as mentioned above, using analytic gradients is as precise as using finite differences due to piecewise linearity. Then, we compute the rank of the ing gradient matrix, where the rank is defined to be the number of singular values that are above 1e-5 of the maximum singular value. In our experiments, we attempt to recover the full span of a 784-by-80 matrix with decreasing layer sizes for varying sample complexity, as specified in the figures. For the MNIST dataset, we use a size 10 vector output and train according to the softmax cross entropy loss, but we only calculate the gradient with respect to the first output node. Our recovery algorithms are GradientsRandom (Algorithm 1), GradientsRandomAda (Algorithm 2), and GradientsM-NIST. GradientsRandom is a direct application of our first span recovery algorithm and calculates gradients via pertur- bations at random points for a random network. GradientsRandomAda uses our adaptive span recovery algorithm for a random network. Finally, GradientsMNIST is an application of GradientsRandom on a network with weights trained on MNIST data. In general, we note that the experimental outcomes are very similar among all three scenarios. Figure 3: Fooling ReLU networks into misclassifying noise as digits by introducing Gaussian noise into the null space after span recovery. The prediction of the network is presented above the images, along with its softmax probability. For networks with very small widths and multiple layers, we see that span recovery deteriorates as depth increases, supporting our theory (see Figure 1). This holds both in the case when the networks are randomly initialized with Gaussian weights or trained on a real dataset (MNIST) and whether we use adaptive or non-adaptive recovery algorithms. However, we note that these small networks have unrealistically small widths (less than 10) and when trained on MNIST, these networks fail to achieve high accuracy, all falling below 80 percent. The small width case is therefore only used to support, with empirical evidence, why our theory cannot possibly guarantee full span recovery under every network architecture. For more realistic networks with moderate or high widths, however, full span recovery seems easy and implies a real possibility for attack (see Figure 2). Although we tried a variety of widths and depths, the are robust to reasonable settings of layer sizes and depths. Therefore, we only present experimental with sub-networks of a network with layer sizes. Note that full span recovery of the first-layer weight matrix with rank 80 is achieved almost immediately in all cases, with less than 100 samples. On the real dataset MNIST, we demonstrate the utility of span recovery algorithms as an attack to fool neural networks to misclassify noisy inputs (see Figure 3). We train a ReLU network (to around 95 percent accuracy) and recover its span by computing the span of the ing gradient matrix. Then, we recover the null space of the matrix and generate random Gaussian noise projected onto the null space. We see that our attack successfully converts images into noisy versions without changing the output of the network, implying that allowing a full (or even partial) span recovery on a classification network can lead to various adversarial attacks despite not knowing the exact weights of the network. We first restate the which have had their proofs omitted, and include their proofs subsequently. Assumption 2 holds with probability 1 − δ. Proof. By Theorem 5.58 of , if the entries A are drawn i.i.d. from some sub-Gaussian isotropic distribution D over R n such that A j 2 = √ n almost surely, then, for some constants c, C > 0 depending only on D ψ2. Since the entries are i.i.d. with variance 1, it follows that the rows of A are isotropic. Moreover, we can always condition on the rows having norm exactly √ n, and pulling out a positive diagonal scaling through the first Relu of M (x), and absorbing this scaling into W d. It follows that the conditions of the theorem hold, and we have for a suitably large re scaling of the constant C. Setting n > Ω(k 3), it follows that κ(A) < (1 + 1/(100k)), which holds immediately if A has orthogonal rows. Now observe that Ag is distributed as a multi-variate Gaussian with co-variance A T A, and is therefore given by the probability density function (pdf) x T x be the pdf of an identity covariance Gaussian N (0, I k). We lower bound p (x)/p(x) for x with x 2 2 ≤ 16k. In this case, we have Thus for any sign pattern S, Pr[sign(Ag) = S: Ag −k, and spherical symmetry of Gaussians, Pr[sign(g) = S: g For the second claim, by an inductive argument, the entries in the rows i ∈ S j of the product W Sj Si is non-zero with probability 1. It follows that w, is the inner product of a non-zero vector with a vector w with continuous, independent entries, and is thus non-zero with probability 1. By a union bound over all possible non-empty sets S j, the desired follows. We now show that the second part of Assumption 2 holds. To do so, first let g ∼ N (0, I n). We demonstrate that Pr W1,W2,...,W d,g [M (x) = 0] ≤ 1 − γδ/100. Here the entries of the W i's are drawn independently but not necessarily identically from a continuous symmetric distribution. To see this, note that we can condition on the value of g, and condition at each step on the non-zero value of y i = φ(W i+1 φ(W i+2 φ(. . . φ(Ag)... ). Then, over the randomness of W i, note that the inner product of a row of W i and y i is strictly positive with probability at least 1/2, and so each coordinate of W i y i is strictly positive independently with probability ≥ 1/2. It follows that φ(W i y i) is non-zero with probability at least 1 − 2 −ki. Thus where the second inequality is by assumption. It follows by our first part that So by Markov's inequality, Thus with probability 1 − δ over the choice of Lemma 3.2 There is an algorithm which, given g ∼ N (0, I k), with probability 1 − exp(−n c) for any constant c > 1 (over the randomness in g), computes ∇M (g) ∈ R n with O(n) queries to the network, and in poly(n) running time. where φ is the ReLU. If ∇M (g) exists, there is an > 0 such that M is differentiable on B (g). We show that with good probability, if g ∼ N (0, I n) (or in fact, almost any continuous distribution), then M (g) is continuous in the ball B (g) = {x ∈ R n | x − g 2 <} for some which we will now compute. First, we can condition on the event that g 2 2 ≤ (nd) 10c, which occurs with probability at least 1 − exp(−(nd) 5c ) by concentration for χ 2 distributions. Now, fix any sign pattern, and let S = (S 1, S 2, . . ., S d+1). We note that we can enforce the constraint that for an input x ∈ R n, the sign pattern of M i (x) is precisely S i. To see this, note that after conditioning on a sign pattern for each layer, the entire network becomes linear. Thus each constraint that (−poly(nd) ) is a value we will later choose. Thus, we obtain a linear program with k + d i=1 k i constraints and n variables. The feasible polytope P represents the set of input points which satisfy the activation patterns S and are η-close to the discontinuity given by the j-th neuron in the i-th layer. We can now introduce the following non-linear constraint on the input that x 2 ≤ (nd) 10c. Let B = B (nd) 10c be the feasible region of this last constraint, and let P * = P ∩ B. We now bound the Lesbegue measure (volume) V (P *) of the region P. First note that V (P *) ≤ V (P), where V (P) is the region defined by the set of points which satisfy: where each coordinate of the vector y ∈ R n is a linear combination of products of the weight matrices W, ≥ i. One can see that the first two constraints for P are also constraints for P *, and the last constraint is precisely B, thus P * ⊂ P which completes the claim of the measure of the latter being larger. Now we can rotate P by the rotation which sends y → y 2 · e 1 ∈ R n without changing the volume of the feasible region. The ing region is contained in the region P given by Finally, note that P ⊂ R n is a Eucledian box with n − 1 side lengths equal to (nd) 10c and one side length of y 2 η, and thus V (P) ≤ y 2 η(nd) 10nc. Now note we can assume that the entries of the weight matrices A, W 1,..., W d are specified in polynomially many (in n) bits, as if this were not the case the output M (x) of the network would not have polynomial bit complexity, and could not even be read in poly(n) time. Equivalently, we can assume that our running time is allowed to be polynomial in the number of bits in any value of M (x), since this is the size of the input to the problem. Given this, since the coordinates of y were linear combinations of products of the coordinates of the weight matrices, and note that each of which is at most 2 n C for some constant C (since the matrices have polynomial bit-complexity), we have that P * ≤ η2 n C (nd) 10nc as needed. Now the pdf of a multi-variate Gaussian is upper bounded by 1, so 10nc. It follows that the probability that a multivariate Gaussian g ∼ N (0, I n) satisfies the sign pattern S and is η close to the boundary for the j-th neuron in the i-th layer. Now since there are at most nd possible combinations of sign patterns S, it follows that the the probability that a multivariate Gaussian g ∈ N (0, I n) is η close to the boundary for the j-th neuron in the i-th layer is at most η2 10nc 2 nd. Union bounding over each of the k i neurons in layer i, and then each of the d layers, it follows that g ∈ N (0, I n) is η close to the boundary for any discontinuity in M (x) is at most η2, it follows that with probability at least 1 − exp(−(nd) c ), the network evaluated at g ∈ N (0, I n) is at least η close from all boundaries (note that C is known to us by assumption). Now we must show that perturbing the point g by any vector with norm at most in a new point g which still has not hit one of the boundaries. Note that M (g) is linear in an open ball around g, so the change that can occur in any intermediate neuron after perturbing g by some v ∈ R n is at most where · 2 is the spectral norm. Now since each entry in the weight matrix can be specified in polynomially many bits, the Frobenius norm of each matrix (and therefore the spectral norm), is bounded by n 2 2 n C for some constant C. Thus and setting = η/β, it follows that M (x) is differentiable in the ball B (x) as needed. We now generate u 1, u 2,..., u n ∼ N (0, I n), which are linearly independent almost surely. We set v i = ui 2 ui 2. Since M (g) is a ReLU network which is differentiable on B (g), it follows that M (g) is a linear function on B (g), and moreover v i ∈ B (g) for each i ∈ [n]. Thus for any c < 1 we have. Finally, since the directional derivative is given by ∇ vi M (x) = ∇M (x), v i / v i 2, and since v 1,..., v n are linearly independent, we can set up a linear system to solve for ∇M (x) exactly in polynomial time, which completes the proof. with at most k/2 non-zeros spanned by the rows of V is at most. In other words, the set Proof. Any vector w in the span of the rows of V can be expressed as a linear combination of at most ck rows of V. So create a variable x i for each coefficient i ∈ [ck] in this linear combination, and let f j (x) be the linear function of the x i s which gives the value of the j-th coordinate of w. Then f (x) = (f 1 (x),..., f k (x)) is a k-tuple of polynomials, each in ck-variables, where each polynomial has degree 1. By Theorem 4.1 of , it follows that the number of sign patterns which contain at most k/2 non-zero entries is at most ck+k/2 ck. Setting c ≤ 1/2, this is at most, where φ is the ReLU, satisfies Assumption 1 and 2. Then the algorithm given in Figure 1 makes O(kn log(k/δ)/γ) queries to M (x) and returns in poly(n, k, 1/γ, log(1/δ))-time a subspace V ⊆ span(A) of dimension at least k 2 with probability 1 − δ. 1/γ repetitions is O(log(1/γ) √ n), and thus the expected running time reduces to the stated bound, which completes the first claim of the Proposition. For the second claim, note that ∇ gi σ V (c * g i) > 0 by construction of the binary search, and since σ V (c * g i) > 0 = 1, by Property 4 with probability γ we have that ∇ gi σ V (g i) > η. Now with probability 1 − γ/2, we have that g i 2 2 ≤ O(n log(1/γ)) (see Lemma 1), so by a union bound both of these occur with probability γ/2. Now since (c * − c)x 2 ≤ 0 2 −N (after rescaling N by a factor of log( g i 2) = O(log(n))), and since 2 N is also an upper bound on the spectral norm of the Hessian of σ by construction, it follows that ∇ gi σ V (cg i) > η/2. Now we set x ← cg i +c 0 2 −N g i /(cg i 2). First note that this increases σ V (cx)−1 by at most 0, so σ(cx)−1 ≤ 2 0, so this does not affect the first claim of the Proposition. But in addition, note that conditioned on the event in the prior paragraph, we now have that σ V (x) > 1 + η 0 2 −N. The above facts can be seen by the fact that 2 N is polynomially larger than the spectral norm of the Hessian of σ, thus perturbing x by 0 2 −N additive in the direction of x will in a positive change of at least 1 2 (η/4)(0 2 −N) in σ. Moreover, by applying a similar argument as in the last paragraph, we will have ∇ x σ V (cx) > η/4 still after this update to x. Lemma 4.2 Fix any, δ > 0, and let N = poly(n, k, where 0 = Θ(2 −N C /) for a sufficiently large constant C = O, and ∇ x σ V (x) > η/2, then with probability We first claim that the c i which achieves this value satisfies c i u i 2 ≤ (10 · 2 −N 0 /η). To see this, first note that by Proposition 4.1, we have ∇ x σ V (x) > η/4 with probability γ. We will condition on this occurring, and if it fails to occur we argue that we can detect this and regenerate x. Now conditioned on the above, we first claim that ∇ ui σ V (x) ≥ η/8, which follows from the fact that we can bound the angle between the unit vectors in the directions of u i and x by cos (angle(u i, x)) = u i u 2, x x 2 ≥ (1 − n/2 −N) > (1 − η/2 −N/2) along with the fact that we have ∇ x σ V (x) > η/4. Since |σ V (x) − 1| < 2 0 < 2 −N C, and since 2 N is an upper bound on the spectral norm of the Hessian of σ, we have that ∇ ui σ V (x + cu i) > η/8 + 2 −N > η/10 for all c < 2 −2N. In other words, if H is the hessian of σ, then perturbing x by a point with norm O(c) ≤ 2 −2N can change the value of the gradient by a vector of norm at most 2 2N H 2 ≤ 2 −N, where H 2 is the spectral norm of the Hessian. It follows that setting c = (10 · 2 −N 0 /η) is sufficient for σ V (x + cu i) < 1, which completes the above claim. Now observe that if after binary searching, the property that c ≤ (10 · 2 −N 0 /η) does not hold, then this implies that we did not have ∇σ(x) > η/4 to begin with, so we can throw away this x and repeat until this condition does hold. By Assumption 4, we must only repeat O(1/γ) times in expectation in order for the assumption to hold. Next, also note that we can bound c i ≥ 0 η2 −N /N, since 2 N again is an upper bound on the norm of the gradient of σ and we know that σ V (x) − 1 > 0 η2 −N. Altogether, we now have that |Ξ(c i u i)| ≤ c poly(n, k, log(κ i), log(1 η), log, log(1 δ)), and returns with probability 1 − δ a subspace V ⊂ R n of dimension k such that for any x ∈ V, we have Proof. We iteratively apply Lemma 4.2, each time appending the output v ∈ R n of the proposition to the subspace V ⊂ R n constructed so far. WLOG we can assume v is a unit vector by scaling it. Note that we have the property at any given point in time k < k that V = Span(v 1, . . ., v k) where each v i satisfies that P Span{v1,...,vi−1} v i 2 ≤. Note that the latter fact implies that v 1,... v k are linearly independent. Thus at the end, we recover a rank k subspace V = Span(v 1, . . ., v k), with the property that P Span(A) v i 2 2 ≥ (1 −) v i 2 2 for each i ∈ [k]. Now let V ∈ R n×n be the matrix with i-th column equal to v i. Fix any unit vector x = Va ∈ V, where a ∈ R n is uniquely determined by x. Let V = V + + V − where V + = P Span(A) V and V − = V − V + Then x = V + a + V − a, and (I n − P Span(A) )x 2 ≤ (I n − P Span(A) )V + a 2 + (I n − P Span(A) )V − a 2 First note that by the construction of the v i's, each column of V − has norm O, thus V Thus σ min (V) ≥ (1 − O), so we have (I n − P Span(A) )x 2 ≤ V − 2 1 σmin(V) ≤ 2 √ n. By the Pythagorean theorem: P Span(A) )x 2 2 = 1 − (I n − P Span(A) )x 2 2 ≥ 1 − O(n 2). Thus we can scale by a factor of Θ(1/ √ n) in the call to Lemma 4.2, which gives the desired of P Span(A) x 2 ≥ 1 −.

We provide a PAC-Bayes based generalization guarantee for uncompressed, deterministic deep networks by generalizing noise-resilience of the network on the training data to the test data.

represent the short summary for finding the corresponding scientific article (long) E.g. We propose a method of transferring knowledge between related RL tasks using visual mappings, and demonstrate its effectiveness on visual variants of the Atari Breakout game and different levels of Road Fighter, a Nintendo car driving game. == Deep Reinforcement Learning has managed to achieve state-of-the-art in learning control policies directly from raw pixels. However, despite its remarkable success, it fails to generalize, a fundamental component required in a stable Artificial Intelligence system. Using the Atari game Breakout, we demonstrate the difficulty of a trained agent in adjusting to simple modifications in the raw image, ones that a human could adapt to trivially. In transfer learning, the goal is to use the knowledge gained from the source task to make the training of the target task faster and better. We show that using various forms of fine-tuning, a common method for transfer learning, is not effective for adapting to such small visual changes. In fact, it is often easier to re-train the agent from scratch than to fine-tune a trained agent. We suggest that in some cases transfer learning can be improved by adding a dedicated component whose goal is to learn to visually map between the known domain and the new one. Concretely, we use Unaligned Generative Adversarial Networks (GANs) to create a mapping function to translate images in the target task to corresponding images in the source task. These mapping functions allow us to transform between various variations of the Breakout game, as well as between different levels of a Nintendo game, Road Fighter. We show that learning this mapping is substantially more efficient than re-training. A visualization of a trained agent playing Breakout and Road Fighter, with and without the GAN transfer, can be seen in \url{https://streamable.com/msgtm} and \url{https://streamable.com/5e2ka}. Transferring knowledge from previous occurrences to new circumstances is a fundamental human capability and is a major challenge for deep learning applications. A plausible requirement for artificial general intelligence is that a network trained on one task can reuse existing knowledge instead of learning from scratch for another task. For instance, consider the task of navigation during different hours of the day. A human that knows how to get from one point to another on daylight will quickly adjust itself to do the same task during night time, while for a machine learning system making a decision based on an input image it might be a harder task. That is because it is easier for us to make analogies between similar situations, especially in the things we see, as opposed to a robot that does not have this ability and its knowledge is based mainly on what it already saw. Deep reinforcement learning has caught the attention of researchers in the past years for its remarkable success in achieving human-level performance in a wide variety of tasks. One of the field's famous achievements was on the Atari 2600 games where an agent was trained to play video games directly from the screen pixels and information received from the game BID20. However, this approach depends on interacting with the environment a substantial number of times during training. Moreover, it struggles to generalize beyond its experience, the training process of a new task has to be performed from scratch even for a related one. Recent works have tried to overcome this inefficiency with different approaches such as, learning universal policies that can generalize between related tasks BID25, as well as other transfer approaches BID7 BID24. In this work, we first focus on the Atari game Breakout, in which the main concept is moving the paddle towards the ball in order to maximize the score of the game. We modify the game by introducing visual changes such as adding a rectangle in the middle of the image or diagonals in the . From a human perspective, it appears that making visual changes that are not significant to the game's dynamics should not influence the score of the game, a player who mastered the original game should be able to trivially adapt to such visual variants. We show that the agent fails to transfer. Furthermore, fine-tuning, the main transfer learning method used today in neural networks, also fails to adapt to the small visual change: the information learned in the source task does not benefit the learning process of the very related target task, and can even decelerate it. The algorithm behaves as if these are entirely new tasks. Our second focus is attempting to transfer agent behavior across different levels of a video game: can an agent trained on the first level of a game use this knowledge and perform adequately on subsequent levels? We explore the Nintendo game Road Fighter, a car racing game where the goal is to finish the track before the time runs out without crashing. The levels all share the same dynamics, but differ from each other visually and in aspects such as road width. Similar to the Breakout , an agent trained to play the first level fails to correctly adapt its past experience, causing the learned policy to completely fail on the new levels. To address the generalization problem, we propose a zero-shot generalization approach, in which the agent learns to transfer between related tasks by learning to visually map images from the target task back to familiar corresponding images from the source task. Such mapping is naturally achieved using Generative Adversarial Networks (GANs) BID9, one of the most popular methods for the image-to-image translation that is being used in computer vision tasks such as style transfer BID15, object transfiguration BID31, photo enhancement BID17 and more recently, video game level generation BID27. In our setup, it is not realistic to assume paired images in both domains, calling for the use of Unaligned GANs BID19 BID15. Using this approach we manage to transfer between similar tasks with no additional learning. Contributions This work presents three main contributions. First, in Section 2, we demonstrate how an agent trained with deep reinforcement learning algorithms fails to adapt to small visual changes, and that the common transfer method of fine-tuning fails as well. Second, in Section 3, we propose to separate the visual mapping from the game dynamics, ing in a new transfer learning approach for related tasks based on visual input mapping. We evaluate this approach on Breakout and Road Fighter, and present the comparing to different baselines. We show that our visual transfer approach is much more sample efficient then the alternatives. Third, in section 5, we suggest an evaluation setup for unaligned GAN architectures, based on their achieved performance on concrete down-stream tasks. Many Breakout variations can be constructed that involve the same dynamics. The main idea is to make modifications that are not critical for a human playing the game but are for the algorithm that relies on visual inputs. We demonstrate the difficulty of deep reinforcement learning to generalize using 4 types of modifications as presented in FIG0. For all the experiments in this section forward we use the Asynchronous Advantage Actor-Critic (A3C) algorithm BID21, taking advantage of being faster than Deep Q-Network (DQN) BID20. The A3C learns the policy and the state-value function using parallel actor-learners exploring different policies for the acceleration and stability of the training. We rescale the image to 80 × 80 and keep the RGB colors for more realistic images. We use 32 actor learners, a discount rate of 0.99, learning rate of 0.0001, 20-step returns, and entropy regularization weight of 0.01. The A3C variation we choose is the LSTM-A3C network. We use the standard high-performance architecture implemented in BID16. The setup mentioned in 2.1 successfully trains on Breakout, reaching a score of over 400 points. However, when a network trained on the original game is presented with the game variants, it fails completely, reaching to a maximum score of only 3 points. This shows that the network does not necessarily learn the game's concepts and heavily relies on the images it receives. The common approach for transferring knowledge across tasks is fine-tuning. We experiment with common techniques used in deep learning models. In each setting, we have a combination of frozen and fine-tuned layers (Partial/Full) as well as layers that are initialized with the target's parameters and layers that are initialized with random values (Random). Our settings are inspired by BID29. We train each one of the tasks (before and after the transformation) for 60 million frames, and our evaluation metric is the total reward the agents collect in an episode averaged by the number of episodes, where an episode ends when the game is terminated or when a number of maximum steps is reached. We periodically compute the average during training. We consider the following settings:• From-Scratch: The game is being trained from scratch on the target game.• Full-FT: All of the layers are initialized with the weights of the source task and are fine-tuned on the target task.• Random-Output: The convolutional layers and the LSTM layer are initialized with the weights of the source task and are fine-tuned on the target task. The output layers are initialized randomly.• Partial-FT: All of the layers are initialized with the weights of the source task. The three first convolutional layers are kept frozen, and the rest are fine-tuned on the target task.• Partial-Random-FT: The three first convolutional layers are initialized with the weights of the source task and are kept frozen, and the rest are initialized randomly. The presented in FIG1 show a complete failure of all the fine-tuning approaches to transfer to the target tasks. In the best scenarios the transfer takes just as many epochs as training from scratch, while in other cases starting from a trained network makes it harder for the network to learn the target task. As the graphs show, some of the modification influenced more than others. For example, FIG1 shows that adding a simple rectangle can be destructive for a trained agent: while training from scratch consistently and reliably achieves scores over 300, the settings starting from a trained agent struggle to pass the 200 points mark within the same number of iterations, and have a very high variance. We noticed that during training the agent learns a strategy to maximize the score with a minimum number of actions. None of the experiments we performed showed better when the layers in the network were fine-tuned, and some showed negative transfer which is a clear indication of an overfitting problem. The A3C model learned the detail and noise in the training data to the extent that it negatively impacted the performance of the model on new data. Our and drawn from them are consistent with the shown when a similar approach was used on Pong BID24. In addition to Breakout, we also experimented transfer between the first and advanced level of Road Fighter, where the s change but the dynamics remains the same. This experiments ed with 0 points on each of the levels, a complete failure of the agent to re-use the driving techniques learned on the first levels on the next ones. An agent capable of performing a task in a source domain is now presented with a new domain. Fine-tuning the agent on the target domain fails to transfer knowledge from the source domain. We propose to separate the visual transfer from the dynamics transfer. To perform well, the agent can try and make analogies from the new domain to the old one: after observing a set of states (images) in the new domain, the agent can learn to map them to similar, familiar states from the source domain, and act according to its source domain policy on the mapped state. More concretely, given a trained policy π(a|s; θ) with trained parameters θ proposing an action a for source domain states s ∈ S, we wish to learn a mapping function G: T → S from target domain states t ∈ T such that interacting with the environment T by applying the policy π(a|G(t); θ) will in a good distribution of actions for the states T, as indicated by high overall scores. In other words, we seek a mapping function G that allows us to re-use the same policy π θ learned for source environment S when interacting with the target environment T.As both the source and target domain items are images, we heuristically learn the function G by collecting sets of images from S and T and learning to visually map between them using Unaligned GAN BID19 BID15. We use the scores obtained from interacting with the environment via π(a|G(t); θ) for the GAN model selection and stopping criteria. In this work, we focus on learning setups that receive only raw image data, without additional domain knowledge about objects or game dynamics. This prohibits us from using supervised paired GANs BID13 for learning the mapping function G: we cannot collect the needed supervision of corresponding (s, t) pairs. Instead, we use unaligned GANs BID19 BID15, in which the learner observes two sets of images, one from each domain, with the goal of learning to translate images in one domain to images in another. All major approaches to the unaligned image-to-image translation use the Cycle-Consistency principle. We have two mapping (encoding) functions DISPLAYFORM0 is a set of images collected from the source task and T = {t j} M j=1 is a set of images collected from the target task. The goal is to generate an image s, for any given t ∈ T where G 1 (t) = s, that is indistinguishable from s ∈ S. The cycle consistency principle relies on the assumption that the two functions, G 1 and G 2 are inverses of each other. It encourages unsupervised mapping by forcing G 2 (G 1 (t)) = t and G 1 (G 2 (s)) = s where s and t are the input images. The second component of the GAN architecture are the discriminators D 1 and D 2 aiming to distinguish between images generated by G 1 and G 2 and the real images from the target and source distributions respectively. In the following experiments, we use the UNIT framework BID19, which we found to perform well for the Breakout tasks (in section 5 we explicitly compare the UNIT and CycleGAN approaches on both the Breakout and Road Fighter transfer tasks). A distinguishing element in the UNIT framework is the shared-latent space assumption, according to which there is a shared-latent space consisting a shared latent code z for any pair of images s and t that can be recovered from this code. This share-latent space is represented as the weights of the last few layers of the encoding network and the few first layers of the decoding networks, and is learned by using Variational Autoencoders (VAEs). This sharing strongly ties the images in the source and target domain to each other, encouraging mappings that preserve similarities across domains. In contrast, the CycleGAN architecture does not make the shared space assumption and instead the generators are trained independently with two separate networks. For further information of the unaligned GAN architectures, see the original papers. Datasets. The Unaligned GAN training dataset requires images from both domains. We collect images from the source domain by running an untrained agent and collecting the observed images, and we do similarly for the target domain. The number of collected images should balance between two objectives: On the one hand, we want to take a small number of images, and on the other hand, it is essential for us to have a diverse dataset. We repeat this procedure for every target task, and create a source-target dataset for each. During training, we further ensure the images pairs are not aligned by randomly picking an image from each set at each iteration. Setup and Analysis. For our experiments we use the same architecture and hyper-parameters proposed in the UNIT paper. We initialize the weights with Xavier initialization BID8, set the batch size to 1 and train the network for a different number of iterations on each task. Some tasks are harder than others, the more changes exist in the frames the harder it is for the GAN to learn the mapping between the domains. However, our evaluation metric, testing the agent with the generated images, is a clear indication of how hard each task is and the number of iterations needed is based on the of this evaluation. Evaluation. We use GAN training to learn a mapping function G. GAN training, and unaligned GANs training in particular, are unstable and it challenging to find a good loss-based stopping criteria for them. A major issue with GANs is the lack of an evaluation metric that works well for all models and architectures, and which can assist in model selection. Different works use different methods that were suitable for their types of data. Our setup suggests a natural evaluation criteria: we run the source agent without any further training while using the model to translate each image of the target task back to the source task and collect the rewards the agent receives during the game when presented with the translated image. We use the total accumulated rewards (the score) the agent collects during the game as our model accuracy. We examine how well the agent does when receiving translated frames generated by the generator trained with GANs. Since we initialized the layers with the values of the trained network, we assume that the success of the agent is dependent on the similarity between the generated and the source task's frames. We first test our approach on Breakout, evaluating its ability to remove the changes added in the images. Second, we challenge our method even more on Road Fighter, where the goal is to transfer between different environments. Our goal in Breakout is removing the modifications in each one of the target tasks and transfer between each to the original game. Although the variations share many similarities, some tasks were more challenging than others, e.g., the lines of the Green Lines variation hide parts of the ball in Table 1: The score and number of frames needed for it of: the source task (Source), target task when initialized with the source task' network parameters with no additional training (Target) and the target task when initialized with the source task' network parameters where every frame is translated to a frame from the source task (Target with GANs). Target Task Target Target DISPLAYFORM0 Figure 3: Illustration of a frame taken from the target task (left) and its matching frame of the source task generated with GANs (right) for each one of the Breakout variations. (a)-(d) demonstrate successes, while (e) and (f) show failure modes of the unaligned GAN. In (e) the ball in the input image is not generated in the output and in (f) not all bricks are generated, and some of the generated bricks appear smudged. some frames. On the opposite side, the Rectangle variation requires less training since the number of pixels changed in the image is small. During testing, we encountered problems with the images generation that we did not observe during the GAN training. The translation task we attempted to perform was supposedly simple -search for the differences between the domains shared by all images, change them to the way they are in the opposite domain and leave everything else the same. Unfortunately, since the network does not have any prior information about objects in the image, it struggles to generate them even if they were not changed. The most common problem we had was that the generator generated bricks that were supposed to be removed from the game, and in some cases, they were very noisy in the generated image (Fig. 3f). Another problem was the ball generation and more specifically, the location of the generated ball. Since the ball is small and changes its position often, it was hard for the generator, trained with unaligned pairs, to decide if and where to locate it in the generated image (Fig. 3e). These issues and others eventually caused the agent to fail in following the policies it learned on the source task. We found that more training leads to better for some of the variations and so the number of iterations needed was different for each variation. In Table 1 we show the of a test game played by the agent with and without GANs. We stop the training after reaching 300 points, which we consider to be a high score. As the show, the source game trained from scratch requires ten of millions of images to achieve such score comparing to the target task trained with GANs that only needs a few hundreds of thousands-a 100x fold increase in sample efficiency. Moreover, the frames the GAN was trained on were limited to the first games in which the A3C network was not trained, and yet it managed to generalize to more advanced stages of the game. While the Breakout variants work well to demonstrate transfer failure cases, they can be considered as "toy examples". We proceed to demonstrate the effectiveness of our transfer method on a "real" task: learning to transfer between different levels of the Road Fighter game. Road Fighter contains 4 different levels FIG2, each with a different where some are more difficult than others. The levels mostly differ visually and all have the same rules and require the same driving techniques. Thus, we believe that these techniques should sustain when playing a new level. We start by training an RL agent to play the first level of the game. To maximize the score, the agent has to acquire 3 main capabilities: driving fast, avoiding collision with obstacles, and if a car collision occurs reacting fast to avoid crashing. We use the A2C algorithm, the synchronous version of the Advantage Actor-Critic which performs better on this game than A3C, reaching over 10, 000 game points on level 1.For the GAN training, we collect 100k 84x84 frames from each of levels 2, 3 and 4 by running an untrained agent repeatedly until we have collected sufficient samples (training an RL agent to reach a score of 10,000 points on Level 1 required observing above 100M frames.). Using the collected images we train a mapping function for each task to map the new level (target task) to the first one (source task). We use the same GAN architecture used for Breakout, but initialize the weights with Orthogonal initialization. Compared to Breakout, these tasks introduce new challenges: rather than removing a mostly static element, the GAN has to be able to change the and road size while keeping the cars in the right position on the road. On the other hand, this setup may be closer to the one unaligned GANs are usually applied in. We restrict ourselves to collecting images from the beginning of the game, before the agent had any training. This restricts the phenomena the GAN can observe, and some target tasks' images do not have a clear corresponding situation in the first level, potentially causing unpredictable behaviors. For example, the generator matches the diagonal shaped roads to one of the first rare and unique images of level 1 (Fig. 5e).Our experiments presented in TAB1 demonstrate how an agent trained to master the first level of the game fails to follow the optimal policies on new levels, reaching 0 points. However, with the GAN-based visual analogies the agent is able to apply some of the abilities it gained when training on the first level, most notably driving fast, staying on the road, avoiding some cars, and, most importantly, recovering from car crashes. The ing agent achieves impressive scores on levels 2, 3 and 4 (5350, 5350 and 2050 points, respectively), with no additional RL training and while observing only a fraction of the frames required for training a corresponding RL agent from scratch for these levels. Limitations. While the GAN works well in generating objects it has seen before, such as the agent's car, it does have limitations on objects it has not seen. As a , it ends up generating differently colored cars all in red, or not generating them at all, as shown in Fig. 5a, 5d and 5f. Colorful cars can be "collected" by the agent and are worth 1000 points each. Generating them in red makes the agent avoid them, losing these extra points and achieving overall lower scores even if finishing the track. When cars are not fully generated, the agent is less likely to avoid them, and eventually crashes. Data Efficiency. We measure the number of frames of game-interaction needed for the analogytransfer method. We collect 100k frames, and then train the GAN for up to 500k iterations, evaluating it every 10, 000 iterations by running the game and observing the score, and pick the best scoring model. This amounts to 100k + 50 * F frames, where F = 3000 is roughly the average number of frames in a game. This amounts to about 250k frames of game interaction for each transfered level, Figure 5: Left: the original frame. Right: GAN generated. Upper row shows the success cases of the GAN while the lower row shows representative failures: in (d) and (f) the only object generated on the road is the player's car and in (e) the diagonal shaped road of level 2 in matched to the starting point of level 1.an order of magnitude fewer interaction frames than training an RL agent to achieve a comparable score. Discussion. The approach is successfully transferring to most tasks, achieving the best scores on levels 2 and 3. In general, as the game progresses, the more obstacles are presented making it harder to play. One of our best is achieved on level 2 where the road is identical to level 1's, reaching the best score after 320k GAN iterations. On level 3, the challenge increases as the road is very narrow, making it harder for a player to avoid crashing. However, the GAN manages to generate the road in the right shape in most frames and position the cars in the matching ratio. Moreover, despite the challenges, due to the agent's ability to avoid crashing when colliding with cars it gets over 5000 points after 450k GAN iterations. In level 4 we get the maximum after 270k iterations. This level is also the most challenging one to play, which might be the reason for the score being the lowest out of all tasks. The main obstacles are the sudden turns in the road causing the car to be very close to the sideways and the increasing dangers a player has to avoid. Theses difficulties make this level much harder than level 1 and might require more training even from a human player. Overall, the agent performs quite well by successfully applying 2 out of 3 main capabilities it gained during training. It is missing the third capability, avoiding collisions and collecting bonus cars, mainly because of bad generation. We believe that these tasks and demonstrate a success of the analogy transfer method for zero-short generalization across different levels of a video game. They also suggest a potential of performing well on additional real world tasks in which visual analogies can be made. Evaluating GAN models and comparing them to each other is a major challenge: working with images and without well-defined objectives, testing the quality of the generator is delegated to human judgment, often using crowdsourcing to evaluate the generated images BID13; BID5. This metric is not stable and can be unreliable due to changing human factors. Others use a linear classifier to evaluate the image representations the GAN learned on supervised datasets, e.g., MNIST and CIFAR-10 . These approaches and others may be good enough for a limited group of test images but do not necessarily reflect the performance of a model in the real world. We note that while unaligned GANs are known to achieve impressive with only few training examples, our seemingly trivial translation cases proved to be quite challenging for the unaligned GAN architectures: producing valid translations that work in the game require training on a substantial number of images. During training, the GAN sees images from the early stages of the game where only a few bricks are missing in Breakout and there are no obstacles in Road Fighter. Our task requires it to generalize by translating images where objects are removed, added or in a different locations. This requires levels of generalization that may not be reflected in existing test-sets. We argue that using simple, well defined yet diverse test cases, situated in the context of a concrete down-stream task like we do here, is an important step forward in the evaluation of GAN models. We propose to evaluate GAN by running a game in which the trained Deep RL network that learned policies based on images is now receiving images from the same domain generated with GAN, and equate a successful GAN model with one ing in a high game score. We use the approach to compare Cycle-GAN and UNIT-GAN BID19.We examine both methods by running the RL agent with each every 1000 GAN training iterations and considering the maximum score after 500k iterations. We present the in TAB2. The UNIT GAN performs better than Cycle-GAN in most Breakout tasks, while Cycle-GAN outperforms UNIT in most Road Fighter tasks while requiring fewer iterations. The main difference between the two methods is the weight-sharing constraint applied in UNIT, making the domains dependent on each other by sharing and updating the weights of one or several decoders and encoders layers. We hypothesize this constraint is an advantage in tasks where the representation of the images in the different domains are similar. Thus, in Breakout, where most pixels are identical in the source and target images, Cycle-GAN often fails where UNIT succeed. However, in tasks such as Road Fighter's where most pixels are different, the agent could benefit from architectures such as Cycle-GAN where the two domains are independent of each other. Transfer Learning (TL) is a machine learning technique used to improve the training speed of a target task with knowledge learned in a source task. Pretraining and fine-tuning was proposed in BID11 and applied to TL in BID0 and BID4. In this procedure, the approach is to train the base network and then copy its first n layers to the first n layers of a target network. One can choose to update the feature layers transferred to the new task with the error backpropagated from its output, or they can be left frozen, meaning that they do not change during training on the new task. Unfortunately, as we have shown, while fine-tuning might have the ability to accelerate the training process is some cases, it can also have a damaging impact on others. Generalization is a key element in training deep learning models with time or data size constraints. Recent discussions on overfitting in Deep RL algorithms BID30 encouraged better evaluation (e.g. OpenAI Retro Contest 1) and generalization methods. In Atari, there are many similarities between the goals and the mechanism of the games. For this reason, there have been many works attempting to transfer between games or between different variations of the same game, one approach trying to do both is the progressive networks BID24. A progressive network is constructed by successively training copies of A3C on each task of interest. In that work, they transferred between different games as well as from different variations of the game Pong. The drawback of this approach is the number of parameters growing quadratically with the number of tasks. However, even if this growth rate was improved, different tasks may require different adjustments and the predefinition of the number of layers and network representation is preventing it. Zero-shot generalization is a discussed and researched topic nowadays. One work transferring between modified versions of the same game using zero-shot transfer is the schema networks BID14. Like us, they also chose to demonstrate their method on the game Breakout, using Object Oriented Markov Decision Process. In contrast, we do not use the representation of the objects in the game, and we wish to preserve the accomplishments of DQN and transfer using only raw data. Other attempted to achieve robust policies using learned disentangled representation of the image BID10, analogies between sets of instructions BID22, interactive replay BID3 while training and learn general policies by training on multiple tasks in parallel BID6 BID26.Finally, the idea of using GANs for transfer learning and domain adaptation was explored for supervised image classification and robotics applications by several authors BID1 BID12 BID18 BID2. In these methods, there is supervised source domain data (x Our RL-based setup is different: first, our coverage of target-domain data is very limited (we can only observe states which are reachable by the un-adapted or untrained agent). Second, we do not have access to supervised gold labels on the source domain, but only to a learned policy network. Third, interactions with the game environment provide very indirect rewards, so using this reward signal to influence the GAN training will be very inefficient. We thus opt for a different strategy: rather than mapping the source to the target domain and training on the projected signal, which is unrealistic an costly in the RL setup, we instead take a pre-trained source model and train an unaligned GAN to map from the target domain back to the source domain, in order to re-use the source model's knowledge and apply it to the target domain data. We believe this form of usage of GAN for transfer learning is novel. We demonstrated the lack of generalization by looking at artificially constructed visual variants of a game (Breakout), and different levels of a game (Road Fighter). We further show that transfer learning by fine-tuning fails. The policies learned using model-free RL algorithms on the original game are not directly transferred to the modified games even when the changes are irrelevant to the game's dynamics. We present a new approach for transfer learning between related RL environments using GANs without the need for any additional training of the RL agent, and while requiring orders of magnitude less interactions with the environment. We further suggest this setup as a way to evaluate GAN architectures by observing their behavior on concrete tasks, revealing differences between the Cycle-GAN and UNIT-GAN architectures. We believe our approach is applicable to cases involving both direct and less direct mapping between environments, as long as an image-to-image translation exist. While we report a success in analogy transfer using Unaligned GANs, we also encountered limitations in the generation process that made it difficult for the agent to maximize the on the Road Fighter's tasks. In future work, we plan to explore a tighter integration between the analogy transfer method and the RL training process, to facilitate better performance where dynamic adjustments are needed in addition to the visual mapping.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.