Question: represent the scientific article to find a short tldr

Question: Despite their popularity and successes, deep neural networks are poorly understood theoretically and treated as'black box' systems. Using a functional view of these networks gives us a useful new lens with which to understand them. This allows us us to theoretically or experimentally probe properties of these networks, including the effect of standard initializations, the value of depth, the underlying loss surface, and the origins of generalization. One key is that generalization from smoothness of the functional approximation, combined with a flat initial approximation. This smoothness increases with number of units, explaining why massively overparamaterized networks continue to generalize well. Deep neural networks, trained via gradient descent, have revolutionized the field of machine learning. Despite their widespread adoption, theoretical understanding of fundamental properties of deep learning -the true value of depth, the root cause of implicit regularization, and the seemingly'unreasonable' generalization achieved by overparameterized networks -remains mysterious. Empirically, it is known that depth is critical to the success of deep learning. Theoretically, it has been proven that maximum expressivity grows exponentially with depth, with a smaller number of trainable parameters . This theoretical capacity may not be used, as recently shown explicitly by . Instead, the number of regions within a trained network is proportional to the total number of hidden units, regardless of depth. Clearly deep networks perform better, but what is the value of depth if not in increasing expressivity? Another major factor leading to the success and widespread adoption of deep learning has been its surprisingly high generalization performance . In contrast to other machine learning techniques, continuing to add parameters to a deep network (beyond zero training loss) tends to improve generalization performance. This is even for networks that are massively overparameterized, wherein according to traditional ML theory they should (over)fit all the training data . How does training deep networks with excess capacity lead to generalization? And how can it be that this generalization error decreases with overparameterization? We believe that taking a functional view allows us a new, useful lens with which to explore and understand these issues. In particular, we focus on shallow and deep fully connected univariate ReLU networks, whose parameters will always in a Continuous Piecewise Linear (CPWL) approximation to the target function. We provide theoretical for shallow networks, with experiments showing that these qualitative hold in deeper nets. Our approach is related to previous work from (; ;) in that we wish to characterize parameterization and generalization. We differ from these other works by using small widths, rather than massively overparamaterized or infinite, and by using a functional parameterization to measure properties such as smoothness. Other prior works such as (; ;) attempt to provide theoretical upper or lower bounds to the number of induced pieces in ReLU networks, whereas we are more interested in the empirical number of pieces in example tasks. Interestingly, also takes a functional view, but is not interested in training and generalization as we are. Previous work has hinted at the importance of small norm initialization, but the functional perspective allows us to prove generalization properties in shallow networks. The main contribution of this work are as follows: -Functional Perspective of Initialization: Increasingly Flat with Depth. In the functional perspective, neural network parameters determine the locations of breakpoints and their delta-slopes (defined in Section 2.1) in the CPWL reparameterization. We prove that, for common initializations, these distributions are mean 0 with low standard deviation. The delta-slope distribution becomes increasingly concentrated as the depth of the network increases, leading to flatter approximations. In contrast, the breakpoint distribution grows wider, allowing deeper network to better approximate over a broader range of inputs. -Value of Depth: Optimization, not Expressivity. Theoretically, depth adds an exponential amount of expressivity. Empirically, this is not true in trained deep networks. We find that expressivity scales with the number of total units, and weakly if at all with depth. However, we find that depth makes it easier for GD to optimize the ing network, allowing for a greater flexibility in the movement of breakpoints, as well as the number of breakpoints induced during training. -Generalization is due to Flat Initialization in the Overparameterized Regime. We find that generalization in overparametrized FC ReLu nets is due to three factors: (i) the very flat initialization, (ii) the curvature-based parametrization of the approximating function (breakpoints and deltaslopes) and (iii) the role of gradient descent (GD) in preserving (i) and regularizing via (ii). In particular, the global, rather than local, impact of breakpoints and delta-slopes helps regularize the approximating function in the large gaps between training data, ing in their smoothness. Due to these nonlocal effects, more overparameterization leads to smoother approximations (all else equal), and thus typically better generalization (; . Consider a fully connected ReLU neural netf θ (x) with a single hidden layer of width H, scalar input x ∈ R and scalar output y ∈ R.f (·; θ) is continuous piecewise linear function (CPWL) since the ReLU nonlinearity is CPWL. We want to understand the function implemented by this neural net, and so we ask: How do the CPWL parameters relate to the NN parameters? We answer this by transforming from the NN parametrization (weights and biases) to two CPWL parametrizations: where the Iversen bracket b is 1 when the condition b is true, and 0 otherwise. Here the NN parameters denote the input weight, bias, and output weight of neuron i, and (·) + max{0, ·} denotes the ReLU function. The first CPWL parametrization is, where β i − bi wi is (the x-coordinate of) the breakpoint (or knot) induced by neuron i, µ i w i v i is the delta-slope contribution of neuron i, and s i sgn w i ∈ {±1} is the orientation of β i (left for s i = −1, right for s i = +1). Intuitively, in a good fit the breakpoints β i will congregate in areas of high curvature in the ground truth function |f (x)| ≥ 0, while deltaslopes µ i will actually implement the needed curvature by changing the slope by µ i from one piece p(i) to the next p(i) + 1. As the number of pieces grows, the approximation will improve, and the delta-slopes (scaled by the piece lengths) approach the true curvature of f: We note that the BDSO parametrization of a ReLU NN is closely related to but different than a traditional roughness-minimizing m-th order spline parametrizationf spline (x) BDSO (i) lacks the base polynomial, and (ii) it has two possible breakpoint orientations s i ∈ {±1} whereas the spline only has one. We note in passing that adding in the base polynomial (for linear case m = 1) into the BDSO ReLU parametrization yields a ReLU ResNet parametrization. We believe this is a novel viewpoint that may shed more light on the origin of the effectiveness of ResNets, but we leave it for future work. The second parametrization is the canonical one for PWL functions: <... < β P is the sorted list of (the x-coordinates of) the P H + 1 breakpoints (or knots), m p, γ p are the slope and y-intercept of piece p. Computing the analogous reparametrization to function space for deep networks is more involved, so we present a basic overview here, and a more detailed treatment in Appendix B. For L ≥ 2 layers with widths H , the neural network's activations are defined as: z for all hidden layers ∈ {1, 2, . . ., L} and for all neurons i is a breakpoint induced by neuron i in layer if it is a zero-crossing of the net input i.e. z Considering these parameterizations (especially the BDSO parameterization) provides a new, useful lens with which to analyze neural nets, enabling us to reason more easily and transparently about the initialization, loss surface, and training dynamics. The benefits of this approach derive from two main properties: that we have'modded out' the degeneracies in the NN parameterization and the loss depends on the NN parameters θ N N only through the BDSO parameters (the approximating function) θ BDSO i.e. (θ N N) = (θ BDSO (θ N N)), analogous to the concept of a minimum sufficient statistic in exponential family models. Much recent related work has also veered in this direction, analyzing function space . We now study the random initializations commonly used in deep learning in function space. These include the independent Gaussian initialization, with, and independent Uniform initialization, with We find that common initializations in flat functions, becoming flatter with increasing depth. Theorem 1. Consider a fully connected ReLU neural net with scalar input and output, and a single hidden layer of width H. Let the weights and biases be initialized randomly according to a zero-mean Gaussian or Uniform distribution. Then the induced distributions of the function space parameters (breakpoints β, delta-slopes µ) are as follows: (a) Under an independent Gaussian initialization, Using this , we can immediately derive marginal and conditional distributions for the breakpoints and delta-slopes. Corollary 1. Consider the same setting as Theorem 1. (a) In the case of an independent Gaussian initialization, where G nm pq (·|·) is the Meijer G-function and K ν (·) is the modified Bessel function of the second kind. (b) In the case of an independent Uniform initialization, where Tri(·; a) is the symmetric triangular distribution with base [−a, a] and mode 0. Implications. Corollary 1 implies that the breakpoint density drops quickly away from the origin for common initializations. If f has significant curvature far from the origin, then it may be far more difficult to fit. We show that this is indeed the case by training a shallow ReLU NN with an initialization that does not match the underlying curvature, with training becoming easier if the initial breakpoint distribution better matches the function curvature. We also show that during training, breakpoint distributions move to better match the underlying function curvature, and that this effect increases with depth (see Section 3, Table 1, and Appendix A.6). This implies that a data-dependent initialization, with a breakpoint distribution near areas of high curvature, could potentially be faster and easier to train. Next, we consider the typical Gaussian He or Glorot (Glorot & Bengio) initializations. In the He initialization, we have σ w = √ 2, σ v = 2/H. In the Glorot initalization, we have σ w = σ v = 2/(H + 1). We wish to consider their effect on the smoothness of the initial function approximation. From here on, we measure the smoothness using a roughness metric, defined as ρ i µ 2 i, where lower roughness indicates a smoother approximation. Theorem 2. Consider the initial roughness ρ 0 under a Gaussian initialization. In the He initialization, we have that the tail probability is given by, where E[ρ 0] = 4. In the Glorot initialization, we have that the tail probability is given by Thus, as the width H increases, the distribution of the roughness of the initial functionf 0 gets tighter around its mean. In the case of the He initialization, this mean is constant; in the Glorot initialization, it decreases with H. In either case, for reasonable widths, the initial roughness is small with high probability. This smoothness has implications for the implicit regularization/generalization phenomenon observed in recent work (see Section 3 for generalization/smoothness analysis during training). Work. Several recent works analyze the random initialization in deep networks. However, there are two main differences, First, they focus on the infinite width case (; ;) and can thus use the Central Limit Theorem (CLT), whereas we focus on finite width case and cannot use the CLT, thus requiring nontrivial mathematical machinery (see Supplement for detailed proofs). Second, they focus on the activations as a function of input whereas we also compute the joint densities of the BDSO parameters i.e. breakpoints and deltaslopes. The latter is particularly important for understanding the non-uniform density of breakpoints away from the origin as noted above. We now consider the mean squared error (MSE) loss as a function of either the NN parameters such that the restriction off BDSO to any piece of this partition, denotedf (·; θ BDSO)| πp, is a linear function. An open question is how many such critical points exist. A starting point is to consider that there are C(N +H, H) (N +H)!/N! H! possible partitions of the data. Not every such partition will admit a piecewise-OLS solution which is also continuous, and it is difficult to analytically characterize such solutions, so we resort to simulation and find a lower bound that suggests the number of critical points grows at least polynomially in N and H (Figure 7). Using Theorem 3, we can characterize growth of global minima in the overparameterized case. Call a partition Π lonely if each piece π p contains at most one datapoint. Then, we can prove the following : Theorem 4. For any lonely partition Π, there are infinitely many parameter settings θ BDSO that induce Π and are global minima with˜ (θ BDSO) = 0. Proof. Note that each linear piece p has two degrees of freedom (slope and intercept). By way of induction, start at (say) the left-most piece. If there is a datapoint in this piece, choose an arbitrary slope and intercept that goes through it; otherwise, choose an arbitrary slope and intercept. At each subsequent piece, we can use one degree of freedom to ensure continuity with the previous piece, and use one degree of freedom to match the data (if there is any). Remark 1. Suppose that the H breakpoints are uniformly spaced and that the N data points are uniformly distributed within the region of breakpoints. Then in the overparametrized regime H ≥ αN 2 for some constant α > 1, the induced partition Π is lonely with high probabilility 1 − e −N 2 /(H+1) = 1 − e −1/α. Furthermore, the total number of lonely partitions, and thus a lower bound on the total number of global minima of˜ is Thus, with only order N 2 units, we can almost guarantee lonely partitions, where the piecewise OLS solution on these lonely paratitions will be the global optimum. Note how simple and transparent the function space explanation is for why overparametrization makes optimization easy, as compared to the weight space explanation , requiring order N 7 units. The above sections argue that overparameterization leads to a flatter initial function approximation, and an easier time reaching a global minima over the training data. However, neural networks also exhibit unreasonably high generalization performance, which must be due to implicit regularization, since the effect is independent of loss function. Here we provide an argument that overparameterization directly leads to this implicit regularization, due to the increasing flatness of the initialization and the non-locality of the delta-slope parameters. Consider a dataset like that shown in Figure 8 with a data gap between regions of two continuous functions f L, f R and consider a breakpoint i with orientation s i in the gap. Starting with a flat initialization, the dynamics of the i-th delta-slope areμ where r 2,s (t), r 3,s (t) are the (negative) net correlation and residual on the active side of i, in this case including data from the function f si but not f −si. Note that the both terms of the gradientμ i have a weak dependence on i through the orientation s i, and the second term additionally depends on i through β i (t). Thus the vector of delta-slopes with orientation s evolves according toμ s = r 2,s (t)1 + r 3,s (t)β s. Now consider the regime of overparametrization H N. It will turn out to be identical to taking a continuum limit f (x, t), the curvature of the approximation (the discrete index i has become a continuous index x) andβ i (t) → 0 (following from Theorem 5, multiplyingβ i (t) by v i (t)/w i (t) and factoring out µ i (t) → 0). Integrating the dynamicsμ s (x, t) = r 2,s (t) + r 3,s (t)x over all time yields µ s (x, t = ∞) = µ s (x, t = 0) + R * 2,s + R * 3,s x, where the curvature µ s (x, t = 0) ≈ 0 (Section 3) and R * j,s ∞ 0 dt r j,s (t) < ∞ (convergence of residuals n (t) and immobility of breakpointsβ i (t) = 0 implies convergence of r j,s (t)). Integrating over space twice(from x = ξ s to x = x) yields a cubic splinef (x, t) = c 0,s + c 1, where c 0,s, c 1,s are integration constants determined by the per-piece boundary conditions (PBCs), thus matching the 0-th and 1st derivatives at the gap endpoints. The other two coefficients c k,s R * k,s, k ∈ {2, 3} and serve to match the 2nd and 3rd derivatives at the gap endpoints. Clearly, matching the training data only requires the two parameters c 0,s, c 1,s; and yet, surprisingly, two unexpected parameters c 2,s, c 3,s emerge that endowf with smoothness in the data gap, despite the loss function not possessing any explicit regularization term. Tracing back to find the origin of these smoothness-inducing terms, we see that they emerge as a consequence of (i) the smoothness of the initial function and (ii) the active half space structure, which in turn arises due to the discrete curvature-based (delta-slope) parameterization. Stepping back, the ReLU net parameterization is a discretization of this underlying continuous 2nd-order ordinary differential equation. In Section 3 we conduct experiments to test this theory. Breaking Bad: Breakpoint densities that are mismatched to function curvature makes optimization difficult We first test our initialization theory against real networks. We initialize fullyconnected ReLU networks of varying depths, according to the popular He initializations . Figure 1 shows experimentally measured densities of breakpoints and delta-slopes. Our theory matches the experiments well. The main points to note are that: (i) breakpoints are indeed more highly concentrated around the origin, and that (ii) as depth increases, delta-slopes have lower variance and thus lead to even flatter initial functions. We next ask whether the standard initializations will experience difficulty fitting functions that have significant curvature away from the origin (e.g. learning the energy function of a protein molecule). We train ReLU networks to fit a periodic function (sin(x)), which has high curvature both at and far from the origin. We find that the standard initializations do quite poorly away from the origin, consistent with our theory that breakpoints are essential for modeling curvature. Probing further, we observe empirically that breakpoints cannot migrate very far from their initial location, even if there are plenty of breakpoints overall, leading to highly suboptimal fits. We additionally show (see Appendix A.6 for details) that breakpoint distributions change throughout training to more accurately match the ground truth curvature. In order to prove that it is indeed the breakpoint density that is causally responsible, we attempt to rescue the poor fitting by using a simple data-dependent initialization that samples breakpoints uniformly over the training data range [x min, x max], achieved by exploiting Eq.. Sine Quadratic Standard 4.096 ± 2.25.1032 ± 0404 Uniform 2.280 ±.457.1118 ±.0248 We train shallow ReLU networks on training data sampled from a sine and a quadratic function, two extremes on the spectrum of curvature. The data shows that uniform breakpoint density rescues bad fits in cases with significant curvature far from the origin, with less effect on other cases, confirming the theory. We note that this could be a potentially useful data-dependent initialization strategy, one that can scale to high dimensions, but we leave this for future work. Explaining and Quantifying the Suboptimality of Gradient Descent. The suboptimality seen above begs a larger question: under what conditions will GD be successful? Empirically, it has been observed that neural nets must be massively overparameterized (relative to the number of parameters needed to express the underlying function), in order to ensure good training performance. Our theory provides a possible explanation for this phenomenon: if GD cannot move breakpoints too far from their starting point, then one natural strategy is to sample as many breakpoints as possible everywhere, allowing us to fit an arbitrary f. The downside of this strategy is that many breakpoints will add little value. In order to test this explanation and, more generally, understand the root causes 55.5 ± 2.9 52 ± 1.414 50 ±.7 49.25 ± 3.3 51.25 ± 6.1 49.25 ± 4.5 4 68 ± 3.1 57.25 ± 6.8 48.5 ± 2.5 42.5 ± 4.8 40.25 ± 3.9 40.25 ± 3.3 5 62.25 ± 15.1 49 ± 3.5 44.5 ± 5.1 38 ± 5.1 33.75 ± 1.1 31.5 ± 1.7 of the GD's difficulty, we focus on the case of a fully connected shallow ReLU network. A univariate input (i) enables us to use our theory, (ii) allows for visualization of the entire learning trajectory, and (iii) enables direct comparison with existing globally (near-)optimal algorithms for fitting PWL functions. The latter include the Dynamic Programming algorithm (DP, ), and a very fast greedy approximation known as Greedy Merge (GM, ). How do these algorithms compare to GD, across different target function classes, in terms of training loss, and the number of pieces/hidden units? We use this metric for the neural network as well, rather than the total number of trainable parameters. Taking the functional approximation view allows us to directly compare neural network performance to these PWL approximation algorithms. For a quadratic function (e.g. with high curvature, requiring many pieces), we find that the globally optimal DP algorithm can quickly reduce training error to near 0 with order 100 pieces. The GM algorithm, a relaxation of the DP algorithm, requires slightly higher pieces, but requires significantly less computational power. On the other hand all variants of GD (vanilla, Adam, SGD w/ BatchNorm) all require far more pieces to reduce error below a target threshold, and may not even monotonically decrease error with number of pieces. Interestingly, we observe a strict ordering of optimization quality with Adam outperforming BatchNorm SGD outperforming Vanilla GD. These (Figure 1) show how inefficient GD is with respect to (functional) parameters, requiring orders of magnitude more for similar performance to exact or approximate PWL fitting algorithms. Learned Expressivity is not Exponential in Depth. In the previous experiment, we counted the number of linear pieces in the CPWL approximation as the number of parameters, rather than the number of weights. Empirically, we know that the greatest successes have come from deep learning. This raises the question: how does the depth of a network affect its expressivity (as measured in the number of pieces)? Theoretically, it is well known that maximum expressivity increases exponentially with depth, which, in a deep ReLU neural network, means an exponential increase in the number of linear pieces in the CPWL approximation. Thus, theoretically the main power of depth is that it allows for more powerful function approximation relative to a fixed budget of parameters compared to a shallow network. However, recent work has called this into question, finding that in realistic networks expressivity does not scale exponentially with depth. We perform a similar experiment here, asking how the number of pieces in the CPWL function approximation of a deep ReLU network varies with depth. The in Table 2 clearly show that the number of pieces does not exponentially scale with depth. In fact, we find that depth only has a weak effect overall, although more study is needed to determine exactly what effect depth has on the number and variability of pieces. These lend more support to the recent findings of From the functional approximation, we know that a unit induces breakpoints only when the ReLU function applied to the unit's input has zero crossings. In layer one, this happens exactly once per unit as the input to each ReLU is just a line over the input space. In deeper layers, the function approximation is learned, allowing for a varying number of new breakpoints. Given our previous on the flatness of the standard initializations, this will generally only happen once per unit, implying that the number of pieces will strongly correlate with the number of units at initialization. Depth helps with Optimization by enabling the Creation, Annihilation and Mobility of Breakpoints. If depth does not strongly increase expressivity, then it is natural to ask whether its value lies with optimization. In order to test this, we examine how the CPWL function approximation develops in each layer during learning, and how it depends on the target function. A good fit requires that breakpoints accumulate at areas of higher curvature in the training data, as these regions require more pieces. We argue that the deeper layers of a network help with this optimization procedure, allowing the breakpoints more mobility as well as the power to create and annihilate breakpoints. One key difference between the deeper layers of a network and the first layer is the ability for a single unit to induce multiple breakpoints. As these units' inputs change during learning, the number of breakpoints induced by deeper units in a network can vary, allowing for another degree of freedom for the network to optimize. Through the functional parameterization of the hidden layers, these "births and deaths" of breakpoints can be tracked as changes in the number of breakpoints induced per layer. Another possible explanation for the value added of depth is breakpoint mobility, or that breakpoints in deeper layers can move more than those in shallow layers. We run experiments comparing how the velocity and number of induced breakpoints varies between layers of a deeper network. Figure 2: Total changes in number of breakpoints induced and average velocity of breakpoints relative to the first layer in each layer of a five layer ReLU network Figure 2 shows the . The number of breakpoints in deeper layers changes more often than in shallow layers. The breakpoint velocity in deeper layers is also higher than the first layer, although not monotonically increasing. Both of these provide support for the idea that later layers help significantly with optimization and breakpoint placement, even if they do not help as strongly with expressivity. Note that breakpoints induced by a layer of the network are present in the basis functions of all deeper layers. Their functional approximations thus become more complex with depth. However the roughness of the basis functions at initialization in the deeper layers is lower than that of the shallow layers. But, as the network learns, for complex functions most of the roughness is in the later layers as seen in Figure 3 (right)., 2018; 2015) has argued that it comes from an implicit regularization inherent in the optimization algorithm itself (i.e. SGD). In contrast, for the case of shallow and deep univariate fully connected ReLU nets, we provide causal evidence that it is due to the specific, very flat CPWL initialization induced by common initialization methods. In order to test this in both shallow and deep ReLU networks, we compare training with the standard flat initialization to a'spiky' initialization. For a shallow ReLU network, we can test a'spiky' initialization by exactly solving for network parameters to generate a given arbitrary CPWL function. This network initialization is then compared against a standard initialization, and trained against a smooth function with a small number of training data points. Note that in a 1D input space we need a small number of training data points to create a situation similar to that of the sparsity caused by high dimensional input, and to allow for testing generalization between data points. We find that both networks fit the training data near perfectly, reaching a global minima of the training loss, but that the'spiky' initialization has much worse generalization error (Table 3). Visually, we find that the initial'spiky' features of the starting point CPWL representation are preserved in the final approximation of the smooth target function (Figures 4 and 6). For a deep ReLU network, it is more difficult to exactly solve for a'spiky' initialization. Instead, we train a network to approximate an arbitrary CPWL function, and call those trained network parameters the'spiky' initialization. Once again, the'spiky' initialization has near identical training performance, hitting all data points, but has noticeably worse generalization performance. Figure 4:'Spiky' (orange) and standard initialization (blue), compared before (left) and after (right) training. Note both cases had similar, very low training set error. It appears that generalization performance it not automatically guaranteed by GD, but instead due to the flat initializations which are then preserved by GD.' Spiky' initializations also have their (higher) curvature preserved by GD. This idea makes sense, as generalization depends on our target function smoothly varying, and a smooth approximation is promoted by a smooth initialization. Variance. Our last experiment examines how smoothness (roughness) depends on the number of units, particularly in the case where there are large gaps in the training data. We use a continuous and discontinuous target function (shown in Figure 8). We trained shallow ReLU networks with varying width H and initial weight variance σ w on these training data until convergence, and measured the total roughness of ing CPWL approximation in the data gaps. Figure 5: Roughness vs. Width (left) and the variance of the initialization (right) for both data gap cases shown in Figure 8. Each data point is the of averaging over 4 trials trained to convergence. Figure 5 shows that roughness in the data gaps decreases with width and increases with initial weight variance, confirming our theory. A spiky (and thus rougher) initialization leads to increased roughness at convergence as well, lending support to the idea that roughness in data gaps can be'remembered' from initialization. On the other hand, higher number of pieces spreads out the curvature work over more units, leading to smaller overall roughness. Taken together, our experiments indicate that smooth, flat initialization is partly (if not wholly) responsible for the phenomenon of implicit regularization in univariate fully connected ReLU nets, and that increasing overparameterization leads to even better generalization. Conclusions. We show in this paper that examining deep networks through the lens of function space can enabled new theoretical and practical insights. We have several interesting findings: the value of depth in deep nets seems to be less about expressivity and more about learnability, enabling GD to finding better quality solutions. The functional view also highlights the importance initialization: a smooth initial approximation seems to encourage a smoother final solution, improving generalization. Fortunately, existing initializations used in practice start with smooth initial approximations, with smoothness increasing with depth. Analyzing the loss surface for a ReLU net in function space gives us a surprisingly simple and transparent view of the phenomenon of overparameterization: it makes clear that increasing width relative to training data size leads w.h.p. to lonely partitions of the data which are global minima. Function space shows us that the mysterious phenomenon of implicit regularization may arise due to a hidden 2nd order differential equation that underlies the ReLU parameterization. In addition, this functional lens suggests new tools, architectures and algorithms. Can we develop tools to help understand how these CPWL functions change across layers or during training? Finally, our analysis shows that bad local minima are often due to breakpoints getting trapped in bad local minima: Can we design new learning algorithms that make global moves in the BDSO parameterization in order to avoid these local minima? A EXPERIMENTAL DETAILS Figure 6:'Spiky' (orange) and standard initialization (blue), compared before training (left) and post-training (right) using a deep network Trained on a deep, 5 layer network, with 4 hidden layers of width 8. Trained on function over the interval [-2,2]. Learning rate = 1e-4, trained via GD over 10000 epochs, with roughness measured every 50 epochs. Roughness per layer was summed over all units within that layer. Shallow version trained on a 21 unit FC ReLU Network. Deep version trained on a deep, 5-layer network with 4 hidden layers of width 8. In both cases, the'spiky' initialization was a 20 -breakpoint CPWL function, with y n ∼ Uniform([−2, 2]). In the deep case, the spiky model was initialized with the same weights as the non-spiky model, and then pre-trained for 10,000 epochs to fit the CPWL. After that, gradient descent training proceeded on both models for 20,000 epochs, with all training having learning rate 1e-4. Training data was 20 random points in the range [-2,2], while the testing data (used to measure generalization) was spaced uniformly at every ∆x =.01 of the target interval of the target function. In the shallow case, there was no pre-training, as the'spiky' model was directly set to be equal to the CPWL. In the shallow model, training occurred for 20,000 epochs. All experiment were run over 5 trials, and values in table are reported as mean ± standard deviation. Base shallow learning rate was 1e-4 using gradient descent method, with learning rate divided by 5 for the spiky case due to the initialization method generating larger weights. Despite differing learning rates, both models had similar training loss curves and similar final training loss values, e.g. for sine, final training loss was.94 for spiky and 1.02 for standard. Functions used were sin(x), arctan(x), a sawtooth function from [-2,2] with minimum value of -1 at the endpoints, and 4 peaks of maximum value 1, cubic 2, and exp(.5x) Note GD was chosen due to the strong theoretical focus of this paper -similar were obtained using ADAM optimizer, in which case no differing learning rates were necessary. We used networks with a total of H = 40 hidden units, spread over L ∈ {1, 2, 3, 4, 5} hidden layers. Training data consiste of uniform samples of function over the interval x ∈ [−3, 3]. Learning rate = 5 · 10 −5, trained via GD over 25, 000 epochs. The target functions tested were sin(πx), a 5-piece polynomial with maximum value of 2 in the domain [−3, 3], a sawtooth with period 3 and amplitude 1, arctan(x), exp(x), and 1 9 x 2. Each value in the table was the average of 5 trials. We use a deep, 6-layer network, with 5 hidden layers of width 8. Training data consists of the'smooth' and'sharp' functions over the interval x ∈ [−3, 3]. Learning rate = 5e-5, trained via GD until convergence, where convergence was defined as when the loss between two epochs changed by less than 10 −8. Breakpoints were calculated every 50 epochs. The velocity of breakpoints was then calculated, and the values seen in the figure are normalized to the velocity of the first layer. Various function classes were trained until convergence on a depth 1 or 4 ReLU network, with 500 total units distributed evenly across layers. Initial and final breakpoint distributions were measured using a kernel density estimate, and compared with the underlying curvature (absolute value of 2nd derivative) of the ground truth function. The cubic spline was a cubic spline fit to a small number of arbitrary data points. Table 4 shows that the breakpoint densities moved over training to become more correlated with the underlying curvature of the ground truth function. This effect was more pronounced with depth. In certain very simple functions (e.g. x 2 or exp(x), not shown), a failure case emerged where there was no real change in correlation over training. Diagnostics appeared to show this was due to the function being so simple as to train almost instantaneously in our overparameterized network, meaning breakpoints had no time to move. Figure 9 shows what happens to the breakpoint densities Table 4: Top: Correlation of the BP distribution before and after training for depth 1 and 4 networks across function classes. Bottom: Change in correlation over training over training -in the shallow case, they are more constrained by the initial condition, and continue to have a higher density near the origin even when not necessary or appropriate. Each neuron of the second hidden layer receives as input the of a CPWL function z i (x) as defined above. The output of this function is then fed through a ReLU, which has two implications: first, every zero crossing of z i is a breakpoint of x i; second, any breakpoints β j ) < 0 will not be breakpoints of x i. Importantly, the number of breakpoints in g θ (x) is now a function of the parameters θ, rather than equal to fixed H as in the L = 1 case; in other words, breakpoints can be dynamically created and annihilated throughout training. This fact will have dramatic implications when we explore how gradient descent optimizes breakpoints in order to model curvature in the training data (see Section 3). But first, due to complexities of depth, we must carefully formalize the notion of a breakpoint for a deep network. Let a π (x) = i∈π a i. Then, β i is active iff there exists some path π such that a π is discontinuous at and: 2 ), and a cubic spline of a few arbitrary data points This gives us Equation, as desired. Let the subscripts p, q denote the parameters sorted by β p value. In this setting, let β 0 −∞, and β H+1 ∞. Then, This gives us Equation, as desired. Lemma 1. Suppose (b i, w i, v i) are initialized independently with densities f B (b i), f W (w i), and f V (v i). Then, the density of (β i, µ i) is given by Then, we can derive the density of (β i, µ i) by considering the invertable continuous transformation given by where J is the Jacobian determinant of g −1. Then, we have J = − sgn w i and |J| = 1. The density of (β i, µ i) is then derived by integrating out the dummy variable u: are independent, this expands to Theorem 1(a). Consider a fully connected ReLU neural net with scalar input and output, and a single hidden layer of width H. Let the weights and biases be initialized randomly according to a zero-mean Gaussian or Uniform distribution. Then, under an independent Gaussian initialization, Proof. Starting with Lemma 1, unknown otherwise but the integrand is even in µ, giving Corollary 1(a). Consider the same setting as Theorem 1. In the case of an independent Gaussian initialization, where G nm pq (·|·) is the Meijer G-function and K ν (·) is the modified Bessel function of the second kind. Proof. Marginalizing out µ from the joint density in Sympy returns the desired f β (β) from above. Sympy cannot compute the other marginal, so we verify it by hand: , Eq. 3.462.20, we have applying this with a = |µ| σ b σvσw and b = σ b, We can then use these densities to derive the conditional: Theorem 1(b). Consider a fully connected ReLU neural net with scalar input and output, and a single hidden layer of width H. Let the weights and biases be initialized randomly according to a zero-mean Gaussian or Uniform distribution. Then, under an independent Uniform initialization, Proof. Starting with Lemma 1, Corollary 1(b). Consider the same setting as Theorem 1. In the case of an independent Uniform initialization, where Tri(·; a) is the symmetric triangular distribution with base [−a, a] and mode 0. Proof. Beginning with the marginal of β i, Remarks. Note that the marginal distribution on µ i is the distribution of a product of two independent random variables, and the marginal distribution on β i is the distribution of the ratio of two random variables. For the Gaussian case, the marginal distribution on µ i is a symmetric distribution with variance σ Proof. Computing the time derivatives of the BDSO parameters and using the loss gradients of the loss with respect to the NN parameters gives us: This completes the proof.

represent the tldr for finding the corresponding scientific article
Example:
Provided: "Secret is a transfer method for RL based on the transfer of credit assignment." Match: "The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient. Our main contribution is Secret, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm. To some, intelligence is measured as the capability of transferring knowledge to unprecedented situations. While the notion of intellect itself is hard to define, the ability to reuse learned information is a desirable trait for learning agents. The coffee test , presented as a way to assess general intelligence, suggests the task of making coffee in a completely unfamiliar kitchen. It requires a combination of advanced features (planning, control and exploration) that would make the task very difficult if not out of scope for the current state-of-the-art Reinforcement Learning (RL) agents to learn. On the other hand, it is solved trivially by humans, who exploit the universally invariant structure of coffee-making: one needs to fetch a mug, find coffee, power the coffee machine, add water and launch the brewing process by pushing the adequate buttons. Thus, to solve the coffee test, transfer learning appears necessary. Were we to possess a random kitchen simulator and a lot of compute, current transfer methods would still fall short of consistently reusing structural information about the task, hence also falling short of efficient adaptation. Credit assignment, which in RL refers to measuring the individual contribution of actions to future rewards, is by definition about understanding the structure of the task. By structure, we mean the relations between elements of the states, actions and environment rewards. In this work, we investigate what credit assignment can bring to transfer. Encouraged by recent successes in transfer based on supervised methods, we propose to learn to assign credit through a separate supervised problem and transfer credit assignment capabilities to new environments. By doing so, we aim at recycling structural information about the underlying task. To this end, we introduce SECRET (SElf-attentional CREdit assignment for Transfer), a transferable credit assignment mechanism consisting of a self-attentive sequence-to-sequence model whose role is to reconstruct the sequence of rewards from a trajectory of agent-environment interactions. It assigns credit for future reward proportionally to the magnitude of attention paid to past state-action pairs. SECRET can be used to incorporate structural knowledge in the reward function without modifying optimal behavior, as we show in various generalization and transfer scenarios that preserve the structure of the task. Existing backward-view credit assignment methods require to add auxiliary terms to the loss function used to train agents, which can have detrimental effects to the learning process (de), and rely on an external memory, which hinder the generality of their approach. SECRET does neither. Also, as we show in Sec. 3.1, the architecture we consider for SECRET has interesting properties for credit assignment. We elaborate about our novelty with respect to prior work in Sec. 4. We insist on the fact that the focus of our work is on transfer and that it is not our point to compete on credit assignment capabilities. We would like to emphasize several aspects about the generality of SECRET: 1) our method does not require any modification to the RL algorithm used to solve the tasks considered, 2) it does not require any modification to the agent architecture either and 3) it does not alter the set of optimal policies we wish to attain. Moreover, our method for credit assignment is offline, and as a , it can use interaction data collected by any mean (expert demonstrations, replay memories , backup agent trajectories... ). We believe that this feature is of importance for real-world use cases where a high number of online interactions is unrealistic but datasets of interactions exist as a byproduct of experimentation. Background We place ourselves in the classical Markov Decision Process (MDP) formalism . An MDP is a tuple (S, A, γ, R, P) where S is a state space, A is an action space, γ is a discount factor (γ ∈), R: S × A × S → R is a bounded reward function that maps state-action pairs to the expected reward for taking such an action in such a state. Note that we choose a form that includes the ing state in the definition of the reward function over the typical R: S × A → R. This is for consistency with objects defined later on. Finally, P: S × A → ∆ S is a Markovian transition kernel that maps state-action pairs to a probability distribution over ing states, ∆ S denoting the simplex over S. An RL agent interacts with an MDP at a given timestep t by choosing an action a t ∈ A and receiving a ing state s t+1 ∼ P (·|s t, a t) and a reward r t = R(s t, a t, s t+1) from the environment. A trajectory τ = (s i, a i, r i) i=1,...,T is a set of state-action pairs and ing rewards accumulated in an episode. A subtrajectory is a portion of trajectory that starts at the beginning of the episode. The performance of an agent is evaluated by its expected discounted cumulative reward E ∞ t=0 γ t r t. In a partially observable MDP (POMDP), the agent receives at each timestep t an observation o t ∼ O(·|s t) that contains partial information about the underlying state of the environment. 2 SECRET: SELF-ATTENTIONAL CREDIT ASSIGNMENT FOR TRANSFER SECRET uses previously collected trajectories from environments in a source distribution. A selfattentive sequence model is trained to predict the final reward in subtrajectories from the sequence of observation-action pairs. The distribution of attention weights from correctly predicted nonzero rewards is viewed as credit assignment. In target environments, the model gets applied to a small set of trajectories. We use the credit assigned to build a denser and more informative reward function that reflects the structure of the (PO)MDP. The case where the target distribution is identical to the source distribution (in which we use held-out environments to assess transfer) will be referred to as generalization or in-domain transfer, as opposed to out-of-domain transfer where the source and the target distributions differ. Credit assignment as offline reward prediction We learn to assign credit through an offline reward prediction task, based on saved trajectories of agent-environment interactions. We create a sequence-to-sequence (seq2seq) model ) that takes as input the sequence of observation-action pairs and has to reconstruct the corresponding sequence of environment rewards. Being offline, the reward prediction task is learned separately from the RL task, and the reward prediction model does not share representations with the agent. This way, the representations learned for credit assignment do not affect or get mixed with the representations learned for control. Operating offline brings several advantages: one can directly interact with the replay memory of agents and even use expert demonstrations or arbitrary saved transitions as a source of supervision, which could be useful in settings where on-policy interactions are costly, such as robotics. We equip our seq2seq model with an attention mechanism and view the attention weights of the reward reconstruction task as our primary source of assigned credit. The motivation to do so is that the seq2seq model looks into the past to find predictive signal in order to reconstruct the reward, so observation-action pairs it attends to should be those which reduce its uncertainty about the future, in other words those that explain future reward and should be credited. On the use of observations In MDPs, environment states follow the Markov property: they summarize the history of previous interactions and are sufficient to predict the future. As such, predictive models are highly biased towards focusing on the current sequence element, which hinders credit assignment. Under that consideration, when dealing with MDPs, we turn states into observations by applying transformations that hide a certain amount of information from states and break the Markov assumption. For instance, in gridworlds with visual states, we crop the image and get a player centered image with a given window size. Doing so encourages the model to look into the past to find predictive signal, and allow us to track the relative importance given to each element to reconstruct the credit assigned. In POMDPs, this might be unnecessary depending on the amount of information shared between observations and true states. Self-attention for credit assignment Unlike other seq2seq architectures, self-attentive models like Transformers have direct computational paths between pairs of sequence elements, due to their representations that depend on projections of all sequence elements. This feature is key to long-term credit assignment. As an illustration, consider an RL task where the terminal reward depends only on the first observation, which is drawn randomly. Predicting the reward correctly requires to remember the first observation, which would be very challenging for a recurrent architecture whose memory goes through O(n) transformations, n being the size of the sequence. On the other hand, a self-attentive model directly accesses the value of the initial observation, which makes credit assignment easier. Reward prediction architecture We use a Transformer decoder with a single self-attention layer and a single attention head. The model input is a sequence of observationaction couples (o t, a t) t=0,...,T. Each observation goes through a series of convolutional layers (for visual inputs) followed by a series of feed-forward layers. Each action representation, a one-hot vector in the discrete action case, is concatenated to the learned observation embedding. Those representations of dimensionality d i are combined with positional encoding (PE), fed to a self-attention layer and then to a position-wise feedforward layer that outputs logits for reward prediction classes. PE encodes the relative positions of sequence elements, see Appendix A for details. Self-attention is an attention mechanism with parameterization (W k, W q, W v), each matrix belonging to R di×d k, that puts sequence elements in relation by computing non-linear similarity scores for all pairs of elements in the sequence. To do so, each sequence element is mapped to a query vector that is matched against keys and values obtained from the previous elements. To be consistent with the goal of assigning credit, the model should not be able to peek into the future. Thus, we restrict the computational window of each sequence element to the information stored in representations of the previous elements in the sequence and its own by applying a causal mask M c to the of the pairwise similarity computations, assigning a value of 0 to masked elements after the softmax. Let X = (x t) t=0,...,T ∈ R T ×di denote the input sequence in a matrix form, x t being the of internal computations of the model on its t th input. In the same fashion, we note Z = (z t) t=0,...,T ∈ R T ×d k the sequence ing from the application of self-attention. We then have where Q = XW q ∈ R T ×d k stores queries, K = XW k ∈ R T ×d k keys, and V = XW v ∈ R T ×d k values as linear projections of the input; d k stands for the dimension of the key vectors, M c ∈ {0, 1} T ×T is a binary matrix that acts as a causal mask (a lower triangular matrix), is the Hadamard product and C is a large constant (10 9 in practice). Notably, the ing observation-action representation can be viewed as a linear combination of the values of previous elements: The vector α t contains the normalized attention weights for the prediction at timestep t and sums to 1. Since observations contain only a portion of their initial information, the fact that the model succeeds in the prediction task indicates that it reconstructed the missing information from its past. Therefore, attention weights themselves can be viewed as a form of credit assignment, and will be used as such in what follows. While performing regression on the rewards could also be an option, our experiments found that regression tends to converge to poor local optima. Consequently, we predict the sign of the experi-enced rewards: q(r) = sign(r) with sign = 0. We chose the sign as the classification target for its invariance to the scale of the rewards. We use a weighted sequential cross-entropy as the loss function over the class-wise model predictions f θ,c, writing τ (o, a) the subtrajectory of τ ending with the observation-action couple (o, a) to translate the effect of the binary mask: We have found class weighting to be very important to keep the variance of prediction metrics across a variety of datasets of sampled trajectories low. Generating trajectories To train SECRET, we generate a dataset of trajectories which contains a certain proportion of successful trajectories. If source environments are simple enough so that the task has sufficient chance to be solved by acting randomly, we use a random policy to generate trajectories. For more complex distributions of environments, we use a RL agent (either trained or in the learning phase) to generate trajectories. We think purely exploratory methods like could have advantages over using an RL agent and leave the study of their use for future work. In this subsection, we explain how we use credit assignment to make learning more sample-efficient. Reward shaping In RL, agents often deal with sparse rewards that make the learning process slow. Reward shaping is a technique that aims at densifying the reward so as to improve sample efficiency. It defines a class of reward functions that can be added to the original environment rewards without modifying the set of optimal policies. For a given MDP M = (S, A, γ, R, P), we define a new MDP M = (S, A, γ, R, P) where R = R + F is the shaped reward and F the shaping. The reward shaping theorem states that if there exists a function φ such that F: (s, a, s) → γφ(s) − φ(s), then M and M admit the same set of optimal policies. φ is called a potential function. With domain knowledge, one can use reward shaping to design more informative reward functions without encouraging unwanted behavior. Nevertheless, shaping rewards requires good priors for the task and the potential function must often be engineered manually. Since SECRET weighs the contribution of observation-action pairs to future reward, we use it to derive a shaped reward that corresponds to the sum of future reward reachable from the underlying state, weighted by the attention calculated by the model. We explain the process in the following. Computing the potential function We define the redistributed return R ← τ of a trajectory τ as: where α i←j is the attention weight on (o i, a i) when predicting the reward r j and s i are environment states. Indeed, SECRET uses observations but we keep the states they are constructed from to compute the potential. In POMDPs, we recover an approximate state from the observation, either manually or through inference. In this work, we use a state constructed manually, see Appendix A for details. To compute the potential function, we generate a set D of trajectories like described in Sec. 2.1. Since we operate on trajectories, the same state-action pair can appear twice in a sequence and benefit from a different amount of attention, which is why we must include the first summation. In the reward shaping formalism, the potential function φ depends only on the state. To stay within its bounds, we define φ as the forwarded redistributed return. It is computed as the following estimate: Note that in practice we only redistribute individual rewards that were successfully predicted. Also, some states are generally missing from the data distribution induced by the set of trajectories used. For those states, we set to potential to 0, which in a −φ(s) additional reward when transitioning to those from the state s. As a , it gives agents incentive to stay on the support of the data distribution unless they encounter high-reward states. Because it relies on reward shaping, SECRET conserves optimal policies. We empirically find that agents learn faster with the ing augmented reward function. A way to look at it is that we densify the learning signal and bias the agent towards behaviors that encourage future rewards. We start by conveying intuition as to why SECRET should transfer to new environments. In fields other than RL, seq2seq models similar to that of SECRET have shown outstanding transfer capabilities (; ;), even in low-resource settings . In transfer scenarios that preserve the structure of the MDP, the optimal finegrained control sequence can vary drastically from one environment to another. This is why credit assignment is an interesting alternative to the transfer of weights: given an underlying environment state and a specific action, their contribution to future rewards is not fundamentally altered. Such scenarios include specific changes in the state (or observation) distribution and changes to the reward function that preserve the optimal policies. These also include changes in the dynamics of the environment, and though it affects credit assignment, we show later on that SECRET adapts surprisingly well to such scenarios. Another point that motivates the use of our method for transfer is the fact that we keep the representations learned for credit assignment separate from the control representations learned by agents. Indeed, de showed that RL representations were not optimal for transfer. Transfer setting We argue that transfer should be considered effective when agents learn to solve target tasks efficiently because efficiency gains in the target domain compound while the cost of training in the source is fixed. Hence, we use the Total Target Time Scenario metric to assess transfer. Nevertheless, collecting trajectories in the source domain can be costly. We report the number of trajectories used to train SECRET in each scenario. As before, SECRET is trained on episodes of interaction sampled from the source distribution. In each target environment, we sample multiple trajectories (see the following section for details about the policies used to generate the trajectories). We then compute the attentional potential function by calculating an estimate of the expected redistributed reward, as described in Sec. 2.2. In this section, we aim to answer the following questions: can SECRET improve the sample efficiency of learning for RL agents? Does it generalize and/or transfer? How does it compare to transfer baselines? Is the credit assigned by SECRET interpretable? The of complementary experiments are presented in Appendix B. The Triggers environment We introduce Triggers, an interpretable and customizable environment that we use to assess the quality of the credit inferred with our method. In Triggers, the agent is located in a two-dimensional bounded grid. Its actions consist solely of moving of one cell in one of the cardinal directions. Any action that would lead the agent outside the boundaries of the environment (as indicated by the walls in the figure) is ignored but still counted as an action taken by the agent. The goal of the agent (represented as a yellow square) is to activate all the switches (red squares) and then collect all the prizes (pink squares). Prizes are the only source of reward and give a −1 penalty unless all switches are activated, in which case they give a +1 bonus. Both prizes and switches disappear once collected. The main feature of Triggers is that every positive reward is conditional to the presence of a known subset of states in the agent history, and thus credit assignment can be assessed in a rigorous way. In that setup, agents benefit from understanding the link between keys and doors. We hypothesized that our credit assignment mechanism might identify this relation and reward the agent for picking up keys. To assert this, we modified the setting so that picking up keys does not provide rewards. Additionally, the visual input is richer than the one from Triggers environments and the average number of steps per episode is extended. Finally, agents move and rotate across the room. Since picking up a key does not require to actually see the key, it can be hard to know whether a key was taken and predict further door opening rewards. Trajectories are generated with a trained agent. Agents used We use tabular Q-learning for in-domain and out-ofdomain experiments in Triggers except for the transfer to environments with modified dynamics where we use Deep Q-Networks (DQN) . We use Proximal Policy Optimization (PPO) agents for in-domain experiments in DMLab. We provide an analysis of the credit inferred by SECRET. The analysis is qualitative and quantitative, since we rely on both visual assessment and binary detection metrics. The process of evaluating credit assignment in Triggers goes as follows: we first generate trajectories and train the model. We then compare the credit assigned by SECRET on trajectories sampled from held-out environments to a ground truth credit assignment. We build that ground truth by exploiting the exact knowledge of where triggers are. It is a vector that is 0 almost everywhere and 1 on state-action couples that precede the activation of a Triggers. By doing so, we explicitly target the state-action couples whose ing state is causally linked to the reward experienced later. We find the redistribution to be near optimal in simple instances of Triggers (see Fig. 3 -left): attention concentrates quasi exclusively on state-action pairs that enable the collection of future reward. This is confirmed by precision-recall analysis: over the distribution of scenarios considered, a simple binarization heuristic over attention values yields an average precision of 0.96 for an average recall of 0.94. More information on the heuristic is in Appendix A. In keys_doors_puzzle, we adopt the same set of experiments. Since the agent can move backward and spin, in some scenarios it takes a key that is not in is line of sight. In addition, the granularity of the state space is such that off-by-one prediction errors are common but do not hinder the credit mechanism: attributing credit to the state-action couple preceding the collection of a key or the previous one leads to imperceptible changes in the ing shaped rewards. Fig. 3 -right shows similar as for Triggers. Appendix A also provides a heatmap for this task that shows that attention concentrates around the keys. We then study how we can leverage the inferred credit and transfer representations that are helpful in new scenarios. We show that agents train faster when using shaped rewards from SECRET. As before, the reward model is trained on episodes of interaction in environments sampled from the source distribution. In transfer environments, we sample multiple trajectories, each using the same maze configuration. We then compute the attentional potential function by calculating an estimate of the expected redistributed reward, as described in Sec. 2.2. To evaluate its effect, we compare agents trained from environment rewards to agents that use the ing shaped reward. In-domain transfer For in-domain transfer, we transfer the representations for credit assignment to held-out instances of the same distribution over MDPs. For the Triggers environment, the RL agents are tabular Q-learners. For the DMLab environment, we use PPO agents and modify the original task: we do not reward the agent for collecting keys but only to open doors so that the attention can focus on the key positions. Note that this makes the task harder. As we display in Fig. 4 agents learn visibly faster to solve tasks when benefitting from SECRET in both environments. Out-of-domain transfer For out-of-domain transfer we use the Triggers environment and consider two scenarios that are hard for standard agents: transfer to bigger environments (see Fig. 5) and transfer to environments with inverted dynamics (see Fig. 6). In the bigger setting, direct weight transfer cannot be used since the visual input has bigger spatial dimensions. On the other hand, SECRET can be used since the transformation we apply to turn states into observations conserves the visual input dimensions. In the inverted dynamics setting, the effect of the agent's actions are inverted, which makes the task hard for transfer methods. In that setting, we compare the transferability figure). The effect of the shaping is exclusively beneficial, while transferring weights from the source task can be detrimental to the learning process. of our mechanism to that of the representations learned by an agent equipped with deep function approximation. To this end we use DQN agents and either train them from scratch in the target environments or start from the set of weights learned in the source environments (WT in Fig. 6). In both settings, shaping the rewards assists the agent in learning to solve the task. We display some in Fig. 5 and Fig. 6. When transferring to bigger environments, the agent benefits very early on from the shaped reward, while also reaching better asymptotical performance. Transfer in RL While a lot of approaches exist in the transfer literature, to the best of our knowledge none explicitly transfer credit assignment capabilities. Previous works aimed at making the training of an agent in the same task more sample-efficient by using a pretrained model as a teacher (a;). We learn to assign credit as a parallel task that does not modify the representations of the RL agent. Others learn auxiliary reward functions in the hope that they will enable transfer by imposing consistency in the reward (; ;). Although we also learn additional reward signal, it is based on a redistribution of rewards from the environment, which ensures consistency with the original reward function. Transfer is also viewed as learning tasks in a sequential way (b;) and this suggests to introduce inductive bias to the neural architectures of agents to reduce catastrophic forgetting. Our method does not require to alter the agent's architecture. Other explicitly address the problem of transfer through the lens of multitask learning while we stick to learning from an initial distribution of environments. Meta-learning approaches aim to train agents on a distribution of tasks or environments so that their learned skills and representations work across the underlying continuum, and allow for fast adaptation of the agents (; ; ; ; ;). In contrast to meta-learning methods, we do not modify the RL algorithm used to train the agent and SECRET is compatible with any core algorithm for RL. Credit assignment Previous works investigated the role of attention mechanisms for credit assignment. propose SAB, a sparse attention mechanism used to derive a modified backpropagation algorithm. We draw inspiration from SAB but operate in the RL context without sparsity assumptions about the attention weights. RUDDER proposes to equip RL agents with an online method for credit assignment based on return decomposition. Our method operates offline and decomposes individual rewards. RUDDER requires a specific exploration scheme, an additional episodic replay buffer, a compute-heavy contribution analysis method and the addition of several auxiliary losses to the objective the RL agent optimizes. In comparison, SECRET is a lightweight method that does not deal with exploration. Crucially, the focus of is on online credit assignment while ours is on transfer. provide an agent with an external memory and the unsupervised task of reconstructing its inputs (both states and rewards). The agent uses memory reads as a way to identify related elements in sequences, and uses those to transfer the value of states providing delayed rewards to the bootstrapping target of contributing elements. In contrast, SECRET makes use of a non-autoregressive architecture, does not reconstruct states, makes use of reward shaping instead of modifying the update function and most importantly does not rely on an external memory. Recall Traces ) use a generative model that goes backward from high-reward states and samples state-action pairs that could have led to that state. SECRET also works backward from high-reward states but creates links to previous states from existing trajectories instead of sampling them. In this work, we investigated the role credit assignment could play in transfer learning and came up with SECRET, a novel transfer learning method that takes advantage of the relational properties of self-attention and transfers credit assignment instead of policy weights. We showed that SECRET led to improved sample efficiency in generalization and transfer scenarios in non-trivial gridworlds and a more complex 3D navigational task. To the best of our knowledge, this is the first line of work in the exciting direction of credit assignment for transfer. We think it would be worth exploring how SECRET could be incorporated into online reinforcement learning methods and leave this for future work. In this section we provide additional details about our experimental setup and the hyperparameters we use. In Triggers, episodes stop after 50 timesteps in 8x8 grids and 100 timesteps in 12x12 grids. Those values were chosen large enough such that the policies used to train the reward predictor have a chance to gather informative rewards. In DMLab, episodes stop after 900 timesteps, as defined by the standard settings. We use the same set of hyperparameters in all our experiments with few variation. In Triggers experiments, we use 128 units per dense layer, 32 convolutional filters and a single convolutional layer to process partial states and actions. We apply dropout at several places in the model as a regularizer. We use a dropout rate of 0.1 after dense layers, a dropout rate of 0.2 in the self-attention mechanism, and a dropout rate of 0.2 in the normalization blocks of the Transformer architecture. Since Transformers create representations via pooling, they need additional information so as to take into account the relative positions of sequence elements. Hence, we use the same positional encoding scheme as in , that is we add sine and cosine signals of varying frequencies to the sequence of input embeddings X emb before the self-attention layer: In these equations, t refers to the timestep in the sequence, i to the dimension of the embedding considered, and ν has a constant value of 10000. In DMLab experiments, we use two convolutional layers, 16 filters for each, and otherwise identical hyperparameters. Typically, we set the class weights in the loss function to w = w(−1) = 0.499, w = 0.02. Fig. 7 gives an overview of the whole architecture. In Sec. 3.1, we compare the attention vectors we get as outputs from the reward prediction model to ideal credit assignment with binary metrics. The ground truth we use is a binary vector of the size of the attention vector. Its values are 0 everywhere and 1 for timesteps that correspond to the activation of a Triggers. To do so, we introduce a simple heuristic to binarize the attention scalars: we consider all values above a threshold α to correspond to events to be credited. Then, we can measure precision and recall as in a binary classification paradigm. The precision and recall reported are the average precision and recall over 4 scenarios: Triggers with a 8x8 grid, 1 trigger and 1 reward; Triggers with a 8x8 grid, 1 trigger and 2 rewards; Triggers with a 8x8 grid, 2 triggers and 2 rewards; and Triggers with a 8x8 grid, 3 triggers and 1 reward. In each scenario, we train the model over a set of 40000 trajectories, each of which is drawn from a randomly sampled maze. Then, we apply it on 5000 trajectories from held-out environments and collect the attention weights corresponding to predictions on timesteps where the agent experiences positive reward. We use a fixed α of 0.2. We collect 40000 trajectories sampled from random policies to train the prediction reward model in all distributions of environments. In experiments involving tabular Q-learning we use online Q-learning with a learning rate of 0.1 and a constant greediness factor also equal to 0.1. For the out-of-domain transfer experiment with modified dynamics, we use a smaller version of the DQN architecture in. The first convolutional layer has 8 filters, a 3x3 kernel size and a stride of 2. The second and the third convolutional layers have both 16 filters, a 3x3 kernel size and a stride of 1. Those are completed by a feed-forward layer with 64 units followed by another feedforward layer with as many units as the number of available actions. The greediness factor is decayed linearly from 1 to 0.01 over 250000 steps in the environment at train time and has a constant value of 0.001 at test time. We use RMSProp as an optimizer with a base learning rate of 0.00025. We update the target network every 2000 steps and initially fill the replay buffer with 5000 transitions sampled following a random policy. The replay buffer has a maximum size of 1000000. We provide additional details about this setup: we train SECRET using 10000 trajectories sampled from a distribution of mazes that are generated randomly. These trajectories are sampled using an agent trained over the same distribution. We do so to increase the proportion of trajectories where rewards are experienced. Indeed, we found that using random policies yielded very few of these. Once the model is trained, we use it to compute the attentional potential function over a fixed maze. 1000 trajectories are sampled on the fixed maze using the same policy as the one that generated the trajectories used to train the reward prediction model. Since consecutive frames can be very similar, we consider a positive reward prediction to be correct (and thus use the corresponding attention weights when estimating the potential) if it happens within 5 frames of a reward actually experienced in the environment. We then compare the performance of agents trained with the original reward function to those trained with the shaped reward. An important point is that we use the knowledge of the position and the keys the agent possesses to build the state used to compute the potential function. This information is not given to the agent. We create an approximate state as such: it is the concatenation of the discretized position and the identifier of the key possessed. The discretized position is the of the euclidean division by a cell size integer c s and is necessary since the DMLab position is continuous. c s is fixed and has the value of 50 in our experiments. We acknowledge that relying on a manually constructed state limits the generality of our approach, but we are confident that this limitation can be addressed in future work by using an estimate of the true state. All agents mentioned in that section are PPO learners with a learning rate of 0.00019, an entropy coefficient of 0.0011, 12 actors, a discount factor γ = 0.99. They use generalized advantage estimation In this section we provide the of additional experiments that are useful to understand several aspects of SECRET. We study the influence of the size of the window in the transformation used to turn states into observations in Triggers. The goal of this experiment is notably to study the effect of the degree of partial observability to the assigned credit, and also the effect of transferring a possibly badly assigned credit to an agent, through reward shaping. We consider the following window types: 3x3, 5x5 and 7x7 (odd numbers because the window is centered on the agent), as well as the full state. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table 1, and the average discounted return of tabular Q-learning agents for each window size in Fig 9a. Notice that the considered reward prediction accuracy (Table 1) is weighted, such that all classes have the same importance. In this setting, we can observe that bigger window sizes are detrimental to the quality of the credit assigned, even if it helps to predict reward (Table 1). This was to be expected: the more "observable" is the sequence of observations used to train the reward predictor, the easier is the reward prediction (in the full state case, it can be predicted solely from the current state-action couple), but also the less likely the assigned credit will be centered on the trigger. However, even with a low credit precision, the shaped reward still accelerates the learning process (Fig. 9a). We stated that class weighting was important to keep the variance of prediction metrics across a variety of datasets of sampled trajectories low, the classes (sign of the reward) being quite imbalanced. Here, we study the influence of the class weights considered in the loss of the reward prediction model in Triggers. We consider the following class weights for the class corresponding to zero rewards: w = 1, w = 0.1, w = 0.01 and w = 0.001. The other class weights are fixed and have the value w = w(−1) = 1. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table 2, and the average discounted return of tabular Q-learning agents for each window size in Fig 9b. In that setting, putting heavier misclassification penalties for under-represented classes (−1 and 1) in improved credit and RL performance. We study the influence of the amount of data used by SECRET in a Triggers task. The objective is to assess how data-demanding is the proposed approach, and the effect of a lack of data on the transfer. There are two types of data: the data from the source domain (used to train its reward prediction component), and the data from the target domain (used to build the potential function). We consider the following number of trajectories to train the self-attentive reward predictor: 500, 1000, 5000, 10000, 25000 and 50000. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table 3, and the average discounted return of tabular Q-learning agents for each number of trajectories in Fig. 9c. We observe that increasing the size of the dataset improves the , both for the efficiency of the reward predictor (Table 3) and for the transfer (Fig. 9c). Having too few data can slow down learning compared to a vanilla agent (Fig. 9c), but it does not prevent from learning to solve the task. Number of source episodes Reward prediction accuracy Credit precision Credit recall 500 0.53 ± 0.11 0.23 ± 0.37 0.09 ± 0.2 1000 0.65 ± 0.07 0.49 ± 0.33 0.52 ± 0.42 5000 0.79 ± 0.08 0.9 ± 0.30 0.9 ± 0.30 10000 0.84 ± 0.07 0.9 ± 0.30 0.9 ± 0.30 25000 0.88 ± 0.09 0.9 ± 0.30 0.9 ± 0.30 50000 0.83 ± 0.11 0.8 ± 0.4 0.8 ± 0.4 We consider the following number of trajectories sampled from the target environment to build the potential function: 100, 500, 1000, 2000, 5000, 10000. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display the average discounted return of tabular Q-learning agents for each number of trajectories in the source domain in Fig.9d. Here again, we observe that the number of episodes in the target distribution is an important parameter for SECRET, but it has less impact than the number of episodes in the source domain. In the main paper, we have only shown the distribution of attention for an in-domain scenario. Here, following the same protocol as in Sec. 3.1, we measure the distribution of attention weights in the two out-of-domain scenarios considered in our experiments: bigger mazes and inverted dynamics. The are reported in Fig. 10. We observe a similar distribution of attention, peaked on the triggers."

A functional approach reveals that flat initialization, preserved by gradient descent, leads to generalization ability.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the scientific article to find a short TLDR

Question: It is well-known that deeper neural networks are harder to train than shallower ones. In this short paper, we use the (full) eigenvalue spectrum of the Hessian to explore how the loss landscape changes as the network gets deeper, and as residual connections are added to the architecture. Computing a series of quantitative measures on the Hessian spectrum, we show that the Hessian eigenvalue distribution in deeper networks has substantially heavier tails (equivalently, more outlier eigenvalues), which makes the network harder to optimize with first-order methods. We show that adding residual connections mitigates this effect substantially, suggesting a mechanism by which residual connections improve training. Practical experience in deep learning suggests that the increased capacity that comes with deeper models can significantly improve their predictive performance. It has also been observed that as the network becomes deeper, training becomes harder. In convolutional neural networks (CNNs), residual connections BID5 are used to alleviate this problem. Various explanations are provided for this phenomenon: BID6 suggests that residual connections reduce the flatness of the landscape, whereas BID3 questions this premise, noting that the extremal eigenvalues of the loss Hessian are much larger when residual connections are present: large Hessian eigenvalues indicate that the curvature of the loss is much sharper, and less flat. In a different line of work, BID0 observes that the gradients with respect to inputs in deeper networks decorrelate with depth, and suggest that residual connections reduce the'shattering' of the gradients. In this paper, we explore the interaction between depth and the loss geometry. We first establish that gradient explosion or vanishing is not responsible for the slowing down of training, as is commonly believed. Searching for an alternative explanation, we study the Hessian eigenvalue density (using the tools introduced in BID3 to obtain estimates of the eigenvalue histogram or density). The classical theory of strongly convex optimization tells us that optimization is slow when the spectrum simultaneously contains very small and very large eigenvalues (i.e., optimization rate is dependent on κ = λ max /λ min). Following this intuition, we focus on examining the relative spread of the Hessian eigenvalues. In particular, we quantify the extent of the large outliers by computing some scale-invariant classical statistics of the Hessian eigenvalues, namely the skewness and kurtosis. Finally, we observe that in comparable models with residual connections, these magnitude of these outliers is substantially mitigated. In BID3, it is hypothesised that batch normalization suppresses large outlier eigenvalues, thereby speeding up training; in this paper, we present evidence that residual connections speed up training through essentially the same channel. Throughout, the dataset of interest is CIFAR-10; we describe the specific model architectures used in Appendix A. It is well-known that deeper CNNs are harder to train than shallower ones. We exhibit training loss curves depicting this for both residual and non-residual (we refer to these as simple) CNNs in Appendix B, at various network depths (20 and 80). The most prevalent explanation for why very deep networks are hard to train is that the gradient explodes or vanishes as the number of layers increase BID4; this explanation has been infrequently challenged (Section 4.1 in BID5), but no definitive experiments have been shown. We study this hypothesis in FIG0, where we compare the gradient norms of a depth 80 residual and non-residual networks. Two things become clear from this this plot. Firstly, there is no exponential increase or decrease in gradient norms (i.e., we would see vastly different gradient norm scales), as hypothesised in gradient explosion explanations. Secondly, residual connections do not consistently increase or decrease the gradient norms. In FIG0, 49.4% of variables have lower gradient norm in residual networks (in comparison to a baseline of non-residual networks), making the exploding/vanishing gradient explanation untenable in this case. Let H ∈ R n×n be the Hessian of the training loss function with respect to the parameters of the model: DISPLAYFORM0 where θ ∈ R n is the parameter vector, and L(θ) is the training loss. The Hessian is a measure of (local) loss curvature. In the convex setting, optimization characteristics are largely determined by the loss curvature, we expect to be able to observe the factors slowing down the training from analyzing the Hessian spectrum along the optimization trajectory. Let λ 1 ≥ λ 2 · · · ≥ λ n be the eigenvalues of the Hessian. The theory of convex optimization suggests that first-order methods such as SGD slow down dramatically when the relative differences among the eigenvalues of the loss Hessian are large; in particular, from convex analysis suggest that as | λi λ1 | becomes smaller, the optimization in the direction of the eigenvectors associated with λ i slows down BID1 BID2 1. Following this intuition, when the distribution of the eigenvalues of H has heavy tails (equivalently large outliers), we expect the network to train slowly as there will be many eigenvalues where λ i /λ 1 is small. FIG1 shows the (smoothed) density of the eigenvalues of the Hessian for a series of simple CNNs with increasing depth. This figure shows two prominent features of the loss Hessian:1. Most of the eigenvalues of the Hessian are concentrated near zero. This means that the loss surface is relatively flat, in agreement with BID8 BID3 and others.2. As the network gets deeper, outliers appear in the spectrum of H. Moreover, the magnitude of these outliers increases with the depth of the network. This means that as the network becomes deeper, DISPLAYFORM1 shrinks for almost all of the directions, making the training challenging. To quantify the magnitude and extent of these outlier eigenvalues, we compute some scale-independent classical statistics of the Hessian eigenvalues. We are primarily interested in skewness and kurtosis defined as: DISPLAYFORM2 The skewness of a distribution measures its asymmetry, and the kurtosis measures how heavy (or nonGaussian) the tails are -a heavy tailed distribution has a kurtosis greater than 3. In our case, we compute these statistics on the Hessian eigenvalues by observing that for v ∼ N (0, I n): DISPLAYFORM3 Due to the rapid concentration of the quadratic form in high dimensions (for concrete bounds, see BID7) we expect extremely accurate approximation of E[λ k] using a few i.i.d. samples of the form v T H k v. Both skewness and kurtosis should dramatically increase as the tails of the eigenvalue density become heavier. Figure 3 shows what happens to these metrics as we increase the depth: both the skewness and kurtosis increase dramatically as we increase the depth of the model. In particular, note that the kurtosis is far from being a Gaussian -these distribution of eigenvalues is extremely heavy tailed. Given that residual connections allow us to train much deeper models, we would predict that the addition of residual connections should prevent the largest eigenvalues from being so extreme. the spectrum of the Hessian for residual networks and their corresponding simple networks (both networks are identical save for the residual connections). We can see that adding residual connections substantially reduces the extent of the outliers in the Hessian spectrum. More quantitatively, in Figure 3, we can see that models with residual connections have substantially lower skewness and kurtosis than models without residual connections. The effects are in substantial: a 90 layer model with residual connections has lower skewness and kurtosis than a non-residual model half its size. In this paper, we have presented qualitative and quantitative evidence that depth increases outlier eigenvalues in the Hessian, and that residual connections mitigate this. We believe that this touches upon some of the fundamental dynamics of optimizing neural networks, and that any theoretical explanation of residual connections needs to explain this. Behrooz Ghorbani was supported by grants NSF-DMS 1418362 and NSF-DMS 1407813. For the purposes of this short exposition, we adopt a class of networks for our study. We consider a standard residual networks trained on CIFAR-10. These types of networks have 6n layers of feature maps of sizes {32 × 32, 16 × 16, 8 × 8} (2n layers for each type) with {16, 32, 64} filters per layer. With the addition of the input convolution layer and the final fully-connected layer, this type of network has 6n + 2 layers. Batch-Normalization is also present in these networks. In our experiments, when we don't include residual connections, we refer to the network'simple-6n + 2' network and when residual connections are included, the network is referred to as ResNet-6n + 2. We use the SGD with momentum with the same learning rate schedule to train both these networks for 100k steps. B Deeper CNNs are harder to train; skip connections help more at depthWe observe that for small n, both simple and ResNet networks train well (FIG6). As the number of layers increase, training simple model becomes slower. Note however that as we increase the depth, that residual

Represent the short summary for finding the corresponding scientific article (long)

Network depth increases outlier eigenvalues in the Hessian. Residual connections mitigate this.

represent the tldr for finding the corresponding scientific article:

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.