Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: In the last few years, deep learning has been tremendously successful in many applications. However, our theoretical understanding of deep learning, and thus the ability of providing principled improvements, seems to lag behind. A theoretical puzzle concerns the ability of deep networks to predict well despite their intriguing apparent lack of generalization: their classification accuracy on the training set is not a proxy for their performance on a test set. How is it possible that training performance is independent of testing performance? Do indeed deep networks require a drastically new theory of generalization? Or are there measurements based on the training data that are predictive of the network performance on future data? Here we show that when performance is measured appropriately, the training performance is in fact predictive of expected performance, consistently with classical machine learning theory. Is it possible to decide the prediction performance of a deep network from its performance in training -as it is typically the case for shallower classifiers such as kernel machines and linear classifiers? Is there any relationship at all between training and test performances? Figure 1a shows that when the network has more parameters than the size of the training set -which is the standard regime for deep nets -the training classification error can be zero and is very different from the testing error. This intriguing lack of generalization was recently highlighted by the surprising and influential observation that the same network that predicts well on normally labeled data (CIFAR10), can fit randomly labeled images with zero classification error in training while its test classification error is of course at chance level, see Figure 1b. The riddle of large capacity and good predictive performance led to many papers, with a variety of claims ranging from "This situation poses a conceptual challenge to statistical learning theory as traditional measures of model complexity struggle to explain the generalization ability of large artificial neural networks... " , to various hypotheses about the role of flat minima;; , about; and to a number of other explanations (e.g. ;) for such unusual properties of deep networks. We start by defining some key concepts. We call "loss" the measure of performance of the network f on a training set S = x 1, y 1, · · ·, x N, y N. The most common loss optimized during training for binary classification is the logistic loss L(f) = 1 N N n=1 ln(1 + e −ynf (xn) ). We call classification "error" 1 N N n=1 H(−y n f (x n)), where y is binary and H is the Heaviside function with H(−yf (x)) = 1 if −yf > 0 which correspond to wrong classification. There is a close relation between the logistic loss and the classification error: the logistic loss is an upper bound for the classification error. Thus minimizing the logistic loss implies minimizing the classification error. The criticism in papers such as refers to the classification error. However, training minimizes the logistic loss. As a first step it seems therefore natural to look at whether logistic loss in training can be used as a proxy for the logistic loss at testing. The second step follows from the following observation. The logistic loss can always be made arbitrarily small for separable data (when f (x n)y n > 0, ∀n) by scaling up the value of f and in fact it can be shown that the norm of the weights of f grows monotonically with time during gradient descent. Notice that multiplying f by any positive number does not affect its sign and thus does not change its behavior at classification. The second step is then to consider the logistic loss of the normalized networkf with f = ρf, with ρ = ρ 1...ρ K where ρ i is the Frobenius norm of the weight matrix of layer i. Now we show that by measuring the performance of a trained network on the training set in terms of the quantities described above, classical generalization holds: the performance of the network on the training set becomes a good qualitative proxy for the expected future performance of the network. The precise procedure involves normalizing the output of the trained network by dividing it by a single number -the product of the Frobenius norms of the weight matrices at each layer -and then measuring the cross-entropy loss. Figure 2a) shows the cross entropy test loss plotted versus the training loss for networks with the same architecture trained with different initializations (this is a way to get networks with different test performance). There is no clear relation between training and test cross-entropy loss in the same way that there is not between classification error in training and testing. Normalizing each network separately yields Figure 2b ): now the test loss is tightly predicted by the training loss. We emphasize that normalization does not change the classification performance of the networks. In fact, since (binary) classification depends only on the sign of f (x), it is not changed by normalization to any ball. The figure suggests that the empirical cross-entropy loss off on the training set predicts rather well how networks will perform on a test set. Impressively, the same normalization process correctly predicts the test loss when the training is on randomly labeled data. This apparent lack of predictivity of the training performance on randomly labeled data was at the core of Zhang et al. criticism of machine learning theory . Figure 2 shows that it is in fact possible to gauge just from the training performance that the network trained on randomly labeled examples will have chance performance on a test set even if its classification error in training is always zero. As shown in Figure 2 the data point corresponding to the randomly trained network still satisfies approximately the same linear relationship, as explained by the classical theory. Thus, while Figure 2 shows that the normalized train cross-entropy can predict the performance of the of the normalized test cross-entropy, Figure 4 We define a deep network with K layers with the usual elementwise scalar activation functions σ(z): R → R as the set of functions where the input is x ∈ R d, the weights are given by the matrices W k, one per layer, with matching dimensions. We use the symbol W as a shorthand for the set of W k matrices k = 1, · · ·, K. For simplicity we consider here the case of binary classification in which f takes scalar values, implying that the last layer matrix W K is W K ∈ R 1,K l and the labels are y n ∈ {−1, 1}. There are no biases apart form the input layer where the bias is instantiated by one of the input dimensions being a constant. The activation function in this paper is the ReLU activation. We denote the network as f = f (W 1, · · ·, W K ; x) where x is the input and the weight matrices W k are the parameters. We callf the network with normalized weights matrices, Consider different asymptotic minima of the empirical loss obtained with the same network architecture on the same training set by minimization of the cross-entropy loss with different initial conditions (see Appendix). We obtain different test losses, depending on initial conditions. The question is whether their test performance can be predicted from empirical properties measured only on the training set. 1 used in our experiments are multiclass problems. In this theory section we discuss for simplicity the simpler case of binary classification. Exactly the same steps in the theoretical arguments below apply to the cross-entropy loss (because Equations 1,3 apply, see Appendix). Consider the structure of the loss for a deep network. The "positive homogeneity" property of ReLU networks implies the following property: where W K = ρ kWK and ||W K || = 1 and ρ 1 · · · ρ K = ρ. 1 MNIST and CIFAR100 are in the appendix This property is valid for layer-wise normalization under any norm. We emphasize again that have the same classification performance on any data set. Furthermore, during gradient descent on separable data -most data sets are separable by overparametrized deep networks -it can be shown (see) that the ρ factor continue to increase with time towards infinity, driving an exponential type loss to zero without affecting the classification performance, which only depends on the sign of y n f (x n)∀n. As we discussed earlier, it seems that different networks corresponding to different empirical minimizers 2 could be evaluated in terms of their normalized formf (x). The intuition is similar to the linear case in which the classifier w T x depends on the unit vectorw = w |w| while |w| diverges to infinity;. 2 In general an overparametrized network may have a large number of global minima, see for instance. To assessf (x) we compute its logistic loss (which is the special case of cross-entropy in the binary case)L Of course for separable data (y n f (x n) > 0, ∀n) the loss off is larger than the loss of f since the negative exponent is smaller. Figure 2 uses L 2 for layer-wise normalization. We are now ready to explain the of Figure 2 in terms of classical machine learning theory. A typical generalization bound that holds with probability at least (1 − δ), ∀f ∈ F has the form: where is the expected loss, R N (F) is the empirical Rademacher average of the class of functions F measuring its complexity; c 1, c 2 are constants that reflect the Lipschitz constant of the loss function and the architecture of the network. We use the bound in Equation 3 and the key observation that the Rademacher complexity satisfies the property, because of homogeneity of the networks. Then, the bound on the cross-entropy loss for the unnormalized network gives sinceL(f) ≈ 0. Considering the corresponding bound for the cross-entropy loss of the normalized network scaled with any desired scaling R gives In our experiments we find that R N (F) is small for the value of N of our datasets and R N (F) is large. Equation 5 implies then that the unnormalized test loss will be quite different from zero and thus different from the unnormalized train loss. On the other hand, in Equation 6 the terms 2N are small, implying that the normalized test loss is very close to the normalized train loss. Thus Equation 6 with R = 1 shows that L(f) is bounded byL(f), predicting a bound in terms of a linear relationship with slope one and a small offset between L(f) andL(f). The prediction is verified experimentally (see Figure 2). Notice that both homogeneity of the ReLU network and separability, that is very small cross-entropy loss, are key assumptions in our argument. The first applies to networks with a linear or ReLU activation but not to networks with other activation functions. The second is usually satisfied by overparametrized networks. Thus Equation 6 shows that the training cross-entropy loss is a very good proxy of the cross-entropy loss at testing, implying generalization for a relatively small number of examples N. Notice that for all the networks in Figure 2, classification performance at training is perfect and that scaling does not change the sign of the networks outputs and therefore their behaviour in terms of classification. In particular, Figure 2 shows that performance on the randomly labeled training set when measured for the normalized network is bad (despite classification being at zero error) predicting correctly bad performance on the test set. As we mentioned, Figure 4 b shows that there is a good correlation between crossentropy loss and classification performance: empirically the ranking between the different classifiers is mostly the same for crossentropy vs classification loss. and of the upper bounds Equations 3 and 6. Lower bounds, however, are not available. As a consequence, the theory does not guarantee that among two (normalized) networks, the one with lower cross-entropy loss in training will always have a lower classification error at test. This difficulty is not specific to deep networks. It is common to approaches using a surrogate loss function. The empirical evidence however supports the claim that there is a roughly monotonic relationship between training (and testing) loss of the normalized network and its expected classification error: Figure 4b shows an approximately monotonic relation between normalized test cross-entropy loss and test classification error. The linear relationship we found means that the generalization error of Equation 3 is small once the complexity of the space of deep networks is "dialed-down" by normalization. It also means that, as expected from the theory of uniform convergence, the generalization gap decreases to zero for increasing size of the training set (see Figure 1). Thus there is indeed asymptotic generalization -defined as training loss converging to test loss when the number of training examples grows to infinity -in deep neural networks, when appropriately measured. The title in "Understanding deep learning requires rethinking generalization" seems to suggest that deep networks are so "magical" to be beyond the reach of existing machine learning theory. This paper shows that this is not the case. On the other hand, the generalization gap for the classification error and for the unnormalized cross-entropy is expected to be small only for much larger N (N must be significantly larger than the number of parameters). However, consistently with classical learning theory, the cross-entropy loss at training predicts well the cross-entropy loss at test when the complexity of the function space is reduced by appropriate normalization. For the normalized case with R = 1 this happens in our data sets for a relatively "small" number N of training examples as shown by the linear relationship of Figure 2. The classical analysis of ERM algorithms studies their asymptotic behavior for the number of data N going to infinity. In this limiting regime, N > W where W is the fixed number of weights; consistency (informally the expected error of the empirical minimizer converges to the best in the class) and generalization (the empirical error of the minimizer converges to the expected error of the minimizer) are equivalent. This note implies that there is indeed asymptotic generalization and consistency in deep networks. However, it has been shown that in the case of linear regression, for instance with kernels, there are situations -depending on the kernel and the data -in which there is simultaneously interpolation of the training data and good expected error. This is typically when W > N and corresponds to the limit for λ = 0 of regularization, that is the pseudoinverse. It is likely that deep nets may have a similar regime, in which case the implicit regularization described here, with its asymptotic generalization effect, is just an important prerequisite for a full explanation for W > N -as it is the case for kernel machines under the square loss. The of this paper strongly suggested that the complexity of the normalized network is controlled by the optimization process. In fact a satisfactory theory of the precise underlying implicit regularization mechanism has now been proposed As expected, the linear relationship we found holds in a robust way for networks with different architectures, different data sets and different initializations. Our observations, which are mostly relevant for theory, yield a recommendation for practitioners: it is better to monitor during training the empirical "normalized" cross-entropy loss instead of the unnormalized cross-entropy loss actually minimized. The former matters in terms of stopping time and predicts test performance in terms of cross-entropy and ranking of classification error. More significantly for the theory of Deep Learning, this paper confirms that classical machine learning theory can describe how training performance is a proxy for testing performance of deep networks. While the theoretical explanation in the main text applies to the case of binary classification, the extension to multi-class case follows straightforwardly. Recall some definitions for neural networks with multiple outputs. Let C be the number of classes -the neural network is then a vector f (W ; x) ∈ R C. The component fj(W ; x) denotes the j-th output. The dataset is again composed of examples xn ∈ R d and labels are now yn ∈ [C]. Note that nothing here changes in regards to homogeneity, and we again can define a normalized network The main theoretical arguments depend on the generalization bounds of the form As the right hand side depends on the neural networks, which do not change in any substantial way, all that remains is understanding the multi-class loss. To transform the outputs of the network into probabilities, the Softmax function is used The cross-entropy loss is then defined simply aŝ It's very easy to see that this reduces to the logistic loss in the binary case. Classification now depends only on the margin ηn = fy n (W ; x) − max j =yn {fj(W ; x)} -if ηn > 0 then the example is correctly classified. This means that, again, classification only cares about the sign of the margin and not the normalization of the neural network. One final property of note is that for separable data, the loss monotonically decreases with increasing ρ. To see this, let us write αnj = fy n (W ; x) − fj(W ; x), which is a positive quantity in the separable case. Additionally define gn = − log j =yn e −α nj = − log j =yn e −ρα nj, which is clearly a monotonic function of ρ if all αnj > 0. We can now rewrite the Cross-entropy loss aŝ which implies that we can drive the loss to 0 by increasing ρ → ∞. Top The top left graph shows testing vs training cross-entropy loss for networks trained on the same data sets but with different initializations. The top right graph shows the testing vs training loss for the same networks, normalized by dividing each weight by the Frobenius norm of its layer. Notice that all points have zero classification error at training. The red point on the top right refers to a network trained on the same CIFAR data set but with randomized labels. It shows zero classification error at training and test error at chance level. The top line is a square-loss regression of slope 1 with positive intercept. The bottom line is the diagonal at which training and test loss are equal. The networks are 3-layer networks; the first layer is convolutional, 64 filters of size 5x5, stride 2, padding 2, no bias, ReLU activation; the second layer is also convolutional, 64 filters of size 5x5, stride 2, padding 2, no bias, ReLU activation; the third layer is fully connected, input size 64*8*8, output size 10, no bias, softmax activation. The training is on the CIFAR-10 dataset with 50k training examples, 10k testing examples. The network used for the point in red was trained similarly except the testing set and training set labels were randomized. No data augmentation was performed, but data were normalized to mean (The bottom graphs show similar experiments for ResNet-56 with 56 layers, demonstrating that our observations hold for state-of-the-art networks (testing errors ranging from 7% to 9%; on this 10-classes classification problem chance performance is 90% error). We again emphasize that the performance in classification of normalized and unnormalized networks is exactly the same. The normalization in the case of ResNet was performed by using the norm of the output of each network on one example from CIFAR-10, because we found it computationally difficult to directly evaluate the effective norm of the residual layers. The networks were trained for 200 epochs with learning rate = 0.01 for the first 100 epochs and learning rate = 0.001 for the second 100 epochs. SGD was used with batch size of 128 and shuffled training data with random crop and horizontal flip data augmentation. First we start with a common observation: even when two networks have the same architecture, same optimization meta parameters and same training loss, they usually have different test performances (i.e. error and loss), because the stochastic nature of the minimization process leads to convergence to different minima among the many existing in the loss landscape;. With standard settings the differences are usually small (though significant, as shown later). We use therefore two approaches to magnify the effect: • Initialize networks with different levels of "random pretraining": the network is pretrained for a specified number of epochs on "corrupted" training data -the labels of a portion of the examples are swapped with each other in a random fashion. • Initialize the weights of the networks with different standard deviations of a diagonal Gaussian distribution. As it turns out, different standard deviations yield different test performance. Similar techniques have been used previously;. We show the of "random pretraining" with networks on CIFAR-10 (Figure 5) and CIFAR-100 (Figure 7) and initialization with different standard deviations on CIFAR-10 (Figure 6) and CIFAR-100 (Figure 8). Our observations are that the linear relationship between train loss and test loss for normalized networks hold in a robust way under several conditions: • Independence from Initialization: The linear relationship is independent of whether the initialization is via pretraining on randomly labeled natural images or whether it is via larger initialization, as shown by Figures 11 and 12. • Independence from Network Architecture: The linear relationship of the test loss and train loss does not depend on the network architectures we tried. • Independence from Data Set: Figures 10, 11, 12, 9 show the linear relationship on CIFAR10 while Figures 17 and 18 show the linear relationship on CIFAR100. • • Normalization is independent of training loss: Figure 10 shows that networks with different cross-entropy training losses (which are sufficiently small to guarantee zero classification error), once normalized, show the same linear relationship between train loss and test loss. One way to formalize the upper bound on classification error by the logistic loss is to consider the excess classification risk R(f) − R *, where R(f) is the classification error associated with f and R * is the Bayes error. Let us call as before L(f) the expected cross-entropy loss and L * the optimal expected cross entropy loss. Then the following bound holds for binary classification in terms of the so-called ψ-transform of the logistic loss: where the ψ function for the logistic is similar to the ψ for the exponential loss which is ψ(x) = 1 − √ 1 − x 2. The key point here is that ψ −1 is monotonic: minimizing the logistic or cross-entropy loss implies minimizing the classification error. Our arguments imply that among two unnormalized minimizers of the exponential loss that achieve the same given small empirical lossL =, the minimizer with higher product of the norms ρ1, · · ·, ρK has the higher capacity and thus the highest expected loss L. Experiments support this claim, see Figure 13. Notice the linear relationship of test loss with increasing capacity on the top right panels of Figure 11, 12, 17, 18. Product of L2 Norms of Weights from All Layers (Unormalized Net) Figure 13 shows that when the capacity of a network (as measured by the product norm of the layers) increases, so does the test error.; This section shows figures replicating the main on the MNIST data set. Figures 15 and 16 show that the linear relationship holds after normalization on the MNIST data set. Figure 16 shows the linear relationship holds after adding the point trained only on random labels. This section is about replicating the main on CIFAR-100. Figure 7 shows how different test performance can be obtained with pretraining on random labels while Figure 8 shows that different increasing initializations are also effective. Figures 17 and 18 show that the linear relationship holds after normalization, regardless of whether the training was done with pretraining on random labels or with large initialization. This section is about replicating the main with ResNets (see main text figures). To avoid using the product of layerwise norms -whose form is not obvious for ResNets -we normalized the different architectures using their output on the same single image, exploting the fact that all the networks only differ in their initialization (for this reason we did not plot the RL point because this shortcut to normalization does not apply in the case of a different training set with random labels). While in this article we use the term generalization when the offset in the difference between training and test losses is small, the technical definition of "generalization" just requires that the offset decreases with increasing N. This means that for all f ∈ F, limN→∞ |L(f) −L(f)| → 0, since in general we expect the Rademacher complexity to decrease as N increases. As Figure 19 shows, |L(f) −L(f)| does in fact decrease with increasing N (notice that L(f) ≈L(f) around N = 50000 for normalization to the unit L2 ball of our network). These arguments do not depend on the specific form of the Rademacher complexity. A typical margin bound for classification is where η is the margin, L binary (f) is the expected classification error, Lsurr(f) is the empirical loss of a surrogate loss such as the logistic or the exponential. For a point x, the margin is η ∼ yρf (x). Since RN (F) ∼ ρ, the margin bound says that the classification error is bounded by 1 η that is by the value off on the "support vectors" -that is on the xi, yi s.t arg minn ynf (xn). We have looked at data showing the test classification error versus the inverse of the margin. The data are consistent with the margin bound in the sense that the error increases with the inverse of the margin and can be bounded by a straight line with appropriate slope and intercept. Since the bound does not directly provide slope and intercept, the data do not represent a very convincing evidence. Why are all the cross-entropy loss values close to chance (e.g. ln 10 ≈ 2.3 for a 10 class data set) in the plots for convnets -bit not for ResNets -showing the linear relationship? This is because most of the (correct) outputs of the normalized neural networks are close to zero as shown by Figure 20. We would expect the norm of the network to be appromimately bounded by |f (W ; x)| |W ||x| = |x|; the data x is usually pre-processed to have mean 0 and a standard deviation of 1. In fact, for the MNIST experiments, the average value f (x) of the most likely class according to the normalized neural network is 0.026683 with a standard deviation 0.007144. This means that significant differences, directly reflecting the predicted class of each point, are between 0.019539 and 0.033827. This in turn implies that the exponentials in the cross-entropy loss are all very close to 1. Before normalization (which of course does not affect the classification performance), the average value f (x) of the most likely class was 60.564373 with standard deviation 16.214078. The model is a 3-layer convolutional ReLU network with the first two layers containing 24 filters of size 5 by 5; the final layer is fully connected; only the first layer has biases. There is no pooling. The network is overparametrized: it has 154, 464 parameters (compared to 50, 000 training examples). The model is the same 3-layer convolutional ReLU network as in section L.1.1 except it had 34 units. The network was still overparametrized: it has 165, 784 parameters (compared to 50, 000 training examples). The model is a 5-layer convolutional ReLU network with (with no pooling). It has in the five layers 32, 64, 64, 128 filters of size 3 by 3; the final layer is fully connected; batch-normalization is used during training. The network is overparametrized with about 188, 810 parameters (compared to 50, 000 training examples).

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the scientific article to find a short tldr

Question: We propose a "plan online and learn offline" framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world. We consider a setting where an agent with limited memory and computational resources is dropped into a world. The agent has to simultaneously act in the world and learn to become proficient in the tasks it encounters. Let us further consider a setting where the agent has some prior knowledge about the world in the form of a nominal dynamics model. However, the state space of the world could be very large and complex, and the set of possible tasks very diverse. This complexity and diversity, combined with limited computational capability, rules out the possibility of an omniscient agent that has experienced all situations and knows how to act optimally in all states, even if the agent knows the dynamics. Thus, the agent has to act in the world while learning to become competent. Based on the knowledge of dynamics and its computational resources, the agent is imbued with a local search procedure in the form of trajectory optimization. While the agent would certainly benefit from the most powerful of trajectory optimization algorithms, it is plausible that very complex procedures are still insufficient or inadmissible due to the complexity or inherent unpredictability of the environment. Limited computational resources may also prevent these powerful methods from real-time operation. While the trajectory optimizer may be insufficient by itself, we show that it provides a powerful vehicle for the agent to explore and learn about the world. Due to the limited capabilities of the agent, a natural expectation is for the agent to be moderately competent for new tasks that occur infrequently and skillful in situations that it encounters repeatedly by learning from experience. Based on this intuition, we propose the plan online and learn offline (POLO) framework for continual acting and learning. POLO is based on the tight synergistic coupling between local trajectory optimization, global value function learning, and exploration. We will first provide intuitions for why there may be substantial performance degradation when acting greedily using an approximate value function. We also show that value function learning can be accelerated and stabilized by utilizing trajectory optimization integrally in the learning process, and that a trajectory optimization procedure in conjunction with an approximate value function can compute near optimal actions. In addition, exploration is critical to propagate global information in value function learning, and for trajectory optimization to escape local solutions and saddle FIG4: Examples of tasks solved with POLO. A 2D point agent navigating a maze without any directed reward signal, a complex 3D humanoid standing up from the floor, pushing a box, and inhand re-positioning of a cube to various orientations with a five-fingered hand. Video demonstration of our can be found at: https://sites.google.com/view/polo-mpc.points. In POLO, the agent forms hypotheses on potential reward regions, and executes temporally coordinated action sequences through trajectory optimization. This is in contrast to strategies like −greedy and Boltzmann exploration that explore at the granularity of individual timesteps. The use of trajectory optimization enables the agent to perform directed and efficient exploration, which in turn helps to find better global solutions. The setting studied in the paper models many problems of interest in robotics and artificial intelligence. Local trajectory optimization becomes readily feasible when a nominal model and computational resources are available to an agent, and can accelerate learning of novel task instances. In this work, we study the case where the internal nominal dynamics model used by the agent is accurate. Nominal dynamics models based on knowledge of physics, or through learning , complements a growing body of work on successful simulation to reality transfer and system identification BID34 BID31; BID23. Combining the benefits of local trajectory optimization for fast improvement with generalization enabled by learning is critical for robotic agents that live in our physical world to continually learn and acquire a large repertoire of skills. The POLO framework combines three components: local trajectory optimization, global value function approximation, and an uncertainty and reward aware exploration strategy. We first present the motivation for each component, followed by the full POLO procedure. We model the world as an infinite horizon discounted Markov Decision Process (MDP), which is characterized by the tuple: M = {S, A, R, T, γ}. S ∈ R n and A ∈ R m represent the continuous (real-valued) state and action spaces respectively. R: S × A → R represents the reward function. T: S × A × S → R + represents the dynamics model, which in general could be stochastic, and γ ∈ is the discount factor. A policy π: S × A → R + describes a mapping from states to actions. The value of a policy at a state is the average discounted reward accumulated by following the policy from the state: DISPLAYFORM0 The overall performance of the policy over some start state distribution β is given by: DISPLAYFORM1 For notational simplicity, we use s to denote the next state visited after (from) s. As described earlier, we consider the setting where an agent is dropped into a complex world. The agent has access to an internal model of the world. However, the world can be complex and diverse, ruling out the possibility of an omniscient agent. To improve its behavior, the agent has to explore and understand relevant parts of the state space while it continues to act in the world. Due to the availability of the internal model, the agent can revisit states it experienced in the world and reason about alternate potential actions and their consequences to learn more efficiently. The optimal value function describes the long term discounted reward the agent receives under the optimal policy. Defining the Bellman operator at state s as: DISPLAYFORM0 the optimal value function V * corresponds to the fixed point: V * (s) = BV * (s) ∀s ∈ S. For small, tabular MDPs, classical dynamic programming algorithms like value iteration can be used to obtain the optimal value function. The optimal policy can be recovered from the value function as: DISPLAYFORM1 For more complex MDPs, computing the optimal value function exactly is not tractable except in a few well known cases like the LQR (Åström &) and LMDPs BID46 BID12. Thus, various approximate techniques have been considered in prior works. One popular approach is fitted value iteration BID9 Munos & Szepesvári, 2008), where a function approximator (e.g. neural network) is used to approximate the optimal value function. The core structure of fitted value iteration considers a collection of states (or a sampling distribution ν), and a parametric value function approximatorV θ. Inspired by value iteration, fitted value iteration updates parameters as: DISPLAYFORM2 where BV θi (s) are targets for the regression problem computed at the specific state s according to Eq.. After sufficient iterations of the procedure in Eq. to get a good approximation, the policy is recovered asπ(s) = arg max a E[r(s, a) + γV θ (s)]. The success and convergence of this overall procedure depends critically on at least two components: the capacity and structure of the function approximator (θ); and the sampling distribution (ν).Lemma 1. BID9 LetV be an approximate value function with ∞ error:= max s |V (s) − V * (s)|. Letπ(s) = arg max a E[r(s, a) + γV (s)] be the induced greedy policy. For all MDPs and β, the bound in Eq. holds. Furthermore, for any size of the state space, there exist MDPs andV for which the bound is tight (holds with equality). DISPLAYFORM3 Intuitively, this suggests that performance ofπ degrades with a dependence on effective problem horizon determined by γ. This can be understood as the policy paying a price of at every timestep. Due to the use of function approximation, errors may be inevitable. In practice, we are often interested in temporally extended tasks where γ ≈ 1, and hence this possibility is concerning. Furthermore, the arg max operation inπ could inadvertently exploit approximation errors to produce a poor policy. The performance of fitted value iteration based methods also rely critically on the sampling distribution to propagate global information (Munos & Szepesvári, 2008), especially in sparse reward settings. For some applications, it may be possible to specify good sampling distributions using apriori knowledge of where the optimal policy should visit (e.g. based on demonstration data). However, automatically generating such sampling distributions when faced with a new task may be difficult, and is analogous to the problem of exploration. Trajectory optimization and model predictive control (MPC) have a long history in robotics and control systems BID15 BID44 1. In MPC, starting from state s t and using the knowledge of the dynamics model, a locally optimal sequence of actions (or policies) up to a moving horizon of H is computed by solving the following optimization problem. DISPLAYFORM0 Here, we use x, u,π as dummy variables for states, actions, and policy to distinguish the "imagined" evolution of the MDP used for the trajectory optimization with the actual states (s) observed in the true evolution of the MDP. Here, r(x, u) represents the running reward which is the same as the MDP reward function, and r f (x t+H) represents a terminal reward function. Let {π * k} be the local time-indexed policies obtained as the solution to the optimization problem in. After solving the optimization problem, the first local time-indexed policy is used asπ M P C (·|s t):=π * t (·|x t). The entire procedure is repeated again in the next time step (t + 1). Note that we have defined the optimization problem over a sequence of feedback policies. However, if the dynamics is deterministic, a sequence of actions {u k} t+H k=t can be optimized and used instead without any loss in performance. See Appendix C for further discussions. This approach has led to tremendous success in a variety of control systems such as power grids, chemical process control BID29, and more recently in robotics BID49. Since MPC looks forward only H steps, it is ultimately a local method unless coupled with a value function that propagates global information. In addition, we also provide intuitions for why MPC may help accelerate the learning of value functions. This synergistic effect between MPC and global value function forms a primary motivation for POLO. Lemma 2. LetV be an approximate value function with ∞ error:= max s |V (s) − V * (s)|. Suppose the terminal reward in Eq. FORMULA6 is chosen as r f (s H) =V (s H), and let the MPC policy bê π M P C (·|s t):=π * t (·|x t) (from Eq. 4). Then, for all MDPs and β, the performance of the MPC policy can be bounded as: DISPLAYFORM0 Proof. The proof is provided in Appendix C.This suggests that MPC (with H > 1) is less susceptible to approximation errors than greedy action selection. Also, without a terminal value function, we have = O(r max /(1 − γ)) in the worst case, which adds an undesirable scaling with the problem horizon. Accelerating convergence of the value function Furthermore, MPC can also enable faster convergence of the value function approximation. To motivate this, consider the H-step Bellman operator: DISPLAYFORM1 In the tabular setting, for any V 1 and V 2, it is easy to verify that |B DISPLAYFORM2 H allows for propagation of global information for H steps, thereby accelerating the convergence due to faster mixing. Note that one way to realize B H is to simply apply B H times, with each step providing a contraction by γ. In the general setting, it is unknown if there exists alternate, cheaper ways to realize B H. However, for problems in continuous control, MPC based on local dynamic programming methods BID18 BID47 provide an efficient way to approximately realize B H, which can be used to accelerate and stabilize value function learning. The ability of an agent to explore the relevant parts of the state space is critical for the convergence of many RL algorithms. Typical exploration strategies like -greedy and Boltzmann take exploratory actions with some probability on a per time-step basis. Instead, by using MPC, the agent can explore in the space of trajectories. The agent can consider a hypothesis of potential reward regions in the state space, and then execute the optimal trajectory conditioned on this belief, ing in a Algorithm 1 Plan Online and Learn Offline (POLO) 1: Inputs: planning horizon H, value function parameters θ 1, θ 2,... θ K, mini-batch size n, number of gradient steps G, update frequency Z 2: for t = 1 to ∞ do 3:Select action a t according to MPC (Eq. 4) with terminal reward r f (s) ≡V (s) from Eq. FORMULA13 4:Add the state experience s t to replay buffer D Sample n states from the replay buffer, and compute targets using Eq. FORMULA14 8:Update the value functions using Eq. (see Section 2.5 for details) 9: end for 10:end if 11: end for temporally coordinated sequence of actions. By executing such coordinated actions, the agent can cover the state space more rapidly and intentionally, and avoid back and forth wandering that can slow down the learning. We demonstrate this effect empirically in Section 3.1.To generate the hypothesis of potentially rewarding regions, we take a Bayesian view and approximately track a posterior over value functions. Consider a motivating setting of regression, where we have a parametric function approximator f θ with prior P(θ). The dataset consists of input-output pairs: DISPLAYFORM0, and we wish to approximate P(θ|D). In the Bayesian linear regression setting with Gaussian prior and noise models, the solution to the following problem generates samples from the posterior BID25 BID4 BID26: DISPLAYFORM1 DISPLAYFORM2 is a noisy version of the target andθ ∼ P(θ) is a sample from the prior. Based on this, BID26 demonstrate the benefits of uncertainty estimation for exploration. Similarly, we use this procedure to obtain samples from the posterior for value function approximation, and utilize them for temporally coordinated action selection using MPC. We consider K value function approximatorsV θ with parameters θ 1, θ 2,... θ K independently trained based on Eq.. We consider the softmax of the different samples as the value at a state: DISPLAYFORM3 Since the above scheme approximates mean + variance for small κ > 0, this procedure encourages the agent to additionally explore parts of the state space where the disagreement between the function approximators is large. This corresponds to the broad notion of optimism in the face of uncertainty BID3 which has been successful in a number of applications BID38 ). To summarize, POLO utilizes a global value function approximation scheme, a local trajectory optimization subroutine, and an optimistic exploration scheme. POLO operates as follows: when acting in the world, the agent uses the internal model and always picks the optimal action suggested by MPC. Exploration is implicitly handled by tracking the value function uncertainties and the optimistic evaluation, as specified in Eq. and. All the experience (visited states) from the world are stored into a replay buffer D, with old experiences discarded if the buffer becomes full. After every Z steps of acting in the world and collecting experience, the value functions are updated by: (a) constructing the targets according to Eq.; (b) performing regression using the randomized prior scheme using Eq. where f θ corresponds to the value function approximator. For state s in the buffer and value network k with parameters θ k, the targets are constructed as: DISPLAYFORM0 which corresponds to solving a N −step trajectory optimization problem starting from state s. As described earlier, using trajectory optimization to generate the targets for fitting the value approximation accelerates the convergence and makes the learning more stable, as verified experimentally in Section 3.3. The overall procedure is summarized in Algorithm 1. Through empirical evaluation, we wish to answer the following questions:1. Does trajectory optimization in conjunction with uncertainty estimation in value function approximation in temporally coordinated exploration strategies?2. Can the use of an approximate value function help reduce the planning horizon for MPC?3. Does trajectory optimization enable faster and more stable value function learning?Before answering the questions in detail, we first point out that POLO can scale up to complex high-dimensional agents like 3D humanoid and dexterous anthropomorphic hand BID23 which are among the most complex control tasks studied in robot learning. Video demonstration can be found at: https://sites.google.com/view/polo-mpc Exploration is critical in tasks where immediate rewards are not well aligned with long-term objectives. As a representative problem, we consider a point mass agent in different 2D worlds illustrated in figure 2: a simple finite size box with no obstacles and a maze. This domain serves to provide an intuitive understanding of the interaction between trajectory optimization and exploration while also enabling visualization of . In the extreme case of no rewards in the world, an agent with only local information would need to continuously explore. We wish to understand how POLO, with its ensemble of value functions tracking uncertainties, uses MPC to perform temporally coordinated actions. Our baseline is an agent that employs random exploration on a per-time-step basis; MPC without a value function would not move due to lack of local extrinsic rewards. Second, we consider an agent that performs uncertainty estimation similar to POLO but selects actions greedily (i.e. POLO with a planning horizon of 1). Finally, we consider the POLO agent which tracks value uncertainties and selects actions using a 32-step MPC procedure. We observe that POLO achieves more region coverage in both point mass worlds compared to alternatives, as quantitatively illustrated in FIG1. The ensemble value function in POLO allows the agent to recognize the true, low value of visited states, while preserving an optimistic value elsewhere. Temporally coordinated action is necessary in the maze world; POLO is able to navigate down all corridors., and inhand manipulation task (middle). POLO was trained for 12000 and 2500 environment timesteps, respectively. We test POLO with the learned terminal value function against pure MPC and compare average reward obtained over 3 trials in the getup task and 1000 steps in the manipulation task. On the right, a value function trained with POLO is used by MPC without per-time-step rewards. The agent's height increases, indicating a task-relevant value function. For comparison, we also include the trace of POLO with dense rewards and multiple trials (dashed vertical lines) Next, we study if value learning helps to reduce the planning horizon for MPC. To this end, we consider two high dimensional tasks: humanoid getup where a 3D humanoid needs to learn to stand up from the ground, and in-hand manipulation where a five-fingered hand needs to re-orient a cube to a desired configuration that is randomized every 75 timesteps. For simplicity, we use the MPPI algorithm BID49 for trajectory optimization. In FIG2, we consider MPC and the full POLO algorithm of the same horizon, and compare their performance after T steps of learning in the world. We find that POLO uniformly dominates MPC, indicating that the agent is consolidating experience from the world into the value function. With even the longest planning horizon, the humanoid getup task has a local solution where it can quickly sit up, but cannot discover a chain of actions required to stand upright. POLO's exploration allows the agent to escape the local solution, and consolidate the experiences to consistently stand up. To further test if the learned value function is task aligned, we take the value function trained with POLO, and use it with MPC without any intermediate rewards. Thus, the MPC is optimizing a trajectory of length H = 64 purely using the value function of the state after 64 steps. We observe, in FIG2, that even in this case, the humanoid is able to consistently increase its height from the floor indicating that the value function has captured task relevant details. We note that a greedy optimization procedure with this value function does not yield good , indicating that the learned value function is only approximate and not good everywhere. While the humanoid getup task presents temporal complexity requiring a large planning horizon, the in-hand manipulation task presents spatial complexity. A large number of time steps are not needed to manipulate the object, and a strong signal about progress is readily received. However, since the targets can change rapidly, the variance in gradient estimates can be very high for function approximation methods BID16. Trajectory optimization is particularly well suited for such types of problems, since it can efficiently compute near-optimal actions conditioned on the instance, facilitating function approximation. Note that the trajectory optimizer is unaware that the targets can change, and attempts to optimize a trajectory for a fixed instance of the task. The value function consolidates experience over multiple target changes, and learns to give high values to states that are not just immediately good but provide a large space of affordances for the possible upcoming tasks. Finally, we study if trajectory optimization can aid in accelerating and stabilizing value function learning. To do so, we again consider the humanoid getup task and study different variants of POLO.In particular, we vary the horizon (N) used for computing the value function targets in Eq.. We observe that as we increase N, the agent learns the value function with fewer interactions with the world, as indicated in FIG3 (a). The benefit of using N −step returns for stable value function learning and actor-critic methods have been observed in numerous works (; BID8, and our experiments reinforce these observations. The use of N −step returns help to traverse the bias-variance trade-off. Furthermore, due to the discounting, the contribution of V (s N) is made weaker and thus the targets are more stable. This mirrors ideas such as target networks commonly used to stabilize training. As discussed earlier, longer horizons make trajectory optimization more tolerant to errors in the value function. To illustrate this, we take the value function trained with POLO on a nominal humanoid model, and perturb the model by changing the size of the head to model value function degradation. FIG3 b) shows that a longer planning horizon can mitigate this degradation. This presents intriguing future possibility of using MPC to improve transfer learning between tasks or robot platforms. Planning and learning: Combining elements of planning and search with approximate value functions has been explored in discrete game domains BID39 BID0 where an MCTS planner is informed by the value function. Alternatively, using prior data to guide the search process in continuous MCTS without explicitly learning a value function has also been explored BID30. Related to this, BID2 uses an offline trajectory library for action selection in real-time, but do not explicitly consider learning parametric value functions. RTDP BID7 considers learning value functions based on states visited by the agent, but does not explicitly employ the use of planning. BID50 consider the setting of learning a value function to help MPC, and found the contribution of value functions to be weak for the relatively simple tasks considered in their work. Approaches such as cost shaping BID22 can also be interpreted as hand specifying an approximate value function, and has been successfully employed with MPC. However, this often require careful human design and task specific expertise. An alternative set of approaches BID35 BID21; BID42 use local trajectory optimization to generate a dataset for training a global policy through imitation learning. These approaches do not use MPC at runtime, and hence may often require retraining for changes in tasks or environment. Furthermore, from this line of work have been demonstrated primarily in settings where trajectory optimization alone can solve the task, or use human demonstration data. In contrast, through our exploration schemes, we are able to improve over the capabilities of MPC and solve tasks where MPC is unsuccessful. Planning and exploration: Exploration is a well-studied and important problem in RL. The importance of having a wide and relevant state distribution has been pointed out in numerous prior works (Munos & Szepesvári, 2008; BID6 BID32 . Strategies such as -greedy or Gaussian exploration have recently been used to successfully solve a large number of dense reward problems. As the reward becomes sparse or heavily delayed, such strategies be-come intractable in high-dimensional settings. Critically, these approaches perform exploration on a per time-step basis, which can lead to back and forth wandering preventing efficient exploration. Parameter-space exploration BID28 BID14 methods do not explore at each time step, but rather generate correlated behaviors based on explored parameters at the start. However, such approaches do not consider exploration as an intentional act, but is rather a deviation from a well defined objective for the agent. Deep exploration strategies BID24 sample a value function from the posterior and use it for greedy action selection. Approaches based on notions of intrinsic motivation and information gain BID11 BID40 BID17 BID27 BID8) also explicitly introduce exploration bonuses into the agent's reward system. However, such approaches critically do not have the element of planning to explore; thus the agent may not actually reach regions of high predicted reward because it does not know how to get there. Our work is perhaps closest to the E3 framework of BID19, which considers altered MDPs with different reward functions, and executes the optimal action under that MDP. However solving these altered MDPs is expensive and their solution is quickly discarded. MPC on the other hand can quickly solve for local instance-specific solutions in these MDPs. Model-free RL: Our work investigates how much training times can be reduced over model-free methods when the internal model is an accurate representation of the world model. As a representative number, BID36 report approximately 5 days of agent experience and 128 CPU core hours for solving tasks such as getting up from the ground. In contrast, POLO requires only 12 CPU core hours and 96 seconds of agent experience. Recently, policy gradient methods were also used for in-hand manipulation tasks BID23, where 3 years of simulated experience and 500 CPU hours were used for object reorientation tasks. For a similar task, POLO only required 1 CPU hour. Of course, model-free methods do not require an accurate internal model, but our suggest that much less experience may be required for the control aspect of the problem. Our work can be viewed as a strong model-based baseline that model-free RL can strive to compete with, as well as a directly useful method for researchers studying simulation to reality transfer. In an alternate line of work, internal models have been used for variance reduction purposes in model-free RL BID13 BID10, in contrast to our use of MPC. Related to this, BID5 consider learning an internal model for discrete action domains like ALE and use short horizon MCTS for planning. learn a dynamics model in simple continuous control tasks and use a random shooting MPC method for action selection. These lines of work consider the interplay between learning dynamics models and planning procedures, and try to improve the quality of internal models. As a consequence, they focus on domains where simple action selection procedures with accurate models obtain near-optimal performance. In our work, we show that we can learn value functions to help real-time action selection with MPC on some of the most high-dimensional continuous control tasks studied recently. Thus, the two lines of work are complementary, and combining POLO with model learning would make for an interesting line of future work. In this work we presented POLO, which combines the strengths of trajectory optimization and value function learning. In addition, we studied the benefits of planning for exploration in settings where we track uncertainties in the value function. Together, these components enabled control of complex agents like 3D humanoid and five-fingered hand. In this work, we assumed access to an accurate internal dynamics model. A natural next step is to study the influence of approximation errors in the internal model and improving it over time using the real world interaction data. The model used for the humanoid experiments was originally distributed with the MuJoCo software package and modified for our use. The model nominally has 27 degrees of freedom, including the floating base. It utilizes direct torque actuation for control, necessitating a small timestep of 0.008 seconds. The actuation input is limited to ±1.0, but the original gear ratios are left unchanged. For POLO, the choice of inputs for the value function involves a few design decisions. We take inspiration from robotics by using only easily observed values. For value function approximation in POLO for the humanoid tasks, we use an ensemble of 6 neural networks, each of which has 2 layers with 16 hidden parameters each; tanh is used for non-linearity. Training is performed with 64 gradient steps on minibatches of size 32, using ADAM with default parameters, every 16 timesteps the agent experiences. In scenarios where the agent resets, we consider a horizon of 600 timesteps with 20 episodes, giving a total agent lifetime of 12000 timesteps or 96 seconds. When we consider no resets, we use the same total timesteps. A control cost is shared for each scenario, where we penalize an actuator's applied force scaled by the inverse of the mass matrix. Task specific rewards are as follows. In the getup scenario, the agent is initialized in a supine position, and is required to bring its root height to a target of 1.1 meters. The reward functions used are as follows. In the non-sparse case, the difficulty in this task is eschewing the immediate reward for sitting in favor of the delayed reward of standing; this sequence is non-trivial to discover. In the walking scenario, the agent is initialized in an upright configuration. We specify a reward function that either penalizes deviation from a target height of 1.1 meters, or penalizes the deviation from both a target speed of 1.0 meters/second and the distance from the world's x-axis to encourage the agent to walk in a straight line. We choose a target speed as opposed to rewarding maximum speed to encourage stable walking gaits. For the box environment, we place a 0.9 meter wide cube in front of the humanoid, which needs to be pushed to a specific point. The friction between the box and ground is very low, however, and most pushes cause the box to slide out of reach; POLO learns to better limit the initial push to

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a framework that incorporates planning for efficient exploration and learning in complex environments.

Represent the short summary for finding the corresponding scientific article (long).

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Modern neural network architectures take advantage of increasingly deeper layers, and various advances in their structure to achieve better performance. While traditional explicit regularization techniques like dropout, weight decay, and data augmentation are still being used in these new models, little about the regularization and generalization effects of these new structures have been studied. Besides being deeper than their predecessors, could newer architectures like ResNet and DenseNet also benefit from their structures' implicit regularization properties? In this work, we investigate the skip connection's effect on network's generalization features. Through experiments, we show that certain neural network architectures contribute to their generalization abilities. Specifically, we study the effect that low-level features have on generalization performance when they are introduced to deeper layers in DenseNet, ResNet as well as networks with'skip connections'. We show that these low-level representations do help with generalization in multiple settings when both the quality and quantity of training data is decreased. Deep models have achieved significant success in many applications. However, deep models are hard to train and require longer times to converge. A solution by construction is copying the learned layers from the shallower model and setting additional layers to identity mapping. Skip connection proposed in the Residual Network BID0, shows the new insight of innovation in network structure for computer vision. In the following years, more new and multi-layer-skipping structures have been proposed and proved to have better performance, among which one typical example is DenseNet BID1. ResNet BID0, HighwayNet (Rupesh Kumar BID3 and FractalNets BID2 have all succeeded by passing the deep information directly to the shallow layers via shortcut connection. Densenet further maximize the benefit of shortcut connections to the extreme. In DenseNet (more accurately in one dense block) every two layers has been linked, making each layer be able to use the information from all its previous layers. In doing this, DenseNet is able to effectively mitigate the problem of gradient vanishing or degradation, making the input features of each layer various and diverse and the calculation more efficient. Concatenation in Dense Block: the output of each layer will concatenate with its own input and then being passed forward to the next layer together. This makes the input characteristics of the next layer diversified and effectively improves the computation and helps the network to integrate shallow layer features to learn discriminative feature. Meanwhile, the neurons in the same Dense block are interconnected to achieve the effect of feature reused. This is why DenseNet does not need to be very wide and can achieve very good . Therefore, shortcut connections form the multi-channel model, making the flow of information from input to output unimpeded. Gradient information can also be fed backward directly from the loss function to the the various nodes. In this paper we make the following contributions:• We design experiments to illustrate that on many occasions it is worth adding some skip connections while sacrificing some of the network width. Every single skip connection replacing some of width is able to benefit the whole network's learning ability. Our'connection-by-connection' adding experiment can indicate this well.• We perform experiments to show that networks that reuse low-level features in subsequent layers perform better than a simple feed-forward model. We degrade both the quantity and the quality of the training data in different settings and compare the validation performances of these models. Our suggest that while all models are able to achieve perfect training accuracy, both DenseNet and ResNet are able to exhibit better generalization performance given similar model complexities.• We investigate solutions learned by the three types of networks in both a regression and classification involving task in low dimensions and compare the effects of both the dense connections and the skip connections. We show that the contribution of the feature maps reintroduced to deeper layers via the connections allow for more representational power. Skip connections are the main features of DenseNet and ResNet, which is convenient to make gradient flow easily and overcome the overfitting. For this reason, it is interesting and necessary to dive more in the effects of skip connections on the performance. In the original DenseNet paper BID1, three dense blocks are connected sequentially and in each dense block all of the convolutional layers have direct paths connected with each other. In our implementation, there are 8 layers which leads to a total of 28 skip connections within each dense block. We increase the number of skip connections from 0 to 28 and test the validation accuracy on CIFAR100 trained with 10k and 50k samples. The total numbers of parameters in these models are set to the same (92k) by controlling the depths of convolutional filters in each dense block. Experiments are conducted to test the effect of skip connections in the next section. We will change the number of skip connections in network smoothly. The increasing process of the number of skip connections can be described as follows: For each layer, first of all the connections to their previous layers which is 2 layers away is added. Then the connections linking layers farther away from each other is added. In this section the generalization performances of three different network structures, Cascade Network(simple layer-wise CNN), ResNet and DenseNet, using MNIST and CIFAR100 dataset are measured. The experiments are done by modifying these datasets either by decreasing the number of training samples or by adding noise to groundtruth labels so as to test which network structure shows better performance under these'harsh training conditions'. Neural networks is prone to overfitting if the training data fed into the network is insufficient. In order to compare generalization performances of the three networks with different connectivity patterns, the appropriate depth and parameters for these networks are selected carefully for better comparisons. The depths of the networks range from 44 to 46 layers due to the fact that it has a fair learning capacity. We haven't chosen the standard DenseNet or ResNet with over a hundred layers as described in the original papers because the plain Cascade Network will suffer from severer gradient vanishing problems as the number of layers increases. The numbers of channels of each layer are carefully picked so that sizes of the three model's parameters are nearly the same (560k to 580k in our experiment) to ensure fairness of our comparisons. In FIG2 (c), the width of each network is specially designed so that when the entire training set is used for training the three networks, similar performance could be reached, which is all around 50%. From the figure it can be noticed that while Cascade Net (Plain Net) has much more parameters, it is the least robust to the decreasing number of training samples. On the contrary, the DenseNet, with more skip connections, has the fewest parameters but the highest robustness. With the decrease number of training samples, the performance of DenseNet drops more moderately than the others shown in FIG2. The architecture can benefit the network's stability and its generalization. As a consequence, the dense connection can be a good method of generalization. The similar have been achieved in MNIST, where the only difference is that the networks contains only fully connected layers without any convolution layers. Considering the potential over fitting problem of deep Linear network and the simplicity of the dataset, we construct just 6 layers for Cascade Net and DenseNet. Same as CIFAR dataset, we vary the widths in order to achieve We can conclude from the above experiments that for deep neural networks both with and without convolution layers, the skip connections can effectively act as an implicit regularizer in network training process, making networks generalize better. To test the skip connections''adapting skills', we have tried to add some noise to training dataset. The noise is added by directly setting some pixels in some channels as 0. The is as the noise grows bigger, the decrease in DenseNet's performance will be smaller than others. As shown in Figure 4, network with dense connections is more robust to noise. In this section we visualize the generalization effects of network structures using the simple 1-dimensional curve fitting problem. The network has to learn a function f that maps scalar value x to scalar value f (x). Network with better regularization capability should have smoother output in the hidden layers and may thus avoid the over-fitting problem, producing greater generalization ability. We use the simple networks which contains only fully-connected layer and the nonlinear layers to fit the Sinc Function defined as: DISPLAYFORM0 in range of (−15, 15). Three kinds of networks are implemented using their main characteristics: the Cascade Net uses a smooth linear model, the ResNet in which several skip connection is added, and the DenseNet which has only one dense block and between every two layers there is one direct connection. In order to analyze the more in detail, we both plot the network's final learned curve and extract the output from each layer. The parameters of all 3 kinds of networks of the same depth are controlled as near as possible, by adjusting the'width' ('growth rate'). For example, when growth rate is 2 for DenseNet and 4 for Plain Net, their parameters are 98 and 113 respectively. Though the Plain Net has more parameters, its are unsatisfactory. Top sequence of subplots in FIG4 are the learning process of 7-layer dense net with a training sample of 30 points. Each subplot is the output of each layer (as the growth rate is 2 there are 2 curves in each plot). Also, while there is only 30 training data points, it can still learn well. In the 7 th subplot, the blue curve is the standard sinc curve and the green one is its learned version where it can be seen that the two are almost identical. The last plot is the training loss changing with the epochs. As for the ordinary Plain Net, the with 30 training data points or even 200 training data points is always very bad without fitting the small waves of both sides. The output 4 curves from each layer is also as simple but the final are bad. It couldn't learn the trivial wave. The thing that is also not satisfactory is in its training loss which drops quickly at the beginning but cannot be smaller later. When more training data are fed to the Plain Net, sometimes it can do a good job. However, most times it still cannot learn more information, like in bottom sequence of subplots in FIG4 with 400 training data, it is still quite hard for this Plain Net to compete with the dense one. • The ResNet is added to them to compare together. At least residual net is of most competitiveness to DenseNet.• Some noise is added to the training data (the training data points are with a small fluctuation from its original sinc curve). With the noise, it is more interesting to see the networks' generalization. The are similar as the circumstance with no noise.• Same experiments are conducted many a time which enables us to plot some statistical . In Figure 6, x-axis is the depth and y-axis is the final loss. The middle point is the mean loss in 10 experiments, the line segments represent the deviation in these same experiments. Still, in these experiments it is designed to make sure that in each depth the parameter number of two kinds of nets are similar through adjusting the growth rate. On one hand, it can be seen that DenseNet is better when deeper. On the other, the Plain Net learns nothing when it is too deep. It is due to the gradient vanishing as will be discussed later. No matter in the mean loss or the deviation, the dense net is smaller than the other two. And It can generalize even much better than residual net relatively when the nets are shallow. We can have a closer look at 15-layer nets of each type in Figure 7. In order to make the parameters numbers of 3 types nearly identical to each other, the Plain Net and the ResNet each has 8 lines of output of previous layer while the DenseNet has only 3 lines (growth rate of each layer). In the figure, only 3 of them are chosen to be displayed to have a more equitable comparison with DenseNet. Instead of showing every layer's output we exhibit the 3 th, 5 th, 7 th, 9 th, 11 th, 13 th and the final layer's output. Also, the training data plotted on the final output is 60 points with trivial deviation from the standard sinc curve, together with the network's learnt curve.• Plain Net:The worst one as can be seen on the top row in Figure 7: gradient vanishing is severe and finally it learns just a little profile of the curve which makes no sense at all.• Net with residual connection: This is much better than the Plain Net (the middle row of Figure 7 . It can fit the curve well in subplot 20. However, the middle output seems a bit complicated and its final fitting is not perfect either. • Net with dense connection: To make number of parameters similar to the two nets above, its growth rate is only 3. It can also fit the curve even better and the output of previous layers is the simplest as the bottom row of Figure 7 .For better investigate the effects of 'skip connection', a more detailed experiment has been conducted. In previous experiments, we only compare no more than 3 kinds of different network architectures, which may not be concrete enough. Thus, we exploit more forms of networks by split the numbers of skip connections. Their 'density' is between the two extreme. The skip connections are added one by one, where each defines a different network architecture. Network widths are chosen accordingly to make the numbers of parameters not differ much. The depth is 7 for all architectures, which means there will be at most 21 'skip connections'. As can be seen in FIG6 (a), the fitting effect is becoming better as the network becomes'denser'. The corresponding loss is displayed in FIG6 (b). From these two figures, the representational power of skip connections is conspicuous. We move on to analyzing the three styles of networks above in a two-dimensional classification problem. The networks have to learn a decision boundary from a non-linearly separable data set. We again restrict our depth to eight layers with the number parameters across the Dense, Residual, and the Plain networks being 614, 712, and 712, respectively. The parameters for the Dense network is controlled by adjusting the growth rate of each layer. Using the same hyper-parameters and number of training epochs, we attain the intermediate decision boundaries across each layer of each network to see the progression of complexity with increasing depth. Similar to the onedimensional experiments, networks that generalize better tend to learn smooth decision boundaries in the presence of noise rather than over-fitting to all data points. To visualize the intermediate of the networks, we feed a grid of evenly separated data points in the vicinity of our original two-dimensional data points and record the raw outputs after being activated by the activation function in each of the layers. Since only the last layer, i.e., the output layer, has two-dimensional outputs, we choose one of the dimensions in each layer to visualize. The top row in Figure 9 shows the progression of a densely connected network in the style of Figure 9: Intermediate decision boundaries for three different models. DenseNet BID1. Notice that in the Dense network, which achieves the lowest lost in the test set, every layer receives inputs from all preceding layers, and is, therefore, able to make use of low-level features even at the last layer stage. The first row of the figure shows the intermediate features received by the eighth layer, which includes the linear features like those from the first layer, all the way to higher-level features from the third and the fifth. We decide to show this last layer for this network since it encompasses learning from all previous stages. The benefits of dense connections, however, is not present in the Residual and the Plain networks. The second and third rows of Figure 9 show the features learned in the first, third, and the fifth layers. By introducing skip connections, modern neural network has proved better performance in computer vision area. This paper investigates how skip connections works in vision task and how they effect the learning power of networks. For this reason, we have design some experiments and verify that networks with skip connections can do the regression best among tested network architectures. It indicates that we can get the insights of this interesting architecture and its tremendous learning power.

Represent the short summary for finding the corresponding scientific article (long)

Our paper analyses the tremendous representational power of networks especially with 'skip connections', which may be used as a method  for better generalization.

An experimental paper that proves the amount of redundant weights that can be freezed from the third epoch only, with only a very slight drop in accuracy.


Question: represent the text to find a very short summary (1-2 sentences max):

Question: One of the challenges in training generative models such as the variational auto encoder (VAE) is avoiding posterior collapse. When the generator has too much capacity, it is prone to ignoring latent code. This problem is exacerbated when the dataset is small, and the latent dimension is high. The root of the problem is the ELBO objective, specifically the Kullback–Leibler (KL) divergence term in objective function. This paper proposes a new objective function to replace the KL term with one that emulates the maximum mean discrepancy (MMD) objective. It also introduces a new technique, named latent clipping, that is used to control distance between samples in latent space. A probabilistic autoencoder model, named $\mu$-VAE, is designed and trained on MNIST and MNIST Fashion datasets, using the new objective function and is shown to outperform models trained with ELBO and $\beta$-VAE objective. The $\mu$-VAE is less prone to posterior collapse, and can generate reconstructions and new samples in good quality. Latent representations learned by $\mu$-VAE are shown to be good and can be used for downstream tasks such as classification. Autoencoders(AEs) are used to learn low-dimensional representation of data. They can be turned into generative models by using adversarial, or variational training. In the adversarial approach, one can directly shape the posterior distribution over the latent variables by either using an additional network called a Discriminator , or using the encoder itself as a discriminator . AEs trained with variational methods are called Variational Autoencoders (VAEs) . Their objective maximizes the variational lower bound (or evidence lower bound, ELBO) of p θ (x). Similar to AEs, VAEs contain two networks: Encoder -Approximate inference network: In the context of VAEs, the encoder is a recognition model q φ (z|x) 1, which is an approximation to the true posterior distribution over the latent variables, p θ (z|x). The encoder tries to map high-level representations of the input x onto latent variables such that the salient features of x are encoded on z. Decoder -Generative network: The decoder learns a conditional distribution p θ (x|z) and has two tasks: i) For the task of reconstruction of input, it solves an inverse problem by taking mapped latent z computed using output of encoder and predicts what the original input is (i.e. reconstruction x ≈ x). ii) For generation of new data, it samples new data x, given the latent variables z. During training, encoder learns to map the data distribution p d (x) to a simple distribution such as Gaussian while the decoder learns to map it back to data distribution p(x) 2. VAE's objective function has two terms: log-likelihood term (reconstruction term of AE objective function) and a prior regularization term 3. Hence, VAEs add an extra term to AE objective function, and approximately maximizes the log-likelihood of the data, log p(x), by maximizing the evidence lower bound (ELBO): Maximizing ELBO does two things: • Increase the probability of generating each observed data x. • Decrease distance between estimated posterior q(z|x) and prior distribution p(z), pushing KL term to zero. Smaller KL term leads to less informative latent variable. Pushing KL terms to zero encourages the model to ignore latent variable. This is especially true when the decoder has a high capacity. This leads to a phenomenon called posterior collapse in literature (; ; van den ; ; Sønderby et al., 2016;). This work proposes a new method to mitigate posterior collapse. The main idea is to modify the KL term of the ELBO such that it emulates the MMD objective . In ELBO objective, minimizing KL divergence term pushes mean and variance parameters of each sample at the output of encoder towards zero and one respectively. This, in turn, brings samples closer, making them indistinguishable. The proposed method replaces the KL term in the ELBO in order to encourage samples from latent variable to spread out while keeping the aggregate mean of samples close to zero. This enables the model to learn a latent representation that is amenable to clustering samples which are similar. As shown in later sections, the proposed method enables learning good generative models as well as good representations of data. The details of the proposal are discussed in Section 4. In the last few years, there have been multiple proposals on how to mitigate posterior collapse. These proposals are concentrated around i) modifying the ELBO objective, ii) imposing a constraint on the VAE architecture, iii) using complex priors, iv) changing distributions used for the prior and the posterior v) or some combinations of these approaches. Modifications of the ELBO objective can be done through annealing the KL term (Sønderby et al., 2016;), lower-bounding the KL term to prevent it from getting pushed to zero , controlling KL capacity by upper bounding it to a pre-determined value or lower-bounding the mutual information by adding skip connections between the latent layer and the layers of the decoder . Proposals that constrain the structure of the model do so by reducing the capacity of the decoder (; ;), by adding skip connections to each layer of the decoder , or by imposing constraints on encoder structure . Taking a different approach, and van den replace simple Gaussian priors with more complex ones such as a mixture of Gaussians. The most recent of these proposals are δ-VAE and SKIP-VAE . δ-VAE imposes a lower bound on KL term to prevent it from getting pushed to zero. One of the drawbacks of this approach is the fact that it introduces yet another hyper-parameter to tune carefully. Also, the model uses dropout to regularize the decoder, reducing the effective capacity of the decoder during training. It is not clear how effective the proposed method is when training more powerful decoders without such regularization. Moreover, the proposal includes an additional constraint on encoder structure, named the anti-causal encoder. SKIP-VAE, on the other hand, proposes to lower bound mutual information by adding skip connections from latent layers to each layer of decoder. One drawback of this approach is that it introduces additional non-linear layer per each hidden layer, ing in more parameters to optimize. Moreover, its advantage is not clear in cases, where one can increase capacity of decoder by increasing number of units in each layer (or number of channels in CNN-based decoders) rather than adding more layers. When we train a VAE model, we ideally want to end up with a model that can reconstruct a given input well and can generate new samples in high quality. Good reconstruction requires extracting the most salient features of data and storing them on latent variable ('Encoder + Latent layer' part of the model). Generating good samples requires a generative model ('Latent layer + Decoder' part) with a model distribution that is a good approximation to actual data distribution. However, there tends to be a trade-off between reconstruction quality of a given input, and quality of new samples. To understand why we have such a trade-off, we can start by looking at ELBO objective function 4: Maximizing this objective function increases p θ (x), the probability of generating each observed data x while decreasing distance between q(z|x) and prior p(z). Pushing q(z|x) closer to p(z) makes latent code less informative i.e. z is influenced less by input data x. The reason why the KL term can be problematic becomes more clear when we look at the KL loss term typically modelled with log of variance during optimization: where D is the dimension of latent variable, and i refers to i th sample. Noting that the mean is in L2 norm, minimizing the KL term leads to pushing the each dimension of the mean,µ d, to zero while pushing σ 2 towards 1. This makes estimated posterior less informative and less dependent on input data. The problem gets worse when dimension of latent variable, D, increases, or when the KL term is multiplied with a coefficient β > 1 . Ideally, we want to be able to distinguish between different input samples. This can be achieved by having distinctive means and variances for clusters of samples. This is where MMD might have advantage over the KL divergence. Matching distributions using MMD can match their sample means although their variance might still differ. We can emulate behaviour of MMD by modifying the KL term. We do so by changing L2 norm of mean, Re-writing it for B samples, we have: It is important to note that we are taking absolute value of sum of sample means. This new formulation in aggregate mean of samples to be zero (i.e. same mean as that of prior distribution) while allowing samples to spread out and enabling model to encode information about input data onto z. It should be noted that this new L1 norm of µ can push individual mean estimates to very high values if it is not constrained. To avoid that, L2 norm of means for each sample is clipped by a pre-determined value during optimization. Based on experiments, it is found that clipping L2 norm of sample means by three times square root of latent dimension works well in general although bigger values might help improve in tasks such as classification: This method will be referred as latent clipping for the rest of this work. In addition, the remaining terms in the KL loss can be kept as is, i.e. exp (log σ 2) − log σ 2 − 1, or we can just optimize for subset of it by using either "log σ 2 ", or " exp (log σ 2) − 1 " term since each method will push log σ 2 towards zero (i.e. variance towards one). log σ 2 is chosen in this work since it is simpler. Finally, the µ-VAE objective function can be formulated as follows: where first term is reconstruction loss, B refers to batch size since aggregated mean is computed over batch samples, J refers to dimension of data, D refers to dimension of latent variable, x is original input, and x is reconstructions. To visualize the implications of the latent clipping, a toy VAE model shown in Table 2 in Appendix A is used. Figure 1 compares three cases, in which a VAE model is trained on MNIST dataset using ReLu, Tanh, and Leaky ReLu activation functions for each case, and the latent layer is visualized to observe clustering of digits. Those three cases are: i) Standard VAE objective with the KL divergence, ii) Standard VAE objective with the KL divergence + latent clipping, and iii) µ-VAE objective function + latent clipping. Two observations can be made: 1. Latent clipping might help improve smoothness of latent space, even in the case of standard VAE objective, ELBO. 2. µ-VAE objective function seems to work well. To test the effectiveness of µ-VAE objective, a CNN-based VAE model is designed and trained on MNIST and MNIST Fashion using same hyper-parameters for both datasets. Centered isotropic Gaussian prior, p(z) ∼ N (0, 1.0), is used and the true posterior is approximated as Gaussian with an approximately diagonal co-variance. No regularization methods such as dropout, or techniques such as batch-normalization is used to avoid having any extra influence on the performance, and to show the advantages of the new objective function. The model is trained with four different objective functions: i) VAE (ELBO objective), ii) β-VAE with β = 4, iii) µ-VAE#1 s.t. µ sample ≤ 3 * √ z dim and iv) µ-VAE#2 s.t. µ sample ≤ 6 * √ z dim, where z dim = 10. Details of architecture, objective functions, hyper-parameters, and training are described in Appendix B. During training of the models, a simple three layer fully connected classifier is also trained over 10 dimensional latent variable to learn to classify data using features encoded on latent variable. Classifier parameters are updated when encoder and decoder parameters are frozen and vice versa so that classifier has no impact on how information is encoded on the latent variable. Evaluation of the generative model is done qualitatively in the form of inspecting quality, and diversity of samples. Posterior collapse is assessed by comparing reconstructions of input data to observe whether the decoder ignores latent code encoded by input data and by comparing the KL divergences obtained for each model. For all three objective functions, the KL divergence is measured using standard KL formula in Equation 3. Moreover, the accuracy of the classifier trained on latent variable is used as a measure of how well the latent variable represents data . Higher classification accuracy reflects a better representation of data and opens doors to use latent representation for downstream tasks such as classification. Figure 2 shows training curves for MNIST Fashion dataset (MNIST can be seen in Appendix C). The new objective function in lower reconstruction loss, higher KL divergence, and higher classification accuracy. Higher KL divergence and classification accuracy can be interpreted as a sign of learning a more informative latent code. β-VAE performs the worst across all metrics as expected. The reason is that β factor encourages latent code to be less informative, and is known to in worse reconstruction quality . of samples obtained using test data for both datasets. VAE seems to able to distinguish all ten digits, but performs worse in MNIST Fashion. β-VAE pushes samples closer as expected, which explains why its performance is low in classification task. µ-VAE, on the other hand, is able to cluster similar samples together in both datasets. Moreover, when upper-bound on µ sample is increased, it spreads out clusters of samples, making them easier to distinguish. Hence, upper-bound used in latent clipping can be a knob to control distance between samples. Also, we should note that we can achieve similar clustering to the Table 1 lists classification accuracy obtained using test datasets. µ-VAE performs the best as expected since it is able to push the clusters apart. Higher upper bound on µ sample in a higher classification accuracy. Also, it should be noted that reported accuracy numbers can be improved, but the purpose of this test was to show that new objective function can reliably be used in downstream tasks such as classification. Figure 4 compares sample distributions obtained at each dimension of latent variable using test dataset of MNIST Fashion for each objective function. β-VAE samples follow N prior very closely, and hence ing in the smallest KL divergence term. Sample distributions from the most dimensions of VAE are also close to prior distribution although some of them show a multi-modal behavior. Sample distributions from both µ-VAE#1 & #2 in zero mean, but they are more spread out as expected. Spread is controlled by upper-bound on µ sample. Similar to VAE, some sample distributions show a multi-modal behavior. Figure 5 shows reconstruction of input using test dataset. β-VAE reconstructions are either blurrier, or wrong, the latter of which is a sign of posterior collapse. VAE performs better, and both versions of µ-VAE gives the best reconstruction quality. Figure 6 shows images generated using random samples drawn from multivariate Gaussian, N(0, σ), where σ = 1 is for VAE and β-VAE while it is 3 for µ-VAE since their samples are more spread out (MNIST can be seen in Appendix C). We can observe that some samples generated from µ-VAE models have dark spots. This is because the model is trying to generate texture on these samples. This can also be observed in samples of VAE model, but it is less pronounced. However, samples from β-VAE do not show any such phenomena since the model perhaps learns global structure of shapes while ignoring local features. Failing to capture local structures is a known problem in latent variable models . N(0, σ).From left to right, model (σ): VAE (σ=1), β-VAE (σ=1), µ-VAE#1 (σ=3), and µ-VAE#2 (σ=3). Higger σ is used for µ-VAE models since their samples are more spread out. that most dimensions of latent variable are not very informative. VAE is slightly better. However, both µ-VAE models learn diverse classes of objects across different dimensions. Moreover, they learn different classes on opposite sides of same dimension. This is encouraging since it shows its power to learn rich representations. In this work, a new objective function is proposed to mitigate posterior collapse observed in VAEs. It is shown to give better reconstruction quality and to learn good representations of data in the form of more informative latent codes. A method, named latent clipping, is introduced as a knob to control distance between samples. Samples can be pushed apart for tasks such as classification, or brought closer for smoother transition between different clusters of samples. Unlike prior work, the proposed method is robust to parameter tuning, and does not constraint encoder, or decoder structure. It can be used as a direct replacement for ELBO objective. Moreover, the proposed method is demonstrated to learn representations of data that can work well in downstream tasks such as classification. Applications of µ-VAE objective with more powerful decoders in various settings can be considered as a future work. Optimization: In all experiments, learning rate of 1e-4 and batch size of 64 are used. Adam algorithm with high momentum (β1 = 0.9, β2 = 0.999) is used as optimizer. High momentum is chosen mainly to let most of previous training samples influence the current update step. For reconstruction loss, mean square error, x−x 2, is used for all cases. As for initialization, since the model consists of convolutional layers with Leaky ReLu in both encoder and decoder, Xavier initialization is used. Thus, initial weights are drawn from a Gaussian distribution with standard deviation (stdev) of 2/N, where N is number of nodes from previous layer. For example, for a kernel size of 3x3 with 32 channels, N = 288, which in stdev of 0.083. Objective functions are shown in Table 3, where µ-VAE objective is written explicitly to avoid any ambiguity in terms of how batch statistics are computed. Table 4 shows model architecture as well as classifier used for all experiments. It consists of CNNbased encoder and decoder while classifier is three layer fully connected neural network. They all use Leaky Relu activation and learning rate of 1e-4.

This paper proposes a new  objective function to replace KL term with one that emulates maximum mean discrepancy (MMD) objective.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.