Question: Stochastic AUC maximization has garnered an increasing interest due to better fit to imbalanced data classification. However, existing works are limited to stochastic AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data. In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model. Building on the saddle point reformulation of a surrogated loss of AUC, the problem can be cast into a {\it non-convex concave} min-max problem. The main contribution made in this paper is to make stochastic AUC maximization more practical for deep neural networks and big data with theoretical insights as well. In particular, we propose to explore Polyak-\L{}ojasiewicz (PL) condition that has been proved and observed in deep learning, which enables us to develop new stochastic algorithms with even faster convergence rate and more practical step size scheme. An AdaGrad-style algorithm is also analyzed under the PL condition with adaptive convergence rate. Our experimental demonstrate the effectiveness of the proposed algorithms. Deep learning has been witnessed with tremendous success for various tasks, including computer vision (; ; ;), speech recognition (; ;), natural language processing (; ;), etc. From an optimization perspective, all of them are solving an empirical risk minimization problem in which the objective function is a surrogate loss of the prediction error made by a deep neural network in comparison with the ground-truth label. For example, for image classification task, the objective function is often chosen as the cross entropy between the probability distribution calculated by forward propagation of a convolutional neural network and the vector encoding true label information (; ;), where the cross entropy is a surrogate loss of the misclassification rate. However, when the data is imbalanced, this formulation is not reasonable since the data coming from minor class have little effect in this case and the model is almost determined by the data from the majority class. To address this issue, AUC maximization has been proposed as a new learning paradigm . Statistically, AUC (short for Area Under the ROC curve) is defined as the probability that the prediction score of a positive example is higher than that of a negative example (; 1983). Compared with misclassification rate and its corresponding surrogate loss, AUC is more suitable for imbalanced data setting . Several online or stochastic algorithms for time based on a new sampled/received training data. Instead of storing all examples in the memory, employ reservoir sampling technique to maintain representative samples in a buffer, based on which their algorithms update the model. To get optimal regret bound, their buffer size needs to be O(√ n), where n is the number of received training examples. design a new algorithm which is not buffer-based. Instead, their algorithm needs to maintain the first-order and second-order statistics of the received data to compute the stochastic gradient, which is prohibitive for high dimensional data. Based on a novel saddle-point reformulation of a surrogate loss of AUC proposed by , there are several studies (; ;) trying to design stochastic primal-dual algorithms. employ the classical primal-dual stochastic gradient and obtain O(1/ √ t) convergence rate. add a strongly convex regularizer, invoke composite mirror descent and achieve O(1/t) convergence rate. leverage the structure of the formulation, design a multi-stage algorithm and achieve O(1/t) convergence rate without strong convexity assumptions. However, all of them only consider learning a linear model, which in a convex objective function. Non-Convex Min-max Optimization. Stochastic optimization of non-convex min-max problems have received increasing interests recently (; ; ; ;). When the objective function is weakly convex in the primal variable and is concave in the dual variable, design a proximal guided algorithm in spirit of the inexact proximal point method , which solves a sequence of convexconcave subproblems constructed by adding a quadratic proximal term in the primal variable with a periodically updated reference point. Due to the potential non-smoothness of objective function, they show the convergence to a nearly-stationary point for the equivalent minimization problem. In the same vein as , design an algorithm by adopting the block alternating minimization/maximization strategy and show the convergence in terms of the proximal gradient. When the objective is weakly convex and weakly concave, propose a proximal algorithm which solves a strongly monotone variational inequality in each epoch and establish its convergence to stationary point. consider non-convex non-concave min-max games where the inner maximization problem satisfies a PL condition, based on which they design a multi-step deterministic gradient descent ascent with convergence to a stationary point. It is notable that our work is different in that (i) we explore the PL condition for the outer minimization problem instead of the inner maximization problem; (ii) we focus on designing stochastic algorithms instead of deterministic algorithms. Leveraging PL Condition for Minimization. PL condition is first introduced by Polyak , which shows that gradient descent is able to enjoy linear convergence to a global minimum under this condition. show that stochastic gradient descent, randomized coordinate descent, greedy coordinate descent are able to converge to a global minimum with faster rates under the PL condition. If the objective function has a finite-sum structure and satisfies PL condition, there are several non-convex SVRG-style algorithms (; ; ; ; ;), which are guaranteed to converge to a global minimum with a linear convergence rate. However, the stochastic algorithms in these works are developed for a minimization problem, and hence is not applicable to the min-max formulation for stochastic AUC maximization. To the best of our knowledge, is the only work that leverages an equivalent condition to the PL condition (namely quadratic growth condition) to develop a stochastic primal-dual algorithm for AUC maximization with a fast rate. However, as mentioned before their algorithm and analysis rely on the convexity of the objective function, which does not hold for AUC maximization with a deep neural network. Finally, we notice that PL condition is the key to many recent works in deep learning for showing there is no spurious local minima or for showing global convergence of gradient descent and stochastic gradient descent methods (; ; ; ; b; a; ; ; ;). Using the square loss, it has also been proved that the PL condition holds globally or locally for deep linear residual network , deep linear network, one hidden layer neural network with Leaky ReLU activation . Several studies (; ; ; b;) consider the trajectory of (stochastic) gradient descent on learning neural networks, and their analysis imply the PL condition in a certain form. For example, Du et al. (2018b) show that when the width of a two layer neural network is sufficiently large, a global optimum would lie in the ball centered at the initial solution, in which PL condition holds. extends this insight further to overparameterized deep neural networks with ReLU activation, and show that the PL condition holds for a global minimum around a random initial solution. Let · denote the Euclidean norm. A function f (x) is ρ-weakly convex if f (x) + ρ 2 x 2 is convex, where ρ is the so-called weak-convexity parameter. A function f (x) satisfies PL condition with 2, where x * stands for the optimal solution of f. Let z = (x, y) ∼ P denote a random data following an unknown distribution P, where x ∈ X represents the feature vector and y ∈ Y = {−1, +1} represents the label. Denote by Z = X × Y and by p = Pr(y = 1) = E y I [y=1], where I(·) is the indicator function. The area under the curve (AUC) on a population level for a scoring function h: X → R is defined as AUC(h) = Pr (h(x) ≥ h(x)|y = 1, y = −1), where z = (x, y) and z = (x, y) are drawn independently from P. By employing the squared loss as the surrogate for the indicator function that is a common choice used by previous studies , the AUC maximization problem can be formulated as min where H denotes a hypothesis class. All previous works of AUC maximization assume h(x) = w x for simplicity. Instead, we consider learning a general nonlinear model parameterized by w, i.e. h(w; x), which is not necessarily linear or convex in terms of w (e.g., h(w; x) can be a score function defined by a neural network with weights denoted by w). Hence, the corresponding optimization problem becomes min The following proposition converts the original optimization problem into a saddle-point problem, which is similar to Theorem 1 in . For completeness, the proof is included in the supplement. Proposition 1. The optimization problem is equivalent to min where z = (x, y) ∼ P, and Remark: It is notable that the min-max formulation is more favorable than the original formulation for developing a stochastic algorithm that updates the model parameters based on one example or a mini-batch of samples. For stochastic optimization of, one has to carefully sample both positive and negative examples, which is not allowed in an online setting. It is notable that in the classical batch-learning setting, p becomes the ratio of positive training examples and the expectation in becomes average over n individual functions. However, our algorithms are applicable to both batch-learning setting and online learning setting. It is clear that min w P (w) = min v φ(v) and P (w) ≤ φ(v) for any v = (w, a, b). The following assumption is made throughout the paper., where µ > 0 and v * is the optimal solution where v * is the global minimum of φ. The first condition is inspired by a PL condition on the objective function P (w) for learning a deep neural network. The following lemma establishes the connection. Algorithm 1 Proximally Guided Algorithm (PGA) 1: Initializev 0 = 0 ∈ R d+2,ᾱ 0 = 0, the global index j = 0 2: for k = 1,..., K do 3: 10: end for 11: Sample τ uniformly randomly from {1, . . ., K} 12: returnv τ,ᾱ τ Lemma 1. Suppose ∇ w h(w; x) ≤L for all w and x. If P (w) satisfies PL condition, i.e. there exists Remark: The PL condition of P (w) could be proved for learning a neural network similar to existing studies, which is not the main focus of this paper. Nevertheless, In Appendix A.7, we provide an example for AUC maximization with one-hidden layer neural network. Warmup. We first discuss the algorithms and their convergence of applied to the considered min-max problem. They have algorithms for problems in batch-learning setting and online learning setting. Since the algorithms for the batch-learning setting have complexities scaling with n, we will concentrate on the algorithm for the online learning setting. The algorithm is presented in Algorithm 1, which is a direct application of Algorithm 2 of to an online setting. Since their analysis requires the domain of the primal and the dual variable to be bounded, hence we add a ball constraint on the primal variable and the dual variable as well. As long as R 1 and R 2 is sufficiently large, they should not affect the solution. The convergence of Algorithm 1 is stated below. Remark: Under the condition φ(v) is smooth and the returned solution is within the added bounded ball constraint, the above implies We can see that this complexity under the PL condition of φ(v) is worse than the typical complexity of stochastic gradient descent method under the PL condition (i.e., O(1/)) . It remains an open problem how to design a stochastic primal-dual algorithm for solving min v max α F (v, α) in order to achieve a complexity of O(1/) in terms of minimizing φ(v). A naive idea is to solve the inner maximization problem of α first and the use SGD on the primal variable v. However, this is not viable since exact maximization over α is a non-trivial task. In this section, we present two primal-dual algorithms for solving the min-max optimization problem with corresponding theoretical convergence . For simplicity, we first assume the positive ratio p is known in advance, which is true in the batch-learning setting. Handling the unknown p in an online learning setting is a simple extension, which will be discussed in Section 4.3. The proposed algorithms follow the same proximal point framework proposed in , i.e., we Draw a minibatch {z j, . . ., solve the following convex-concave problems approximately and iteratively: where γ < 1/L to ensure that the new objective function becomes convex and concave, and v 0 is periodically updated. Algorithm 2. Similar to Algorithm 1, it has a nested loop, where the inner loop is to approximately solve a regularized min-max optimization problem using stochastic primal-dual gradient method, and the outer loop updates the reference point and learning rate. One key difference is that PPD-SG uses a geometrically decaying step size scheme, while Algorithm 1 uses a polynomially decaying step size scheme. Another key difference is that at the end of k-th outer loop, we update the dual variableᾱ k in Step 12, which is motivated by its closed-form solution givenv k. In particular, the givenv k, the dual solution that optimizes the inner maximization problem is given by: In the algorithm, we only use a small number of samples in Step 11 to compute an estimation of the optimal α givenv k. These differences are important for us to achieve lower iteration complexity of PPD-SG. Next, we present our convergence of PPD-SG. where E k−1 stands for the conditional expectation conditioning on all the stochastic events until v k−1 is generated. Theorem 2. Suppose the same conditions in Lemma 2 hold. Set 2 ) Lη0 in Algorithm 2, where O(·) hides logarithmic factor of L, µ,, G, σ. Remark: The above complexity is similar to that of for solving nonconvex minimization problem under the PL condition up to a logarithmic factor. Compared with the complexity of Algorithm 1 discussed earlier, i.e., O(1/(µ 3 3)), the above complexity in the ) is much better -it not only improves the dependence on but also improves the dependence on µ. Our second algorithm named Proximal Primal-Dual Adagrad (PPD-Adagrad) is a AdaGrad-style algorithm. Since it only differs from PPD-SG in the updates of the inner loop, we only present the inner loop in Algorithm 3. The updates in the inner loop are similar to the adaptive updates of traditional AdaGrad . We aim to achieve an adaptive convergence by using PPD-AdaGrad. The analysis of PPD-AdaGrad is inspired by the analysis of AdaGrad for non-convex minimization problems . The key difference is that we have to carefully deal with the primal-dual updates for the non-convex min-max problem. We summarize the convergence of PPD-AdaGrad below. where E k−1 stands for the conditional expectation conditioning on all the stochastic events until v k−1 is generated. Theorem 3. Suppose the same conditions as in Lemma 3 hold. Set The number of iterations is at most O, and the required number of samples is at, where O(·) hides logarithmic factors of L, µ,, δ. Remark: When the cumulative growth of stochastic gradient is slow, i.e., α < 1/2, the number of iterations is less than that in Theorem 2, which exhibits adaptive iteration complexity. It is notable that the setting of η k, T k, m k depends on unknown parameters µ, L, etc., which are typically unknown. One heuristic to address this issue is that we can decrease η k by a constant factor larger than 1 (e.g., 2 or 5 or 10), and similarly increase T k and m k by a constant factor. Another heuristic is to decrease the step size by a constant factor when the performance on a validation data saturates . Variants when p is unknown. In the online learning setting when p is unknown, the stochastic gradients of f in both v and α are not directly available. To address this issue, we can keep unbiased estimators for both p and p(1 − p) which are independent of the new arrived data, and update these estimators during the optimization procedure. All values depending on p and p(1 − p) (i.e., F, g v, g α) are estimated by substituting p and p(1 − p) by p and p(1 − p) (i.e.,F,ĝ v,ĝ α) respectively. The approach for keeping unbiased estimatorp and p(1 − p) during the optimization is described in Algorithm 4, where j is the global index, and m is the number of examples received. Extensions to multi-class problems. In the previous analysis, we only consider the binary classification problem. We can extend it to the multi-class setting. To this end, we first introduce the definition of AUC in this setting according to . Suppose there are c classes, we have c scoring functions for each class, namely h(w 1 ; x),..., h(w c ; x). We assume that these scores are normalized such that c k=1 h(w c ; x) = 1. Note that if these functions are implemented by a deep neural network, they can share the lower layers and have individual last layer of connections. The AUC is defined as Similar to Proposition 1, we can cast the problem into where ij. Then we can modify our algorithms to accommodate the multiple class pairs. We can also add another level of sampling of class pairs into computing the stochastic gradients. In this section, we present some empirical to verify the effectiveness of the proposed algorithms. We compare our algorithms (PPD-SG and PPD-AdaGrad) with three baseline methods including PGA (Algorithm 1), Online AUC method (OAUC) that directly employs the standard primal-dual stochastic gradient method with a decreasing step size for solving the min-max formulation, and the standard stochastic gradient descent (SGD) for minimizing cross-entropy loss. Comparing with PGA and OAUC allows us to verify the effectiveness of the proposed algorithms for solving the same formulation, and comparing with SGD allows us to verify the effectiveness of maximizing AUC for imbalanced data. We use a residual network with 20 layers to implement the deep neural network for all algorithms. We use the stagewise step size strategy as in for SGD, i.e. the step size is decreased by 10 times at 40K, 60K. For PPD-SG and PPD-AdaGrad, we set We conduct the comparisons on four benchmark datasets, i.e., Cat&Dog (C2), CIFAR10 (C10), CIFAR100 (C100), STL10. STL10 is an extension of CIFAR10 and the images are acquired from ImageNet. Cat&Dog is from Kaggle containing 25,000 images of dogs and cats and we choose an 80:20 split to construct training and testing set. We use 19k/1k, 45k/5k, 45k/5k, 4k/1k training/validation split on C2, C10, C100, and STL10 respectively. For each dataset, we construct multiple binary classification tasks with varying imbalanced ratio of number negative examples to number of positive examples. For details of construction of binary classification tasks, please refer to the Appendix A.8. We report the convergence of AUC on testing data in Figure 1, where the title shows the ratio of the majority class to the minority class. The about the convergence of AUC versus the time in seconds are also presented in Figure 3. From the we can see that for the balanced settings with ratio equal to 50%, SGD performs consistently better than other methods on C2 and CIFAR10 data. However, it is worse than AUC optimization based methods on CIFAR100 and STL10. For imbalanced settings, AUC maximization based methods are more advantageous than SGD in most cases. In addition, PPD-SG and PPD-AdaGrad are mostly better than other baseline algorithms. In certain cases, PPD-AdaGrad can be faster than PPD-SG. Finally, we observe even better performance (in Appendix) by a mixed strategy that pre-trains the model with SGD and then switchs to PPD-SG. In this paper, we consider stochastic AUC maximization problem when the predictive model is a deep neural network. By abuilding on the saddle point reformulation and exploring Polyak-Łojasiewicz condition in deep learning, we have proposed two algorithms with state-of-the-art complexities for stochastic AUC maximization problem. We have also demonstrated the efficiency of our proposed algorithms on several benchmark datasets, and the experimental indicate that our algorithms converge faster than other baselines. One may consider to extend the analysis techniques to other problems with the min-max formulation. Proof. It suffices to prove that Note that the optimal values of a, b, α are chosen as a * 2, (c) comes from the standard analysis of primal-dual stochastic gradient method. Denote E k−1 by taking the conditional expectation conditioning on all the stochastic events until v k−1 is generated. Taking E k−1 on both sides and noting thatĝ k t is an unbiased estimator of g k t for ∀t, k, we have By the update ofᾱ k−1, 2L-Lipschitz continuity of E [h(w; x)|y = −1] − E [h(w; x)|y = 1], and noting that α, then we have We can see that φ k (v) is convex and smooth function since γ ≤ 1/L. The smoothness parameter of φ k isL = L+γ −1. Define s k = arg min v∈R d+2 φ k (v). According to Theorem 2.1.5 of , we have Combining with Lemma 2 yields Note that φ k (v) is (γ −1 − L)-strongly convex, and γ = 1 2L, we have Plugging in s k into Lemma 2 and combining yield 2 ), rearranging the terms, and noting that Combining and yields Taking expectation on both sides over all randomness untilv k−1 is generated and by the tower property, we have is L-smooth and hence is L-weakly convex, so we have where (a) and (b) hold by the definition of φ k. Rearranging the terms in yields where (a) holds by using a, b ≤ 1 2 (a 2 + b 2), and (b) holds by the PL property of φ. Combining and, we can see that As a , we have Published as a conference paper at ICLR 2020 2 ), by the setting of η k, we set The required number of samples is A.4 PROOF OF LEMMA 3 2, (c) holds by Jensen's inequality. Now we bound I and II separately. Define Combining and, we have By Lemma 4 of and setting δ ≥ max t ĝ k t ∞, we know that T k 2, and hence Denote E k−1 by taking the conditional expectation conditioning on filtration F k−1, where F k−1 is the σ-algebra generated by all random variables untilv k−1 is generated. Taking E k−1 on both sides of, and employing yields where the equality holds sincev k−1 − s k is measurable with respect to F k−1. Note that where (By setting, then T k is a stopping time which is bounded almost surely. By stopping time argument, we have E k−1 (II) = 0, and hence A.5 PROOF OF THEOREM 3 We can see that φ k (v) is convex and smooth function since γ ≤ 1/L. The smoothness parameter of φ k isL = L+γ −1. Define s k = arg min v∈R d+2 φ k (v). According to Theorem 2.1.5 of , we have Combining with Lemma 3 yields Note that Plugging in s k into Lemma 3 and combining yield, rearranging the terms, and noting that Combining and yields Taking expectation on both sides over all randomness untilv k−1 is generated and by the tower property, we have Note that φ(v) is L-smooth and hence is L-weakly convex, so we have where (a) and (b) hold by the definition of φ k. Rearranging the terms in yields where (a) holds by using a, b ≤ 1 2 (a 2 + b 2), and (b) holds by the PL property of φ. Combining and, we can see that which implies that As a , we have, and note that when τ ≥ 1,, and hence, we can see that the total iteration complexity is. The required number of samples is A.6 PROOF OF LEMMA 1 Proof. For any fixed w, define (a * w, b * w) = arg min a,b φ(w, a, b) (φ(w, a, b) is strongly convex in terms of (a, b), so the argmin is well-defined and unique). Note that we can write σ(w x) and σ(w x) as aw x and bw x respectively, and it is obvious that a 2 ≥ min(c We construct the datasets in the following ways: For CIFAR10/STL10, we label the first 5 classes as negative ("-") class and the last 5 classes as positive ("+") class, which leads to a 50/50 class ratio. For CIFAR100, we label the first 50 classes as negative ("-") class and the last 50 classes as positve ("+") class. For the imbalanced cases, we randomly remove 90%, 80%, 60% data from negative samples on all training data, which lead to 91/9, 83/17, 71/29 ratio respectively. For testing data, we keep them unchanged. Model pretraining is effective in many deep learning tasks, and thus we further evaluate the performance of the proposed methods on pretrained models. We first train the model using SGD up to 2000 iterations with an initial step size of 0.1, and then continue training using PPD-SG. We denote this method as PPD-SG+pretrain and the are shown in Figure 2. The parameters are tuned in the same range as in Section 5. It is observed that pretraining model helps the convergence of model and it can achieve the better performance in terms of AUC in most cases. To investigate the effects of labeling order, we also attempt to randomly partition the classes as positive or negative equally. For CIFAR10 and STL10 dataset, we randomly partition the 10 classes into two labels (i.e., randomly select 5 classes as positive label and other 5 classes as negative label). For CIFAR100 dataset, we randomly partition the 100 classes into two labels (i.e., randomly select 50 classes as positive label and other 50 classes as negative label). After that we randomly remove 95%, 90%, from negative samples on all training data, which lead to 20:1, 10:1 ratios respectively. For testing data, we keep them unchanged. We also add AdaGrad for minimizing cross-entropy loss as a new baseline. The corresponding experimental are included in Figure 3. We can see that PPD-Adagrad and PPD-SG converge faster than other baselines. and STL10 dataset, we randomly partition the 10 classes into two labels (i.e., randomly select 5 classes as positive label and other 5 classes as negative label). For CIFAR100 dataset, we randomly partition the 100 classes into two labels (i.e., randomly select 50 classes as positive label and other 50 classes as negative label).

Represent the short summary for finding the corresponding scientific article (long)

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the article for finding a single-sentence TLDR.

Question: Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might take a very long time to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, our method can be used when only trajectories without expert actions are available, which can leverage kinestetic or third person demonstration. Reinforcement Learning (RL) has shown impressive in a plethora of simulated tasks, ranging from attaining super-human performance in video-games BID18 BID35 and board-games , to learning complex locomotion behaviors BID34 BID4. Nevertheless, these successes are shyly echoed in real world robotics (Riedmiller et BID36 . This is due to the difficulty of setting up the same learning environment that is enjoyed in simulation. One of the critical assumptions that are hard to obtain in the real world are the access to a reward function. Self-supervised methods have the power to overcome this limitation. A very versatile and reusable form of self-supervision for robotics is to learn how to reach any previously observed state upon demand. This problem can be formulated as training a goal-conditioned policy BID14 BID27 that seeks to obtain the indicator reward of having the observation exactly match the goal. Such a reward does not require any additional instrumentation of the environment beyond the sensors the robot already has. But in practice, this reward is never observed because in continuous spaces like the ones in robotics, the exact same observation is never observed twice. Luckily, if we are using an off-policy RL algorithm BID17 BID11, we can "relabel" a collected trajectory by replacing its goal by a state actually visited during that trajectory, therefore observing the indicator reward as often as we wish. This method was introduced as Hindsight Experience Replay BID0 or HER.In theory these approaches could learn how to reach any goal, but the breadth-first nature of the algorithm makes that some areas of the space take a long time to be learned BID7 . This is specially challenging when there are bottlenecks between different areas of the statespace, and random motion might not traverse them easily BID5 . Some practical examples of this are pick-and-place, or navigating narrow corridors between rooms, as illustrated in Fig. 5 in appendix depicting the diverse set of environments we work with. In both cases a specific state needs to be reached (grasp the object, or enter the corridor) before a whole new area of the space is discovered (placing the object, or visiting the next room). This problem could be addressed by engineering a reward that guides the agent towards the bottlenecks, but this defeats the purpose of trying to learn without direct reward supervision. In this work we study how to leverage a few demonstrations that traverse those bottlenecks to boost the learning of goal-reaching policies. Learning from Demonstrations, or Imitation Learning (IL), is a well-studied field in robotics BID15 BID25 BID2. In many cases it is easier to obtain a few demonstrations from an expert than to provide a good reward that describes the task. Most of the previous work on IL is centered around trajectory following, or doing a single task. Furthermore it is limited by the performance of the demonstrations, or relies on engineered rewards to improve upon them. In this work we study how IL methods can be extended to the goal-conditioned setting, and show that combined with techniques like HER it can outperform the demonstrator without the need of any additional reward. We also investigate how the different methods degrade when the trajectories of the expert become less optimal, or less abundant. Finally, the method we develop is able to leverage demonstrations that do not include the expert actions. This is very convenient in practical robotics where demonstrations might have been given by a motion planner, by kinestetic demonstrations (moving the agent externally, and not by actually actuating it), or even by another agent. To our knowledge, this is the first framework that can boost goal-conditioned policy learning with only state demonstrations. We define a discrete-time finite-horizon discounted Markov decision process (MDP) by a tuple M = (S, A, P, r, ρ 0, γ, H), where S is a state set, A is an action set, P: S × A × S → R + is a transition probability distribution, γ ∈ is a discount factor, and H is the horizon. Our objective is to find a stochastic policy π θ that maximizes the expected discounted reward within the DISPLAYFORM0 We denote by τ = (s 0, a 0, ...,) the entire state-action trajectory, where s 0 ∼ ρ 0 (s 0), a t ∼ π θ (a t |s t), and s t+1 ∼ P(s t+1 |s t, a t). In the goal-conditioned setting that we use here, the policy and the reward are also conditioned on a "goal" g ∈ S. The reward is r(s t, a t, s t+1, g) = 1 s t+1 == g, and hence the return is the γ h, where h is the number of time-steps to the goal. Given that the transition probability is not affected by the goal, g can be "relabeled" in hindsight, so a transition (s t, a t, s t+1, g, r = 0) can be treated as (s t, a t, s t+1, g = s t+1, r = 1). Finally, we also assume access to D trajectories (s DISPLAYFORM1 that were collected by an expert attempting to reach a goal g j sampled uniformly among the feasible goals. Those trajectories must be approximately geodesics, meaning that the actions are taken such that the goal is reached as fast as possible. In this section we describe the different algorithms we compare to pure Hindsight Experience Replay BID0 . See the Appendix to prior work on adding a Be- havioral Cloning loss to the policy update as in BID19 . Here we propose a novel expert relabeling technique, we formulate for the first time a goal-conditioned GAIL algorithm, and propose a method to train it with state-only demonstrations. The expert trajectories are collected by asking the expert to reach a specific goal g j . But they are also valid trajectories to reach any other state visited within the demonstration! This is the key motivating insight to propose a new type of relabeling: if we have the transitions (s DISPLAYFORM0 t+k) as also coming from the expert! This can be understood as a type of data augmentation leveraging the assumption that the tasks we work on are quasi-static. It will be particularly effective when not many demonstrations are available. In FIG2 we compare the final performance of two agents for Four Rooms environment, one trained with pure Behavioral Cloning, and the other one also using expert relabeling. The compounding error in Behavioral Cloning might make the policy deviate arbitrarily from the demonstrations, and it requires too many demonstrations when the state dimension increases. The first problem is less severe in our goalconditioned case because in fact we do want to visit and be able to purposefully reach all states, even the ones that the expert did not visited. But the second drawback will become pressing when attempting to scale this method to practical robotics tasks where the observations might be high-dimensional sensory input like images. Both problems can be mitigated by using other Imitation Learning algorithms that can leverage additional rollouts collected by the learning agent in a self-supervised manner, like GAIL BID13. In this section we extend the formulation of GAIL to tackle goal-conditioned tasks, and then we detail how it can be combined with HER BID0, which allows to outperform the demonstrator and generalize to all goals. We call this algorithm goal-GAIL.First of all, the discriminator needs to also be conditioned on the goal D ψ (a, s, g). Once the discriminator is fitted, we can run our favorite RL algorithm on the DISPLAYFORM0 In our case we used the offpolicy algorithm DDPG BID17 to allow for the relabeling techniques outlined above. In the goalconditioned case we also supplement with the indicator reward r DISPLAYFORM1 h. This combination is slightly tricky because now the fitted Q φ does not have the same clear interpretation it has when only one of the two rewards is used BID6. Nevertheless, both rewards are pushing the policy towards the goals, so it shouldn't be too conflicting. Furthermore, to avoid any drop in final performance, the weight of the reward coming from GAIL (δ GAIL) can be annealed. See Appendix for details. Both Behavioral Cloning and GAIL use state-action pairs from the expert. This limits the use of the methods, combined or not with HER, to setups where the exact same agent was actuated to reach different goals. Nevertheless, much more data could be cheaply available if the action was not required. For example, kinestetic demonstration or third-person imitation . The main insight we have here is that we can replace the action in the GAIL formulation by the next state s, and in most environments this should be as informative as having access to the action directly. Intuitively, given a desired goal g, it should be possible to determine if a transition s → s is taking the agent in the right direction. The loss function to train a discriminator able to tell apart the current agent and demonstrations (always transitioning towards the goal) is simply: DISPLAYFORM0 We are interested in answering the following questions:1. Can the use of demonstrations accelerate the learning of goal-conditioned tasks without reward? 2. Is the Expert Relabeling an efficient way of doing dataaugmentation on the demonstrations? We evaluate these questions in two different simulated robotic goal-conditioned tasks that are detailed in the next subsection. All the use 20 demonstrations. All curves have 5 random seeds and the shaded area is one standard deviation Experiments are conducted in two continuous environments in MuJoCo BID33. The performance metric we use in all our experiments is the percentage of goals in the feasible goal space the agent is able to reach. A point mass is placed in an environment with four rooms connected through small openings. The action space is continuous and specifies the desired change in state space which corresponds to the goal space. Pick and Place: A fetch robot needs to pick a block and place it in a desired point in space as described in BID19. The control is four-dimensional, corresponding to a change in position of the end-effector and a change in gripper opening. The goal space is the position of the block. In goal-conditioned tasks, HER BID0 should eventually converge to a policy able to reach any desired goal. Nevertheless, this might take a long time, specially in environments where there are bottlenecks that need to be traversed before accessing a whole new area of the goal space. In this section we show how the methods introduced in the previous section can leverage a few demonstrations to improve the convergence speed of HER. This was already studied for the case of Behavioral Cloning by BID19, and in this work we show we also get a benefit when using GAIL as the Imitation Learning algorithm. In both environments, we observe that running GAIL with relabeling (GAIL+HER) considerably outperforms running each of them in isolation. HER alone has a very slow convergence, although as expected it ends up reaching the same final performance if run long enough. On the other hand GAIL by itself learns fast at the beginning, but its final performance is capped. This is because despite collecting more samples on the environment, those come with no reward of any kind indicating what is the task to perform (reach the given goals). Therefore, once it has extracted all the information it can from the demonstrations it cannot keep learning and generalize to goals further from the demonstrations. This is not an issue anymore when combined with HER, as our show. Here we show that the Expert Relabeling technique introduced in Section 3.1 is beneficial in the goal-conditioned imitation learning framework. As shown in FIG4, our expert relabeling technique brings considerable performance boosts for both Behavioral Cloning methods and goal-GAIL in both environments. We also perform a further analysis of expert relabeling in the four-rooms environment. We see in FIG2 that without the expert relabeling, the agent fails to learn how to reach many intermediate states visited in the middle of a demonstration. Behavioral Cloning and standard GAIL rely on the stateaction (s, a) tuples from the expert. Nevertheless there are many cases in robotics where we only have access to observation-only demonstrations. In this section we want to emphasize that all the obtained with our goal-GAIL method and reported in FIG3 and FIG4 do not require actions that the expert took. Surprisingly, in the four rooms environment, despite the more restricted information goal-GAIL has access to, it outperforms BC combined with HER. This might be due to the superior imitation learning performance of GAIL, and also to the fact that these tasks might be possible to solve by only matching the state-distribution of the expert. With GAIL conditioned only on current state but not action (as also done in other non-goal-conditioned works BID8), we observe that the discriminator learns a very well shaped reward that encourages the agent to go towards the goal, as pictured in Fig. 6 in appendix. See the Appendix for more details. In the above sections we assumed access to optimal experts. Nevertheless, in practical applications the experts might have a more erratic behavior. In this section we study how the different methods perform with a sub-optimal expert. To do so we collect trajectories attempting goals g by modifying our optimal expert π * (a|s, g) in two ways: We add noise α to the optimal actions and make it be -greedy. The sub-optimal expert is then a = 1[DISPLAYFORM0) and u is a uniformly sampled random action. In FIG5 we observe that approaches that copy the action of the expert, like Behavioral Cloning, greatly suffer under a sub-optimal expert. On the other hand, discriminator-based methods are able to leverage noisier experts. A possible explanation is that a discriminator approach can give a positive signal as long as the transition is "in the right direction", without trying to exactly enforce a single action. Under this lens, having some noise in the expert might actually improve the performance of these adversarial approaches, as it has been observed in many generative models literature (Goodfellow et al.). Hindsight relabeling can be used to learn useful behaviors without any reward supervision for goal-conditioned tasks, but they are inefficient when the state-space is large or includes exploration bottlenecks. In this work we show how only a few demonstrations can be leveraged to improve the convergence speed of these methods. We introduce a novel algorithm, goal-GAIL, that converges faster than HER and to a better final performance than a naive goal-conditioned GAIL. We also study the effect of doing expert relabeling as a type of data augmentation on the provided demonstrations, and demonstrate it improves the performance of our goal-GAIL as well as goal-conditioned Behavioral Cloning. We emphasize that our goal-GAIL method only needs state demonstrations, without using expert actions like other Behavioral Cloning methods. Finally, we show that goal-GAIL is robust to sub-optimalities in the expert behavior. Imitation Learning can be seen as an alternative to reward crafting to train desired behaviors. There are many ways to leverage demonstrations, from Behavioral Cloning BID22 ) that directly maximizes the likelihood of the expert actions under the training agent policy, to Inverse Reinforcement Learning that extracts a reward function from those demonstrations and then trains a policy to maximize it BID38 BID3 BID8. Another formulation close to the later introduced by BID13 is Generative Adversarial Imitation Learning (GAIL), explained in details in the next section. Originally, the algorithms used to optimize the policy were on-policy methods like Trust Region Policy Optimization BID29, but recently there has been a wake of works leveraging the efficiency of off-policy algorithms without loss in stability BID1 BID26 BID28 BID16. This is a key capability that we are going to exploit later on. Unfortunately most work in the field cannot outperform the expert, unless another reward is available during training BID34 BID9 BID32, which might defeat the purpose of using demonstrations in the first place. Furthermore, most tasks tackled with these methods consist on tracking expert state trajectories BID37 BID21, but can't adapt to unseen situations. In this work we are interested in goal-conditioned tasks, where the objective is to be able to reach any state upon demand. This kind of multi-task learning are pervasive in robotics, but challenging if no reward-shaping is applied. Relabeling methods like Hindsight Experience Replay BID0 unlock the learning even in the sparse reward case BID6. Nevertheless, the inherent breath-first nature of the algorithm might still make very inefficient learning to learn complex policies. To overcome the exploration issue we investigate the effect of leveraging a few demonstrations. The closest prior work is by BID19, where a Behavioral Cloning loss is used with a Q-filter. We found that a simple annealing of the Behavioral Cloning loss BID23 works better. Furthermore, we also introduce a new relabeling technique of the expert trajectories that is particularly useful when only few demonstrations are available. We also experiment with Goal-conditioned GAIL, leveraging the recently shown compatibility with off-policy algorithms. For a more comprehensive review of related work, please see Appendix. The most direct way to leverage demonstrations DISPLAYFORM0 is to construct a data-set D of all state-action-goal tuples (s j t, a j t, g j), and run a supervised regression algorithm. In the goal-conditioned case and assuming a deterministic policy π θ (s, g), the loss is: DISPLAYFORM1 This loss and its gradient are computed without any additional environments samples from the trained policy π θ. This makes it particularly convenient to combine a gradient descend step based on this loss with other policy updates. In particular we can use a standard offpolicy Reinforcement Learning algorithm like DDPG BID17, where we fit the Q φ (a, s, g), and then estimate the gradient of the expected return as: DISPLAYFORM2 ). The improvement guarantees with respect to the task reward are lost when we combine the BC and the deterministic policy gradient updates, but this can be side-stepped by either applying a Q-filter: 1 Q(s t, a t, g) > Q(s t, π(s t, g), g) to the BC loss as proposed in BID19, or by annealing it as we do in our experiments, which allows the agent to eventually outperform the expert. All possible variants we study are detailed in Algorithm 1 as presented in appendix. In particular, α = 0 falls back to pure Behavioral Cloning, β = 0 removes the BC component, p = 0 doesn't relabel agent trajectories, δ GAIL = 0 removes the discriminator output from the reward, and EX-PERT RELABEL indicates whether the here explained expert relabeling should be performed. In the two environments, i.e. Four Rooms environment and Fetch Pick & Place, the task horizons are set to 300 and 100 respectively. The discount factors are γ = 1 − 1 H. In all experiments, the Q function, policy and discriminator are paramaterized by fully connected neural networks with two hidden layers of size 256. DDPG is used for policy optimization and hindsight probability is set to p = 0.8. The initial value of the behavior cloning loss weight β is set to 0.1 and is annealed by 0.9 per 250 rollouts collected. The initial value of the discriminator reward weight δ GAIL is set to 0.1. We found empirically that there is no need to anneal δ GAIL.t, g h ))27: Anneal δ GAIL and β 28: end while For experiments with sub-optimal expert in section 4.5, is set to 0.4 and 0.5, and σ α is set to 1.5 and 0.3 respectively for Four Rooms environment and Fetch Pick & Place. We trained the discriminator in three settings:• current state and goal: (s, g)• current state, next state and goal: (s, s, g)• current state, action and goal: (s, a, g)We compare the three different setups in Fig. 7 and 8.

We tackle goal-conditioned tasks by combining Hindsight Experience Replay and Imitation Learning algorithms, showing faster convergence than the first and higher final performance than the second.

Represent the TLDR for finding the corresponding scientific article

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.


Question:
represent the scientific article to find a short tldr!
E.g. "Zero-Shot Learning (ZSL) is a classification task where some classes referred as unseen classes have no labeled training images. Instead, we only have side information (or description) about seen and unseen classes, often in the form of semantic or descriptive attributes. Lack of training images from a set of classes restricts the use of standard classification techniques and losses, including the popular cross-entropy loss. The key step in tackling ZSL problem is bridging visual to semantic space via learning a nonlinear embedding. A well established approach is to obtain the semantic representation of the visual information and perform classification in the semantic space. In this paper, we propose a novel architecture of casting ZSL as a fully connected neural-network with cross-entropy loss to embed visual space to semantic space. During training in order to introduce unseen visual information to the network, we utilize soft-labeling based on semantic similarities between seen and unseen classes. To the best of our knowledge, such similarity based soft-labeling is not explored for cross-modal transfer and ZSL. We evaluate the proposed model on five benchmark datasets for zero-shot learning, AwA1, AwA2, aPY, SUN and CUB datasets, and show that, despite the simplicity, our approach achieves the state-of-the-art performance in Generalized-ZSL setting on all of these datasets and outperforms the state-of-the-art for some datasets. Supervised classifiers, specifically Deep Neural Networks, need a large number of labeled samples to perform well. Deep learning frameworks are known to have limitations in fine-grained classification regime and detecting object categories with no labeled data; ). On the contrary, humans can recognize new classes using their previous knowledge. This power is due to the ability of humans to transfer their prior knowledge to recognize new objects . Zero-shot learning aims to achieve this human-like capability for learning algorithms, which naturally reduces the burden of labeling. In zero-shot learning problem, there are no training samples available for a set of classes, referred to as unseen classes. Instead, semantic information (in the form of visual attributes or textual features) is available for unseen classes (; 2014). Besides, we have standard supervised training data for a different set of classes, referred to as seen classes along with the semantic information of seen classes. The key to solving zero-shot learning problem is to leverage trained classifier on seen classes to predict unseen classes by transferring knowledge analogous to humans. Early variants of ZSL assume that during inference, samples are only from unseen classes. Recent observations; realize that such an assumption is not realistic. Generalized ZSL (GZSL) addresses this concern and considers a more practical variant. In GZSL there is no restriction on seen and unseen classes during inference. We are required to discriminate between all the classes. Clearly, GZSL is more challenging because the trained classifier is generally biased toward seen classes. In order to create a bridge between visual space and semantic attribute space, some methods utilize embedding techniques (; ; ; ; ; ; ; ; ; ; ;) and the others use semantic similarity between seen and unseen classes . Semantic similarity based models represent each unseen class as a mixture of seen classes. While the embedding based models follow three various directions; mapping visual space to semantic space (; ; ; ; ;), mapping semantic space to the visual space (; ;), and finding a latent space then mapping both visual and semantic space into the joint embedding space;;; ). The loss functions in embedding based models have training samples only from the seen classes. For unseen classes, we do not have any samples. It is not difficult to see that this lack of training samples biases the learning process towards seen classes only. One of the recently proposed techniques to address this issue is augmenting the loss function with some unsupervised regularization such as entropy minimization over the unseen classes. Another recent methodology which follows a different perspective is deploying Generative Adversarial Network (GAN) to generate synthetic samples for unseen classes by utilizing their attribute information; ). Although generative models boost the significantly, it is difficult to train these models. Furthermore, the training requires generation of large number of samples followed by training on a much larger augmented data which hurts their scalability. The two most recent state-of-the-art GZSL methods, CRnet and COSMO , both employ a complex mixture of experts approach. CRnet is based on k-means clustering with an expert module on each cluster (seen class) to map semantic space to visual space. The output of experts (cooperation modules) are integrated and finally sent to a complex loss (relation module) to make a decision. CRnet is a multi-module (multi-network) method that needs end-to-end training with many hyperparameters. Also COSMO is a complex gating model with three modules: a seen/unseen classifier and two expert classifiers over seen and unseen classes. Both of these methods have many modules, and hence, several hyperparameters; architectural, and learning decisions. A complex pipeline is susceptible to errors, for example, CRnet uses k-means clustering for training and determining the number of experts and a weak clustering will lead to bad . Our Contribution: We propose a simple fully connected neural network architecture with unified (both seen and unseen classes together) cross-entropy loss along with soft-labeling. Soft-labeling is the key novelty of our approach which enables the training data from the seen classes to also train the unseen class. We directly use attribute similarity information between the correct seen class and the unseen classes to create a soft unseen label for each training data. As a of soft labeling, training instances for seen classes also serve as soft training instance for the unseen class without increasing the training corpus. This soft labeling leads to implicit supervision for the unseen classes that eliminates the need for any unsupervised regularization such as entropy loss in. Soft-labeling along with crossentropy loss enables a simple MLP network to tackle GZSL problem. Our proposed model, which we call Soft-labeled ZSL (SZSL), is simple (unlike GANs) and efficient (unlike visual-semantic pairwise embedding models) approach which achieves the state-of-the-art performance in Generalized-ZSL setting on all five ZSL benchmark datasets and outperforms the state-of-the-art for some of them. In zero-shot learning problem, a set of training data on seen classes and a set of semantic information (attributes) on both seen and unseen classes are given. The training dataset includes n samples where x i is the visual feature vector of the i-th image and y i is the class label. All samples in D belong to seen classes S and during training there is no sample available from unseen classes U. The total number of classes is C = |S| + |U|. Semantic information or attributes a k ∈ R a, are given for all C classes and the collection of all attributes are represented by attribute matrix A ∈ R a×C. In the inference phase, our objective is to predict the correct classes (either seen or unseen) of the test dataset D. The classic ZSL setting assumes that all test samples in D belong to unseen classes U and tries to classify test samples only to unseen classes U. While in a more realistic setting i.e. GZSL, there is no such an assumption and we aim at classifying samples in D to either seen or unseen classes S ∪ U. 3 PROPOSED METHODOLOGY 3.1 NETWORK ARCHITECTURE As Figure 1 illustrates our architecture, We map visual space to semantic space, then compute the similarity score (dot-product) between true attributes and the attribute/semantic representation of the input (x). Finally, the similarity score is fed into a Softmax, and the probability of all classes are computed. For the visual features as the input, in all five benchmark datasets, we use the extracted visual features by a pre-trained ResNet-101 on ImageNet provided by. We do not fine-tune the CNN that generates the visual features unlike model in. In this sense, our proposed model is also fast and straightforward to train. In ZSL problem, we do not have any training instance from unseen classes, so the output nodes corresponding to unseen classes are always inactive during learning. Standard supervised training with cross entropy loss biases the network towards seen classes only. The true labels (hard labels) used for training only represent seen classes so the cross entropy cannot penalize unseen classes. Moreover, the available similarity information between the seen and unseen attributed is never utilized. We propose soft labeling based on the similarity between semantic attributes. For each seen sample, we represent its relationship to unseen categories by obtaining semantic similarity (dot-product) using the seen class attribute and all the unseen class attributes. In the simplest form, for every training data, we can find the nearest unseen class to the correct seen class label and assign a small probability q (partial membership or soft label) of this instance to be from the closest unseen class. Note, each training sample only contains a label which comes from the set of seen classes. With soft labeling, we enrich the label with partial assignments to unseen classes and as shows, soft labels act as a regularizer which allows each training case to enforce much more constraint on weights. In a more general soft labeling approach, we propose assigning a probability to all the unseen classes. A natural choice is to transform seen-to-unseen similarities to probabilities (soft labels) shown in Equation. The unseen distribution is obtained for each seen class by calculating dot-product of seen class attribute and all unseen classes attributes and squashing all these dot-product values by Softmax to acquire probabilities. In this case, we distribute the probability q among all unseen classes based on the obtained unseen distribution. This proposed strategy in a soft label for each seen image during training, which as we show later helps the network to learn unseen categories. In order to control the flatness of the unseen distribution, we utilize temperature parameter τ. Higher temperature in flatter distribution over unseen categories and lower temperature creates a more ragged distribution with peaks on nearest unseen classes. A small enough temperature basically in the nearest unseen approach. The Impact of temperature τ on unseen distribution is depicted in Figure 3.a for a particular seen class. Soft labeling implicitly introduces unseen visual features into the network without generating fake unseen samples as in generative methods; ). Hence our proposed approach is able to reproduce same effect as in generative models without the need to create fake samples and train generative models that are known to be difficult to train. Below is the formal description of temperature Softmax: where a i is the i-th column of attribute matrix A ∈ R a×C which includes both seen and unseen class attributes: And s i,j is the true similarity score between two classes i, j based on their attributes. τ and q are temperature parameter and total probability assigned to unseen distribution, respectively. Also y u i,k is the soft label (probability) of unseen class k for seen class i. It should be noted that q is the sum of all unseen soft labels i.e. k∈U y u i,k = q. The proposed method is a multi-class probabilistic classifier that produces a C-dimensional vector of class probabilities p for each sample x i as p(C-dimensional vector of all similarity scores of an input sample. The predicted similarity score between semantic representation of sample x i and attribute a k isŝ i,k g w (x i), a k. Each element of vector p, represents an individual class probability that can be shown below: This Softmax as the activation function of the last layer of the network is calculated on all classes. An established choice to train a multi-class probabilistic classifier is the cross-entropy loss which we later show naturally integrates our idea of soft labeling. Inspired by , in addition to the cross-entropy loss over soft targets, we also consider cross entropy-loss over true labels (hard labels) to improve the performance. During training, we aim at learning the nonlinear mapping g w i.e. obtaining network weights W through: where λ and γ are regularization factors which are obtained through hyperparameter tuning, and L(x i) is the weighted sum of cross-entropy loss over soft labels (L soft) and cross-entropy loss over hard labels (L hard) for each sample as shown below: where α ∈ is a hyperparameter. For better understanding, the hard-loss and soft-loss terms for each sample x i (or x for simplicity) are expanded and elaborated. The hard-loss term is a conventional cross-entropy loss L hard (x) = − C k=1 z k log(p k), where z k is the hard label. Clearly, hard-loss term alone does not work in ZSL regime since it does not penalize unseen classes. The soft-loss term is expanded to seen and unseen terms as follows: Utilizing Equation Hence the first two terms of L soft (x) is the weighted sum of cross-entropy of seen classes and crossentropy of unseen classes. In particular, first term penalizes and controls the relative (normalized) probabilities within all seen classes and the second term acts similarly within unseen classes. We also require to penalize the total probability of all seen classes (1 −q) and total probability of all unseen classes (q). This is accomplished through the last two terms of Equation which is basically a binary cross entropy loss. Intuitively soft-loss in Equation works by controlling the balance within seen/unseen classes (first two terms) as well as the balance between seen and unseen classes (last two terms). As we have shown in Equation, soft-loss enables the classifier to learn unseen classes by only being exposed to samples from seen classes. Hyperparameter q acts as a trade-off coefficient between seen and unseen cross-entropy losses (Figure 2). We can see that the regularizer is a weighted cross entropy on unseen class, which leverages similarity structure between attributes compared to uniform entropy function of DCN. DCN and all prior works use uniform entropy as regularizer, which does not capitalize on the known semantic similarity information between seen and unseen class attributes. At the inference time, our proposed SZSL method works the same as a conventional classifier, we only need to provide the test image and the network will produce class probabilities for all seen and unseen classes. We conduct comprehensive comparison of our proposed SZSL model with the state-of-the-art methods for GZSL settings on five benchmark datasets (Table 1). We present the detailed description of datasets in Appendix A. Our model outperforms the state-of-the-art methods on GZSL setting for all benchmark datasets. For the purpose of validation, we employ the validation splits provided along with the Proposed Split (PS) to perform cross-validation for hyper-parameter tuning. The main objective of GZSL is to simultaneously improve seen samples accuracy and unseen samples accuracy i.e. imposing a trade-off between these two metrics. As the , the standard GZSL evaluation metric is harmonic average of seen and unseen accuracy. This metric is chosen to encourage the network not be biased toward seen classes. Harmonic average of accuracies is defined as where A S and A U are seen and unseen accuracies, respectively. 11.3 74.6 19.6 8.0 73.9 14.4 3.7 55.7 6.9 23.5 59.2 33.6 14.7 30.5 19.8 ConSE 0.4 88.6 0.8 0.5 90.6 1.0 0.0 91.2 0.0 1.6 72.2 3.1 6.8 39.9 11.6 Sync 8.9 87.3 16.2 10.0 90.5 18.0 7.4 66.3 13.3 11.5 70.9 19.8 7.9 43.3 13.4 DeViSE 13.4 68.7 22.4 17.1 74.7 27.8 4.9 76.9 9.2 23.8 53.0 32.8 16.9 27.4 20.9 CMT 0.9 87.6 1.8 8.7 89.0 15.9 1.4 85.2 2.8 7.2 49.8 12.6 8.1 21.8 11.8 Generative Models f-CLSWGAN 57.9 61.4 59.6 ------43.7 57.7 49.7 42.6 36.6 39.4 SP-AEN 23.3 90.9 37.1 ---13.7 63.4 13.7 34.7 70.6 46.6 24.9 38.6 30.3 cycle-UWGAN 59.6 63.4 59. To evaluate SZSL, we follow the popular experimental framework and the Proposed Split (PS) in for splitting classes into seen and unseen classes to compare GZSL/ZSL methods. Utilizing PS ensures that none of the unseen classes have been used in the training of ResNet-101 on ImageNet. The input to the model is the visual features of each image sample extracted by a pre-trained ResNet-101 on ImageNet provided by. The dimension of visual features is 2048. We utilized Keras with TensorFlow back-end to implement our model. We used proposed unseen classes for validation (3-fold CV) and added 20% of train samples (seen classes) as seen validation samples to obtain GZSL validation sets. We crossvalidate τ ∈ [10 −2, 10], mini-batch size ∈ {64, 128, 256, 512, 1024}, q ∈, α ∈, hidden layer size ∈ {128, 256, 512, 1024, 1500} and activation function ∈{tanh, sigmoid, hard-sigmoid, relu} to tune our model. To obtain statistically consistent , the reported accuracies are averaged over 5 trials (using different initialization) after tuning hyper-parameters with cross-validation. Also we ran our experiments on a machine with 56 vCPU cores, Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHZ and 2 NVIDIA-Tesla P100 GPUs each with 16GB memory. The code is provided in the supplementary material. To demonstrate the effectiveness of SZSL model in GZSL setting, we comprehensively compare our proposed method with state-of-the-art GZSL models in Table 2. Since we use the standard proposed split, the published of other GZSL models are directly comparable. As reported in Table 2, accuracies of our model achieves the state-of-the-art GZSL performance on all five benchmark datasets and outperforms the state-of-the-art on AwA2 and aPY datasets. It is exciting and motivating while our architecture is much simpler compared to recently proposed CRnet and COSMO, yet, we achieve similar or better accuracies compared to them. We have only one simple fully connected neural network with 2 trainable layers, compared to CRnet with K mixture of experts followed by relation module with complex loss functions (pairwsie). Soft labeling employed in SZSL gives the model new flexibility to trade-off between seen and unseen accuracies during training and attain a higher value of harmonic accuracy A H, which is the standard metric for GZSL. Assigned unseen soft labels (unseen probability q) enables the classifier to gain more confidence in recognizing unseen classes, which in turn in considerably higher unseen accuracy A U. As the classifier is now discriminating between more classes we get marginally lower seen accuracy A S. However, balancing A S and A U with the cost of deteriorating A S leads to much higher A H. This trade-off phenomenon is depicted in Figure 2 for all datasets. The flexibility provided by soft labeling is examined by obtaining accuracies for different values of q. In Figure 2.a and 2.b, by increasing total unseen probability q, A U increases and A S decreases as expected. From the trade-off curves, there is an optimal q where A H takes its maximum value as shown in Figure 2. Maximizing A H is the primary objective in a GZSL problem that can be achieved by semantic similarity based soft labeling and the trade-off knob, q. It should be noted that both AwA and aPY datasets (Figure 2 .a and 2.b) are coarse-grained class datasets. In contrast, CUB and SUN datasets are fine-grained with hundreds of classes and highly unbalanced seen-unseen split, and hence their accuracies have different behavior concerning q, as shown in Figure 2.c and 2.d. However, harmonic average curve still has the same behavior and possesses a maximum value at an optimal q. We illustrate the intuition with AwA dataset , a ZSL benchmark dataset. Consider a seen class squirrel. We compute closest unseen classes to the class squirrel in terms of attributes. We naturally find that the closest class is rat and the second closest is bat, while other classes such as horse, dolphin, sheep, etc. are not close (Figure 3.a). This is not surprising as squirrel and rat share several attribute. It is naturally desirable to have a classifier that gives rat higher probability than other classes. If we force this softly, we can ensure that classifier is not blind towards unseen classes due to lack of any training example. From a learning perspective, without any regularization, we cannot hope classifier to classify unseen classes accurately. This problem was identified in, where they proposed entropy- based regularization in the form of Deep Calibration Network (DCN). DCN uses cross-entropy loss for seen classes, and regularize the model with entropy loss on unseen classes to train the network. Authors in DCN postulate that minimizing the uncertainty (entropy) of predicted unseen distribution of training samples, enables the network to become aware of unseen visual features. While minimizing uncertainty is a good choice of regularization, it does not eliminate the possibility of being confident about the wrong unseen class. Clearly in DCN's approach, for the above squirrel example, the uncertainty can be minimized even when the classifier gives high confidence to a wrong unseen class dolphin on an image of seen class squirrel. Utilizing similarity based soft-labeling implicitly regularizes the model in a supervised fashion. The similarity values naturally has information of how much certainty we want for specific unseen class. We believe that this supervised regularization is the critical difference why our model outperforms DCN with a significant margin. Figure 3 shows the effect of τ and the consequent assigned unseen distribution on accuracies for AwA dataset. Small τ enforces q to be concentrated on nearest unseen class, while large τ spread q over all the unseen classes and basically does not introduce helpful unseen class information to the classifier. The optimal value for τ is 0.2 for AwA dataset as depicted in Figure 3.b. The impact of τ on the assigned distribution for unseen classes is shown in Figure 3.a when seen class is squirrel in AwA dataset. Unseen distribution with τ = 0.2, well represents the similarities between seen class (squirrel) and similar unseen classes (rat, bat, bobcat) and basically verifies the of Figure 3.b where τ = 0.2 is the optimal temperature. While in the extreme cases, when τ = 0.01, distribution on unseen classes in mostly focused on the nearest unseen class, rat, and consequently the other unseen classes' similarities are ignored. Also τ = 10 flattens the unseen distribution which in high uncertainty and does not contribute helpful unseen class information to the learning. We proposed a discriminative GZSL classifier with visual-to-semantic mapping and cross-entropy loss. During training, while SZSL is trained on a seen class, it simultaneously learns similar unseen classes through soft labels based on semantic class attributes. We deploy similarity based soft labeling on unseen classes that allows us to learn both seen and unseen signatures simultaneously via a simple architecture. Our proposed soft-labeling strategy along with cross-entropy loss leads to a novel regularization via generalized similarity-based weighted cross-entropy loss that can successfully tackle GZSL problem. Soft-labeling offers a trade-off between seen and unseen accuracies and provides the capability to adjust these accuracies based on the particular application. We achieve state-of-the-art performance, in GZSL setting, on all five ZSL benchmark datasets while keeping the model simple, efficient and easy to train." == "How to use cross-entropy loss for zero shot learning with soft labeling on unseen classes : a simple and effective solution that achieves state-of-the-art performance on five ZSL benchmark datasets." != "This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network. "

Question: Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to `effortlessly' extract desired representations from many large-scale datasets. However, generalization bounds for this setting is still missing. In this paper, we present a new PAC-Bayesian generalization bound for the negative log-likelihood loss which utilizes the \emph{Herbst Argument} for the log-Sobolev inequality to bound the moment generating function of the learners risk. Deep neural networks are ubiquitous across disciplines and often achieve state of the art (e.g., ; ;). Albeit neural networks are able to encode highly complex input-output relations, in practice, they do not tend to overfit . This tendency to not overfit has been investigated in numerous works on generalization bounds (; ; a; 2019; ; ;). Indeed, many generalization bounds apply to neural networks. However, most of these bounds assume that the loss function is bounded (a; ;). Unfortunately, this assumption excludes the popular negative log-likelihood (NLL) loss, which is instrumental to Bayesian neural networks that have been used extensively to calibrate model performance and provide uncertainty measures to the model prediction. In this work we introduce a new PAC-Bayesian generalization bound for NLL loss of deep neural networks. Our work utilizes the Herbst argument for the logarithmic-Sobolev inequality in order to bound the moment-generating function of the model risk. Broadly, our PACBayesian bound is comprised of two terms: The first term is dominated by the norm of the gradients with respect to the input and it describes the expressivity of the model over the prior distribution. The second term is the KL-divergence between the learned posterior and the prior, and it measures the complexity of the learning process. In contrast, bounds for linear models or bounded loss functions lack the term that corresponds to the expressivity of the model over the prior distribution and therefore are the same when applied to shallow and deep models. We empirically show that our PAC-Bayesian bound is tightest when we learn the mean and variance of each parameter separately, as suggested by in the context of Bayesian neural networks (BNNs). We also show that the proposed bound holds different insights regarding model architecture, optimization and prior distribution selection. We demonstrate that such optimization minimizes the gap between risk and the empirical risk compared to the standard Bernoulli dropout and other Bayesian inference approximation while being consistent with the theoretical findings. Additionally, we explore in-distribution and out-of-distribution examples to show that such optimization produces better uncertainty estimates than the baseline. PAC-Bayesian bounds for the NLL loss function are intimately related to learning Bayesian inference . Recently many works applied various posteriors in Bayesian neural networks.; introduce a Bayesian inference approximation using Monte Carlo (MC) dropout, which approximates a Gaussian posterior using Bernoulli dropout. introduced Gaussian dropout which effectively creates a Gaussian posterior that couples between the mean and the variance of the learned parameters. explored the relation of this posterior to log-uniform priors, while suggests to take a full Bayesian perspective and learn separately the mean and the variance of each parameter. Our work uses the bridge between PAC-Bayesian bounds and Bayesian inference, as described by , to find the optimal prior parameters in PAC-Bayesian setting and apply it in the Bayesian setting. Most of the literature regarding Bayesian modeling involves around a two-step formalism : a prior is specified for the parameters of the deep net; given the training data, the posterior distribution over the parameters is computed and used to quantify predictive uncertainty. Since exact Bayesian inference is computationally intractable for neural networks, approximations are used, including; Hernández-;;;. In this study we follow this two-step formalism, particularly we follow a similar approach to in which we learn the mean and standard deviation for each parameter of the model using variational Bayesian practice. Our experimental validation emphasizes the importance of learning both the mean and the variance. Generalization bounds provide statistical guarantees on learning algorithms. They measure how the learned parameters w perform on test data given their performance on the training data S = {(x 1, y 1),..., (x m, y m)}, where x i is the data instance and y i is its corresponding label. The performance of the learning algorithm is measured by a loss function (w, x, y). The risk of a learner is its average loss, when the data instance and its label are sampled from their true but unknown distribution D. We denote the risk by L D (w) = E (x,y)∼D (w, x, y). The empirical risk is the average training set loss L S (w) = 1 m m i=1 (w, x i, y i). PAC-Bayesian theory bounds the risk of a learner E w∼q L D (w) when the parameters are averaged over the learned posterior distribution q. The parameters of the posterior distribution are learned from the training data S. In our work we focus on the following PAC-Bayesian bound: Theorem 1 . Let KL(q||p) = q(w) log(q(w)/p(w))dw be the KLdivergence between two probability density functions p, q. For any λ > 0 and for any δ ∈ and for any prior distribution p, with probability at least 1 − δ over the draw of the training set S, the following holds simultaneously for any posterior distribution q: PAC-Bayesian theory is intimately connected to Bayesian inference when considering the negative log-likelihood loss function (w, x, y) = − log p(y|x, w) and λ = m. proved that the optimal posterior in this setting is q(w) = p(w|S). Bayesian inference considers the posterior p(y|x, S) = p(w|S)p(y|x, w)dw, at test time for a data instance x, which corresponds to the risk of the optimal posterior. Unfortunately, the optimal posterior is rarely available, and PAC-Bayes relies on the approximated posterior q. Coincidently, the approximated posterior and its KL-divergence from the prior distribution are instrumental to the evidence lower bound (ELBO), which is extensively used in Bayesian neural networks (BNNs) to bound the log-likelihood While the right hand side of a PAC-Bayesian bound, with the negative log-likelihood loss and λ = m, is identical to the right hand side of the ELBO bound in term of learning, they serve different purposes. One is used for bounding the risk while the other is used for bounding the marginal loglikelihood. Nevertheless, the same algorithms can be used to optimize BNNs and PAC-Bayesian intuitions and components can influence the practice of Bayesian neural networks. It is challenging to derive a PAC-Bayesian bound for the negative log-likelihood (NLL) loss as it requires a bound on the log-partition function log E w∼p, In cases where the loss function is uniformly bounded by a constant, e.g., the zero-one loss, the log-partition function is bounded as well. Unfortunately, the NLL loss is unbounded, even when y is discrete. For instance, consider fully connected case, where the input vector of the (k)-th layer is a function of the parameters of all previous layers, i.e., x k (W 0, . . ., W k−1). The entries of x k are computed from the response of its preceding layer, i.e., W k−1 x k, followed by a transfer function σ(·), i.e.,, if the rows in W k consist of the vector rx k then the NLL loss increases with r, and is unbounded when r → ∞. Our main theorem shows that for smooth loss functions, the log-partition function is bounded by the expansion of the loss function, i.e., the norm of its gradient with respect to the data x. This property is appealing since these gradients often decay rapidly for deep neural networks, as we demonstrate in our experimental evaluation. Consequently deep networks enjoy tighter generalization bounds than shallow networks. Our proof technique follows the Herbst Argument for bounding the log-partition function using the Log-Sobolev inequality for Gaussian distributions . Theorem 2. Assume (x, y) ∼ D and x given y follows the Gaussian distribution. Let (w, x, y) be a smooth loss function (e.g., the negative log-likelihood loss). For any δ ∈ and for any real number λ > 0, with probability at least 1 − δ over the draw of the training set S the following holds simultaneously for any posterior probability density function: The Gaussian assumption for the data generating distribution D can be relaxed to any log-concave distribution, using , Corollary 2.5. We use the Gaussian assumption to avoid notational overhead. Broadly, the proposed bound is comprised of two terms: The first term is the log-partition function which is dominated by the norm of the gradients with respect to the input, namely E (x,ŷ)∼D e α(− (w,x,ŷ)) dα, and it describes the expressivity of the model over the prior distribution. The second term is the KL-divergence between the learned posterior and the prior, and it measures the complexity of the learning process. The proof starts with Eq. and uses the Herbst Argument and the Log-Sobolev inequality to bound the moment-generating function Specifically, the proof consists of three steps. First we use the statistical independence of the training samples to decompose the moment generating function Then we use the Herbst argument to bound the function M (] and obtain the following bound: Finally we use the log-Sobolev inequality for Guassian distributions, The above theorem can be extended to settings for which x is sampled from any log-concave distribution, e.g., the Laplace distribution. The log-concave setting modifies the gradient norm and the log-Sobolev constant 2 in Eq. that corresponds to Gaussian distributions, cf.. We avoid this generalization to simplify our mathematical derivations. A detailed description of the proof can be found on Section 8.1 in the Appendix. The bound in Theorem 2 is favorable when applied to deep networks since their gradients w.r.t. data often decay rapidly. Nevertheless we can also apply our technique to shallow nets trained with NLL loss. We obtain PAC-Bayesian bounds for multi-class logistic regression. The NLL loss for multiclass logistic regression takes the form:, where x ∈ R d is the data instance, y ∈ {1, . . ., k} are the possible labels, and W ∈ R k×d is the matrix of parameters. The bound in Theorem 2 takes the form: given y follows the Gaussian distribution. Let (w, x, y) = − log p(y|x, w) be the negative log-likelihood loss for k−class logistic regression. For any δ ∈, for any λ > 0 and for any prior density function with variance σ 2 p ≤ m/16λ 2, with probability at least 1 − δ over the draw of the training set S the following holds simultaneously for any posterior probability density function: Full proof can be found on Section 8.2 in the Appendix, while we sketch the main steps of the proof below. The above corollary shows that PAC-Bayesian bound for classification using the NLL loss can achieve rate of λ = m. This augments the PAC-Bayesian for regression using the NLL loss for regression, i.e., the square loss, of. The PAC-Bayesian bound for logistic regression is derived by applying Theorem 2. We begin by realizing the gradient of log p(y|x, w) with respect to x. We denote by w y the y−th row of the parameter matrix W. Thus ∇ x log p(y|w, x) = ŷ p(ŷ|x, w)(w y − wŷ), and the gradient norm is upper bounded as follows: ∇ x log p(y|w, x) 2 ≤ 2 y w y 2. Plugging this into Eq. we obtain the following bound: Finally, whenever λσ p ≤ m/8 we derive the bound A detailed description of the proof can be found on Section 8.2 in the Appendix. In this section we study the derived bound empirically. We start with an ablation study of the proposed bound using classification and regression models. Next, we present our for multiclass classification tasks using different datasets and different architectures. We conclude the section, with an analysis of the models' uncertainty estimates using for in-distribution examples and outof-distribution examples. All suggested models follows the a Bayesian Neural Networks (BNN) perspective, in which we learn the mean and standard deviation for each learnable parameter in the network where we define N (0, σ 2 p I) to be the prior over weights. 6.1 ABLATION Effect of σ p. We start by exploring the effect of σ p on the models' performance and the proposed generalization bound. For that, we trained several models using σ p ∈ {0.05, 0.1, 0.2, 0.3} using the MNIST and datasets. All were obtained using fully connected layers with ReLU as non-linear activation function. We optimized the NLL loss function using Stochastic Gradient Descent (SGD) for 50 epochs with a learning rate of 0.01 and momentum of 0.9. For each model we compute the average train and test loss and accuracy together with the absolute difference between the training loss and the test loss, denoted as Generalization Loss. Moreover, we compute the generalization bound as stated in Eq. for all settings. Results are summarized in Table 1. Although σ p = 0.2 reaches slightly better generalization bound on MNIST dataset, σ p = 0.1 performs better over all calculated metrics, i.e., average loss and accuracy, both on MNIST and Fashion-MNIST. Notice, for Fashion-MNIST we observed slightly better generalization gap while using σ p = 0.05, however, its loss and accuracy are worse comparing to σ p = 0.1. Effect of λ. Recall, we bound the moment generating function using the norm of the functions' gradient with respect to the data x (Eq.). To construct tighter generalization bounds, we would like to set λ → m. However, in Eq. λ appears in both numerator and denominator. It is hence not clear whether the bound will converge, which depends on the model architecture, which is represented by the norm of its gradient. In other words, models with lower gradient norm could benefit from larger values of λ, hence tighter generalization bounds. To further explore this property we trained five different models with different number of layers. We look into both classification models while optimizing the NLL loss function, and regression tasks while optimizing the Mean Squared Error (MSE) loss function. For classification we used MNIST and Fashion-MNIST datasets, while for regression we use the Boston Housing dataset (for the regression models, were obtained using 5-fold cross validation). Except for the linear models, we force all models to have roughly the same numbers of parameters (∼80K for MNIST, ∼800K for Fashion-MNIST, ∼1500 for regression). For all models we set ReLU as non-linear activation functions. We optimize all models for 50 epochs using SGD with learning rate of 0.01 and momentum of 0.9. Based on of the prior paragraph, in all reported settings we set σ p = 0.1. Results are reported in Table 2. It can be seen that deeper models produce tighter generalization bounds on all three datasets. When considering model performance on down-stream classification task we notice that in general, models with better generalization bounds perform slightly better in terms of loss and accuracy. One possible explanation is that deeper models have smaller gradients w.r.t. the input. To validate that we further computed the average squared gradient norm w.r.t. the input as a function of the model depth, for both MNIST and Fashion-MNIST datasets. It can be seen from Figure 1a that indeed the gradients decay rapidly as we add more layers to the network. Next, we present in Figure 1b the generalization bound as a function of λ for MNIST models. We explored λ ∈ [√ m, m] and stopped the plot once the bound can no longer be computed. Experiments using Fashion-MNIST produce similar plot and can be found on Section 8.7 in the Appendix. Weights visualization. Since we consider Bayesian Neural Networks (BNNs) and optimize the KLdivergence between the prior and the posterior over the weights, we can visualize the average mean and standard deviation (STD) of the posterior as a function of the model depth. Figure 2a presents this for MNIST and Fashion-MNIST models using four and five depth levels. As expected, we can see that the average mean over the weights is zero for all layers while weights STD approaches 0.1. For the MNIST models (Figure 2a top row), we observed the standard deviation are ∼0.7 and not 0.1. We suspect this behaviour is due to fast optimization, hence the models do not have much signal to push the model towards the prior distribution. Notice, in all settings the average STD of the model weights decreases on the last layer. We observed a similar behavior also for the other models. 6.2 CLASSIFICATION Next, we compare BNN models against two commonly used baselines. The first baseline is a softmax model using the same architecture as the BNN while adding dropout layers. The second base- line is a Bayesian approximation using Monte Carlo Dropout , denoted as MC-Dropout, using different dropout rates and weight decay value of 1e-5. To evaluate these approaches we conducted multi-class classification experiments using three classification benchmarks: MNIST, Fashion-MNIST, and CIFAR-10 . We report train set and test set loss and accuracy, together with their generalization gaps (e.g., the difference between the test and training loss and accuracy). Notice, as oppose to our are reported for multi-class classification and not for binary classification. For completeness, we report binary classification on Section 8.4 in the Appendix. The premise beyond these type of experiments is to preset the benefit of learning the mean and STD separately for each of the models' parameters. Results are reported in Table 7. For BNN and MC-Dropout models, we sample 20 times from the posterior distribution and average their outputs to produce the model output. We also sampled more times, however, we did not see any significant differences. We observe that BNN models achieve comparable to both baselines but with lower loss and accuracy generalization gaps. Throughout the experiments we use dropout value of 0.3 for Softmax and MC-Dropout models and σ p = 0.1 for BNN models. We chose these values after grid search over different dropout values for all baseline models. A detailed description of all implementation details together with for more dropout rates can be found on Sections 8.4, 8.5, and 8.3 in the Appendix. Lastly, we evaluated the uncertainty estimates of BNN models against softmax models and MCDropout models. We experimented with both in-distribution and out-of-distribution examples. The purpose of the following experiments is to demonstrate that following the Bayesian approach together with the carefully picked prior can lead to better uncertainty estimates. In-Distribution Examples. In the context of in-distribution examples we follow the suggestion of and calculate the Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) for all three models. Figure 2b provides visual representation of the . Results suggest that BNNs produce better calibrated outputs for all settings, with two exception of ECE for MNIST and MCE for CIFAR10. Out-of-Distribution Examples. Next, we evaluated the uncertainty estimates using OOD examples. We apply a model trained using dataset A to OOD examples from dataset B. We trained models on MNIST, Fashion-MNIST and CIFAR-10 and assess prediction confidence using OOD examples from MNIST, Fashion-MNIST, NotMNIST , and SVHN . Results are summarized in Table 8. More OOD experiments using different dropout and prior rates can be found on Sections 8.3, 8.6 in the Appendix. All models performed at the chance level (∼ 10% for 10 classes) for both OOD train and test sets. When considering the loss, we observe significantly higher values for the softmax and MC-Dropout models. These two findings imply that the softmax and MC-Dropout models are overly confident and tend to output a high probability for the max label. Hence, we measure the average entropy for all models. We expect BNNs to have higher entropy, due to the fact that it produces better uncertainty estimates, i.e., its' predictions for OOD samples are closer to a uniform distribution. Indeed, reported in Table 8 confirm this intuition. In the following study we present a new PAC-Bayesian generalization bound for learning a deep net using the NLL loss function. The proof relies on bounding the log-partition function using the squared norm of the gradients with respect to the input. Experimental validation shows that the ing bound provides insight for better model optimization and prior distribution search. We demonstrate that learning the mean and STD for all parameters together with optimize prior over the parameters leads to better uncertainty estimates over the baselines and makes it harder to overfit. Proof. We begin by using the statistical independence of the training samples to decompose the following function: Next we represent the moment generating function M (where the last equality follows from a change of integration variable and the integral limits. K (α) refers to the derivative at α. We then compute K (α) and K: Concluding the Herbst argument we obtain the following equality: Combining Eq. with Eq. we derive: Finally we apply the log-Sobolev inequality for Gaussian distributions (cf. , Chapter 2), as described in Eq.. To complete the proof we combine Eq. with Eq. to obtain: Proof. To apply Theorem 2 we start by realizing the gradient of log p(y|x, w) with respect to x. We denote by w y the y−th row of the parameter matrix W. Thus ∇ x log p(y|w, x) = ŷ p(ŷ|x, w)(w y − wŷ). Using the convexity of the norm function we upper bound the gradient norm: Next we use the fact that the gradient norm upper bound is independent of x to simplify the moment generating function bound in Theorem 2. Since (w, x, y) = − log p(ŷ|x, w), we use the bound in Eq.: Thus we are able to simplify Theorem 2 as follows Finally, we recall that p is the prior density function N (0, σ 2 p). Since the parameters are statistically independent, this expectation decomposes to its kd parameters: And the follows from the fact for √ 2 and the follows. The architechtures described in this sub-section are used for the multi-class, binary, and uncertainty estimates experiments. We use multilayer perceptrons for the MNIST dataset, while we use convolutional neural networks (CNNs) for both Fashion-MNIST and CIFAR-10. A detailed description of the architectures is available in Table 5. We optimize the NLL loss function using SGD with a learning rate of 0.01 and a momentum value of 0.9 in all settings. We use mini-batches of size 128 and did not use any learning rate scheduling. For the MC-Dropout models we experienced with different weight decay values, however found that 1e-5 provides the best validation loss, hence choose this value. 8.4 BINARY CLASSIFICATION Experiments in this sub-section were conducted to show consistency with. We follow the same setting in which we use the MNIST dataset, where we group digits into label zero, and labels into label one. All experiments in this subsection were conducted using multilayer perceptrons with one hidden layer consisting of 300 hidden neurons. We use the Rectified Linear Unit (ReLU) as our activation function . We optimize the negative log-likelihood loss function using stochastic gradient descent (SGD) with a learning rate of 0.1 and a momentum value of 0.9. We did not use any learning rate scheduling. SGD is run in mini-batches of size 128. Each model was trained for 20 epochs. We compared BNN to softmax models with dropout rates chosen from the set {0.0, 0.3, 0.5}. Hereby, a dropout with a rate of 0.0 means no dropout at all. In addition to the training set and test set loss and accuracy, we measure the generalization loss, while setting L D (w) to be the average test set loss. In the same manner, we measure the generalization accuracy, while using the zero-one loss instead of the negative log-likelihood loss. Table 6 summarizes the . All models achieve comparable accuracy levels, however the softmax models suffer from larger generalization errors both in terms of loss and accuracy. Notice, as expected, using higher Bernoulli-dropout rates mitigates the generalization gap. Here we report for multi-class classification for BNN and the baselines. Table 7 summarizes the . The main purpose of these additional experiments is to explore more dropout and σ p values for different models. Figure 3: Analysis of the proposed bound as a function network depth. We report the the generalization bound as a function of λ for different deep net depth levels using the Fashion-MNIST dataset.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We derive a new PAC-Bayesian Bound for unbounded loss functions (e.g. Negative Log-Likelihood).

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.