Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The point estimates of ReLU classification networks, arguably the most widely used neural network architecture, have recently been shown to have arbitrarily high confidence far away from the training data. This architecture is thus not robust, e.g., against out-of-distribution data. Approximate Bayesian posteriors on the weight space have been empirically demonstrated to improve predictive uncertainty in deep learning. The theoretical analysis of such Bayesian approximations is limited, including for ReLU classification networks. We present an analysis of approximate Gaussian posterior distributions on the weights of ReLU networks. We show that even a simplistic (thus cheap), non-Bayesian Gaussian distribution fixes the asymptotic overconfidence issue. Furthermore, when a Bayesian method, even if a simple one, is employed to obtain the Gaussian, the confidence becomes better calibrated. This theoretical motivates a range of Laplace approximations along a fidelity-cost trade-off. We validate these findings empirically via experiments using common deep ReLU networks. As neural networks have been successfully applied in ever more domains, including safety-critical ones, the robustness of their predictions and the calibration of their predictive uncertainty have moved into focus, subsumed under the notion of AI safety . A principal goal of uncertainty calibration is that learning machines (and neural networks in particular) should assign low confidence to test cases not explained well by the training data or prior information . The most obvious such instance are test points that lie "far away" from the training data. Many methods to achieve this goal have been proposed, both Bayesian (; ;) and non-Bayesian (; ;). ReLU networks are currently among the most widely used neural architectures. This class comprises any network that can be written as a composition of linear layers (including fully-connected, convolutional, and residual layers) and a ReLU activation function. But while ReLU networks often achieve high accuracy, the uncertainty of their predictions has been shown to be miscalibrated . demonstrated that ReLU networks are always overconfident "far away from the data": scaling a training point x (a vector in a Euclidean input space) with a scalar δ yields predictions of arbitrarily high confidence in the limit δ → ∞. This means ReLU networks are susceptible to adversarial or out-of-distribution (OOD) examples. Bayesian methods have long been known empirically to improve predictive uncertainty calibration. demonstrated empirically that the predictive uncertainty of Bayesian neural networks will naturally be high in regions not covered by training data. Results like this raise the hope that the overconfidence problem of ReLU networks, too, might be mitigated by the use of Bayesian methods. This paper offers a theoretical analysis of the binary classification case of ReLU networks with logistic output layer. We show that equipping such networks with virtually any Gaussian probability distribution (i.e. regardless of whether it is motivated in a Bayesian fashion or not) mitigates the aforementioned theoretical problem, so that predictive confidence far away from the training data approaches a known constant, bounded away from one, whose value is controlled by the covariance (cf. Figure 1). At the same time, this treatment does not change the decision boundary of the trained network, so it has no negative effect on the predictive performance. Figure 1: Binary classification on a toy dataset using a MAP estimate (a) and various Gaussian approximations over the weights, sorted by their complexity of inverting the precision matrix. These approximations are carried out only at the last layer of the network and d denotes the number of hidden units at that layer. The shade of color represents the confidence of the prediction (darker shade means higher confidence). The decision boundary is in thick black. Even an arbitrary (i.e. nonBayesian) isotropic (b) or diagonal (c) covariance makes the confidence bounded away from one. Using the data in a more Bayesian fashion (d) calibrates the uncertainty further, in particular in regions close to the data. A central aspect of our is that asymptotic overconfidence can be mitigated with an essentially arbitrary Gaussian distribution on the weight space, including one of simple diagonal or even scalar covariance, and one whose covariance need not even depend on the training data. Achieving calibration at finite distances from the training data requires increasing levels of fidelity towards full Bayesian inference, for which our also give some quantification. Our thus answer a question about "how Bayesian" one needs to be to achieve certain levels of calibration. This is valuable because even approximate Bayesian treatments of deep learning, such as through Laplace approximations, can have high computational cost. We empirically validate our through a simple Laplace approximation to only the last layer of deep ReLU architectures, and find that this cheap procedure is already competitive to recently proposed non-Bayesian methods specifically constructed to overcome the overconfidence problem of ReLU networks. We also show that this cheap Bayesian approach yields good performance in the multi-class classification setting, indicating that our analysis may carry over to this case. Section 2 begins with a rigorous problem statement and assumptions, then develops the main theoretical . We discuss related work in Section 3, while empirical are in Section 4. Definitions We call a function f: R n → R piecewise affine if there exists a finite set of polytopes {Q r} R r=1, referred to as linear regions of f, such that ∪ R r=1 Q r = R n and f | Qr is an affine function for every Q r. ReLU networks are networks that in piecewise affine classifier functions which include networks with fully-connected, convolutional, and residual layers where just ReLU or leaky-ReLU are used as activation functions and max or average pooling are used as a convolution layer. be a dataset, where the targets t i ∈ {0, 1} or t i ∈ {1, . . ., k} for the binary and multi-class case, respectively. We define the logistic (sigmoid) function as σ(z):= 1/(1 + exp(−z)) for z ∈ R and the softmax function as softmax(z, i):= exp(z i)/ j exp(z j) for z ∈ R k. Given a linear classifier, 1 we will consider probability distributions p(w|D) or p(W|D) over the weight vector and matrix, respectively. We call these distributions posterior if they arose from Bayes' theorem or an approximation thereof. The predictive distribution (also called the marginalized prediction) is for the binary and multi-class cases, respectively. For Euclidean spaces we use the standard inner product and norm. Finally, λ i (·), λ max (·), and λ min (·) return the ith, maximum, and minimum eigenvalue (which are assumed to exist) of their matrix argument, respectively. Problem statement The following theorem from shows that ReLU networks exhibit arbitrarily high confidence far away from the training data: If a training point x ∈ R n is scaled by a sufficiently large scalar δ > 0, the input δx attains arbitrarily high confidence. Q r and f (x) = V r x + a r be the piecewise affine representation of the output of a ReLU network on Q r. Suppose that V r does not contain identical rows for all r = 1,..., R, then for almost any x ∈ R n and > 0 there exists an δ > 0 and a class i ∈ {1, . . ., K} such that it holds softmax(f (δx), i) ≥ 1 −. Moreover, lim δ→∞ softmax(f (δx), i) = 1. For binary classification tasks, it is standard to treat neural networks as probabilistic models of the conditional distribution p(y|x, w). Standard deep training involves assigning a maximum a posteriori (MAP) value w MAP to the weights. Doing so ignores potential uncertainty on w. We will show that this lack of uncertainty is the primary cause of the overconfidence discussed in. Unfortunately, there is generally no analytic solution for eq.. But for the logistic link function, good approximations exist when the distribution over the weights is Gaussian p(w|D) = N (w; µ, Σ) with mean µ and covariance Σ. One such approximation is constructed by scaling the input of the probit function 2 Φ by a constant λ = π/8. Using this approximation and the Gaussian assumption, if we let a:= w T x, we get where the last step uses the approximation Φ(π/8 x) ≈ σ(x) a second time, with In the case of µ = w MAP, eq. can be seen as the "softened" version of the MAP prediction of the classifier, using the covariance of the Gaussian. The principal aspect of interest of in this paper will be not so much any philosophical point about Bayesian inference, but that the approximate probabilistic Gaussian formalism as outlined in eqs. and introduces the second set of parameters in the form of Σ. We will find that at least asymptotic overconfidence problems can be fixed by setting Σ to virtually any sensible value, regardless of whether they are motivated in a Bayesian fashion or not. As a first notable property of this approximation, we show below that, in contrast to some other methods for uncertainty quantification (e.g. Monte Carlo dropout ) it preserves the decision boundary induced by the MAP estimate. Moreover, this property still holds even if we use any feature map φ and define the linear classifier on the image of this map instead. The implication is important in practice, as this gives a guarantee that if we apply this approximation to the last layer of any MAP pre-trained neural networks, then the classification accuracy of the marginalized prediction is exactly the same as the MAP classification accuracy.. The confidence of the marginalized prediction of a linear classifier is the highest in the direction of the lowest curvature, as described by Σ. Then we can obtain a Gaussian approximation p(w|D) ≈ N (w|µ, Σ) of the posterior by setting µ = w MAP and, the inverse Hessian of the negative log-posterior. In our binary classification case, p(y|x, w) is assumed to be Bernoulli(σ(w T x)) while p(w) is assumed to be N (w|0, σ 2 0 I), leading to the standard 2 -regularized binary cross-entropy loss. As our central theoretical contribution, we show that, far away from the training points, z(x) goes to a quantity that only depends on the mean and covariance of the Gaussian over the weights. This implies that we can make p(y = 1|x, D) closer to one-half far away from the training points if we can make z(x) closer to zero by controlling the Gaussian. Proposition 2.3 below shows this in the case of linear classifiers (also cf. Figure 2), while Theorem 2.4 shows that the analysis actually also holds in the case of ReLU networks. Proposition 2.3. Let f: R n → R be a binary linear classifier defined by f (x):= w T x and p(w|D):= N (w|µ, Σ) be the distribution over w. Then for any x ∈ R n, Furthermore, if x ∈ R n then as δ > 0 goes to infinity Proof. See Proposition A.3 in Appendix A. Recall from the definition, φ is a piecewise affine function. Thus, we can write the input space as R n = ∪ R r=1 Q r and for every Q r, the restriction φ| Qr: is an affine function φ| Qr (x):= V r x + a r for some V r ∈ R d×n and a r ∈ R d. Note that if i, j ∈ {1, . . ., M} with i = j then in general V i = V j and a i = a j. Using this definition, we can also show a similar to Proposition 2.3, in the case when x is replaced by any feature vector in the image of φ. ReLU network and let p(w|D):= N (w|µ, Σ) be the distribution over w. Then for any x ∈ R n, where V ∈ R d×n and a ∈ R d are some matrix and vector that depend on x. Furthermore, as δ > 0 goes to infinity Proof. See Theorem A.5 in Appendix A. Given a target upper bound on the logit and confidence values of a ReLU network, we can concretely pick the covariance Σ that respects the asymptotic bound of Theorem 2.4. Corollary 2.5 (Σ from a desired upper confidence bound on ReLU networks). Let f • φ, with φ: and N (w|µ, Σ) be the distribution over w where the mean µ is fixed and Σ is any SPD matrix. Then: (i) For any > 0 there exists Σ such that for any x ∈ R n far away from the training data, we have that |z • φ(x)| ≤. (ii) For any 0.5 < p < 1 there exists Σ such that for any x ∈ R n far away from the training data, we have that σ(|z Proof. See Corollary A.6 in Appendix A. Proposition 2.3 and Theorem 2.4 imply that the confidence of a binary linear classifier with ReLU features can be bound closer to one-half by increasing the minimum eigenvalue of the posterior covariance. In this section, we will move towards the Bayesian setting (i.e. using an explicit prior and likelihood, not just an imposed probability measure on the weights). Specifically, we will present a way to control the posterior through the prior in a Laplace approximation. Concretely, the following proposition and its immediate corollary point out that the eigenvalues of the posterior covariance can be increased (bringing |z(δx)| closer to zero) by increasing the prior variance.. Let p(w|D):= N (w|µ, Σ) be the posterior over w, obtained via a Laplace approximation with prior N (w|0, σ 2 0 I). Suppose H is the Hessian w.r.t. w at µ of the negative log-likelihood of the model. Then (ii) For each i = 1,..., d, the ith eigenvalue λ i (Σ) of Σ is a non-decreasing function of σ Proof. See Proposition A.7 in Appendix A. We get an immediate corollary from Proposition 2.6 that relates its to Theorem 2.4.. Let p(w|D):= N (w|µ, Σ) be the posterior over w, obtained via a Laplace approximation with prior N (w|0, σ is a non-increasing function of σ 2 0 with limits where H is as defined in (i) of Proposition 2.6. Proof. See Corollary A.8 in Appendix A. Lastly, the following corollary formalizes the intuition that the marginalized prediction with the inverse empirical features covariance C −1 as Σ will naturally have high uncertainty far away from the training data. Furthermore, this property can also be observed for Laplace approximation if the spectral properties of the Hessian ((i) of Proposition 2.6) are not too different to those of C. (ii) Σ is obtained via a Laplace approximation w.r.t. a prior N (w|0, σ 2 0 I) with σ 2 0 → ∞ and suppose H defined in (i) of Proposition 2.6 is invertible and the ordering of its eigenvalues is the same as that of C, while the eigenvectors are the same as those of C, then on any level set of µ T φ(x), the confidence decreases faster in the direction where the training data are sparser in the feature space R d. Proof. See Corollary A.9 in Appendix A. Similar statements for multi-class classifiers are not as straight-forward due to the lack of a good closed-form approximation of the integral of softmax under a Gaussian measure. However, as can be seen in Appendix C, at least the application of the above analysis can easily be generalized to the multi-class case. In fact, in the experiments (Section 4), we mainly use multi-class classifiers and show empirically that they are effective in mitigating issues that arise from the overconfidence problem. The overconfidence problem of deep neural networks, and thus ReLU networks, has long been known in the deep learning community . However, only recently this issue was demonstrated formally . Many methods have been proposed to combat or at least detect this issue. Post-hoc heuristics based on temperature or Platt scaling are unable to detect inputs with arbitrarily high confidence far away from the training data . proposed enhanced training objectives based on robust optimization to mitigate this issue. Bayesian methods have long been thought to mitigate the overconfidence problem on any neural network . Empirical evidence supporting this intuition has also been presented (; , etc.). Our complement these with a theoretical justification for the ReLU-logistic case. But while our work is theoretical in nature, we believe its application has practical value since it shows that a full Bayesian (expensive) treatment is not necessary if one is only worried about overconfidence. Indeed, fully Bayesian neural networks are often intractable and crude approximations have to be used, ing in undesirable . In this section we validate our theoretical by applying a Laplace approximation only to the last layer of various widely used ReLU networks and call this method last-layer Laplace approximation (LLLA). We refer the reader to Appendix C for details. Note that since LLLA is the simplest Laplace approximation that we can apply to deep networks, our should also hold for more general Laplace methods, e.g. Kronecker-factored Laplace (KFLA) , where not only the linear classifier's posterior but also the posterior of the feature map is approximated. Note however, these fully-Bayesian methods are significantly more expensive and require a significant amount of implementation effort. We will present our empirical on (i) a 2D toy classification task and (ii) out-of-distribution (OOD) data detection experiments. For the OOD experiment, we find the optimal prior variance σ 2 0 via a heuristic that follows directly from Corollary 2.7. Concretely, we pick the largest positive integer that makes the drop on the mean maximum confidence (MMC) of the in-distribution dataset to be within around 0.03 of the MAP's MMC. Thus, we only set this once without seeing any of the OOD datasets. The dataset is constructed by sampling the input points from k Gaussians. The corresponding targets indicate from which Gaussian the point was sampled. We use a 5-layer ReLU network with 100 hidden units at each layer as the feature map φ. The classifier, along with this feature map is trained jointly. We show the for the binary and multi-class (k = 4) case in Figure 3. As we can see, the MAP predictions have high confidence (low entropy) everywhere except at the region close to the decision boundary. The widely used MC-dropout does not remedy this issue. While ACET remedies the overconfidence issue, it is expensive and in general does not preserve the decision boundary. In contrast, LLLA yields better calibrated predictions: high confidence close to the training points and high uncertainty otherwise, while maintaining the MAP's decision boundary. We furthermore show the zoomed-out version of LLLA prediction we have presented in Figure 1d, along with the contour of the denominator of z (eq.) in Figure 4. We see that the covariance acts as a "moderator" for the MAP predictions: As a test point moves away from the training data, the denominator of z becomes larger and the marginalized prediction goes to a constant close to one-half. Table 1, LLLA yields competitive performance compared to both CEDA and ACET. We have shown that even an extremely approximate and virtually non-Bayesian probabilistic Gaussian treatment mitigates the most extreme aspects of overconfidence in ReLU networks. Our analytical bound the confidence of the Bayesian prediction of linear classifiers and ReLU networks far away from the training data away from one. This motivates a spectrum of approximations, from ad-hoc isotropic to "full Bayesian" Laplace approximations. In the Laplace approximation case, the bound asymptotically converges to a constant whose value can be controlled via the prior. We validated our experimentally by constructing a simple Laplace method that can still capture the properties we have shown, specifically by only approximating the last-layer's posterior distribution. In contrast to other approximations, this method is cheap and simple to implement, yet already yields competitive performance compared to the more expensive, recently proposed non-Bayesian method for combating the overconfidence problem. While more elaborate Laplace approximations can improve fidelity the further, our provide virtually any ReLU network with a simple and computationally lightweight way to mitigate overconfidence. 1/2 = 0. Notice, the denominator of the l.h.s. is positive. Thus, it follows that µ f must be 0, implying that σ(µ f) = 0.5. Lemma A.2. Let x ∈ R n be a vector and A ∈ R n×n be an SPD matrix. If λ min (A) is the minimum eigenvalue of A, then x T Ax ≥ λ min x 2. Proof. Since A is SPD, it admits an eigendecomposition A = QΛQ T and Λ = Λ 1 2 Λ 1 2 makes sense. Therefore, by keeping in mind that Q T x is a vector in R n, we have where the last equality is obtained as Q T x 2 = x T Q T Qx and noting that Q is an orthogonal matrix. Proposition A.3. Let f: R n → R be a binary linear classifier defined by f (x):= w T x and p(w|D):= N (w|µ, Σ) be the distribution over w. Then for any x ∈ R n, Furthermore, if x ∈ R n then as δ > 0 goes to infinity Proof. The first follows directly from Lemma A.2 and by noting that the denominator of eq. is positive since Σ is symmetric positive-definite (SPD) by definition. For the second , let x ∈ R n be arbitrary. By computation and again since the denominator of eq. is positive, we have We would like to inspect the asymptotic behavior of z(δx) with respect to δ. First, for the sake of completeness, we can compute that lim δ→0 |z(δx)| = 0. This reflects the case when δx goes to the decision boundary. Now, for the case when δ → ∞, we can see that since 1/δ 2 → 0 as δ → ∞. Therefore, using Lemma A.2 and Cauchy-Schwarz inequality, we have thus the proof is complete. Under review as a conference paper at ICLR 2020 Lemma A.4 . Let {Q i} R l=1 be the set of linear regions associated to the ReLU network φ: R n → R n. For any x ∈ R n there exists α ∈ R with α > 0 and t ∈ {1, . . ., R} such that δx ∈ Q t for all β ≥ α. Furthermore, the restriction of φ to Q t can be written as an affine function. Theorem A.5. Let f: R d → R be a binary linear classifier defined by f • φ(x):= w T φ(x) where φ: R n → R d is a ReLU network and let p(w|D):= N (w|µ, Σ) be the distribution over w. Then for any x ∈ R n, where V ∈ R d×n and a ∈ R d are some matrix and vector that depend on x. Furthermore, as δ > 0 goes to infinity such that x ∈ Q and φ| Q (x):= Vx + a. Applying eq. to φ| Q (x) and following the proof of Proposition 2.3 yield thus the first is obtained., such that for any δ ≥ α, we have that δx ∈ R and the restriction φ| R can be written as Ux + c. Therefore, for any such δ, Now, notice that as δ → ∞, 1/δ 2 and 1/δ goes to zero. So, in the limit, we have that Again, following the proof of Proposition 2.3 (i.e. using Cauchy-Schwarz and Lemma A.2), we can upper-bound this limit with which concludes the proof. Corollary A.6 (λ min (Σ) from a desired upper confidence bound on ReLU networks). Let f • φ, with φ: R n → R d and f: R d → R, be a ReLU network defined by f • φ(x):= w T φ(x) and N (w|µ, Σ) be the distribution over w where the mean µ is fixed and Σ is any SPD matrix. Then: (i) For any > 0 there exists Σ such that for any x ∈ R n far away from the training data, we have that |z • φ(x)| ≤. (ii) For any 0.5 < p < 1 there exists Σ such that for any x ∈ R n far away from the training data, we have that σ(|z • φ(x)|) ≤ p. Proof. We begin with (i). Let > 0 and δ = 8 π µ 2. Pick any Σ SPD with λ min (Σ) = δ. Then, by eq. of Theorem 2.4 and our choice of λ min (Σ), for any z ∈ R n, asymptotically we have that which is the desired . For (ii), let 0.5 < p < 1 be arbitrary. Observe that the inverse logistic function is given by σ −1 (x):= log x/(1 − x) for 0 < x < 1 and it is positive for 0.5 < x < 1. Therefore by setting in (i) with 2 and verify that for any x ∈ R n this gives |z(x)| ≤ σ −1 (p). Thus, for any x ∈ R n far away from the training data, since σ is monotonic, we have that and the proof is complete.. Let p(w|D):= N (w|µ, Σ) be the posterior over w, obtained via a Laplace approximation with prior N (w|0, σ 2 0 I). Suppose H is the Hessian w.r.t. w at µ of the negative log-likelihood of the model. Then (ii) For each i = 1,..., d, the ith eigenvalue λ i (Σ) of Σ is a non-decreasing function of σ Proof. The negative log-likelihood of Bernoulli distribution is given by Now, observing that σ (x) = σ(x)(1 − σ(x)) for all x ∈ R, we can compute T. T, since t ∈ {0, 1} by assumption. By considering all x, t ∈ D, we get (i). For (ii), first we assume that all Hessians mentioned below are w.r.t. w. We note that the assumption on the prior implies − log p(w) = 1/2 w T (1/σ 2 0 I)w + const, which has Hessian 1/σ 2 0 I. Thus, the Hessian of the negative log posterior − log p(w|D) = − log p(w) − log x,t∈D p(y|x, w) is 1/σ 2 0 I + H. This implies that the posterior covariance Σ of the Laplace approximation is given by Therefore, the ith eigenvalue of Σ for any i = 1,..., n is. For all i = 1,..., n, the derivative of λ i (Σ) w.r.t. σ T for some saturating h: R → R, which has lim x→∞ h(x) = l. Let also φ: R n → R d defined as φ(x):= g(Vx + a) for some V ∈ R d×n and a ∈ R d be a feature map. Suppose p(w|D):= N (w|µ, Σ) is the distribution over w. Then for any x ∈ D, as δ > 0 goes to infinity Proof. By definition, By definition of g, lim δ→∞ g(δVx + a) = (l, . . ., l) T =: l, which implies The theoretical in the main text essentially tell us that if we have a Gaussian approximate posterior that comes from a Laplace approximation, then using eq. (and eq.) when making a prediction can remedy the overconfidence problem on any ReLU network. In this section we describe a simple Laplace method that can still capture the properties that we have presented in Section 2. Concretely, we apply the Laplace approximation only to the linear last layer of ReLU networks, that have been trained via MAP estimation. For the sake of clarity, we omit the bias in the following and revisit the case where the bias is included at the end of this section. For the binary classification case, let g: R n → R be a MAP-trained deep ReLU neural network with a linear last-layer. We can decompose g into a feature map φ: R n → R d and a linear classifier f:. Based on Proposition 2.6, we can simply perform a Laplace approximation to get the posterior of the weight of the linear classifier f, i.e. p(w|D) = N (w|w MAP, H −1) where H is the Hessian of the negative log-posterior w.r.t. w at w MAP. This Hessian could be obtained via automatic differentiation or via the explicit formula stated in (i) of Proposition 2.6. We emphasize that we only deal with the weight at the last layer of g, i.e. the weight of f, and not the weight of the whole network, thus the inversion of H is rarely a problem. For instance, large models such as DenseNet-201 and ResNet-152 (a) In the case of multi-class classification, we now have f:. We obtain the posterior over a random matrix W ∈ R k×d in the form N (vec(W)|vec(W MAP), Σ) for some Σ ∈ R dk×dk SPD. The procedure is still similar to the one described above, since the exact Hessian of the linear multi-class classifier can still be easily and efficiently obtained via automatic differentiation. Note that in this case we need to invert a dk × dk matrix, which, depending on the size of k, can be quite large. For a more efficient procedure, we can make a further approximation to the posterior in the multiclass case by assuming the posterior is a matrix Gaussian distribution. We can use the Kroneckerfactored Laplace approximation (KFLA) , but only for the last layer of the network. That is, we find the Kronecker factorization of the Hessian Then by definition of a matrix Gaussian , we immediately 4 Based on the implementations available in the TorchVision package. 5 For example, the ImageNet dataset has k = 1000. 6 In practice, we take the running average of the Kronecker factors of the Hessian over the mini-batches. obtain the posterior MN (W|W MAP, U, V). The distribution of the latent functions is Gaussian, since f:= Wφ(x) and p(W|D) = MN (W|W MAP, U, V) imply where the last equality follows since (φ(x) T Vφ(x)) is a scalar. We then have the following integral which can be approximated via MC-integration. While one can always assume that the bias trick is already used, i.e. it is absorbed in the weight matrix/vector, in practice when dealing with pre-trained networks, one does not have such liberty. In this case, one can simply assume that the bias b or b is independent of the weight w or W, respectively in the two-and multi-class cases. By using the same Laplace approximation procedure, one can easily get p(b|D):. This implies w T φ(x)+b =: f and Wφ(x) + b =: f are also Gaussians given by. Similarly, in the case when the Kronecker-factored approximation is used, we have Because of the construction above, which is simply done by applying Laplace approximation on the last layer of a ReLU network, we call this method last layer Laplace approximation or LLLA for short. We present the pseudocodes of LLLA in Algorithms 1 and 2. Algorithm 1 LLLA with exact Hessian for binary classification. We train all networks we use in Table 1 for 100 epochs with batch size of 128. The initial learning rates are 0.001 and 0.1 for MNIST and CIFAR-10 experiments, respectively, and we divide them by 10 at epoch 50, 75, and 95. We use ADAM and SGD with 0.9 momentum, respectively. Standard data augmentations, i.e. random crop and standardization are also used for training the network on CIFAR-10. Meanwhile, for LLLA, we use the Kronecker-factored Hessian. Algorithm 2 LLLA with Kronecker-factored Hessian for multi-class classification. A pre-trained network f • φ with W MAP as the weight of f, (averaged) cross-entropy loss L, training set D train, test set D test, mini-batch size m, number of samples s, running average weighting ρ, and prior precision τ 0 = 1/σ 2 0. Predictions P containing p(y = i|x, D train) ∀x ∈ D test ∀i ∈ {1, . . ., k}. We further compare the OOD detection performance of LLLA to the temperature scaling method. To find the optimal temperature, we follow the method of. In particular, we use the implementation provided by https://github.com/JonathanWenger/pycalib.

We argue theoretically that by simply assuming the weights of a ReLU network to be Gaussian distributed (without even a Bayesian formalism) could fix this issue; for a more calibrated uncertainty, a simple Bayesian method could already be sufficient.

Represent the TLDR for finding the corresponding scientific article

Using linear programming we show that the computational complexity of approximate Deep Neural Network training depends polynomially on the data size for several architectures


Question: Represent the article for finding a single-sentence TLDR

Question: Word alignments are useful for tasks like statistical and neural machine translation (NMT) and annotation projection. Statistical word aligners perform well, as do methods that extract alignments jointly with translations in NMT. However, most approaches require parallel training data and quality decreases as less training data is available. We propose word alignment methods that require little or no parallel data. The key idea is to leverage multilingual word embeddings – both static and contextualized – for word alignment. Our multilingual embeddings are created from monolingual data only without relying on any parallel data or dictionaries. We find that traditional statistical aligners are outperformed by contextualized embeddings – even in scenarios with abundant parallel data. For example, for a set of 100k parallel sentences, contextualized embeddings achieve a word alignment F1 that is more than 5% higher (absolute) than eflomal. Word alignment is essential for statistical machine translation and useful in NMT, e.g., for imposing priors on attention matrices (; ;) or for decoding . Further, word alignments have been successfully used in a range of tasks such as typological analysis (; Östling, 2015), annotation projection (; Padó and) and creating multilingual embeddings . Statistical word aligners such as the IBM models and their successors (e.g., fastalign , GIZA++ , eflomal (Östling and) ) are widely used for alignment. With the rise of NMT , attempts have been made to interpret attention matrices as soft word alignments . Several methods create alignments from attention matrices (; ;) or pursue a multitask approach for alignment and translation . However, most systems require parallel data and their performance deteriorates when parallel text is scarce (cf. Tables 1-2 in ). Recent unsupervised multilingual embedding algorithms that use only monolingual data provide high quality static and contextualized embeddings (; ;). Our key idea is to leverage these embeddings for word alignments -without relying on parallel data. Requiring no or little parallel data is advantageous in many scenarios, e.g., in the low-resource case and in domain-specific settings without parallel data. A lack of parallel data cannot be easily remedied: mining parallel sentences is possible (cf. ) but assumes that monolingual corpora contain parallel sentences. Contributions: We propose two new alignment methods based on the matrix of embedding similarities. We propose two post-processing algorithms that handle null words and integrate positional information. We show that word alignments obtained from multilingual BERT outperform strong statistical word aligners like eflomal. We investigate the differences between word and subword processing for alignments and find subword processing to be preferable. Upon acceptance we will publish the source code. Consider parallel sentences s (e), s (f), with lengths l e, l f in languages e, f. Assume we have access to some embedding function E that assigns each word in a sentence a d-dimensional vector, i.e., E(s (k) ) ∈ R l k ×d for k ∈ {e, f}. Let E(s (k) ) i denote the vector of the i-th word in sentence s (k). We define the similarity matrix as the matrix S ∈ le×l f induced by the embeddings where S ij:= sim E(s (e) ) i, E(s (f) ) j is some normalized measure of similarity, e.g., cosine-similarity normalized to be between 0 and 1. We now introduce methods for extracting alignments from S, i.e., obtaining a binary matrix A ∈ {0, 1} le×l f. Argmax. A simple baseline is to align each word in sentence s (e) with the most similar word in s (f) and vice versa. That is, we define A ij = 1 if and only if and A ij = 0 else. Similar methods have been applied to Dice coefficients (cf. ) and attention matrices . Match. Argmax finds a local, not a global optimum. To address this, we frame alignment as an assignment problem: we search for a maximumweight maximal matching (cf. ) in the bipartite weighted graph induced by the similarity matrix. This optimization problem is given by A ij S ij subject to A being a valid maximal matching (i.e., every word in the shorter sentence is aligned). There are known algorithms to solve the above problem in polynomial time (cf.). Note that alignments generated with the matching method are inherently bidirectional and do not require any symmetrization as post-processing. Distortion Correction [Dist]. Distortion, as introduced in IBM Model 2, is essential for alignments based on non-contextualized embeddings since the similarity of two words is solely based on their surface form, independent of position. To penalize high distortion, we multiply the similarity matrix S componentwise with where κ is a hyperparameter to scale the matrix between [(1 − κ), 1]. We use κ = 0.5. 1 We can interpret this as imposing a locality-preserving prior: 1 See supplementary for different values. given a choice, a word should be aligned to a word with a similar relative position ((i/l e − j/l f) 2 close to 0) rather than a more distant word (large Null. Null words model untranslated words and are an important part of alignment models. We remove alignment edges when the normalized entropy of the similarity distribution is above a threshold τ, a hyperparameter. 2 Intuitively if a word is not similar to any of the words in the target sentence, we do not align it. That is, we set A ij = 0 if and only if Traditional word alignment models create forward and backward alignments and then symmetrize them . We compared grow-diagfinal-and (GDFA) and intersection and found them to perform comparably. 4 We use GDFA throughout the paper. We investigate both subword segmentations such as BPE/wordpiece Our test data are three language pairs in different domains. We use Europarl gold alignments 7 for English-German, Bible gold alignments for English-French and gold alignments by for English-Persian (domain: books). We select additional parallel training data that is consistent with the target domain where available: Europarl for English-German and Parallel Bible Corpus (PBC) for English-French and EnglishPersian. For fast-align, GIZA++ and eflomal we add 10,000 parallel sentences (simulating a midresource setting) to the gold standard as training data. We show the effect of adding more or less training data in Figure 1. Since mBERT is pretrained on Wikipedia, we train fastText embeddings on Wikipedia as well. For hyperparameters of all models see supplementary. Our evaluation measures are precision, recall, F 1 and alignment error rate (AER). Overall. Table 1 shows that mBERT performs consistently best. Eflomal has high precision and is on par for ENG-FRA. Surprisingly, fastText outperforms fast-align in two out of three languages (e.g., F 1 65 vs. 61 for ENG-DEU) despite not having access to parallel data. Among the statistical baselines, eflomal outperforms GIZA++, while GIZA++ is better than fast-align, as expected. Parallel Data. Figure 1 shows that fast-align and eflomal get better with more training data with eflomal outperforming fast-align, as expected. However, even with 10 6 parallel sentences mBERT outperforms both statistical baselines. fastText becomes competitive for fewer than 1000 parallel sentences. The main takeaway is that mBERT-based alignments, a method that does not need any parallel training data, outperform state-of-the-art aligners, even in the high resource case. Word vs. Subword. In Table 1 subword processing mainly benefits fast-align, GIZA++ and eflomal (except for ENG-FRA). fastText is harmed by subword processing. We use VecMap to match (sub)word distributions across languages. We hypothesize that it is harder to match subword than word distributions -this effect is strongest for Persian, probably due to different scripts and thus different subword distributions. For mBERT words and subwords are about the same. Table 4 compares alignment and postprocessing methods. Argmax generally yields higher precision whereas Match has higher recall. For fastText using Argmax with Dist yields best F 1 on two languages. Adding a distortion prior boosts performance for static embeddings, e.g., from.46 to.61 for ENG-FRA F 1. Null-word processing increases precision, e.g., from.91 to.96 for ENG-DEU, but does not increase F 1. For mBERT Argmax performs best in two out of three language pairs. Dist has little and sometimes harmful effects on mBERT indicating that mBERT's contextualized representations already match well across languages. mBERT Layers. Figure 2 shows a parabolic trend with layer 8 of mBERT yielding the best performance. This is consistent with other work : in the first layers the contextualization is too weak for high-quality alignments while last layers are too specialized on the pretraining task (masked language modeling). introduced the IBM models, the best known statistical word aligners. More recent aligners, often based on IBM models, include fastalign , GIZA++ and eflomal (Östling and). All of these models are trained on parallel text. Our method instead aligns based on embeddings that are induced from monolingual data only. Prior work on using learned representations for alignment includes compute similarity matrices of encoder-decoder representations that are leveraged for word alignments, together with supervised learning which requires manually annotated alignment. In contrast to our work, they all require parallel data. We presented word aligners based on contextualized (resp. static) embeddings that perform better than (resp. comparably with) statistical word aligners. Our method is the first that does not require parallel data and is particularly useful for scenarios where a medium number of parallel sentences need to be aligned, but no additional parallel data is available. For a set of 100k parallel sentences, contextualized embeddings achieve an alignment F 1 that is 5% higher (absolute) than eflomal. Given a set of predicted alignment edges A and a set of sure (possible) gold standard edges S (P) we computed our evaluation measures as follows: where | · | denotes the cardinality of a set. This is the usual way of evaluating alignments . For asymmetric alignments different symmetrization methods exist. fast-align provides an overview and implementation for these methods, which we use. We compare intersection and grow-diag-final-and (GDFA) in Table 3. In terms of F1 GDFA performs better (Intersection wins once, GDFA five times, three ties). As expected, Intersection yields higher precision while GDFA yields higher recall. Thus intersection is preferable for tasks like annotation projection, whereas GDFA is typically used in statistical machine translation. The analogous numbers from Table 1 in the main paper on word-level can be found in Table 4. Again Distortion is essential for fastText and not necessary for mBERT. Adding Null helps especially for mBERT. Overall the takeaways are consistent with the from subword-level. We provide a list of customized hyperparameters used in our computations in hyperparameters we used default values as provided in the corresponding implementation (see respective links to the code repositories). In the main paper we introduced the hyperparameter κ. In Figure 3 we plot the performance for different values of κ. We observe that introducing distortion indeed helps (i.e., κ > 0) but the actual value is not decisive for performance. This is rather intuitive, as a small adjustment to the similarities is sufficient while larger adjustments do not necessarily hurt or change the Argmax or the optimal point in the Matching Algorithm. In the main paper we have chosen κ = 0.5. For τ in the null-word post-processing we need to use high values as the similarity distributions tend to be quite uniform in the high-dimensional spaces. See Figure 4 for different values of τ. As expected, for τ = 1 no edges are removed and thus the performance is not changed compared to not having a null-word post-processing. With decreasing τ the precision increases and recall goes down. We use τ = 0.999 for fastText and τ = 0.9995 for mBERT. Upon acceptance we will publish the code together with instructions on how to reproduce the . Table 6 provides an overview on the data used in the main paper together with download links.

Represent the short summary for finding the corresponding scientific article (long)

We use representations trained without any parallel data for creating word alignments.

We introduce a novel method to train Seq2Seq models with language models that converge faster, generalize better and can almost completely transfer to a new domain using less than 10% of labeled data.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent studies have shown the vulnerability of reinforcement learning (RL) models in noisy settings. The sources of noises differ across scenarios. For instance, in practice, the observed reward channel is often subject to noise (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible as a . Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively. Designing a suitable reward function plays a critical role in building reinforcement learning models for real-world applications. Ideally, one would want to customize reward functions to achieve application-specific goals . In practice, however, it is difficult to design a function that produces credible rewards in the presence of noise. This is because the output from any reward function is subject to multiple kinds of randomness:• Inherent Noise. For instance, sensors on a robot will be affected by physical conditions such as temperature and lighting, and therefore will report back noisy observed rewards.• Application-Specific Noise. In machine teaching tasks BID13 ), when an RL agent receives feedback/instructions from people, different human instructors might provide drastically different feedback due to their personal styles and capabilities. This way the RL agent (machine) will obtain reward with bias.• Adversarial Noise. Adversarial perturbation has been widely explored in different learning tasks and shows strong attack power against different machine learning models. For instance, has shown that by adding adversarial perturbation to each frame of the game, they can mislead RL policies arbitrarily. Assuming an arbitrary noise model makes solving this noisy RL problem extremely challenging. Instead, we focus on a specific noisy reward model which we call perturbed rewards, where the observed rewards by RL agents are generated according to a reward confusion matrix. This is not a very restrictive setting to start with, even considering that the noise could be adversarial: Given that arbitrary pixel value manipulation attack in RL is not very practical, adversaries in the real-world have high incentives to inject adversarial perturbation to the reward value by slightly modifying it. For instance, adversaries can manipulate sensors via reversing the reward value. In this paper, we develop an unbiased reward estimator aided robust framework that enables an RL agent to learn in a noisy environment with observing only perturbed rewards. Our solution framework builds on existing reinforcement learning algorithms, including the recently developed DRL ones (Q-Learning BID19 BID18, Cross-Entropy Method (CEM) BID11, Deep SARSA BID10, Deep Q-Network (DQN) (; BID6, Dueling DQN (DDQN) BID17, Deep Deterministic Policy Gradient (DDPG) , Continuous DQN (NAF) and Proximal Policy Optimization (PPO) BID4 The main challenge is that the observed rewards are likely to be biased, and in RL or DRL the accumulated errors could amplify the reward estimation error over time. We do not require any assumption on knowing the true distribution of reward or adversarial strategies, other than the fact that the generation of noises follow an unknown reward confusion matrix. Instead, we address the issue of estimating the reward confusion matrices by proposing an efficient and flexible estimation module. provided preliminary studies for the noisy reward problem and gave some general negative . The authors proved a No Free Lunch theorem, which is, without any assumption about what the reward corruption is, all agents can be misled. Our do not contradict with the therein, as we consider a specific noise generation model (that leads to a set of perturbed rewards). We analyze the convergence and sample complexity for the policy trained based on our proposed method using surrogate rewards in RL, using Q-Learning as an example. We conduct extensive experiments on OpenAI Gym (AirRaid, Alien, Carnival, MsPacman, Pong, Phoenix, Seaquest) and show that the proposed reward robust RL method achieves comparable performance with the policy trained using the true rewards. In some cases, our method even achieves higher cumulative reward -this is surprising to us at first, but we conjecture that the inserted noise together with our noisy-removal unbiased estimator adds another layer of exploration, which proves to be beneficial in some settings. This merits a future study. Our contributions are summarized as follows: FORMULA2 We adapt and generalize the idea of defining a simple but effective unbiased estimator for true rewards using observed and perturbed rewards to the reinforcement learning setting. The proposed estimator helps guarantee the convergence to the optimal policy even when the RL agents only have noisy observations of the rewards. We analyze the convergence to the optimal policy and finite sample complexity of our reward robust RL methods, using Q-Learning as the running example. Extensive experiments on OpenAI Gym show that our proposed algorithms perform robustly even at high noise rates. Robust Reinforcement Learning It is known that RL algorithms are vulnerable to noisy environments . Recent studies (; ;) show that learned RL policies can be easily misled with small perturbations in observations. The presence of noise is very common in real-world environments, especially in robotics-relevant applications. Consequently, robust (adversarial) reinforcement learning (RRL/RARL) algorithms have been widely studied, aiming to train a robust policy which is capable of withstanding perturbed observations BID12; ) or transferring to unseen environments BID1 ). However, these robust RL algorithms mainly focus on noisy vision observations, instead of the observed rewards. A couple of recent works (; BID3 have also looked into a rather parallel question of training robust RL algorithms with uncertainty in models. Learning with Noisy Data Learning appropriately with biased data has received quite a bit of attention in recent machine learning studies ; BID7 BID6 ; BID9 BID16 ; . The idea of above line of works is to define unbiased surrogate loss function to recover the true loss using the knowledge of the noises. We adapt these approaches to reinforcement learning. Though intuitively the idea should apply in our RL settings, our work is the first one to formally establish this extension both theoretically and empirically. Our quantitative understandings will provide practical insights when implementing reinforcement learning algorithms in noisy environments. In this section, we define our problem of learning from perturbed rewards in reinforcement learning. Throughout this paper, we will use perturbed reward and noisy reward interchangeably, as each time step of our sequential decision making setting is similar to the "learning with noisy data" setting in supervised learning (; BID7 BID6 BID9 . In what follows, we formulate our Markov Decision Process (MDP) problem and the reinforcement learning (RL) problem with perturbed (noisy) rewards. Our RL agent interacts with an unknown environment and attempts to maximize the total of his collected reward. The environment is formalized as a Markov Decision Process (MDP), denoting as M = S, A, R, P, γ. At each time t, the agent in state s t ∈ S takes an action a t ∈ A, which returns a reward r(s t, a t, s t+1) ∈ R (which we will also shorthand as r t), and leads to the next state s t+1 ∈ S according to a transition probability kernel P, which encodes the probability Pa(st, st+1). Commonly P is unknown to the agent. The agent's goal is to learn the optimal policy, a conditional distribution π(a|s) that maximizes the state's value function. The value function calculates the cumulative reward the agent is expected to receive given he would follow the current policy π after observing the current state DISPLAYFORM0 where 0 ≤ γ ≤ 1 1 is a discount factor. Intuitively, the agent evaluates how preferable each state is given the current policy. From the Bellman Equation, the optimal value function is given by V * (s) = maxa∈A s t+1 ∈S Pa(st, st+1) [rt + γV * (st+1)]. It is a standard practice for RL algorithms to learn a state-action value function, also called the Q-function. Q-function denotes the expected cumulative reward if agent chooses a in the current state and follows π thereafter: DISPLAYFORM1 In many practical settings, our RL agent does not observe the reward feedback perfectly. We consider the following MDP with perturbed reward, denoting asM = S, A, R, C, P, γ 2: instead of observing r t ∈ R at each time t directly (following his action), our RL agent only observes a perturbed version of r t, denoting asr t ∈R. For most of our presentations, we focus on the cases where R,R are finite sets; but our generalize to the continuous reward settings. The generation ofr follows a certain function C: S × R →R. To let our presentation stay focused, we consider the following simple state-independent 3 flipping error rates model: if the rewards are binary (consider r + and r −),r(s t, a t, s t+1) (r t) can be characterized by the following noise rate parameters e +, e −: e+ = P(r(st, at, st+1) = r−|r(st, at, st+1) = r+), e− = P(r(st, at, st+1) = r+|r(st, at, st+1) = r−). When the signal levels are beyond binary, suppose there are M outcomes in total, denoting as [R 0, R 1, · · ·, R M −1].r t will be generated according to the following confusion matrix C M ×M where each entry c j,k indicates the flipping probability for generating a perturbed outcome: c j,k = P(r t = R k |r t = R j). Again we'd like to note that we focus on settings with finite reward levels for most of our paper, but we provide discussions in Section 3.1 on how to handle continuous rewards with discretizations. In the paper, we do not assume knowing the noise rates (i.e., the reward confusion matrices), which is different from the assumption of knowing them as adopted in many supervised learning works. Instead we will estimate the confusion matrices (Section 3.3). In this section, we first introduce an unbiased estimator for binary rewards in our reinforcement learning setting when the error rates are known. This idea is inspired by , but we will extend the method to the multi-outcome, as well as the continuous reward settings. With the knowledge of noise rates (reward confusion matrices), we are able to establish an unbiased approximation of the true reward in a similar way as done in. We will call such a constructed unbiased reward as a surrogate reward. To give an intuition, we start with replicating the for binary reward R = {r −, r +} in our RL setting: Lemma 1. Let r be bounded. Then, if we define, r(st, at, st+1):=(1−e −)·r + −e + ·r − 1−e + −e − (r(st, at, st+1) = r+) DISPLAYFORM0 we have for any r(s t, a t, s t+1), Er |r [r(s t, a t, s t+1)] = r(s t, a t, s t+1).In the standard supervised learning setting, the above property guarantees convergence -as more training data are collected, the empirical surrogate risk converges to its expectation, which is the same as the expectation of the true risk (due to unbiased estimators). This is also the intuition why we would like to replace the reward terms with surrogate rewards in our RL algorithms. The above idea can be generalized to the multi-outcome setting in a fairly straight-forward way. DefineR:= [r(r = R 0),r(r = R 1),...,r(r = R M −1)], wherer(r = R m) denotes the value of the surrogate reward when the observed reward is DISPLAYFORM1 DISPLAYFORM2 we have for any r(s t, a t, s t+1), Er |r [r(s t, a t, s t+1)] = r(s t, a t, s t+1).Continuous reward When the reward signal is continuous, we discretize it into M intervals and view each interval as a reward level, with its value approximated by its middle point. With increasing M, this quantization error can be made arbitrarily small. Our method is then the same as the solution for the multi-outcome setting, except for replacing rewards with discretized ones. Note that the finerdegree quantization we take, the smaller the quantization error -but we would suffer from learning a bigger reward confusion matrix. This is a trade-off question that can be addressed empirically. So far we have assumed knowing the confusion matrices, but we will address this additional estimation issue in Section 3.3, and present our complete algorithm therein. We now analyze the convergence and sample complexity of our surrogate reward based RL algorithms (with assuming knowing C), taking Q-Learning as an example. Convergence guarantee First, the convergence guarantee is stated in the following theorem:Theorem 1. Given a finite MDP, denoting asM = S, A,R, P, γ, the Q-learning algorithm with surrogate rewards, given by the update rule, DISPLAYFORM0 converges w.p.1 to the optimal Q-function as long as t α t = ∞ and t α 2 t < ∞.Note that the term on the right hand of Eqn. includes surrogate rewardr estimated using Eqn. FORMULA2 and Eqn.. Theorem 1 states that that agents will converge to the optimal policy w.p.1 with replacing the rewards with surrogate rewards, despite of the noises in observing rewards. This is not surprising -though the surrogate rewards introduce larger variance, we are grateful of their unbiasedness, which grants us the convergence. In other words, the addition of the perturbed reward does not destroy the convergence guarantees of Q-Learning. Sample complexity To establish our sample complexity , we first introduce a generative model following previous literature (; 2000;). This is a practical MDP setting to simplify the analysis. Definition 1. A generative model G(M) for an MDP M is a sampling model which takes a stateaction pair (s t, a t) as input, and outputs the corresponding reward r(s t, a t) and the next state s t+1 randomly with the probability of P a (s t, s t+1), i.e., s t+1 ∼ P(·|s, a).Exact value iteration is impractical if the agents follow the generative models above exactly . Consequently, we introduce a phased Q-Learning which is similar to the ones presented in; for the convenience of proving our sample complexity . We briefly outline phased Q-Learning as follows -the complete description (Algorithm 2) can be found in Appendix A. Definition 2. Phased Q-Learning algorithm takes m samples per phase by calling generative model G(M). It uses the collected m samples to estimate the transition probability P and update the estimated value function per phase. Calling generative model G(M) means that surrogate rewards are returned and used to update value function per phase. The sample complexity of Phased Q-Learning is given as follows: DISPLAYFORM1 be bounded reward, C be an invertible reward confusion matrix with det(C) denoting its determinant. For an appropriate choice of m, the Phased Q-Learning algorithm calls the generative model DISPLAYFORM2 times in T epochs, and returns a policy such that for all state s ∈ S, DISPLAYFORM3 Theorem 2 states that, to guarantee the convergence to the optimal policy, the number of samples needed is no more than O(1/det(C)2 ) times of the one needed when the RL agent observes true rewards perfectly. This additional constant is the price we pay for the noise presented in our learning environment. When the noise level is high, we expect to see a much higher 1/det(C)2; otherwise when we are in a low-noise regime, Q-Learning can be very efficient with surrogate reward . Note that Theorem 2 gives the upper bound in discounted MDP setting; for undiscounted setting (γ = 1), the upper bound is at the order of O. Lower bound is omitted due to the lack of space. The idea of constructing MDP in which learning is difficult and the algorithm must make |S||A|T log 1 δ calls to G(M), is similar to.While the surrogate reward guarantees the unbiasedness, we sacrifice the variance at each of our learning steps, and this in turn delays the convergence (as also evidenced in the sample complexity bound). It can be verified that the variance of surrogate reward is bounded when C is invertible, and it is always higher than the variance of true reward. This is summarized in the following theorem: Theorem 3. Let r ∈ [0, R max] be bounded reward and confusion matrix C is invertible. Then, the variance of surrogate rewardr is bounded as follows: DISPLAYFORM4 To give an intuition of the bound, when we have binary reward, the variance for surrogate reward bounds as follows: DISPLAYFORM5 (1−e+−e−) 2. As e − + e + → 1, the variance becomes unbounded and the proposed estimator is no longer effective, nor will it be well-defined. In practice, there is a trade-off question between bias and variance by tuning a linear combination of R andR, i.e., R proxy = ηR + (1 − η)R, and choosing an appropriate η ∈. In Section 3.1 we have assumed the knowledge of reward confusion matrices, in order to compute the surrogate reward. This knowledge is often not available in practice. Estimating these confusion matrices is challenging without knowing any ground truth reward information; but we'd like to note that efficient algorithms have been developed to estimate the confusion matrices in supervised learning settings BID0;; ). The idea in these algorithms is to dynamically refine the error rates based on aggregated rewards. Note this approach is not different from the inference methods in aggregating crowdsourcing labels, as referred in the literature (; ;). We adapt this idea to our reinforcement learning setting, which is detailed as follows. At each training step, the RL agent collects the noisy reward and the current state-action pair. Then, for each pair in S × A, the agent predicts the true reward based on accumulated historical observations of reward for the corresponding state-action pair via, e.g., averaging (majority voting). Finally, with the predicted true reward and the accuracy (error rate) for each state-action pair, the estimated reward confusion matricesC are given bỹ DISPLAYFORM0 where in above # [·] denotes the number of state-action pair that satisfies the condition [·] in the set of observed rewardsR(s, a) (see Algorithm 1 and 3);r(s, a) andr(s, a) denote predicted true rewards (using majority voting) and observed rewards when the state-action pair is (s, a). The above procedure of updatingc i,j continues indefinitely as more observation arrives. Algorithm 1 Reward Robust RL (sketch) DISPLAYFORM1 Initialize value function Q(s, a) arbitrarily. while Q is not converged do Initialize state s ∈ S while s is not terminal do Choose a from s using policy derived from Q Take action a, observe s and noisy rewardr if collecting enoughr for every S × A pair then Get predicted true rewardr using majority voting Estimate confusion matrixC based onr andr (Eqn. 4) Obtain surrogate rewardṙ DISPLAYFORM2 Our final definition of surrogate reward replaces a known reward confusion C in Eqn. FORMULA4 with our estimated oneC. We denote this estimated surrogate reward asṙ. We present (Reward Robust RL) in Algorithm 1 4. Note that the algorithm is rather generic, and we can plug in any exisitng RL algorithm into our reward robust one, with only changes in replacing the rewards with our estimated surrogate rewards. In this section, reward robust RL is tested in different games, with different noise settings. Due to space limit, more experimental can be found in Appendix D. Environments and RL Algorithms To fully test the performance under different environments, we evaluate the proposed robust reward RL method on two classic control games (CartPole, Pendulum) and seven Atari 2600 games (AirRaid, Alien, Carnival, MsPacman, Pong, Phoenix, Seaquest), which encompass a large variety of environments, as well as rewards. Specifically, the rewards could be unary (CartPole), binary (most of Atari games), multivariate (Pong) and even continuous (Pendulum). A set of state-of-the-art reinforcement learning algorithms are experimented with while training under different amounts of noise (See TAB4 5 . For each game and algorithm, three policies are trained based on different random initialization to decrease the variance. Reward Post-Processing For each game and RL algorithm, we test the performances for learning with true rewards, learning with noisy rewards and learning with surrogate rewards. Both symmetric and asymmetric noise settings with different noise levels are tested. For symmetric noise, the confusion matrices are symmetric. As for asymmetric noise, two types of random noise are tested: 1) rand-one, each reward level can only be perturbed into another reward; 2) rand-all, each reward could be perturbed to any other reward, via adding a random noise matrix. To measure the amount of noise w.r.t confusion matrices, we define the weight of noise ω in Appendix B.2. The larger ω is, the higher the noise rates are. CartPole The goal in CartPole is to prevent the pole from falling by controlling the cart's direction and velocity. The reward is +1 for every step taken, including the termination step. When the cart or pole deviates too much or the episode length is longer than 200, the episode terminates. Due to the unary reward {+1} in CartPole, a corrupted reward −1 is added as the unexpected error (e − = 0). As a , the reward space R is extended to {+1, −1}. Five algorithms , , , and are evaluated. In FIG1, we show that our estimator successfully produces meaningful surrogate rewards that adapt the underlying RL algorithms to the noisy settings, without any assumption of the true distribution of rewards. With the noise rate increasing (from 0.1 to 0.9), the models with noisy rewards converge slower due to larger biases. However, we observe that the models always converge to the best score 200 with the help of surrogate rewards. DISPLAYFORM0 In some circumstances (slight noise -see FIG8, 6b, 6c, 6d), the surrogate rewards even lead to faster convergence. This points out an interesting observation: learning with surrogate reward even outperforms the case with observing the true reward. We conjecture that the way of adding noise and then removing the bias introduces implicit exploration. This implies that for settings even with true reward, we might consider manually adding noise and then remove it in expectation. Pendulum The goal in Pendulum is to keep a frictionless pendulum standing up. Different from the CartPole setting, the rewards in pendulum are continuous: r ∈ (−16.28, 0.0]. The closer the reward is to zero, the better performance the model achieves. Following our extension (see Section 3.1), the (−17, 0] is firstly discretized into 17 intervals: (−17, −16], (−16, −15], · · ·, (−1, 0], with its value approximated using its maximum point. After the quantization step, the surrogate rewards can be estimated using multi-outcome extensions presented in Section 3.1. We experiment two popular algorithms, and in this game. In FIG2, both algorithms perform well with surrogate rewards under different amounts of noise. In most cases, the biases were corrected in the long-term, even when the amount of noise is extensive (e.g., ω = 0.7). The quantitative scores on CartPole and Pendulum are given in TAB1, where the TAB2. Our reward robust RL method is able to achieve consistently good scores. Atari We validate our algorithm on seven Atari 2600 games using the state-of-the-art algorithm PPO BID4. The games are chosen to cover a variety of environments. The rewards in the Atari games are clipped into {−1, 0, 1}. We leave the detailed settings to Appendix B. Results for PPO on Pong-v4 in symmetric noise setting are presented in FIG3. Due to limited space, more on other Atari games and noise settings are given in Appendix D.3. Similar to previous , our surrogate estimator performs consistently well and helps PPO converge to the optimal policy. TAB2 shows the average scores of PPO on five selected Atari games with different amounts of noise (symmetric & asymmetric). In particular, when the noise rates e + = e − > 0.3, agents with surrogate rewards obtain significant amounts of improvements in average scores. We do not present the for the case with unknown C because the state-space (image-input) is very large for Atari games, which is difficult to handle with the solution given in Section 3.3. DISPLAYFORM1 Only an underwhelming amount of reinforcement learning studies have focused on the settings with perturbed and noisy rewards, despite the fact that such noises are common when exploring a realworld scenario, that faces sensor errors or adversarial examples. We adapt the ideas from supervised Er |r (r) = Pr |r (r =r −)r − + Pr |r (r =r +)r +.When r = r +, from the definition in Lemma 1:Pr |r (r =r −) = e +, Pr |r (r =r +) = 1 − e +. Taking the definition of surrogate rewards Eqn. FORMULA2 DISPLAYFORM0 Similarly, when r = r −, it also verifies Er |r [r(s t, a t, s t+1)] = r(s t, a t, s t+1).Proof of Lemma 2. The idea of constructing unbiased estimator is easily adapted to multi-outcome reward settings via writing out the conditions for the unbiasedness property (s.t. Er |r [r] = r.). For simplicity, we shorthandr(r = R i) asR i in the following proofs. Similar to Lemma 1, we need to solve the following set of functions to obtainr: DISPLAYFORM1 whereR i denotes the value of the surrogate reward when the observed reward is R i. Define R:= [R 0 ; R 1 ; · · · ; R M −1], andR:= [R 0,R 1, ...,R M −1], then the above equations are equivalent to: R = C ·R. If the confusion matrix C is invertible, we obtain the surrogate reward: DISPLAYFORM2 According to above definition, for any true reward level R i, i = 0, 1, · · ·, M − 1, we have DISPLAYFORM3 Furthermore, the probabilities for observing surrogate rewards can be written as follows: DISPLAYFORM4 wherep i = j p j c j,i, andp i, p i represent the probabilities of occurrence for surrogate rewardR i and true reward R i respectively. Corollary 1. Letp i and p i denote the probabilities of occurrence for surrogate rewardr(r = R i) and true reward R i. Then the surrogate reward satisfies, DISPLAYFORM5 Proof of Corollary 1. From Lemma 2, we have, DISPLAYFORM6 Consequently, DISPLAYFORM7 To establish Theorem 1, we need an auxiliary (Lemma 3) from stochastic process approximation, which is widely adopted for the convergence proof for Q-Learning (; BID14 . Lemma 3. The random process {∆ t} taking values in R n and defined as DISPLAYFORM8 converges to zero w.p.1 under the following assumptions: DISPLAYFORM9 Here F t = {∆ t, ∆ t−1, · · ·, F t−1 · · ·, α t, · · ·} stands for the past at step t, α t (x) is allowed to depend on the past insofar as the above conditions remain valid. The notation || · || W refers to some weighted maximum norm. Proof of Lemma 3. See previous literature (; BID14 .Proof of Theorem 1. For simplicity, we abbreviate s t, s t+1, Q t, Q t+1, r t,r t and α t as s, s, Q, Q, r,r, and α, respectively. Subtracting from both sides the quantity Q * (s, a) in Eqn.: DISPLAYFORM10 In consequence, DISPLAYFORM11 Finally, DISPLAYFORM12 Becauser is bounded, it can be clearly verified that DISPLAYFORM13 for some constant C. Then, due to the Lemma 3, ∆ t converges to zero w.p.1, i.e., Q (s, a) converges to Q * (s, a).The procedure of Phased Q-Learning is described as Algorithm 2: DISPLAYFORM14 DISPLAYFORM15 Note thatP here is the estimated transition probability, which is different from P in Eqn. FORMULA22.To obtain the sample complexity , the range of our surrogate reward needs to be known. Assuming reward r is bounded in [0, R max], Lemma 4 below states that the surrogate reward is also bounded, when the confusion matrices are invertible:Lemma 4. Let r ∈ [0, R max] be bounded, where R max is a constant; suppose C M ×M, the confusion matrix, is invertible with its determinant denoting as det(C). Then the surrogate reward satisfies DISPLAYFORM16 Proof of Lemma 4. From Eqn. FORMULA4, we have, DISPLAYFORM17 where adj(C) is the adjugate matrix of C; det(C) is the determinant of C. It is known from linear algebra that, DISPLAYFORM18 where M ji is the determinant of the (M − 1) × (M − 1) matrix that from deleting row j and column i of C. Therefore, M ji is also bounded: DISPLAYFORM19 where the sum is computed over all permutations σ of the set {0, 1, · · ·, M − 2}; c is the element of M ji; sgn(σ) returns a value that is +1 whenever the reordering given by σ can be achieved by successively interchanging two entries an even number of times, and −1 whenever it can not. Consequently, DISPLAYFORM20 Proof of Theorem 2. From Hoeffding's inequality, we obtain: DISPLAYFORM21 In the same way,r t is bounded by M det(C) · R max from Lemma 4. We then have, DISPLAYFORM22 Further, due to the unbiasedness of surrogate rewards, we have st+1∈S P a (s t, s t+1)r t = st+1∈S;rt∈R P a (s t, s t+1,r t)r t.As a , DISPLAYFORM23 In the same way, DISPLAYFORM24 Recursing the two equations in two directions (0 → T), we get DISPLAYFORM25 Combining these two inequalities above we have: DISPLAYFORM26 For arbitrarily small, by choosing m appropriately, there always exists 1 = 2 =(1−γ) 2(1+γ) such that the policy error is bounded within. That is to say, the Phased Q-Learning algorithm can converge to the near optimal policy within finite steps using our proposed surrogate rewards. Finally, there are |S||A|T transitions under which these conditions must hold, where | · | represent the number of elements in a specific set. Using a union bound, the probability of failure in any condition is smaller than DISPLAYFORM27 We set the error rate less than δ, and m should satisfy that DISPLAYFORM28 In consequence, after m|S||A|T calls, which is, O DISPLAYFORM29, the value function converges to the optimal one for every state s, with probability greater than 1 − δ. The above bound is for discounted MDP setting with 0 ≤ γ < 1. For undiscounted setting γ = 1, since the total error (for entire trajectory of T time-steps) has to be bounded by, therefore, the error for each time step has to be bounded by T. Repeating our anayslis, we obtain the following upper bound: DISPLAYFORM30 Proof of Theorem 3. DISPLAYFORM31 Using the CauchySchwarz inequality, DISPLAYFORM32 So we get, Var(r) − Var(r) ≥ 0. In addition, DISPLAYFORM33 We set up our experiments within the popular OpenAI baselines BID4 and kerasrl framework. Specifically, we integrate the algorithms and interact with OpenAI Gym environments TAB4. A set of state-of-the-art reinforcement learning algorithms are experimented with while training under different amounts of noise, including Q-Learning BID19 BID18, Cross-Entropy Method (CEM) BID11, Deep SARSA BID10, Deep Q-Network (DQN) (; BID6, Dueling DQN (DDQN) BID17, Deep Deterministic Policy Gradient (DDPG) , Continuous DQN (NAF) and Proximal Policy Optimization (PPO) BID4 algorithms. For each game and algorithm, three policies are trained based on different random initialization to decrease the variance in experiments. We explore both symmetric and asymmetric noise of different noise levels. For symmetric noise, the confusion matrices are symmetric, which means the probabilities of corruption for each reward choice are equivalent. For instance, a confusion matrix DISPLAYFORM0 says that r 1 could be corrupted into r 2 with a probability of 0.2 and so does r 2 (weight = 0.2).As for asymmetric noise, two types of random noise are tested: 1) rand-one, each reward level can only be perturbed into another reward; 2) rand-all, each reward could be perturbed to any other reward. To measure the amount of noise w.r.t confusion matrices, we define the weight of noise as follows: DISPLAYFORM1, where ω controls the weight of noise; I and N denote the identity and noise matrix respectively. Suppose there are M outcomes for true rewards, N writes as: DISPLAYFORM2 where for each row i, 1) rand-one: randomly choose j, s.t n i,j = 1 and n i,k = 0 if k = j; 2) randall: generate M random numbers that sum to 1, i.e., j n i,j = 1. For the simplicity, for symmetric noise, we choose N as an anti-identity matrix. As a , c i,j = 0, if i = j or i + j = M. To obtain an intuitive view of the reward perturbation model, where the observed rewards are generated based on a reward confusion matrix, we constructed a simple MDP and evaluated the performance of robust reward Q-Learning (Algorithm 1) on different noise ratios (both symmetric and asymmetric). The finite MDP is formulated as FIG6: when the agent reaches state 5, it gets an instant reward of r + = 1, otherwise a zero reward r − = 0. During the explorations, the rewards are perturbed according to the confusion matrix C 2×2 = [1 − e −, e − ; e +, 1 − e +]. There are two experiments conducted in this setting: 1) performance of Q-Learning under different noise rates TAB5; 2) robustness of estimation module in time-variant noise (FIG6). As shown in TAB5, Q-Learning achieved better consistently with the guidance of surrogate rewards and the confusion matrix estimation algorithm. For time-variant noise, we generated varying amount of noise at different training stages: 1) e − = 0.1, e + = 0.3 (0 to 1e 4 steps); 2) e − = 0.2, e + = 0.1 (1e 4 to 3e 4 steps); 3) e − = 0.3, e + = 0.2 (3e 4 to 5e 4 steps); 4) e − = 0.1, e + = 0.2 (5e 4 to 7e 4 steps). In FIG6, we show that Algorithm 1 is robust against time-variant noise, which dynamically adjusts the estimatedC after the noise distribution changes. Note that we set a maximum memory size for collected noisy rewards to let the agents only learn with recent observations. CartPole and Pendulum The policies use the default network from keras-rl framework. which is a five-layer fully connected network 6. There are three hidden layers, each of which has 16 units and followed by a rectified nonlinearity. The last output layer is activated by the linear function. For CartPole, We trained the models using Adam optimizer with the learning rate of 1e −3 for 10,000 steps. The exploration strategy is Boltzmann policy. For DQN and Dueling-DQN, the update rate of target model and the memory size are 1e −2 and 50, 000. For Pendulum, We trained DDPG and NAF using Adam optimizer with the learning rate of 5e −4 for 150, 000 steps. the update rate of target model and the memory size are 1e −3 and 100, 000.Atari Games We adopt the pre-processing steps as well as the network architecture from. Specifically, the input to the network is 84×84×4, which is a concatenation of the last 4 frames and converted into 84 × 84 gray-scale. The network comprises three convolutional layers and two fully connected layers 7. The kernel size of three convolutional layer are 8 × 8 with stride 4 (32 filters), 4 × 4 with stride 2 (64 filters) and 3 × 3 with stride 1 (64 filters), respectively. Each hidden layer is followed by a rectified nonlinearity. Except for Pong where we train the policies for 3e 7 steps, all the games are trained for 5e 7 steps with the learning rate of 3e −4. Note that the rewards in the Atari games are discrete and clipped into {−1, 0, 1}. Except for Pong game, in which r = −1 means missing the ball hit by the adversary, the agents in other games attempt to get higher scores in the episode with binary rewards 0 and 1. C.1 REWARD ROBUST RL ALGORITHMS As stated in Section 3.3, the confusion matrix can be estimated dynamically based on the aggregated answers, similar to previous literature in supervised learning . To get a concrete view, we take Q-Learning for an example, and the algorithm is called Reward Robust Q-Learning (Algorithm 3). Note that is can be extended to other RL algorithms by plugging confusion matrix estimation steps and the computed surrogate rewards, as shown in the experiments FIG8 ). Get predicted true rewardr(s, a) using majority voting in everyR(s, a) Estimate confusion matrixC based onr(s, a) andr(s, a) (Eqn. FORMULA11) Empty all the sets of observed rewardsR(s, a) Obtain surrogate rewardṙ(s, a) using DISPLAYFORM0 In Algorithm 3, the predicted true rewardr(s, a) is derived from majority voting in collected noisy setsR(s, a) for every state-action pair (s, a) ∈ S × A, which is a simple but efficient way of leveraging the expectation of aggregated rewards without assumptions on prior distribution of noise. In the following, we adopt standard Expectation-Maximization (EM) idea in the our estimation framework (arguably a simple version of it), inspired by previous works BID20.Assuming the observed noisy rewards are independent conditional on the true reward, we can compute the posterior probability of true reward from the Bayes' theorem: DISPLAYFORM0 where P(r = R j) is the prior of true rewards, and P(r = R k |r = R j) is estimated by current estimated confusion matrixC: P(r = R k |r = R j) =c j,i. Note that the inference should be conducted for each state-action pair (s, a) ∈ S × A in every iteration, i.e., P(r(s, a) = R i |r(s, a, 1) = R 1, · · ·,r(s, a, n) = R n ), abbreviated as P(r(s, a) = R i ), which requires relatively greater computation costs compared to the majority voting policy. It also points out an interesting direction to check online EM algorithms for our perturbed-RL problem. After the inference steps in Eqn., the confusion matrixC is then updated based on the posterior probabilities:c DISPLAYFORM1 where P(r(s, a) = R i ) denotes the inference probabilities of true rewards based on collected noisy rewards setsR(s, a). To utilize EM algorithms in the robust reward algorithms (e.g., Algorithm 3), we need to replace Eqn. by Eqn. for the estimation of reward confusion matrix. In previous sections, to let our presentation stay focused, we consider the state-independent perturbed reward environments, which share the same confusion matrix for all states. In other words, the noise for different states is generated within the same distribution. More generally, the generation ofr follows a certain function C: S × R →R, where different states may correspond to varied noise distributions (also varied confusion matrices). However, our algorithm is still applicable except for maintaining different confusion matrices C s for different states. It is worthy to notice that Theorem 1 holds because the surrogate rewards produce an unbiased estimation of true rewards for each state, i.e., Er |r,st [r(s t, a t, s t+1)] = r(s t, a t, s t+1). Furthermore, Theorem 2 and 3 can be revised as:Theorem 4. (Upper bound) Let r ∈ [0, R max] be bounded reward, C s be invertible reward confusion matrices with det(C s) denoting its determinant. For an appropriate choice of m, the Phased Q-Learning algorithm calls the generative model DISPLAYFORM0 times in T epochs, and returns a policy such that for all state s ∈ S, |V π (s) − V * (s)| ≤, > 0, w.p. ≥ 1 − δ, 0 < δ < 1.Theorem 5. Let r ∈ [0, R max] be bounded reward and all confusion matrices C s are invertible. Then, the variance of surrogate rewardr is bounded as follows: DISPLAYFORM1 As illustrated in Theorem 3, our surrogate rewards introduce larger variance while conducting unbiased estimation which are likely to decrease the stability of RL algorithms. Apart from the linear combination idea (appropriate trade-off), some variance reduction techniques in statistics (e.g., correlated sampling) can also be applied into our surrogate rewards. Specially, BID2 proposed to a reward estimator to compensate for stochastic corrupted reward signals. It is worthy to notice that their method is designed for variance reduction under stochastic (zero-mean) noise, which is no longer efficacious in more general perturbed-reward setting. However, it is potential to integrate their method with our robust-reward RL framework because surrogate rewards guarantee unbiasedness in reward expectation. To verify this idea, we repeated the experiments of Cartpole in Section 4.2 but included variance reduction step for estimated surrogate rewards. Following BID2, we adopted sample mean as a simple approximator during the training and set sequence length as 100. As shown in Figure 5, the models with only variance reduction technique (red lines) suffer from huge biases when the noise is large, and cannot converge to the optimal policies like those under noisy rewards. Nevertheless, they benefits from variance reduction for surrogate rewards (purple lines), which achieve faster convergence or better performance in many cases (e.g., Figure 5a (ω = 0.7), 5b (ω = 0.3)).It is also not surprising that the integrated algorithm (purple lines) outperforms better as the noise rate increases (indicating larger variance from Theorem 3, e.g., ω = 0.9). Similarly, TAB6 provides quantitative which show that our surrogate benefits from variance reduction techniques ("ours + VRT"), especially when the noise rate is large. DISPLAYFORM0 Figure 5: Learning curves from five reward robust RL algorithms (see Algorithm 3) on CartPole game with true rewards (r), noisy rewards (r) (η = 1), sample-mean noisy rewards (η = 1), estimated surrogate rewards (ṙ) and sample-mean estimated surrogate rewards. Note that confusion matrices C are unknown to the agents here. From top to the bottom, the noise rates are 0.1, 0.3, 0.7 and 0.9. Here we repeated each experiment 10 times with different random seeds and plotted 10% to 90% percentile area with its mean highlighted. To validate the effectiveness of robust reward algorithms (like Algorithm 3), where the noise rates are unknown to the agents, we conduct extensive experiments in CartPole. It is worthwhile to notice that the noisy rates are unknown in the explorations of RL agents. Besides, we discretize the Figure 6 provides learning curves from five algorithms with different kinds of rewards. The proposed estimation algorithms successfully obtain the approximate confusion matrices, and are robust in the unknown noise environments. From FIG9, we can observe that the estimation of confusion matrices converges very fast. The are inspiring because we don't assume any additional knowledge about noise or true reward distribution in the implementation. and estimated surrogate rewards (ṙ). Note that confusion matrices C are unknown to the agents here. From top to the bottom, the noise rates are 0.1, 0.3, 0.7 and 0.9. Here we repeated each experiment 10 times with different random seeds and plotted 10% to 90% percentile area with its mean highlighted.

A new approach for learning with noisy rewards in reinforcement learning

represent the tldr for finding the corresponding scientific article

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.