Question: Represent the scientific article to find a short TLDR

Question: In recent years several adversarial attacks and defenses have been proposed. Often seemingly robust models turn out to be non-robust when more sophisticated attacks are used. One way out of this dilemma are provable robustness guarantees. While provably robust models for specific $l_p$-perturbation models have been developed, we show that they do not come with any guarantee against other $l_q$-perturbations. We propose a new regularization scheme, MMR-Universal, for ReLU networks which enforces robustness wrt $l_1$- \textit{and} $l_\infty$-perturbations and show how that leads to the first provably robust models wrt any $l_p$-norm for $p\geq 1$. The vulnerability of neural networks against adversarial manipulations is a problem for their deployment in safety critical systems such as autonomous driving and medical applications. In fact, small perturbations of the input which appear irrelevant or are even imperceivable to humans change the decisions of neural networks. This questions their reliability and makes them a target of adversarial attacks. To mitigate the non-robustness of neural networks many empirical defenses have been proposed, e.g. by;;;;; , but at the same time more sophisticated attacks have proven these defenses to be ineffective (; ;), with the exception of the adversarial training of. However, even these l ∞ -adversarially trained models are not more robust than normal ones when attacked with perturbations of small l p -norms with p = ∞ (; ; b;). The situation becomes even more complicated if one extends the attack models beyond l p -balls to other sets of perturbations (; ; ;). Another approach, which fixes the problem of overestimating the robustness of a model, is provable guarantees, which means that one certifies that the decision of the network does not change in a certain l p -ball around the target point. Along this line, current state-of-theart methods compute either the norm of the minimal perturbation changing the decision at a point (e.g. ;) or lower bounds on it (; ;). Several new training schemes like (; ; ; ; a; ;) aim at both enhancing the robustness of networks and producing models more amenable to verification techniques. However, all of them are only able to prove robustness against a single kind of perturbations, typically either l 2 -or l ∞ -bounded, and not wrt all the l p -norms simultaneously, as shown in Section 5. Some are also designed to work for a specific p , and it is not clear if they can be extended to other norms. The only two papers which have shown, with some limitations, non-trivial empirical robustness against multiple types of adversarial examples are and Tramèr & Boneh In this paper we aim at robustness against all the l p -bounded attacks for p ≥ 1. We study the non-trivial case where none of the l p -balls is contained in another. If p is the radius of the l p -ball for which we want to be provably robust, this requires: q > p > q for p < q and d being the input dimension. We show that, for normally trained models, for the l 1 -and l ∞ -balls we use in the experiments none of the adversarial examples constrained to be in the l 1 -ball (i.e. of an l 1 -attack) belongs to the l ∞ -ball, and vice versa. This shows that certifying the union of such balls is significantly more complicated than getting robust in only one of them, as in the case of the union the attackers have a much larger variety of manipulations available to fool the classifier. We propose a technique which allows to train piecewise affine models (like ReLU networks) which are simultaneously provably robust to all the l p -norms with p ∈ [1, ∞]. First, we show that having guarantees on the l 1 -and l ∞ -distance to the decision boundary and region boundaries (the borders of the polytopes where the classifier is affine) is sufficient to derive meaningful certificates on the robustness wrt all l p -norms for p ∈ (1, ∞). In particular, our guarantees are independent of the dimension of the input space and thus go beyond a naive approach where one just exploits that all l p -metrics can be upper-and lower-bounded wrt any other l q -metric. Then, we extend the regularizer introduced in Croce et al. (2019a) so that we can directly maximize these bounds at training time. Finally, we show the effectiveness of our technique with experiments on four datasets, where the networks trained with our method are the first ones having non-trivial provable robustness wrt l 1 -, l 2 -and l ∞ -perturbations. It is well known that feedforward neural networks (fully connected, CNNs, residual networks, DenseNets etc.) with piecewise affine activation functions, e.g. ReLU, leaky ReLU, yield continuous piecewise affine functions (see e.g. ;). Croce et al. (2019a) exploit this property to derive bounds on the robustness of such networks against adversarial manipulations. In the following we recall the guarantees of Croce et al. (2019a) wrt a single l p -perturbation which we extend in this paper to simultaneous guarantees wrt all the l p -perturbations for p in [1, ∞]. Let f: R d → R K be a classifier with d being the dimension of the input space and K the number of classes. The classifier decision at a point x is given by arg max r=1,...,K f r (x). In this paper we deal with ReLU networks, that is with ReLU activation function (in fact our approach can be easily extended to any piecewise affine activation function e.g. leaky ReLU or other forms of layers leading to a piecewise affine classifier as in Croce et al. (2019b) Denoting the activation function as σ (σ(t) = max{0, t} if ReLU is used) and assuming L hidden layers, we have the usual recursive definition of f as where n l is the number of units in the l-th layer (For the convenience of the reader we summarize from the description of the polytope Q(x) containing x and affine form of the classifier f when restricted to Q(x). We assume that x does not lie on the boundary between polytopes (this is almost always true as faces shared between polytopes are of lower dimension). Let This allows us to write f (l) (x) as composition of affine functions, that is which we simplify as A forward pass through the network is sufficient to compute V (l) and b (l) for every l. The polytope Q(x) is given as intersection of N = L l=1 n l half spaces defined by Let q be defined via.., K and s = c, which represent the N l p -distances of x to the hyperplanes defining the polytope Q(x) and the K − 1 l p -distances of x to the hyperplanes defining the decision boundaries in Q(x). Finally, we define as the minimum values of these two sets of distances (note that d The l p -robustness r p (x) of a classifier f at a point x, belonging to class c, wrt the l p -norm is defined as the optimal value of the following optimization problem where is S a set of constraints on the input, e.g. pixel values of images have to be in. The l p -robustness r p (x) is the smallest l p -distance to x of a point which is classified differently from c. Thus, r p (x) = 0 for misclassified points. The following theorem from Croce et al. (2019a), rephrased to fit the current notation, provides guarantees on r p (x). Although Theorem 2.1 holds for any l p -norm with p ≥ 1, it requires to compute d B p (x) and d D p (x) for every p individually. In this paper, exploiting this and the geometrical arguments presented in Section 3, we show that it is possible to derive bounds on the robustness r p (x) for any p ∈ (1, ∞) using only information on r 1 (x) and r ∞ (x). In the next section, we show that the straightforward usage of standard l p -norms inequalities does not yield meaningful bounds on the l p -robustness inside the union of the l 1 -and l ∞ -ball, since these bounds depend on the dimension of the input space of the network. Figure 1: Visualization of the l 2 -ball contained in the union resp. the convex hull of the union of l 1 -and l ∞ -balls in R 3. First column: co-centric l 1 -ball (blue) and l ∞ -ball (black). Second: in red the largest l 2 -ball completely contained in the union of l 1 -and l ∞ -ball. Third: in green the convex hull of the union of the l 1 -and l ∞ -ball. Fourth: the largest l 2 -ball (red) contained in the convex hull. The l 2 -ball contained in the convex hull is significantly larger than that contained in the union of l 1 -and l ∞ -ball. 3 Minimal l p -norm of the complement of the union of l 1 -and l ∞ -ball and its convex hull By the standard norm inequalities it holds, for every x ∈ R d, that and thus a naive application of these inequalities yields the bound However, this naive bound does not take into account that we know that x 1 ≥ 1 and x ∞ ≥ ∞. Our first yields the exact value taking advantage of this information. Thus a guarantee both for l 1 -and l ∞ -ball yields a guarantee for all intermediate l p -norms. However, for affine classifiers a guarantee for B 1 and B ∞ implies a guarantee wrt the convex hull C of their union B 1 ∪ B ∞. This can be seen by the fact that an affine classifier generates two half-spaces, and the convex hull of a set A is the intersection of all half-spaces containing A. Thus, inside C the decision of the affine classifier cannot change if it is guaranteed not to change in B 1 and B ∞, as C is completely contained in one of the half-spaces generated by the classifier (see Figure 1 for illustrations of B 1, B ∞, their union and their convex hull). With the following theorem, we characterize, for any p ≥ 1, the minimal l p -norm over R d \ C. where α = 5) (red) and its naive lower bound (green). We fix ∞ = 1 and show the varying 1 ∈ (1, d), for d = 784 and d = 3072. We plot the value (or a lower bound in case of) of the minimal x 2, depending on 1, given by the different approaches (first and third plots). The red curves are almost completely hidden by the green ones, as they mostly overlap, but can be seen for small values of x 1. Moreover, we report (second and fourth plots) the ratios of the minimal The values provided by are much larger than those of. Note that our expression in Theorem 3.1 is exact and not just a lower bound. Moreover, the minimal l p -distance of R d \ C to the origin in Equation is independent from the dimension d, in contrast to the expression for the minimal l p -norm over R d \ U 1,∞ in and its naive lower bound in, which are both decreasing for increasing d and p > 1. In Figure 1 we compare visually the largest l 2 -balls (in red) fitting inside either U 1,∞ or the convex hull C in R 3, showing that the one in C is clearly larger. In Figure 2 we provide a quantitative comparison in high dimensions. We plot the minimal l 2 -norm over R d \ C (blue) and over R d \ U 1,∞ (red) and its naive lower bound (green). We fix x ∞ = ∞ = 1 and vary e. the dimensions of the input spaces of MNIST and CIFAR-10. One sees clearly that the blue line corresponding to is significantly higher than the other two. In the second and fourth plots of Figure 2 we show, for each 1, the ratio of the l 2 -distances given by and. The maximal ratio is about 3.8 for d = 784 and 5.3 for d = 3072, meaning that the advantage of increases with d. These two examples indicate that the l p -balls contained in C can be a few times larger than those in U 1,∞. Recall that we deal with piecewise affine networks. If we could enlarge the linear regions on which the classifier is affine so that it contains the l 1 -and l ∞ -ball of some desired radii, we would automatically get the l p -balls of radii given by Theorem 3.1 to fit in the linear regions. The next section formalizes the ing robustness guarantees. Combining the of Theorems 2.1 and 3.1, in the next theorem we derive lower bounds on the robustness of a continuous piecewise affine classifier f, e.g. a ReLU network, at a point x wrt any l p -norm with p ≥ 1 using only d). for any p ∈ (1, ∞), with α = It tries to push the k B closest hyperplanes defining Q(x) farther than γ B from x and the k D closest decision hyperplanes farther than γ D from x both wrt l p -metric. In other words, MMR-l p aims at widening the linear regions around the training points so that they contain l p -balls of radius either γ B or γ D centered in the training points. Using MMR-l p wrt a fixed l p -norm, possibly in combination with the adversarial training of , leads to classifiers which are empirically resistant wrt l p -adversarial attacks and are easily verifiable by state-of-the-art methods to provide lower bounds on the true robustness. where We stress that, even if the formulation of MMR-Universal is based on MMR-l p, it is just thanks to the novel geometrical motivation provided by Theorem 3.1 and its interpretation in terms of robustness guarantees of Theorem 4.1 that we have a theoretical justification of MMR-Universal. Moreover, we are not aware of any other approach which can enforce simultaneously l 1 -and l ∞ -guarantees, which is the key property of MMR-Universal. The loss function which is minimized while training the classifier f is then, with {( being the training set and CE the cross-entropy loss, During the optimization our regularizer aims at pushing both the polytope boundaries and the decision hyperplanes farther than γ 1 in l 1 -distance and farther than γ ∞ in l ∞ -distance from the training point x, in order to achieve robustness close or better than γ 1 and γ ∞ respectively. According to Theorem 4.1, this enhances also the l p -robustness for p ∈ (1, ∞). Note that if the projection of x on a decision hyperplane does not lie inside is just an approximation of the signed distance to the true decision surface, in which case Croce et al. (2019a) argue that it is an approximation of the local Cross-Lipschitz constant which is also associated to robustness (see). The regularization parameters λ 1 and λ ∞ are used to balance the weight of the l 1 -and l ∞ -term in the regularizer, and also wrt the cross-entropy loss. Note that the terms of MMR-Universal involving the quantities d (x) penalize misclassification, as they take negative values in this case. Moreover, we take into account the k B closest hyperplanes and not just the closest one as done in Theorems 2.1 and 4.1. This has two reasons: first, in this way the regularizer enlarges the size of the linear regions around the training points more quickly and effectively, given the large number of hyperplanes defining each polytope. Second, pushing many hyperplanes influences also the neighboring linear regions of Q(x). This comes into play when, in order to get better bounds on the robustness at x, one wants to explore also a portion of the input space outside of the linear region Q(x), which is where Theorem 4.1 holds. As noted in; Croce et al. (2019a); , established methods to compute lower bounds on the robustness are loose or completely fail when using normally trained models. In fact, their effectiveness is mostly related to how many ReLU units have stable sign when perturbing the input x within a given l p -ball. This is almost equivalent to having the hyperplanes far from x in l p -distance, which is what MMR-Universal tries to accomplish. This explains why in Section 5 we can certify the models trained with MMR-Universal with the methods of and. We compare the models obtained via our MMR-Universal regularizer to state-of-the-art methods for provable robustness and adversarial training. As evaluation criterion we use the robust test error, defined as the largest classification error when every image of the test set can be perturbed within a fixed set (e.g. an l p -ball of radius p). We focus on the l p -balls with p ∈ {1, 2, ∞}. Since computing the robust test error is in general an NP-hard problem, we evaluate lower and upper bounds on it. The lower bound is the fraction of points for which an attack can change the decision with perturbations in the l p -balls of radius p (adversarial samples), that is with l p -norm smaller than p. For this task we use the PGD-attack . In choosing the values of p for p ∈ {1, 2, ∞}, we try to be consistent with previous literature (e.g. ; Croce et al. (2019a) ) for the values of ∞ and 2. Equation provides, given 1 and ∞, a value at which one can expect l 2 -robustness (approximately 2 = √ 1 ∞). Then we fix 1 such that this approximation is slightly larger than the desired 2. We show in Table 1 the values chosen for p, p ∈ {1, 2, ∞}, and used to compute the robust test error in Table 2. Notice that for these values no l p -ball is contained in the others. Moreover, we compute for the plain models the percentage of adversarial examples given by an l 1 -attack (we use the PGD-attack) with budget 1 which have also l ∞ -norm smaller than or equal to ∞, and vice versa. These percentages are zero for all the datasets, meaning that being (provably) robust in the union of these l p -balls is much more difficult than in just one of them (see also C.1). We train CNNs on MNIST, Fashion-MNIST (et al. (2019a), either alone or with adversarial training (MMR+AT) and the training with our regularizer MMR-Universal. We use AT, KW, MMR and MMR+AT wrt l 2 and l ∞, as these are the norms for which such methods have been used in the original papers. More details about the architecture and models in C.3. In Table 2 we report test error (TE) computed on the whole test set and lower (LB) and upper (UB) bounds on the robust test error obtained considering the union of the three l p -balls, indicated by l 1 + l 2 + l ∞ (these statistics are on the first 1000 points of the test set). The lower bounds l 1 + l 2 + l ∞ -LB are given by the fraction of test points for which one of the adversarial attacks wrt l 1, l 2 and l ∞ is successful. The upper bounds l 1 + l 2 + l ∞ -UB are computed as the percentage of points for which at least one of the three l p -balls is not certified to be free of adversarial examples (lower is better). This last one is the metric of main interest, since we aim at universally provably robust models. In C.2 we report the lower and upper bounds for the individual norms for every model. MMR-Universal is the only method which can give non-trivial upper bounds on the robust test error for all datasets, while almost all other methods aiming at provable robustness have l 1 + l 2 + l ∞ -UB close to or at 100%. Notably, on GTS the upper bound on the robust test error of MMR-Universal is lower than the lower bound of all other methods except AT-(l 1, l 2, l ∞), showing that MMR-Universal provably outperforms existing methods which provide guarantees wrt individual l p -balls, either l 2 or l ∞, when certifying the union l 1 +l 2 +l ∞. The test error is slightly increased wrt the other methods giving provable robustness, but the same holds true for combined adversarial training AT-(l 1, l 2, l ∞) compared to standard adv. training AT-l 2 /l ∞. We conclude that MMR-Universal is the only method so far being able to provide non-trivial robustness guarantees for multiple l p -balls in the case that none of them contains any other. We have presented the first method providing provable robustness guarantees for the union of multiple l p -balls beyond the trivial case of the union being equal to the largest one, establishing a baseline for future works. Without loss of generality after a potential permutation of the coordinates it holds |x d | = x ∞. Then we get, which finishes the proof. Proof. We first note that the minimum of the l p -norm over R d \ C lies on the boundary of C (otherwise any point on the segment joining the origin and y and outside C would have where (1 − α) ∞ < ∞ and Thus S would not span a face as a convex combination intersects the interior of C. This implies that if 1 e j is in S then all the vertices v of B ∞ in S need to have v j = ∞, otherwise S would not define a face of C. Analogously, if − 1 e j ∈ S then any vertex v of B ∞ in S has v j = − ∞. However, we note that out of symmetry reasons we can just consider faces of C in the positive orthant and thus we consider in the following just sets S which contain vertices of "positive type" 1 e j. Let now S be a set (not necessarily defining a face of C) containing h ≤ k vertices of B 1 and d − h vertices of B ∞ and P the matrix whose columns are these points. The matrix P has the form −h is a matrix whose entries are either ∞ or − ∞. If the matrix P does not have full rank then the origin belongs to any hyperplane containing S, which means it cannot be a face of C. This also implies A has full rank if S spans a face of C. We denote by π the hyperplane generated by the affine hull of S (the columns of P) assuming that A has full rank. Every point b belonging to the hyperplane π generated by S is such that there exists a unique a ∈ R d which satisfies where 1 d1,d2 is the matrix of size d 1 × d 2 whose entries are 1. The matrix (P, b) ∈ R d+1,d+1 need not have full rank, so that and then the linear system P a = b has a unique solution. We define the vector v ∈ R d as solution of, which is unique as P has full rank. From their definitions we have P a = b and 1 T a = 1, so that and thus noticing that this also implies that any vector b ∈ R d such that b, v = 1 belongs to π (suppose that ∃q / ∈ π with q, v = 1, then define c as the solution of P c = q and then 1 = q, v = P c, v = c, P T v = c, 1 which contradicts that q / ∈ π). Applying Hölder inequality to we get for any b ∈ π, where 1 p + 1 q = 1. Moreover, as p ∈ (1, ∞) there exists always a point b * for which holds as equality. In the rest of the proof we compute v q for any q > 1 when S is a face of C and then yields the desired minimal value of b p over all b lying in faces of C. which implies Moreover, we have Furthermore v 2 is defined as the solution of We note that all the entries of are either 1 or −1, so that the inner product between each row of A T and v 2 is a lower bound on the l 1 -norm of v 2. Since every entry of the r.h.s. of the linear system is, which combined with leads to In order to achieve equality u, v = v 1 it has to hold u i = sgn(v i) for every v i = 0. If at least two components of v were non-zero, the corresponding columns of A T would be identical, which contradicts the fact that A T has full rank. Thus v 2 can only have one non-zero component which in absolute value is equal to Thus, after a potential reordering of the components, v has the form From the second condition in, we have This means that, in order for S to define a face of C, we need h = k if α > 0, h ∈ {k − 1, k} if α = 0 (in this case choosing h = k − 1 or h = k leads to the same v, so in practice it is possible to use simply h = k for any α). Once we have determined v, we can use again and to see that Finally, for any v there exists b * ∈ π for which equality is achieved in. Suppose that this b * does not lie in a face of C. Then one could just consider the line segment from the origin to b * and the point intersecting the boundary of C would have smaller l p -norm contradicting the just derived inequality. Thus the b * realizing equality in lies in a face of C. Restricting the analysis to p = 2 for simplicity, we get and one can check that δ * is indeed a maximizer. Moreover, at δ * we have a ratio between the two bounds We observe that the improvement of the robustness guarantee by considering the convex hull instead of the union is increasing with dimension and is ≈ 3. Therefore the interior of the l 1 -ball of radius ρ 1 (namely, B 1 (x, ρ 1)) and of the l ∞ -ball of radius ρ ∞ (B ∞ (x, ρ ∞)) centered in x does not intersect with any of those hyperplanes. This implies that {π j} j are intersecting the closure of In Table 3 we compute the percentage of adversarial perturbations given by the PGD-attack wrt l p with budget p which have l q -norm smaller than q, for q = p (the values of p and q used are those from Table 1). We used the plain model of each dataset. The most relevant statistics of Table 3 are about the relation between the l 1 -and l ∞ -perturbations (first two rows). In fact, none of the adversarial examples wrt l 1 is contained in the l ∞ -ball, and vice versa. This means that, although the volume of the l 1 -ball is much smaller, even because of the intersection with the box constraints d, than that of the l ∞ -ball in high dimension, and most of it is actually contained in the l ∞ -ball, the adversarial examples found by l 1 -attacks are anyway very different from those got by l ∞ -attacks. The choice of such p is then meaningful, as the adversarial perturbations we are trying to prevent wrt the various norms are non-overlapping and in practice exploit regions of the input space significantly diverse one from another. Moreover, one can see that also the adversarial manipulations wrt l 1 and l 2 do not overlap. Regarding the case of l 2 and l ∞, for MNIST and F-MNIST it happens that the adversarial examples wrt l 2 are contained in the l ∞ -ball. However, as one observes in Table 4, being able to certify the l ∞ -ball is not sufficient to get non-trivial guarantees wrt l 2. In fact, all the models trained on these datasets to be provably robust wrt the l ∞ -norm, that is KW-l ∞, MMR-l ∞ and MMR+AT-l ∞, have upper bounds on the robust test error in the l 2 -ball larger than 99%, despite the values of the lower bounds are small (which means that the attacks could not find adversarial perturbations for many points). Such analysis confirms that empirical and provable robustness are two distinct problems, and the interaction of different kinds of perturbations, as we have, changes according to which of these two scenarios one considers. In Table 4 we report, for each dataset, the test error and upper and lower bounds on the robust test error, together with the p used, for each norm individually. It is clear that training for provable l p -robustness (expressed by the upper bounds) does not, in general, yield provable l q -robustness for q = p, even in the case where the lower bounds are small for both p and q. In order to compute the upper bounds on the robust test error in Tables 2 and 4 we use the method of for all the three l p -norms and that of only for the l ∞ -norm. This second one exploits a reformulation of the problem in in terms of mixed integer programming (MIP), which is able to exactly compute the solution of for p ∈ {1, 2, ∞}. However, such technique is strongly limited by its high computational cost. The only reason why it is possible to use it in practice is the exploitation of some presolvers which are able to reduce the complexity of the MIP. Unfortunately, such presolvers are effective just wrt l ∞. On the other hand, the method of applies directly to every l p -norm. This explains why the bounds provided for l ∞ are tighter than those for l 1 and l 2. The convolutional architecture that we use is identical to , which consists of two convolutional layers with 16 and 32 filters of size 4 × 4 and stride 2, followed by a fully connected layer with 100 hidden units. The AT-l ∞, AT-l 2, KW, MMR and MMR+AT training models are those presented in Croce et al. (2019a) and available at https: //github.com/max-andr/provable-robustness-max-linear-regions. We trained the AT-(l 1, l 2, l ∞) performing for each batch of the 128 images the PGD-attack wrt the three norms (40 steps for MNIST and F-MNIST, 10 steps for GTS and CIFAR-10) and then training on the point realizing the maximal loss (the cross-entropy function is used), for 100 epochs. For all experiments with MMR-Universal we use batch size 128 and we train the models for 100 epochs. Moreover, we use Adam optimizer of with learning rate of 5 × 10 −4 for MNIST and F-MNIST, 0.001 for the other datasets. We also reduce the learning rate by a factor of 10 for the last 10 epochs. On CIFAR-10 dataset we apply random crops and random mirroring of the images as data augmentation. For training we use MMR-Universal as in with k B linearly (wrt the epoch) decreasing from 20% to 5% of the total number of hidden units of the network architecture. We also use a training schedule for λ p where we linearly increase it from λ p /10 to λ p during the first 10 epochs. We employ both schemes since they increase the stability of training with MMR. In order to determine the best set of hyperparameters λ 1, λ ∞, γ 1, and γ ∞ of MMR, we perform a grid search over them for every dataset. In particular, we empirically found that the optimal values of γ p are usually between 1 and 2 times the p used for the evaluation of the robust test error, while the values of λ p are more diverse across the different datasets. Specifically, for the models we reported in Table 4 the following values for the (λ 1, λ ∞) have been used: (3.0, 12.0) for MNIST, (3.0, 40.0) for F-MNIST, (3.0, 12.0) for GTS and (1.0, 6.0) for CIFAR-10. In Tables 2 and 4, while the test error which is computed on the full test set, the statistics regarding upper and lower bounds on the robust test error are computed on the first 1000 points of the respective test sets. For the lower bounds we use the FAB-attack with the Figure 3: We show, for each dataset, the evolution of the test error (red), upper bound (UB) on the robust test error wrt l 1 (black), l 2 (cyan) and l ∞ (blue) during training. Moreover, we report in green the upper bounds on the test error when the attacker is allowed to exploit the union of the three l p -balls. The statistics on the robustness are computed at epoch 1, 2, 5, 10 and then every 10 epochs on 1000 points with the method of , using the models trained with MMR-Universal. original parameters, 100 iterations and 10 restarts. For PGD we use also 100 iterations and 10 restarts: the directions for the update step are the sign of the gradient for l ∞, the normalized gradient for l 2 and the normalized sparse gradient suggested by Tramèr & with sparsity level 1% for MNIST and F-MNIST, 10% for GTS and CIFAR-10. Finally we use the Liner Region Attack as in the original code. For MIP we use a timeout of 120s, that means if no guarantee is obtained by that time, the algorithm stops verifying that point. We show in Figure 3 the clean test error (red) and the upper bounds on the robust test error wrt l 1 (black), l 2 (cyan), l ∞ (blue) and wrt the union of the three l p -balls (green), evaluated at epoch 1, 2, 5, 10 and then every 10 epochs (for each model we train for 100 epochs) for the models trained with our regularizer MMR-Universal. For each dataset used in Section 5 the test error is computed on the whole test set, while the upper bound on the robust test error is evaluated on the first 1000 points of the test set using the method introduced in (the thresholds 1, 2, ∞ are those provided in Table 1). Note that the statistics wrt l ∞ are not evaluated additionally with the MIP formulation of as the in the main paper which would improve the upper bounds wrt l ∞. For all the datasets the test error keeps decreasing across epochs. The values of all the upper bounds generally improve during training, showing the effectiveness of MMR-Universal. We here report the robustness obtained training with MMR-l p +AT-l q with p = q on MNIST. This means that MMR is used wrt l p, while adversarial training wrt l q. In particular we test p, q ∈ {1, ∞}. In Table 5 we report the test error (TE), lower (LB) and upper bounds (UB) on the robust test error for such model, evaluated wrt l 1, l 2, l ∞ and l 1 + l 2 + l ∞ as done in Section 5. It is clear that training with MMR wrt a single norm does not suffice to get provable guarantees in all the other norms, despite the addition of adversarial training. In fact, for both the models analysed the UB equals 100% for at least one norm. Note that the statistics wrt l ∞ in the plots do not include the of the MIP formulation of.

represent the tldr for finding the corresponding scientific article

We introduce a method to train models with provable robustness wrt all the $l_p$-norms for $p\geq 1$ simultaneously.

represent the short summary for finding the corresponding scientific article (long)
------
Fewshot example: "Position paper proposing rebellious and deceptive explanations for agents." == "As the area of Explainable AI (XAI), and Explainable AI Planning (XAIP), matures, the ability for agents to generate and curate explanations will likewise grow. We propose a new challenge area in the form of rebellious and deceptive explanations. We discuss how these explanations might be generated and then briefly discuss evaluation criteria. Explanations as a research area in AI (XAI) has been around for several decades BID7 BID5 BID10 BID45 BID12 BID40 BID44 BID28. It has additionally gained momentum recently as evidenced by the increasing number of workshops and special tracks covering it in various conferences (e.g., VIS-xAI, FEAPAI4Fin, XAIP, XAI, OXAI, MAKE-eXAI, ICCBR-19 Focus area).While still growing in use, there have been some approaches to formalizing XAI. BID11 stated that anything calling itself XAI should address the following questions:• Why did the agent do that and not something else?• When does the agent succeed and when does it fail?• When can I trust the agent?However, less thought out is the idea of explanations that are deceptive or rebellious in nature. These forms of explanation can be an entirely new area of discussion and use for certain autonomous agents. The study of deception and rebellion are both rich fields, and many aspects of both that are studied in civilian and military capacities. For example, the area of deception detection works on finding ways to detect inconsistencies BID41 BID22 BID2. BID17 discuss a number of ways why deception is an important topic for autonomous agents. Studies of rebellion and resistance have investigated how, why, when it does, and doesn't, happen (Martí and Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. BID24 BID33. The use of both has also been studied BID34 BID1 BID21 BID18 BID30.The idea of pairing deception and rebellion with explanations may not be intuitive initially. However, in addition to being areas of rich study, deception and rebellion offer key conditions that are of interest to agent reasoning. Predominately, it requires multiple actors (i.e., An actor deceives another actor, or an actor rebels against a coordinator). Additionally, there needs to be some sort of conflict or misalignment between the actors. Either something needs to be in contention for an actor to rebel, or something needs to be in conflict for the actor to use deception. b Rebellion in agents has been a growing area of interest BID9 BID3 BID8. This area is focused on finding models in which agents can rebel from directives given in certain circumstances. This can include having more upto-date knowledge that would affect the plan, finding opportunities to exploit but may be off-mission, or solving problems or roadblocks before they become an issue even if it is off-mission. discuss three ways in which rebellion could manifest in agents. The expression of a rebellion can consist of either an explicit or implicit act. The focus is either inward or outward facing. Lastly, the interaction initiation can either be reactive or proactive. Deception in agents has been progressing over the last decade, with many discussions on formalizing deception. The majority of this formalism is on the topic of lying BID42 BID38 BID43. There has also been inroads for more encompassing deception as described by BID39 and BID37. Of interest here, BID37 defined Quantitative & Qualitative Maxims for Dishonesty as the following maxims:1. Lie, Bullshit (BS), or withhold information as little as possible to achieve your objective.2. Never lie if you can achieve your objective by BS.3. Never lie nor BS if you can achieve your objective by withholding Information.4. Never lie, BS, nor withhold information if you can achieve your objective with a half-truth. A particular topic that has received attention is deceptive, or dishonest, agents in negotiations BID31 BID47.With these concepts in mind, we will pursue research to answer the following:What kind of reasoning models are required to generate explanations of a deceptive or rebellious nature? Research into rebellion and deception have been studied for decades in other fields. We summarize some of these studies in this section, as they will form the foundation and context of our approach. Dissent assumes a challenge to some system of power or belief BID26. The author goes further to differentiate between dissent and rebellion. Martin claims that a basic distinction between dissent and rebellion is that dissenters still believe in the process that has been established whereas rebels do not. A number of real-world studies on dissent and whistleblowing in organizations showcase the importance of this topic and area BID15 BID19 BID26 BID32 BID25. Many of these discuss the outcomes and towards both the actor and the agent in these situations. BID25 discuss organizational responses to whistle-blowing and dissent. Manager reprisals may include building a damaging record towards them, threats against them, isolation, prosecution, or setting them up for failure at their job. In a number of instances, it seems that giving dissent, or becoming a whistle-blower, can quickly turn into an adversarial or antagonistic situation. Closely tied to dissent is the idea of cynicism. BID23 defines it as: a belief that there is a gap between desired and observed organizational identity; a negative affect toward the organization or organizational change (strategy); and tendencies to disparaging and/or critical behaviors toward the organization that are consistent with those beliefs and affect. They additionally give examples of cynicism as: pessimism, emotional/narrative expressions or outbursts, frustration, irony, accusations, neglect, negative coping behaviors and expressions, and aggression. Observed actions of resistance to change have been studied in a number of ways. Once such study BID46 noted that there was a tendency towards "informal or mundane" types of resistance compared to an open, direct, and explicit form of objection. When faced with managerial initiatives, the following expressions of resistance were noted: "careful carelessness", humor, cynicism and skepticism, nostalgic talk (i.e., the "Good ol' Days"), alternative articulations of self-hood, and simulation of productivity. While these expressions of resistance were present, workers were also camouflaging this dissent with a goodhumored appearance. It was noted that hiding dissent allowed for behind-the-scenes inaction to stifle any new directives and also avoided many conversations that workers deemed futile. Similar studies include BID14 BID27 BID35.Along with giving a number of examples of resistance, BID24 While overt forms of resistance usually did not work, the stories of that resistance remained and were used later to continue forms of resistance. There have been several studies on how deception in society works. A very good resource is (Whiten and Byrne 1988) which defines a number of deceptive actions observed among primates in the wild. They are categorized as either concealment, distraction, use of a tool, or use of another agent. Examples include hiding things from sight, looking at things to have others avoid looking at something else, and getting other agents to take the blame for something. In addition to primates, there have been deceptive studies for cephalopods BID4 ) and dolphins BID16.In studies of deception in warfare BID34, there have been noticeable benefits to using deception. This includes minimizing the amount of resources for a task or ensuring the opponents mis-allocate their resources. Also, most of the example deceptive acts required an extended duration of time to be executed and prove successful. This included preparations for and performing the deceptive actions. A good case study from (Dunin-Keplicz and Verbrugge 2011) describes an ecological disaster scenario. We present it now to provide context for explanations discussed in Section 4. In the ecological disaster scenario, two poisons have breached containment in a large area. Each poison is highly dangerous on its own. However, when combined they could be explosive. Robots on the ground have the ability to neutralize specific poisons in small areas. An unmanned air vehicle (UAV) can survey large areas to identify high concentrations of poison. A coordinator can issue commands to either the UAV or the ground robots. A robot can gauge the type and quantity of poisons in the small area around it, and can move across the ground. The UAV can scan larger areas to identify large concentrations of individual poisons, though it reports only the highest quantity. Additionally, the UAV cannot scan areas that are obscured by ceilings or coverings. The coordinator receives the information from the UAV and the Robots, but does not have a view of their own. Examples of rebellion here could be robots not following commands to enter areas that would explode or the UAV deciding to survey a different area to help robots on the ground compared to an area that the coordinator instructed the UAV to survey. Let us now consider explanations that are rebellious or deceptive within the context given in Section 3. Some of the recurring reasons for generating these explanations include "makes it simpler" or "avoids a larger conversation". These can limit conversations but can also cause misinterpretations. It also has the ability to avoid conversation bottlenecks, letting agents continue to perform other actions. During a disaster, it may come to pass that an agent with the ability to administer first-aid or medical attention, perhaps they are equipped with air masks or oxygen, encounters a victim who will only let them near if they do not inform any law enforcement of their position. In this instance, the agent would administer the first aid or medical treatment and, when asked by the Coordinator to explain their activities, would say they are performing a different activity or in a different location. A robot could be traversing an area trying to help people evacuate or administer aid and they have informed the Coordinator they are performing this task. However, suppose there is a person who wishes to remain anonymous or would refuse help. In this case, the robot could explain its actions but leave out details that would identify the person in later reports or debriefs. This would help save the person and increase that victim's trust in the robot to help them out of the area. A version of an explanation with a half-truth could be as follows: a medical agent has found and is administering aid to a victim, however the victim is too far gone. Keeping the victim calm with a half-truth or "white lie" explanation would be beneficial to the victim. An example of a protest-based explanation could come from the following contingency. An exploratory or search agent has encountered an area of the environment that it deems too hazardous to continue in. The Coordinator asks why it isn't moving forward anymore. The agent responds with,"I will not move forward until the hazardous area has been secured and is safe to pass through. " As discussed in Section 2.3, cynicism is a bit odd. It is usually used as a form of soft-resistance. The agent still performs the action or command, but may not do it optimally. An example could be that the Coordinator assigns a robot that has a full amount of neutralizing and medical equipment to survey an area. This might take a while for the robot to execute, so the Coordinator might ask why progress is slow, and the explanation could be " If I could fly this would go much quicker." Alternatively, asking to release all of its equipment so that it can be lighter to perform the survey is another example. For this instance, the agent is in some sort of situation in which it will not continue an objective given by the Coordinator. Perhaps the Coordinator has tasked an agent with neutralizing an area under a fallen concrete roof. However the agent has noticed a victim in the area being treated by another agent. In that instance the agent could respond to the Coordinator, "I will not neutralize that area, there is a victim in the vicinity. Please assign me a different objective." In order to generate possible deceptive explanations as suggested above an agent would require a few things in its models to properly generate a model. An agent would require an internal model of the domain so that it can reason about possible actions, tasks, and goals. It would also require an internal model of the external agent's domain model. This is required so that when generating the "deceptive" aspect, it can be validated against the presumed model of that agent. In addition to these models, a few conditions should be met as well. Notably, a discrepancy must be noticed between the external agent's model and the internal model in relation to the query asked. There needs to be a specific condition or contingency in which the truth would not maximize an overall objective benefit for the agent in question. Likewise, rebellious explanations require similar things such as internal models for both the domain and objectives along with noticing a discrepancy between the objectives, the domain, and the agent's internal model. Of great interest is the work in model reconciliation BID6. This is focused on maintaining (at least) two separate models, an internal one to the agent, and an external one for someone interacting with the agent. The idea is for the agent to reconcile any differences between these two versions to formulate an explanation. This approach is promising in regards to expanding it towards rebel or deceptive tasks in explanation. In the case of either deceptive or rebellious explanations, a discrepancy is required. This has been an active research focus. survey work on discrepancy detection. Useful to this thread of research, BID29 ) discusses it in the context of goal reasoning. In terms of reasoning models, BID36 ) discuss some interesting concepts in relation to Goal Networks. A life cycle for goals is also discussed. Combining these goal networks and the temporal nature of goal life cycles, a goal timeline develops. This timeline structure can represent the reasoning needed for some of the explanation models once discrepancies have been detected. Utilizing models of the world that are not in the explainer's original model is both challenging and novel to pursue for several reasons. It requires an agent to distinguish between different viewpoints of explanations. Introduces reasoning over viable explanations that can be generated. Requires a conversation concerning the ethics of deciding when an agent can decide to deceive. Finally, it opens up the area of XAI to new sets of scenarios -namely those that are deceptive or rebellious. To facilitate the development of deceptive or rebellious explanations, we will need a way to evaluate them. We propose a few areas that may be suitable for such testing. One such testing ground is the RoboCup Rescue. This is a popular disaster simulation BID20 that can be leveraged to simulate examples similar to those given in Section 3. Various games and game simulations may prove useful to test for these explanations. Some game options include Minecraft, One Night Ultimate Werewolf, Secret Hitler, Clue, and Diplomacy. Other relevant domains may include those that involve unmanned air and underwater vehicles. These vehicles require a large amount of autonomy and can be utilized in areas where discrepancies between an operator's situation awareness and the vehicle's belief state differ dramatically. Along with testing simulations, we can also look at measures of explanation effectiveness. Some of these measures can include clarity, timeliness, or correctness. Did the explanation answer the query? How easy was the explanation to understand? Was the time it took to respond seen as adequate? Is the user's attitude toward the agent lower or higher given this form of explanation?"

Using linear programming we show that the computational complexity of approximate Deep Neural Network training depends polynomially on the data size for several architectures


Question: Represent the scientific article to find a short TLDR

Question: In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels in one model. The proposed control problem contains a restoration dynamics which is modeled by an RNN. The moving endpoint, which is essentially the terminal time of the associated dynamics, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage. Image restoration, including image denoising, deblurring, inpainting, etc., is one of the most important areas in imaging science. Its major purpose is to obtain high quality reconstructions of images corrupted in various ways during imaging, acquisiting, and storing, and enable us to see crucial but subtle objects that reside in the images. Image restoration has been an active research area. Numerous models and algorithms have been developed for the past few decades. Before the uprise of deep learning methods, there were two classes of image restoration approaches that were widely adopted in the field: transformation based approach and PDE approach. The transformation based approach includes wavelet and wavelet frame based methods BID11 BID3, dictionary learning based methods BID0, similarity based methods BID2 BID10, low-rank models BID21 BID18, etc. The PDE approach includes variational models BID31 BID35 BID1, nonlinear diffusions BID33 BID6 BID38, nonlinear hyperbolic equations BID32, etc. More recently, deep connections between wavelet frame based methods and PDE approach were established BID4 BID12.One of the greatest challenge for image restoration is to properly handle image degradations of different levels. In the existing transformation based or PDE based methods, there is always at least one tuning parameter (e.g. the regularization parameter for variational models and terminal time for nonlinear diffusions) that needs to be manually selected. The choice of the parameter heavily relies on the degradation level. Recent years, deep learning models for image restoration tasks have significantly advanced the state-of-the-art of the field. BID20 proposed a convolutional neural network (CNN) for image denoising which has better expressive power than the MRF models by BID22. Inspired by nonlinear diffusions, BID9 designed a deep neural network for image denoising and BID40 improves the capacity by introducing a deeper neural network with residual connections. use the CNN to simulate a wide variety of image processing operators, achieving high efficiencies with little accuracy drop. However, these models cannot gracefully handle images with varied degradation levels. Although one may train different models for images with different levels, this may limit the application of these models in practice due to lack of flexibility. Taking blind image denoising for example. BID40 designed a 20-layer neural network for the task, called DnCNN-B, which had a huge number of parameters. To reduce number of parameters, BID24 proposed the UNLNet 5, by unrolling a projection gradient algorithm for a constrained optimization model. However, BID24 also observed a drop in PSNR comparing to DnCNN. Therefore, the design of a light-weighted and yet effective model for blind image denoising remains a challenge. Moreover, deep learning based models trained on simulated gaussian noise images usually fail to handle real world noise, as will be illustrated in later sections. Another example is JPEG image deblocking. JPEG is the most commonly used lossy image compression method. However, this method tend to introduce undesired artifacts as the compression rate increases. JPEG image deblocking aims to eliminate the artifacts and improve the image quality. Recently, deep learning based methods were proposed for JPEG deblocking BID13 BID40. However, most of their models are trained and evaluated on a given quality factor. Thus it would be hard for these methods to apply to Internet images, where the quality factors are usually unknown. In this paper, we propose a single image restoration model that can robustly restore images with varied degradation levels even when the degradation level is well outside of that of the training set. Our proposed model for image restoration is inspired by the recent development on the relation between deep learning and optimal control. The relation between supervised deep learning methods and optimal control has been discovered and exploited by BID39; BID26 BID7 BID16. The key idea is to consider the residual block x n+1 = x n + f (x n) as an approximation to the continuous dynamicsẊ = f (X). In particular, BID26 BID16 demonstrated that the training process of a class of deep models (e.g. ResNet by BID19, PolyNet by BID42, etc.) can be understood as solving the following control problem: DISPLAYFORM0 Here x 0 is the input, y is the regression target or label,Ẋ = f (X, w) is the deep neural network with parameter w(t), R is the regularization term and L can be any loss function to measure the difference between the reconstructed images and the ground truths. In the context of image restoration, the control dynamicẊ = f (X(t), ω(t)), t ∈ (0, τ) can be, for example, a diffusion process learned using a deep neural network. The terminal time τ of the diffusion corresponds to the depth of the neural network. Previous works simply fixed the depth of the network, i.e. the terminal time, as a fixed hyper-parameter. However BID30 showed that the optimal terminal time of diffusion differs from image to image. Furthermore, when an image is corrupted by higher noise levels, the optimal terminal time for a typical noise removal diffusion should be greater than when a less noisy image is being processed. This is the main reason why current deep models are not robust enough to handle images with varied noise levels. In this paper, we no longer treat the terminal time as a hyper-parameter. Instead, we design a new architecture (see Fig. 3) that contains both a deep diffusion-like network and another network that determines the optimal terminal time for each input image. We propose a novel moving endpoint control model to train the aforementioned architecture. We call the proposed architecture the dynamically unfolding recurrent restorer (DURR).We first cast the model in the continuum setting. Let x 0 be an observed degraded image and y be its corresponding damage-free counterpart. We want to learn a time-independent dynamic systeṁ X = f (X(t), w) with parameters w so that X = x and X(τ) ≈ y for some τ > 0. See Fig. 2 for an illustration of our idea. The reason that we do not require X(τ) = y is to avoid over-fitting. For varied degradation levels and different images, the optimal terminal time τ of the dynamics may vary. Therefore, we need to include the variable τ in the learning process as well. The learning of the dynamic system and the terminal time can be gracefully casted as the following moving endpoint control problem: DISPLAYFORM1 Different from the previous control problem, in our model the terminal time τ is also a parameter to be optimized and it depends on the data x. The dynamic systemẊ = f (X(t), w) is modeled by a recurrent neural network (RNN) with a residual connection, which can be understood as a residual network with shared weights BID25. We shall refer to this RNN as the restoration unit. In order to learn the terminal time of the dynamics, we adopt a policy network to adaptively determine an optimal stopping time. Our learning framework is demonstrated in Fig. 3. We note that the above moving endpoint control problem can be regarded as the penalized version of the well-known fixed endpoint control problem in optimal control BID15, where instead of penalizing the difference between X(τ) and y, the constraint X(τ) = y is strictly enforced. In short, we summarize our contribution as following:• We are the first to use convolutional RNN for image restoration with unknown degradation levels, where the unfolding time of the RNN is determined dynamically at run-time by a policy unit (could be either handcrafted or RL-based).• The proposed model achieves state-of-the-art performances with significantly less parameters and better running efficiencies than some of the state-of-the-art models.• We reveal the relationship between the generalization power and unfolding time of the RNN by extensive experiments. The proposed model, DURR, has strong generalization to images with varied degradation levels and even to the degradation level that is unseen by the model during training (Fig. 1).• The DURR is able to well handle real image denoising without further modification. Qualitative have shown that our processed images have better visual quality, especially sharper details compared to others. The proposed architecture, i.e. DURR, contains an RNN (called the restoration unit) imitating a nonlinear diffusion for image restoration, and a deep policy network (policy unit) to determine the terminal time of the RNN. In this section, we discuss the training of the two components based on our moving endpoint control formulation. As will be elaborated, we first train the restoration unit to determine ω, and then train the policy unit to estimate τ (x). If the terminal time τ for every input x i is given (i.e. given a certain policy), the restoration unit can be optimized accordingly. We would like to show in this section that the policy used during training greatly influences the performance and the generalization ability of the restoration unit. More specifically, a restoration unit can be better trained by a good policy. The simplest policy is to fix the loop time τ as a constant for every input. We name such policy as "naive policy". A more reasonable policy is to manually assign an unfolding time for each degradation level during training. We shall call this policy the "refined policy". Since we have not trained the policy unit yet, to evaluate the performance of the trained restoration units, we manually pick the output image with the highest PSNR (i.e. the peak PSNR).We take denoising as an example here. The peak PSNRs of the restoration unit trained with different policies are listed in Table. 1. the refined policy, the noise levels and the associated loop times are,. For the naive policy, we always fix the loop times to 8. As we can see, the refined policy brings the best performance on all the noise levels including 40. The restoration unit trained for specific noise level (i.e. σ = 40) is only comparable to the one with refined policy on noise level 40. The restoration unit trained on multiple noise levels with naive policy has the worst performance. These indicate that the restoration unit has the potential to generalize on unseen degradation levels when trained with good policies. According to FIG2, the generalization reflects on the loop times of the restoration unit. It can be observed that the model with steeper slopes have stronger ability to generalize as well as better performances. According to these , the restoration unit we used in DURR is trained using the refined policy. More specifically, for image denoising, the noise level and the associated loop times are set to,,, and. For JPEG image deblocking, the quality factor (QF) and the associated loop times are set to and. We discuss two approaches that can be used as policy unit:Handcraft policy: Previous work BID30 has proposed a handcraft policy that selects a terminal time which optimizes the correlation of the signal and noise in the filtered image. This criterion can be used directly as our policy unit, but the independency of signal and noise may not hold for some restoration tasks such as real image denoising, which has higher noise level in the low-light regions, and JPEG image deblocking, in which artifacts are highly related to the original image. Another potential stopping criterion of the diffusion is no-reference image quality assessment BID28, which can provide quality assessment to a processed image without the ground truth image. However, to the best of our knowledge, the performances of these assessments are still far from satisfactory. Because of the limitations of the handcraft policies, we will not include them in our experiments. Reinforcement learning based policy: We start with a discretization of the moving endpoint problem on the dataset {(x i, y i)|i = 1, 2, · · ·, d}, where {x i} are degraded observations of the damage-free images {y i}. The discrete moving endpoint control problem is given as follows: DISPLAYFORM0 DISPLAYFORM1 Here, DISPLAYFORM2 is the forward Euler approximation of the dynamicsẊ = f (X(t), w). The terminal time {N i} is determined by a policy network P (x, θ), where x is the output of the restoration unit at each iteration and θ the set of weights. In our experiment, we simply set r = 0, i.e. doesn't introduce any regularization which might bring further benefit but is beyond this paper's scope of discussion. In other words, the role of the policy network is to stop the iteration of the restoration unit when an ideal image restoration is achieved. The reward function of the policy unit can be naturally defined by DISPLAYFORM3 In order to solve the problem (2.2), we need to optimize two networks simultaneously, i.e. the restoration unit and the policy unit. The first is an restoration unit which approximates the controlled dynamics and the other is the policy unit to give the optimized terminating conditions. The objective function we use to optimize the policy network can be written as DISPLAYFORM4 where π θ denotes the distribution of the trajectories X = {X i n, n = 1, . . ., N i, i = 1, . . ., d} under the policy network P (·, θ). Thus, reinforcement learning techniques can be used here to learn a neural network to work as a policy unit. We utilize Deep Q-learning BID29 as our learning strategy and denote this approach simply as DURR. However, different learning strategies can be used (e.g. the Policy Gradient). In all denoising experiments, we follow the same settings as in BID9 BID40; BID24. All models are evaluated using the mean PSNR as the quantitative metric on the BSD68 BID27 ). The training set and test set of BSD500 (400 images) are used for training. Six gaussian noise levels are evaluated, namely σ = 25, 35, 45, 55, 65 and 75. Additive noise are applied to the image on the fly during training and testing. Both the training and evaluation process are done on gray-scale images. The restoration unit is a simple U-Net BID34 style fully convolutional neural network. For the training process of the restoration unit, the noise levels of 25, 35, 45 and 55 are used. Images are cut into 64 × 64 patches, and the batch-size is set to 24. The Adam optimizer with the learning rate 1e-3 is adopted and the learning rate is scaled down by a factor of 10 on training plateaux. The policy unit is composed of two ResUnit and an LSTM cell. For the policy unit training, we utilize the reward function in Eq.4. For training the policy unit, an RMSprop optimizer with learning rate 1e-4 is adopted. We've also tested other network structures, these tests and the detailed network structures of our model are demonstrated in the supplementary materials. In all JPEG deblocking experiments, we follow the settings as in BID40. All models are evaluated using the mean PSNR as the quantitative metric on the LIVE1 dataset BID36. Both the training and evaluation processes are done on the Y channel (the luminance channel) of the YCbCr color space. The PIL module of python is applied to generate JPEG-compressed images. The module produces numerically identical images as the commonly used MATLAB JPEG encoder after setting the quantization tables manually. The images with quality factors 20 and 30 are used during training. De-blocking performances are evaluated on four quality factors, namely QF = 10, 20, 30, and 40. All other parameter settings are the same as in the denoising experiments. We select DnCNN-B BID40 and UNLNet 5 BID24 for comparisons since these models are designed for blind image denoising. Moreover, we also compare our model with non-learning-based algorithms BM3D BID10 and WNNM BID18. The noise levels are assumed known for BM3D and WNNM due to their requirements. Comparison are shown in TAB2.Despite the fact that the parameters of our model (1.8 × 105 for the restoration unit and 1.0 × 10 5 for the policy unit) is less than the DnCNN (approximately 7.0 × 10 5), one can see that DURR outperforms DnCNN on most of the noise-levels. More interestingly, DURR does not degrade too much when the the noise level goes beyond the level we used during training. The noise level σ = 65, 75 is not included in the training set of both DnCNN and DURR. DnCNN reports notable drops of PSNR when evaluated on the images with such noise levels, while DURR only reports small drops of PSNR (see the last row of TAB2 and Fig. 6). Note that the reason we do not provide the of UNLNet 5 in TAB2 is because the authors of BID24 has not released their codes yet, and they only reported the noise levels from 15 to 55 in their paper. We also want to emphasize that they trained two networks, one for the low noise level (5 ≤ σ ≤ 29) and one for higher noise level (30 ≤ σ ≤ 55). The reason is that due to the use of the constraint ||y − x|| 2 ≤ by , we should not expect the model generalizes well to the noise levels surpasses the noise level of the training set. For qualitative comparisons, some restored images of different models on the BSD68 dataset are presented in Fig. 5 and Fig. 6. As can be seen, more details are preserved in DURR than other models. It is worth noting that the noise level of the input image in Fig. 6 is 65, which is unseen by both DnCNN and DURR during training. Nonetheless, DURR achieves a significant gain of nearly 1 dB than DnCNN. Moreover, the texture on the cameo is very well restored by DURR. These clearly indicate the strong generalization ability of our model. More interestingly, due to the generalization ability in denoising, DURR is able to handle the problem of real image denoising without additional training. For testing, we test the images obtained from BID23. We present the representative in Fig. 7 and more are listed in the supplementary materials. We also train our model for blind color image denoising, please refer to the supplementary materials for more details. For deep learning based models, we select DnCNN-3 BID40 for comparisons since it is the only known deep model for multiple QFs deblocking. As the AR-CNN BID13 is a commonly used baseline, we re-train the AR-CNN on a training set with mixed QFs and denote this model as AR-CNN-B. Original AR-CNN as well as a non-learning-based method SA-DCT are also tested. The quality factors are assumed known for these models. Quantitative are shown in TAB3. Though the number of parameters of DURR is significantly less than the DnCNN-3, the proposed DURR outperforms DnCNN-3 in most cases. Specifically, considerable gains can be observed for our model on seen QFs, and the performances are comparable on unseen QFs. A representative on the LIVE1 dataset is presented in FIG4. Our model generates the most clean and accurate details. More experiment details are given in the supplementary materials. Figure 7: Denoising on a real image from BID23. Our model can be easily extended to other applications such as deraining, dehazing and deblurring. In all these applications, there are images corrupted at different levels. Rainfall intensity, haze density and different blur kernels will all effect the image quality. In this paper, we proposed a novel image restoration model based on the moving endpoint control in order to handle varied noise levels using a single model. The problem was solved by jointly optimizing two units: restoration unit and policy unit. The restoration unit used an RNN to realize the dynamics in the control problem. A policy unit was proposed for the policy unit to determine the loop times of the restoration unit for optimal . Our model achieved the state-of-the-art in blind image denoising and JPEG deblocking. Moreover, thanks to the flexibility of the given policy, DURR has shown strong abilities of generalization in our experiments.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We propose a novel method to handle image degradations of different levels by learning a diffusion terminal time. Our model can generalize to unseen degradation level and different noise statistic.

Represent the TLDR for finding the corresponding scientific article

A method to binarize both weights and activations of a deep neural network that is efficient in computation and memory usage and performs better than the state-of-the-art.


Question: represent the article for finding a single-sentence tldr

Question: The Wasserstein distance received a lot of attention recently in the community of machine learning, especially for its principled way of comparing distributions. It has found numerous applications in several hard problems, such as domain adaptation, dimensionality reduction or generative models. However, its use is still limited by a heavy computational cost. Our goal is to alleviate this problem by providing an approximation mechanism that allows to break its inherent complexity. It relies on the search of an embedding where the Euclidean distance mimics the Wasserstein distance. We show that such an embedding can be found with a siamese architecture associated with a decoder network that allows to move from the embedding space back to the original input space. Once this embedding has been found, computing optimization problems in the Wasserstein space (e.g. barycenters, principal directions or even archetypes) can be conducted extremely fast. Numerical experiments supporting this idea are conducted on image datasets, and show the wide potential benefits of our method. The Wasserstein distance is a powerful tool based on the theory of optimal transport to compare data distributions with wide applications in image processing, computer vision and machine learning BID29. In a context of machine learning, it has recently found numerous applications, e.g. domain adaptation, or word embedding BID22. In the context of deep learning, the Wasserstein appeared recently to be a powerful loss in generative models BID2 and in multi-label classification BID19. Its power comes from two major reasons: i) it allows to operate on empirical data distributions in a non-parametric way ii) the geometry of the underlying space can be leveraged to compare the distributions in a geometrically sound way. The space of probability measures equipped with the Wasserstein distance can be used to construct objects of interest such as barycenters BID0 or geodesics BID34 that can be used in data analysis and mining tasks. More formally, let X be a metric space endowed with a metric d X. Let p ∈ (0, ∞) and P p (X) the space of all Borel probability measures µ on X with finite moments of order p, i.e. X d X (x, x 0) p dµ(x) < ∞ for all x 0 in X. The p-Wasserstein distance between µ and ν is defined as: DISPLAYFORM0 Here, Π(µ, ν) is the set of probabilistic couplings π on (µ, ν). As such, for every Borel subsets A ⊆ X, we have that µ(A) = π(X × A) and ν(A) = π(A × X). It is well known that W p defines a metric over P p (X) as long as p ≥ 1 (e.g. BID39), Definition 6.2).When p = 1, W 1 is also known as Earth Mover's distance (EMD) or Monge-Kantorovich distance. The geometry of (P p (X), W 1 (X)) has been thoroughly studied, and there exists several works on computing EMD for point sets in R k (e.g. BID35). However, in a number of applications the use of W 2 (a.k.a root mean square bipartite matching distance) is a more natural distance arising in computer vision BID6, computer graphics BID5 BID16 BID36 BID7 or machine learning BID14. See BID16 for a discussion on the quality comparison between W 1 and W 2.Yet, the deployment of Wasserstein distances in a wide class of applications is somehow limited, especially because of an heavy computational burden. In the discrete version of the above optimisation problem, the number of variables scale quadratically with the number of samples in the distributions, and solving the associated linear program with network flow algorithms is known to have a cubical complexity. While recent strategies implying slicing technique BID6 BID26, entropic regularization BID13 BID3 BID37 or involving stochastic optimization BID21, have emerged, the cost of computing pairwise Wasserstein distances between a large number of distributions (like an image collection) is prohibitive. This is all the more true if one considers the problem of computing barycenters BID14 BID3 or population means. A recent attempt by Staib and colleagues BID38 use distributed computing for solving this problem in a scalable way. We propose in this work to learn an Euclidean embedding of distributions where the Euclidean norm approximates the Wasserstein distances. Finding such an embedding enables the use of standard Euclidean methods in the embedded space and significant speedup in pairwise Wasserstein distance computation, or construction of objects of interests such as barycenters. The embedding is expressed as a deep neural network, and is learnt with a strategy similar to those of Siamese networks BID11. We also show that simultaneously learning the inverse of the embedding function is possible and allows for a reconstruction of a probability distribution from the embedding. We first start by describing existing works on Wasserstein space embedding. We then proceed by presenting our learning framework and give proof of concepts and empirical on existing datasets. Metric embedding The question of metric embedding usually arises in the context of approximation algorithms. Generally speaking, one seeks a new representation (embedding) of data at hand in a new space where the distances from the original space are preserved. This new representation should, as a positive side effect, offers computational ease for time-consuming task (e.g. searching for a nearest neighbor), or interpretation facilities (e.g. visualization of high-dimensional datasets). More formally, given two metrics spaces (X, d X) and (Y, d y) and D ∈ [1, ∞), a mapping φ: X → Y is an embedding with distortion at most D if there exists a coefficient α DISPLAYFORM0 Here, the α parameter is to be understood as a global scaling coefficient. The distortion of the mapping is the infimum over all possible D such that the previous relation holds. Obviously, the lower the D, the better the quality of the embedding is. It should be noted that the existence of exact (isometric) embedding (D = 1) is not always guaranteed but sometimes possible. Finally, the embeddability of a metric space into another is possible if there exists a mapping with constant distortion. A good introduction on metric embedding can be found in BID30.Theoretical on Wasserstein space embedding Embedding Wasserstein space in normed metric space is still a theoretical and open questions BID31. Most of the theoretical guarantees were obtained with W 1. In the simple case where X = R, there exists an isometric embedding with L 1 between two absolutely continuous (wrt. the Lebesgue measure) probability measures µ and ν given by their by their cumulative distribution functions F µ and F ν, i.e. W 1 (µ, ν) = R |F µ (x) − F ν (x)|dx. This fact has been exploited in the computation of sliced Wasserstein distance BID6 BID28. Conversely, there is no known isometric embedding for pointsets in [n] k = {1, 2, . . ., n} k, i.e. regularly sampled grids in R k, but best known distortions are between O(k log n) and Ω(k + √ log n) BID10 BID23 BID24. Regarding W 2, recent BID1 have shown there does not exist meaningful embedding over R 3 with constant approximation. Their show notably that an embedding of pointsets of size n into L 1 must incur a distortion of O(√ log n). Regarding our choice of W 2 2, there does not exist embeddability up to our knowledge, but we show that, for a population of locally concentrated measures, a good approximation can be obtained with our technique. We now turn to existing methods that consider local linear approximations of the transport problem. Linearization of Wasserstein space Another line of work BID40 BID27 also considers the Riemannian structure of the Wasserstein space to provide meaningful linearization by projecting onto the tangent space. By doing so, they notably allows for faster computation of pairwise Wasserstein distances (only N transport computations instead of N (N − 1)/2 with N the number of samples in the dataset) and allow for statistical analysis of the embedded data. They proceed by specifying a template element and compute, from particle approximations of the data, linear transport plans with this template element, that allow to derive an embedding used for analysis. Seguy and Cuturi BID34 also proposed a similar pipeline, based on velocity field, but without relying on an implicit embedding. It is to be noted that for data in 2D, such as images, the use of cumulative Radon transform also allows for an embedding which can be used for interpolation or analysis BID6 BID26, by exploiting the exact solution of the optimal transport in 1D through cumulative distribution functions. Our work is the first to propose to learn a generic embedding rather than constructing it from explicit approximations/transformations of the data and analytical operators such as Riemannian Logarithm maps. As such, our formulation is generic and adapts to any type of data. Finally, since the mapping to the embedded space is constructed explicitly, handling unseen data does not require to compute new optimal transport plans or optimization, yielding extremely fast computation performances, with similar approximation performances. We discuss here how our method, coined DWE for Deep Wasserstein Embedding, learns in a supervised way a new representation of the data. To this end we need a pre-computed dataset that consists of pairs of histograms {x Wasserstein distance DISPLAYFORM0 ..,n . One immediate way to solve the problem would be to concatenate the samples x 1 and x 2 and learn a deep network that predicts y. This would work in theory but it would prevent us from interpreting the Wasserstein space and it is not by default symmetric which is a key property of the Wasserstein distance. Another way to encode this symmetry and to have a meaningful embedding that can be used more broadly is to use a Siamese neural network BID8 . Originally designed for metric learning purpose and similarity learning (based on labels), this type of architecture is usually defined by replicating a network which takes as input two samples from the same learning set, and learns a mapping to new space with a contrastive loss. It has mainly been used in computer vision, with successful applications to face recognition BID11 or one-shot learning for example BID25. Though its capacity to learn meaningful embeddings has been highlighted in BID41, it has never been used, to the best of our knowledge, for mimicking a specific distance that exhibits computation challenges. This is precisely our objective here. We propose to learn and embedding network φ that takes as input a histogram and project it in a given Euclidean space of R p. In practice, this embedding should mirror the geometrical property of the Wasserstein space. We also propose to regularize the computation of this embedding by adding a reconstruction loss based on a decoding network ψ. This has two important impacts: First we observed empirically that it eases the learning of the embedding and improves the generalization performance of the network (see experimental in appendix) by forcing the embedded representation to catch sufficient information of the input data to allow a good reconstruction. This type of autoencoder regularization loss has been discussed in BID43 in the different context of embedding learning. Second, using a decoder network allows the interpretation of the , which is of prime importance in several data-mining tasks (discussed in the next subsection). An overall picture depicting the whole process is given in FIG1. The global objective function reads min DISPLAYFORM1 where λ > 0 weights the two data fitting terms and KL(,) is the Kullbach-Leibler divergence. This choice is motivated by the fact that the Wasserstein metric operates on probability distributions. Once the functions φ and ψ have been learned, several data mining tasks can be operated in the Wasserstein space. We discuss here the potential applications of our computational scheme and its wide range of applications on problems where the Wasserstein distance plays an important role. Though our method is not an exact Wasserstein estimator, we empirically show in the numerical experiments that it performs very well and competes favorably with other classical computation strategies. Wasserstein barycenters BID0 BID14 BID7. Barycenters in Wasserstein space were first discussed by BID0. Designed through an analogy with barycenters in a Euclidean space, the Wasserstein barycenters of a family of measures are defined as minimizers of a weighted sum of squared Wasserstein distances. In our framework, barycenters can be obtained as DISPLAYFORM0 where x i are the data samples and the weights α i obeys the following constraints: i α i = 1 and α i > 0. Note that when we have only two samples, the barycenter corresponds to a Wasserstein interpolation between the two distributions with α = [1 − t, t] and 0 ≤ t ≤ 1 BID33. When the weights are uniform and the whole data collection is considered, the barycenter is the Wasserstein population mean, also known as Fréchet mean BID4.Principal Geodesic Analysis in Wasserstein space BID34 BID4. PGA, or Principal Geodesic Analysis, has first been introduced by Fletcher et al. BID18. It can be seen as a generalization of PCA on general Riemannian manifolds. Its goal is to find a set of directions, called geodesic directions or principal geodesics, that best encode the statistical variability of the data. It is possible to define PGA by making an analogy with PCA. Let x i ∈ R n be a set of elements, the classical PCA amounts to i) find x the mean of the data and subtract it to all the samples ii) build recursively a subspace V k = span(v 1, · · ·, v k) by solving the following maximization problem: DISPLAYFORM1 Fletcher gives a generalization of this problem for complete geodesic spaces by extending three important concepts: variance as the expected value of the squared Riemannian distance from mean, Geodesic subspaces as a portion of the manifold generated by principal directions, and a projection operator onto that geodesic submanifold. The space of probability distribution equipped with the Wasserstein metric (P p (X), W 2 2 (X)) defines a geodesic space with a Riemannian structure BID33, and an application of PGA is then an appealing tool for analyzing distributional data. However, as noted in BID34 BID4, a direct application of Fletcher's original algorithm is intractable because P p (X) is infinite dimensional and there is no analytical expression for the exponential or logarithmic maps allowing to travel to and from the corresponding Wasserstein tangent space. We propose a novel PGA approximation as the following procedure: i) find x the approximate Fréchet mean of the data as x = 1 N N i φ(x i) and subtract it to all the samples ii) build recursively a subspace V k = span(v 1, · · ·, v k) in the embedding space (v i being of the dimension of the embedded space) by solving the following maximization problem: DISPLAYFORM2 which is strictly equivalent to perform PCA in the embedded space. Any reconstruction from the corresponding subspace to the original space is conducted through ψ. We postpone a detailed analytical study of this approximation to subsequent works, as it is beyond the goals of this paper. Other possible methods. As a matter of facts, several other methods that operate on distributions can benefit from our approximation scheme. Most of those methods are the transposition of their Euclidian counterparts in the embedding space. Among them, clustering methods, such as Wasserstein k-means BID14, are readily adaptable to our framework. Recent works have also highlighted the success of using Wasserstein distance in dictionary learning BID32 or archetypal Analysis BID42. In this section we evaluate the performances of our method on grayscale images normalized as histograms. Images are offering a nice testbed because of their dimensionality and because large datasets are frequently available in computer vision. The framework of our approach as shown in FIG1 of an encoder φ and a decoder ψ composed as a cascade. The encoder produces the representation of input images h = φ(x). The architecture used for the embedding φ consists in 2 convolutional layers with ReLU activations: first a convolutional layer of 20 filters with a kernel of size 3 by 3, then a convolutional layer of 5 filters of size 5 by 5. The convolutional layers are followed by two linear dense layers respectively of size 100 and the final layer of size p = 50. The architecture for the reconstruction ψ consists in a dense layer of output 100 with ReLU activation, followed by a dense layer of output 5*784. We reshape the layer to map the input of a convolutional layer: the output vector is 3D-tensor. Eventually, we invert the convolutional layers of φ with two convolutional layers: first a convolutional layer of 20 filters with ReLU activation and a kernel of size 5 by 5, followed by a second layer with 1 filter, with a kernel of size 3 by 3. Eventually the decoder outputs a reconstruction image of shape 28 by 28. In this work, we only consider grayscale images, that are normalized to represent probability distributions. Hence each image is depicted as an histogram. In order to normalize the decoder reconstruction we use a softmax activation for the last layer. All the dataset considered are handwritten data and hence holds an inherent sparsity. In our case, we cannot promote the output sparsity through a convex L1 regularization because the softmax outputs positive values only and forces the sum of the output to be 1. Instead, we apply a p p pseudo -norm regularization with p = 1/2 on the reconstructed image, which promotes sparse output and allows for a sharper reconstruction of the images BID20 and DWE given as average number of W 2 2 computation per seconds for different configurations. Dataset and training. Our first numerical experiment is performed on the well known MNIST digits dataset. This dataset contains 28×28 images from 10 digit classes In order to create the training dataset we draw randomly one million pairs of indexes from the 60 000 training samples and compute the exact Wasserstein distance with a squared Euclidean ground metric using the POT toolbox. All those pairwise distances can be computed in an embarrassingly parallel scheme (1h30 on 1 CPU). Among this million, 700 000 are used for learning the neural network, 200 000 are used for validation and 100 000 pairs are used for testing purposes. The DWE model is learnt on a GTX TitanX Maxwell 980 GPU node and takes around 1h20 with a stopping criterion computed from on a validation set. Numerical precision and computational performance The true and predicted values for the Wasserstein distances are given in FIG2. We can see that we reach a good precision with a test MSE of 0.4 and a relative MSE of 2e-3. The correlation is of 0.996 and the quantiles show that we have a very small uncertainty with only a slight bias for large values where only a small number of samples is available. This show that a good approximation of the W 2 2 can be performed by our approach (≈1e-3 relative error). Now we investigate the ability of our approach to compute W 2 2 efficiently. To this end we compute the average speed of Wasserstein distance computation on test dataset to estimate the number of W 2 2 computations per second in the Table of FIG2. Note that there are 2 ways to compute the W 2 2 with our approach denoted as Indep and Pairwise. This comes from the fact that our W 2 2 computation is basically a squared Euclidean norm in the embedding space. The first computation measures the time to compute the W 2 2 between independent samples by projecting both in the embedding and computing their distance. The second computation aims at computing all the pairwise W 2 2 between two sets of samples and this time one only needs to project the samples once and compute all the pairwise distances, making it more efficient. Note that the second approach would be the one used in a retrieval problem where one would just embed the query and then compute the distance to all or a selection of the dataset to find a Wasserstein nearest neighbor for instance. The speedup achieved by our method is very impressive even on CPU with speedup of x18 and x1000 respectively for Indep and Pairwise. But the GPU allows an even larger speedup of respectively x1000 and x500 000 with respect to a state-of-the-art C compiled Network Flow LP solver of the POT Toolbox BID5. Of course this speed-up comes at the price of a time-consuming learning phase, which makes our method better suited for mining large scale datasets and online applications. Wasserstein Barycenters Next we evaluate our embedding on the task of computing Wasserstein Barycenters for each class of the MNIST dataset. We take 1000 samples per class from the test dataset and compute their uniform weight Wasserstein Barycenter using Eq. 3. The ing barycenters and their Euclidean means are reported in FIG3. Note that not only those barycenters are sensible but also conserve most of their sharpness which is a problem that occurs for regularized barycenters BID37 BID3. The computation of those barycenters is also very efficient since it requires only 20ms per barycenter (for 1000 samples) and its complexity scales linearly with the number of samples. Figure 4: Principal Geodesic Analysis for classes 0,1 and 4 from the MNIST dataset for squared Euclidean distance (L2) and Deep Wasserstein Embedding (DWE). For each class and method we show the variation from the barycenter along one of the first 3 principal modes of variation. Principal Geodesic Analysis We report in Figure 4 the Principal Component Analysis (L2) and Principal Geodesic Analysis (DWE) for 3 classes of the MNIST dataset. We can see that using Wasserstein to encode the displacement of mass leads to more semantic and nonlinear subspaces such as rotation/width of the stroke and global sizes of the digits. This is well known and has been illustrated in BID34. Nevertheless our method allows for estimating the principal component even in large scale datasets and our reconstruction seems to be more detailed compared to BID34 maybe because our approach can use a very large number of samples for subspace estimation. Datasets The Google Doodle dataset is a crowd sourced dataset that is freely available from the web 1 and contains 50 million drawings. The data has been collected by asking users to hand draw with a mouse a given object or animal in less than 20 seconds. This lead to a large number of examples for each class but also a lot of noise in the sens that people often get stopped before the end of their drawing.We used the numpy bitmaps format proposed on the quick draw github account. Those are made of the simplified drawings rendered into 28x28 grayscale images. These images are aligned to the center of the drawing's bounding box. In this paper we downloaded the classes Cat, Crab and Faces and tried to learn a Wasserstein embedding for each of these classes with the same architecture as used for MNIST. In order to create the training dataset we draw randomly 1 million pairs of indexes from the training samples of each categories and compute the exact Wasserstein distance with squared Euclidean ground metric using the POT toolbox The numerical performances of the learned models on each of the doodle dataset is reported in the diagonal of Table 1. Those datasets are much more difficult than MNIST because they have not been curated and contain a very large variance due to numerous unfinished doodles. An interesting comparison is the cross comparison between datasets where we use the embedding learned on one dataset to compute the W 2 2 on another. The cross performances is given in Table 1 and shows that while there is definitively a loss in accuracy of the prediction, this loss is limited between the doodle datasets that all have an important variety. Performance loss across doodle and MNIST dataset is larger because the latter is highly structured and one needs to have a representative dataset to generalize well which is not the case with MNIST. This also clearly highlights that our method finds a data-dependent embedding that is specific to the geometry of the learning set. Wasserstein interpolation Next we qualitatively evaluate the subspace learned by DWE by comparing the Wasserstein interpolation of our approach with the true Wasserstein interpolation estimated by solving the OT linear program and by using regularized OT with Bregman projections BID3. The interpolation for all those methods and the Euclidean interpolation are available in FIG4. The LP solver takes a long time (20 sec/interp) and leads to a "noisy" interpolation as already explained in. The regularized Wasserstein barycenter is obtained more rapidly (4 sec/interp) but is also very smooth at the risk of loosing some details, despite choosing a small regularization that prevents numerical problems. Our reconstruction also looses some details due to the Auto-Encoder error but is very fast and can be done in real time (4 ms/interp). In this work we presented a computational approximation of the Wasserstein distance suitable for large scale data mining tasks. Our method finds an embedding of the samples in a space where the Euclidean distance emulates the behavior of the Wasserstein distance. Thanks to this embedding, numerous data analysis tasks can be conducted at a very cheap computational price. We forecast that this strategy can help in generalizing the use of Wasserstein distance in numerous applications. However, while our method is very appealing in practice it still raises a few questions about the theoretical guarantees and approximation quality. First it is difficult to foresee from a given network architecture if it is sufficiently (or too much) complex for finding a successful embedding. It can be conjectured that it is dependent on the complexity of the data at hand and also the locality of the manifold where the data live in. Second, the theoretical existence on such Wasserstein embedding with constant distortion are still lacking. Future works will consider these questions as well as applications of our approximation strategy on a wider range of ground loss and data mining tasks. Also, we will study the transferability of one database to another (i.e. leveraging on previously computed embedding) to diminish the computational burden of computing Wasserstein distances on numerous pairs for the learning process, by considering for instance domain adaptation strategies between embeddings. We discuss here the role of the decoder, not only as a matter of interpreting the , but rather as a regularizer. We train our DWE on MNIST with and without the decoder and compares the learning curves of the MSE on the validation set. In FIG5, DWE achieves a lower MSE with the decoder, which enforces the use of a decoder into our framework. We illustrate here the plurality of examples found in this dataset by drawing random excerpts in FIG6. There exist also a lot of outlier images (scribblings, texts, etc.). As discussed in the main text several drawings are unfinished and/or do not represent correctly the required class. We then compute the Wasserstein interpolation between four samples of each datasets in FIG7. Note that these interpolation might not be optimal w.r.t. the objects but we clearly see a continuous displacement of mass that is characteristic of optimal transport. This leads to surprising artefacts for example when the eye of a face fuse with the border while the nose turns into an eye. Also note that there is no reason for a Wasserstein barycenter to be a realistic sample. In FIG9 we show the quantitative evaluation for DWE on the three datasets, that correspond to Table 1 in the paper. The reported MSE performances correspond to the ones in the diagonal of Table 1. We can see that the deviation is larger for large values of W We report in FIG1 a nearest neighbor walk (sequential jumps to the nearest, in the sense of the considered metric, image that has not already been seen) on a subset of 10000 test samples starting with the same image but using either the L2 distance in the input or DWE embedded space. Note that the L2 in input space is here very sensible to outliers (black squares) that are rare in the dataset but have a L2 distance rather small to all other examples (most sequences converge to those samples). Conversely the DWE neighbors follow a smooth trajectory along the examples. This illustrates the advantage of W 2 2 for image retrieval, which is made computationally possible with DWE.

We show that it is possible to fastly approximate Wasserstein distances computation by finding an appropriate embedding where Euclidean distance emulates the Wasserstein distance

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We show that the learnability of different neural architectures can be characterized directly by computable measures of data complexity.