Question: Represent the text to find a very short summary (1-2 sentences max)

Question: There are two major paradigms of white-box adversarial attacks that attempt to impose input perturbations. The first paradigm, called the fix-perturbation attack, crafts adversarial samples within a given perturbation level. The second paradigm, called the zero-confidence attack, finds the smallest perturbation needed to cause misclassification, also known as the margin of an input feature. While the former paradigm is well-resolved, the latter is not. Existing zero-confidence attacks either introduce significant approximation errors, or are too time-consuming. We therefore propose MarginAttack, a zero-confidence attack framework that is able to compute the margin with improved accuracy and efficiency. Our experiments show that MarginAttack is able to compute a smaller margin than the state-of-the-art zero-confidence attacks, and matches the state-of-the-art fix-perturbation attacks. In addition, it runs significantly faster than the Carlini-Wagner attack, currently the most accurate zero-confidence attack algorithm. Adversarial attack refers to the task of finding small and imperceptible input transformations that cause a neural network classifier to misclassify. White-box attacks are a subset of attacks that have access to gradient information of the target network. In this paper, we will focus on the white-box attacks. An important class of input transformations is adding small perturbations to the input. There are two major paradigms of adversarial attacks that attempt to impose input perturbations. The first paradigm, called the fix-perturbation attack, tries to find perturbations that are most likely to cause misclassification, with the constraint that the norm of the perturbations cannot exceed a given level. Since the perturbation level is fixed, fix-perturbation attacks may fail to find any adversarial samples for inputs that are far away from the decision boundary. The second paradigm, called the zero-confidence attack, tries to find the smallest perturbations that are guaranteed to cause misclassification, regardless of how large the perturbations are. Since they aim to minimize the perturbation norm, zero-confidence attacks usually find adversarial samples that ride right on the decision boundaries, and hence the name "zero-confidence". The ing perturbation norm is also known as the margin of an input feature to the decision boundary. Both of these paradigms are essentially constrained optimization problems. The former has a simple convex constraint (perturbation norm), but a non-convex target (classification loss or logit differences). In contrast, the latter has a non-convex constraint (classification loss or logit differences), but a simple convex target (perturbation norm).Despite their similarity as optimization problems, the two paradigms differ significantly in terms of difficulty. The fix-perturbation attack problem is easier. The state-of-the-art algorithms, including projected gradient descent (PGD) BID10 and distributional adversarial attack , can achieve both high efficiency and high success rate, and often come with theoretical convergence guarantee. On the other hand, the zero-confidence attack problem is much more challenging. Existing methods are either not strong enough or too slow. For example, DeepFool BID11 and fast gradient sign method (FGSM) BID3 BID7 b) linearizes the constraint, and solves the simplified optimization problem with a simple convex target and a linear constraint. However, due to the linearization approximation errors, the solution can be far from optimal. As another extreme, L-BFGS BID18 and Carlini-Wagner (CW) BID1 convert the optimization problem into a Lagrangian, and the Lagrangian multiplier is determined through grid search or binary search. These attacks are generally much stronger and theoretically grounded, but can be very slow. The necessity of developing a better zero-confidence attack is evident. The zero-confidence attack paradigm is a more realistic attack setting. More importantly, it aims to measure the margin of each individual token, which lends more insight into the data distribution and adversarial robustness. Motivated by this, we propose MARGINATTACK, a zero-confidence attack framework that is able to compute the margin with improved accuracy and efficiency. Specifically, MARGINATTACK iterates between two moves. The first move, called restoration move, linearizes the constraint and solves the simplified optimization problem, just like DeepFool and FGSM; the second move, called projection move, explores even smaller perturbations without changing the constraint values significantly. By construction, MARGINATTACK inherits the efficiency in DeepFool and FGSM, and improves over them in terms of accuracy with a convergence guarantee. Our experiments show that MARGINAT-TACK attack is able to compute a smaller margin than the state-of-the-art zero-confidence attacks, and matches the state-of-the-art fix-perturbation attacks. In addition, it runs significantly faster than CW, and in some cases comparable to DeepFool and FGSM. In addition to the aforementioned state-of-the-art attacks, there are a couple of other works that attempt to explore the margin. Jacobian-based saliency map attack BID13 is among the earliest works that apply gradient information to guide the crafting of adversarial examples. It chooses to perturb the input features whose gradient is consistent with the adversarial goal. Onepixel attack BID17 finds adversarial examples by perturbing only one pixel, which can be regarded as finding the 0 margin of the inputs. BID5 converts PGD into a zeroconfidence attack by searching different perturbation levels, but this again can be time-consuming because it needs to solve multiple optimization subproblems. Weng et al. proposed a metric called CLEVER , which estimates an upper-bound of the margins. Unfortunately, recent work BID2 has shown that CLEVER can overestimate the margins due to gradient masking BID14. The above are a just a small subset of white-box attack algorithms that are relevant to our work. For an overview of the field, we refer readers to BID0.The MARGINATTACK framework is inspired by the Rosen's algorithm BID15 for constraint optimization problems. However, there are several important distinctions. First, the Rosen's algorithm rests on some unrealistic assumptions for neural networks, e.g. continuously differentiable constraints, while MARGINATTACK has a convergence guarantee with a more realistic set of assumptions. Second, the Rosen's algorithm requires a step size search for each iteration, which can be time-consuming, whereas MARGINATTACK will work with a simple diminishing step size scheme. Most importantly, as will be shown later, MARGINATTACK refers to a large class of attack algorithms depending on how the two parameters, a (k) and b (k), are set, and the Rosen's algorithm only fits into one of the settings, which only works well under the 2 norm. For other norms, there exist other parameter settings that are much more effective. As another highlight, the convergence guarantee of MARGINATTACK holds for all the settings that satisfy some moderate assumptions. In this section, we will formally introduce the algorithm and discuss its convergence properties. In the paper, we will denote scalars with non-bolded letters, e.g. a or A; column vectors with lowercased, bolded letters, e.g. a; matrix with upper-cased, bolded letters, e.g. A; sets with upper-cased double-stoke letters, e.g. A; gradient of a function f (x) evaluated at x = x 0 as ∇f (x 0). Given a classifier whose output logits are denoted as l 0 (x), l 1 (x), · · ·, l C−1 (x), where C is the total number of classes, for any data token (x 0, t), where x 0 is an n-dimensional input feature vector, and t ∈ {0, · · ·, C − 1} is its label, MARGINATTACK computes DISPLAYFORM0 where d(·) is a norm. In this paper we only consider 2 and ∞ norms, but the proposed method is generalizable to other norms. For non-targeted adversarial attacks, the constraint is defined as DISPLAYFORM1 where ε is the offset parameter. As a common practice, ε is often set to a small negative number to ensure that the adversarial sample lies on the incorrect side of the decision boundary. In this paper, we will only consider non-targeted attack, but all the discussions are applicable to targeted attacks (i.e. c(x) = max i =a l i (x) − l a (x) − ε for a target class a). MARGINATTACK alternately performs the restoration move and the projection move. Specifically, denote the solution after the k-th iteration as x (k). Then the two steps are:Restoration Move: The restoration move tries to hop to the constraint boundary, i.e. c(x) = 0 with the shortest hop. Formally, it solves: DISPLAYFORM0 where α (k) is the step size within. Notice that the left hand side of the constraint in Eq. is the first-order Taylor approximation of c(z DISPLAYFORM1, so this constraint tries to move point closer to c(x) = 0 by α (k). It can be shown, from the dual-norm theory, 1 that the solution to is DISPLAYFORM2. Specifically, noticing that the dual norm of the p norm is the (1−p −1) −1 norm, we have DISPLAYFORM3 As mentioned, Eq. is similar to DeepFool under 2 norm, and to FGSM under ∞ norm. Therefore, we can expect that the restoration move should effectively hop towards the decision boundary, but the hop direction may not be optimal. That is why we need the next move. Projection Move: The projection move tries to move closer to x 0 while ensuring that c(x) will not change drastically. Formally, DISPLAYFORM4 where β (k) is the step size within; a (k) and b (k) are two scalars, which will be specified later. As an intuitive explanation on Eq., notice that the second term, which we will call the distance reduction term, reduces the distance to x 0, whereas the third term, which we will call the constraint reduction term, reduces the the constraint (because s(z (k) ) and ∇c(z (k) ) has a positive inner product). Therefore, the projection move essentially strikes a balance between reduction in distance and reduction in constraint. a (k) and b (k) can have two designs. The first design is to ensure the constraint values are roughly the same after the move, i.e. c(z DISPLAYFORM5 whose solution is DISPLAYFORM6 Another design is to ensure the perturbation norm reduces roughly by DISPLAYFORM7 . By Taylor approximation, we have DISPLAYFORM8 whose solution is DISPLAYFORM9 It should be noted that Eqs. FORMULA8 and FORMULA0 are just two specific choices for a (k) and b (k). It turns out that MARGINATTACK will work with a convergence guarantee for a wide range of bounded a (k) s and b (k) s that satisfy some conditions, as will be shown in section 3.4. Therefore, MARGINAT-TACK provides a general and flexible framework for zero-confidence adversarial attack designs. In practice, we find that Eq. works better for 2 norm, and Eq. works better for ∞ norm. FIG1 illustrates a typical convergence path of MARGINATTACK using 2 norm and Eq. FORMULA8 as an example. The red dots on the right denote the original inputs x 0 and its closest point on the decision boundary, x *. Suppose after iteration k, MARGINATTACK reaches x (k), denoted by the green dot on the left. The restoration move travels directly towards the decision boundary by finding the normal direction to the current constraint contour. Then, the projection move travels along the tangent plane of the current constraint contour to reduce the distance to x 0 while preventing the constraint value from deviating much. As intuitively expected, the iteration should eventually approach x *. FIG2 plots an empirical convergence curve of the perturbation norm and constraint value of MARGINATTACK-2 on a randomly chosen CIFAR image. Each move from a triangle to a circle dot is a restoration move, and from circle to triangle a projection move. The red line is the smoothed version. As can be seen, a restoration move reduces the constraint value while slightly increasing the constraint norm, and a projection move reduces the perturbation norm while slightly affecting the constraint value. Both curves can eventually converge. The constraint function c(x) in Eq. FORMULA1 is nonconvex, thus the convergence analysis for MARGINAT-TACK is limited to the vicinity of a unique local optimum, as stated in the following theorem. Theorem 1. Denote x * as one local optimum for Eq.. Assume ∇c(x *) exists. Define projection matrices DISPLAYFORM0 Consider the neighborhood B = {x : DISPLAYFORM1 2 ≤ X, |c(x)| ≤ C} that satisfies the following assumptions:1. (Differentiability) ∀x ∈ B, ∇c(x) exists, but can be discontinuous, i.e. all the discontinuity points of the gradient in B are jump discontinuities; DISPLAYFORM2 6. (Unique Optimality) x * is the only global optimum within B; DISPLAYFORM3 Then we have the convergence guarantee lim k→∞ DISPLAYFORM4 The proof will be presented in the appendix. Here are a few remarks. First, assumption 1 allows jump discontinuities in ∇c(x) almost everywhere, which is a very practical assumption for deep neural networks. Most neural network operations, such as ReLU and max-pooling, as well as the max operation in Eq., introduce nothing beyond jump discontinuities in gradient. Second, assumption 3 does require the constraint gradient to be lower bounded, which may lead to concerns that MARGINATTACK may fail in the presence of gradient masking BID14. However, notice that the gradient boundedness assumption is only imposed in B, which is in the vicinity of the decision boundary, whereas gradient masking is most likely to appear away from the decision boundary and where the input features are populated. Besides, as will be discussed later, a random initialization as in PGD will be adopted to bypass regions with gradient masking. Experiments on adversarially trained models also verify the robustness of MARGINATTACK.Finally, assumption 5 essentially stipulates that c(x) is convex or "not too concave" in B (and thus so is the constraint set c(x) ≤ 0), so that the first order optimality condition can readily imply local minimum instead of a local maximum. In fact, it can be shown that assumption 5 can be implied if c(x) is convex in B. There are a few additional implementation details as outlined below. Box Constraint: In many applications, each dimension of the input features should be bounded, i.e. x ∈ [x min, x max] n. To impose the box constraint, the restoration move problem as in Eq. FORMULA2 is modified as DISPLAYFORM5 whose solution is DISPLAYFORM6 Proj(·) is an operator that projects the vector in its argument onto the subset in its subscript. I is a set of indices with which the elements inz (k) satisfy the box constraint, and I C is its complement. I is determined by running Eq. iteratively and updating I after each iterations. Unlike other attack algorithms that simply project the solution onto the constraint box, MARGINAT-TACK incorporates the box constraint in a principled way, such that any local optimal solution x * will be an invariant point of the restoration move. Thus the convergence is faster. Target Scan: According to Eq., each restoration move essentially approaches the adversarial class with the highest logit, but the class with the highest logit may not be the closest. To mitigate the problem, we follow a similar approach adopted in DeepFool, which we call target scan. Target scan performs a target-specific restoration move towards each class, and chooses the move with the shortest distance. Formally, target scan introduces a set of target-specific constraints {c i (x) = l t (x) − l i (x) − ε}. A restoration move with target scan solves DISPLAYFORM7 where z (k,i) is the solution to Eqs. FORMULA2 or FORMULA0 with c(x (k) ) replaced with c i (x (k) ), and thus is equal to Eqs. or with c(DISPLAYFORM8 A is a set of candidate adversarial calsses, which can be all the incorrect classes if the number of classes is small, or which can be a subset of the adversarial classes with the highest logits otherwise. Experiments show that target scan is necessary only in the first few restoration moves, when the closest and highest adversarial classes are likely to be distinct. Therefore, the computation cost will not increase too much. Initialization: The initialization of x can be either deterministic or random as follows DISPLAYFORM9 where U{[−u, u] n } denotes the uniform random distribution in [−u, u] n. Similar to PGD, we can perform multiple trials with random initialization to find a better local optimum. Final Tuning MARGINATTACK can only cause misclassification when c(x) ≤ ε. To make sure the attack is successful, the final iterations of MARGINATTACK consists of restoration moves only, DISPLAYFORM10 and no projection moves, until a misclassification is caused. This can also ensure the final solution satisfies the box constraint (because only the restoration move incorporates the box constraint).Summary: Alg. 1 summarizes the MARGINATTACK procedure. As for the complexity, each restoration move or projection move requires only one backward propagation, and thus the computational complexity of each move is comparable to one iteration of most attack algorithms. This section compares MARGINATTACK with several state-of-the-art adversarial attack algorithms in terms of the perturbation norm and computation time on image classification benchmarks. Three regularly trained models are evaluated on.• MNIST : The classifier is a stack of two 5 × 5 convolutional layers with 32 and 64 filters respectively, followed by two fully-connected layers with 1,024 hidden units.• CIFAR10 BID6 ): The classifier is a pre-trained ResNet32 BID4 provided by TensorFlow. 5.• ImageNet BID16: The classifier is a pre-trained ResNet50 BID4 provided by TensorFlow Keras 6. Evaluation is on a validation subset containing 10,000 images. The range of each pixel is for MNIST, and for CIFAR10 and ImageNet. The settings of MARGINATTACK and baselines are listed below. Unless stated otherwise, the baseline algorithms are implemented by cleverhans BID14. The hyperparameters are set to defaults if not specifically stated.• CW BID1: The target and evaluation norm is 2. The learning rate is set to 0.05 for MNIST, 0.001 for CIFAR10 and 0.01 for ImageNet, which are tuned to its best performance. The number of binary steps for multiplier search is 10. • DeepFool : The evaluation norm is 2.• FGSM BID3: FGSM is implemented by authors. The step size is searched to achieve zero-confidence attack. The evaluation distance metric is ∞.• PGD BID10: The target and evaluation norm are ∞. The learning rate is set to 0.01 for MNIST, and 0.05 for CIFAR10 and 0.1 for ImageNet.• MARGINATTACK: Two versions of MARGINATTACK are implemented, whose target and evaluation norms are 2, and ∞, respectively. The hyperparmeters are detailed in TAB4 in the appendix. The first 10 restoration moves are with target scan, and the last 20 moves are all restoration moves. The number of iterations/moves is set to 2,000 for CW, 200 with 10 random starts for PGD and MARGINATTACK (except for ImageNet where there is only one random run), and 200 for the rest. Except for PGD, all the other attacks are zero-confidence attacks. For these attacks, we plot the CDF of the margins of the validation data, which can also be interpreted as the percentage success rate of these attacks as a function of perturbation level. FIG3 plots the success rate curves, where the upper panel shows the 2 attacks, and the lower one shows ∞ attacks. As can be observed, the MARGINATTACK curves are above all other algorithms at all perturbation levels and in all datasets. CW is very close to MARGINATTACK on MNIST and CIFAR10, but MARGINATTACK maintains a 3% advantage on MNIST and 1% on CIFAR10. It seems that CW is unable to converge well within 2,000 iterations on ImageNet, although the learning rate has been tuned to maximize its performance. MARGINATTACK, on the other hand, converges more efficiently and consistently. To obtain a success rate curve for PGD, we have to run the attack again and again for many different perturbation levels, which can be time-consuming for large datasets (this shows an advantage of zero-confidence attacks over fix-perturbation attacks). Instead, we choose four perturbation levels for each attack scenario to compare. The perturbation levels are chosen to roughly follow the 0.2, 0.4, 0.6 and 0.8 quantiles of the MARGINATTACK margins. TAB1 compares the success rates under the chosen quantiles among the ∞ attacks. We can see that MARGINATTACK outperforms PGD under all the perturbation levels, and that both significantly dominate FGSM. We also evaluate MARGINATTACK on the MNIST Adversarial Examples Challenge 7, which is a challenge of attacking an MNIST model adversarially trained using PGD with 0.3 perturbation level. Same as the PGD baseline listed, MARGINATTACK is run with 50 random starts, and the initialization perturbation range u = 0.3. The number of moves is 500. The target norm is ∞. b n = 5 and a n is set as in Eq.. The rest of the configuration is the same as in the previous experiments. Table 2 lists the success rates of different attacks under 0.3 perturbation level. The baseline algorithms are all fix-perturbation attacks, and their are excerpted from the challenge white-box attack leaderboard. As can be seen, MARGINATTACK, as the only zero-confidence attack algorithm, has the second best , which shows that it performs competitively against the state-of-the-art fix-perturbation attacks. We would like to revisit the convergence plot of the constraint value c(x) and perturbation norm d(x) of as in FIG2. We can see that MARGINATTACK converges very quickly. In the example shown in the figure, it is able to converge within 20 moves. Therefore, MARGINATTACK can be greatly accelerated. If margin accuracy is the priority, a large number of moves, e.g. 200 as in our experiment, would help. However, if efficiency is the priory, a small number of moves, e.g. 30, suffices to produce a decent attack. To further assess the efficiency of MARGINATTACK, Tab. 3 compares the running time (in seconds) of attacking one batch of images, implemented on a single NVIDIA TESLA P100 GPU. The batch size is 200 for MNIST and CIFAR10, and 100 for ImageNet. The settings are the same as stated in section 4.1, except that for a better comparison, the number of iterations of CW is cut down to 200, and PGD and MARGINATTACK runs one random pass, so that all the algorithms have the same iteration/moves. Only the 2 versions of MARGINATTACK are shown because the other versions have similar run times. As shown, running time of MARGINATTACK is much shorter than CW, and is comparable to DeepFool and PGD. CW is significantly slower that the other algorithms because it has to run multiple trials to search for the best Lagrange multiplier. Note that DeepFool and CW enable early stop, but MARGINATTACK does not. Considering MARGINATTACK's fast convergence rate, the running time can be further reduced by early stop. We have proposed MARGINATTACK, a novel zero-confidence adversarial attack algorithm that is better able to find a smaller perturbation that in misclassification. Both theoretical and empirical analyses have demonstrated that MARGINATTACK is an efficient, reliable and accurate adversarial attack algorithm, and establishes a new state-of-the-art among zero-confidence attacks. What is more, MARGINATTACK still has room for improvement. So far, only two settings of a (k) and b (k) are developed, but MARGINATTACK will work for many other settings, as long as assumption 5 is satisfied. Authors hereby encourage exploring novel and better settings for the MARGINATTACK framework, and promote MARGINATTACK as a new robustness evaluation measure or baseline in the field of adversarial attack and defense. This supplementary material aims to prove Thm. 1. Without the loss of generality, K in Eq. in set to 0. Before we prove the theorem, we need to introduce some lemmas. Lemma 1.1. If assumption 3 in Thm. 1 holds, then ∀x ∈ B DISPLAYFORM0 Proof. According to Eq., for 2 norm, DISPLAYFORM1 for ∞ norm, DISPLAYFORM2 Lemma 1.2. Given all the assumptions in Thm. 1, where DISPLAYFORM3 and assuming DISPLAYFORM4 where DISPLAYFORM5 A and B are defined in Eq..According to assumption 8, this implies DISPLAYFORM6 at the rate of at least 1/n ν.Proof. As a digression, the second term in Eq. FORMULA0 is well defined, because DISPLAYFORM7 is upper bounded by Lem. 1.1 and assumptions 3.Back to proving the lemma, we will prove that each restoration move will bring c(x (k) ) closer to 0, while each projection move will not change c(x (k) ) much. First, for the restoration move DISPLAYFORM8 The first line is from the generalization of Mean-Value Theorem with jump discontinuities, and ξ = tz (k) + (1 − t)x (k) and t is a real number in. The second line is from Eq.. The last line is from assumptions 4 and 7 and Eq..Next, for the projection move DISPLAYFORM9 The first line is from the fact that assumption 3 implies that c(x) is M -Lipschitz continuous. DISPLAYFORM10 for some M d and M s. To see this, for 2 norm DISPLAYFORM11 where b is defined as the maximum perturbation norm within B, i.e. DISPLAYFORM12 which is well defined because B is a tight set. For ∞ norm, DISPLAYFORM13 Note that Eq. also holds for other norms. With Eq. and assumption 8, Eq. FORMULA1 becomes DISPLAYFORM14 Combining Eqs. FORMULA1 and FORMULA2 we have DISPLAYFORM15 where DISPLAYFORM16 According to assumption 7, 0 < A < 1. Also, according to Eq. FORMULA1, DISPLAYFORM17 and thus DISPLAYFORM18 If DISPLAYFORM19 Otherwise, Eq. implies DISPLAYFORM20 This concludes the proof. Lemma 1.3. Given all the assumptions in Thm. 1, and assuming DISPLAYFORM21 Proof. First, for restoration move DISPLAYFORM22 δm 2 Line 4 is given by Eq.. Line 5 is derived from Lem. 1.1. The last line is from Lem. 1.2. δm 2 where DISPLAYFORM0 It can easily be shown that DISPLAYFORM1 Therefore DISPLAYFORM2 Combining Eqs. FORMULA2 and FORMULA1, we have DISPLAYFORM3 Step Case: Assume Eq. holds ∀k ≤ K, then Eqs. and holds ∀k ≤ K.• Proving |c(x (K+1) )| ≤ C: DISPLAYFORM4 where the last inequality is given by Eq.. DISPLAYFORM5 Notice that from Eq., 0 DISPLAYFORM6 where the last inequality is given by Eq.. DISPLAYFORM7 Lemma 1.5. Under the assumptions in Thm. 1 DISPLAYFORM8 Proof. From Thm. 2, a solution, denoted as x, to min DISPLAYFORM9 would satisfy DISPLAYFORM10 If P [x * − x 0] = 0, there are two possibilities. The first possibility is that x * is not a solution to Eq., which contradicts with the first order optimality condition that x * must satisfy. The second possibility is there are multiple solutions to the problem in Eq. FORMULA1, and x and x * are both its solutions. This can happen if d(·) is 1 or ∞ norm. By definition DISPLAYFORM11 * is a local minimum to Eq., ∃j ∈ I, ε < 1, ∀δ < ε, DISPLAYFORM12 65) Otherwise, if c(x δ) ≤ 0, then x δ is a feasible solution to the problem in Eq. FORMULA0 and DISPLAYFORM13 which contradicts with the assumption that x * is a unique local optimum in B. DISPLAYFORM14 takes discrete values. Therefore, to satisfy assumption 2, s j (x δ) = s j (x *), which implies DISPLAYFORM15 The first inequality is because DISPLAYFORM16 Eqs. FORMULA6 and FORMULA6 cause a contradiction. Now we are ready to prove Thm. 1.Proof of Thm. 1. From Lems. 1.2, 1.3 and 1.4, we can established that Eqs. FORMULA0 and FORMULA2 holds under all the assumptions in Thm. 1. The only thing we need to prove is that Eqs. FORMULA0 and FORMULA2 necessarily implies lim k→ x (k) − x 0 2 = 0.First, from Lem. 1.5 DISPLAYFORM17 Then, ∀x ∈ B s.t. P [x − x 0] 2 2 = 0, we have x − x = λs(x). From assumption 4, we know that c(x) is monotonic along x − x = λs(x). Therefore, x * is the only point in B that satisfies P [x − x 0] 2 2 = 0 and c(x) = 0. Also, notice that P [x − x 0] and c(x) are both continuous mappings. This concludes the proof. Notice that the product of λ and ∇ T c(x) y is constant, so if λ is to be minimized, then ∇ T c(x)y needs to be maximized. Namely, y can be determined by solving which is the definition of dual norm. Therefore y = s(x) Plug Eq. into the constraint in Eq., we can solve for λ. This concludes the proof. As a remark, Thm. 2 is applicable to the optimization problems in Eqs. FORMULA2 and FORMULA1 Proof. Since c(x) is convex in B, we have ∀x DISPLAYFORM18 Further, assume x satisfies ∇ T c(x *)(x − x *) = 0 Then we have P T P (x − x *) = P (x − x *) = x − x * where the first equality is from the fact that P is an orthogonal projection matrix under 2 norm; the second equality is from the fact that the projection subspace of P is orthogonal to ∇c(x *) by construction. Also, from Lem. 1.5, we have DISPLAYFORM19 Plug Eqs. FORMULA7 to FORMULA7 into FORMULA6, we have DISPLAYFORM20 On the other hand, let γ = m a /2 x * − x 0 2, then ∀x satisfying Eq. and DISPLAYFORM21 where the second line comes from Eq. and the fact that x * is the optimal solution to the problem in Eq.. Combining Eqs. FORMULA8 and FORMULA0, we know that assumption 5 holds with strict inequality for x satisfying Eq. and x = x *.∇ T c(x)P T P (x − x *), ∇ T d(x − x 0)P T P (x − x 0) and γ(x − x 0) T P T P (x − x 0) are continuous functions, and therefore ∃B where assumption 5 also holds. This concludes the proof.

Represent the TLDR for finding the corresponding scientific article

This paper introduces MarginAttack, a stronger and faster zero-confidence adversarial attack.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the article for finding a single-sentence tldr.

Question: Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field. Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly. The degree of variation and diversity of inputs makes this a difficult task. Existing methods either learn a feedforward predictor, which is not itself totally immune to the scale variation it is meant to counter, or select scales by a fixed algorithm, which cannot learn from the given task and data. We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity. We propose a novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input. Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter. The world is infinite in its variations, but our models are finite. While inputs differ in many dimensions and degrees, a deep network is only so deep and wide. To nevertheless cope with variation, there are two main strategies: static enumeration and dynamic adaptation. Static enumeration defines a set of variations, processes them all, and combines the . For example, pyramids enumerate scales and group-structured filters enumerate orientations . Dynamic adaptation selects a single variation, conditioned on the input, and transforms processing accordingly. For example, scale-space search selects a scale transformation from input statistics and end-to-end dynamic networks select geometric transformations , parameter transformations , and feature transformations directly from the input. Enumeration and adaptation both help, but are limited by computation and supervision, because the sets enumerated and ranges selected are bounded by model size and training data. Deep networks for vision exploit enumeration and adaptation, but generalization is still limited. Networks are enumerative, by convolving with a set of filters to cover different variations then summing across them to pool the variants (; ;). For scale variation, image pyramids and feature pyramids (; enumerate scales, process each, and combine the outputs. However, static models have only so many filters and scales, and may lack the capacity or supervision for the full data distribution. Dynamic models instead adapt to each input . The landmark scale invariant feature transform extracts a representation adapted to scales and orientations predicted from input statistics. Dynamic networks, including spatial transformers and deformable convolution , make these predictions and transformations end-to-end. Predictive dynamic inference is however insufficient: the predictor may be imperfect in its architecture or parameters, or may not generalize to data it was not designed or optimized for. Bottom-up prediction, with only one step of adaptation, can struggle to counter variations in scale and other factors that are too large or unfamiliar. To further address the kinds and degrees of variations, including extreme out-of-distribution shifts, we devise a complementary third strategy: unsupervised optimization during inference. We define an unsupervised objective and a constrained set of variables for effective gradient optimization. Our novel inference objective minimizes the entropy of the model output to optimize for confidence. The variables optimized over are task parameters for pixel-wise classification and structure parameters Accuracy is high and prediction entropy is low for training and testing at the same scale (left). Accuracy drops and entropy rises when tested at 3x the training scale, even when the network is equipped with dynamic receptive fields to adapt to scale variation (middle). Previous approaches are limited to one-step, feedforward scale prediction, and are unable to handle a 3x shift. In contrast our iterative gradient optimization approach is able to adapt further (right), and achieve higher accuracy by minimizing entropy with respect to task and scale parameters. for receptive field adaptation, which are updated together to compensate for scale shifts. This optimization functions as top-down feedback to iteratively adjust feedforward inference. In effect, we update the trained model parameters to tune a custom model for each test input. Optimization during inference extends dynamic adaptation past the present limits of supervision and computation. Unsupervised optimization boosts generalization beyond training by top-down tuning during testing. Iterative updates decouple the amount of computation, and thus degree of adaptation, from the network architecture. Our main is to demonstrate that adaptation by entropy optimization improves accuracy and generalization beyond adaptation by prediction (see Figure 1), which we show for semantic segmentation by inference time optimization of a dynamic Gaussian receptive field model on the PASCAL VOC dataset. Our approach extends dynamic scale inference from one-step prediction to multi-step iteration through optimization. For optimization during inference, we require an objective to optimize and variables to optimize over. Lacking task or scale supervision during inference, the objective must be unsupervised. For variables, there are many choices among parameters and features. Our main contribution is an unsupervised approach for adapting task and structure parameters via gradient optimization to minimize prediction entropy. Note that our inference optimization is distinct from the training optimization. We do not alter training in any way: the task loss, optimizer, and model are entirely unchanged. In the following, optimization refers to our inference optimization scheme, and not the usual training optimization. To optimize inference, a base dynamic inference method is needed. For scale, we choose local receptive field adaptation , because scale varies locally even within a single image. In particular, we adopt dynamic Gaussian receptive fields that combine Gaussian scale-space structure with standard "free-form" filters for parameter-efficient spatial adaptation. These methods rely on feedforward regression to infer receptive fields that we further optimize. Figure 2 illustrates the approach. Optimization is initialized by feedforward dynamic inference of Gaussian receptive fields . At each following step, the model prediction and its entropy are computed, and the objective is taken as the sum of pixel-wise entropies. Model parameters are iteratively updated by the gradient of the objective, ing in updated predictions and entropy. Optimization of the parameters for the Gaussian receptive fields is instrumental for adapting to scale. (top) is optimized according to the output (bottom) at test time. We optimize receptive field scales and filter parameters to minimize the output entropy (middle). Optimizing during inference makes iterative updates shown from left to right: receptive field scale adapts, entropy is reduced, and accuracy is improved. This gives a modest refinement for training and testing at the same scale, and generalization improves for testing at different scales. Unsupervised inference objectives can be bottom-up, based on the input, or top-down, based on the output. To augment already bottom-up prediction, we choose the top-down objective of entropy minimization. In essence, the objective is to reduce model uncertainty. More precisely, for the pixel-wise outputŶ ∈ C×H×W for C classes and an image of height H and width W, we measure uncertainty by the Shannon entropy : for each pixel at index i, j to yield pixel-wise entropy of the same spatial dimensions as the output. Entropy is theoretically motivated and empirically supported. By inspection, we see that networks tend to be confident on in-distribution data from the training regime. (Studying the probabilistic calibration of networks confirms this.) In our case, this holds for testing scales similar to the training scales, with high entropy on segment contours. On out-of-distribution data, such as scale shifts, the output entropy is higher and less structured. This objective is severe, in that its optimum demands perfect certainty (that is, zero entropy). As a more stable alternative, we consider adaptively thresholding the objective by the average entropy across output pixels. We calculate the mean entropy at each iteration, and only take the gradient of pixels with above-average entropy. This mildly improves accuracy. Our final objective is then: where S is the set of pixels with entropy above the average H µ. At each step, we re-calculate the average entropy and re-select the set of violating pixels. In this way, optimization is focused on updating predictions where the model is the most uncertain. We need to pick the variables to optimize over so that there are enough degrees of freedom to adapt, but not so many that overfitting sets in. Furthermore, computation time and memory demand a minimal set of variables for efficiency. Choosing parameters in the deepest layers of the network satisfy these needs: capacity is constrained by keeping most of the model fixed, and computation is reduced by only updating a few layers. The alterantive of choosing all the parameters, and optimizing end-to-end during inference, is ineffective and inefficient: inference is slower and less accurate than feedforward prediction. We select the task parameters θ score of the output classification filter, for mapping from features to classes, and the structure parameters θ scale of the scale regression filter, for mapping from features to receptive field scales. Optimizing over these parameters indirectly optimizes over the local predictions for classification scoresŶ and scalesΣ. Why indirectly optimize the outputs and scales via these parameters, instead of direct optimization? First, dimensionality is reduced for regularization and efficiency: the parameters are shared across the local predictions for the input image and have fixed dimension. Additionally, this preserves dependence on the data: optimizing directly over the classification predictions admits degenerate solutions that are independent of the input. Initialization The unaltered forward pass of the base network gives scoresŶ and scalesΣ. Iteration For each step t, the loss is the sum of thresholded entropies of the pixel-wise predictionŝ Y (t). The gradient of the loss is taken for the parameters θ scale. Given the new parameters, a partial forward pass re-infers the local scales and predictions forŶ (t+1) andΣ (t+1). This efficient computation is a small fraction of the initialization forward pass. Termination The number of iterations is set and fixed to control the amount of inference computation. We do so for simplicity, but note that in principle convergence rules such as relative tolerance could be used with the loss, output, or parameter changes each iteration for further adaptivity. Figure 3 shows the progress of our inference optimization across iterations. We experiment with extending from predictive to iterative dynamic inference for semantic segmentation, because this task has a high degree of appearance and scale variation. In particular, we show for iterative optimization of classifier and scale parameters in a dynamic Gaussian receptive field model on the PASCAL VOC dataset. By adapting both task and structure parameters, our approach improves accuracy on in-distribution inputs and generalizes better on out-of-distribution scale shifts. We ablate which variables to optimize and for how many steps, and analyze our choices by oracle and adversary . These experiments establish the efficacy of entropy minimization during inference for scale adaptation, while oracle show opportunity for further progress. Data and Metric PASCAL VOC is a well-established semantic segmentation benchmark with 20 semantic classes and a class. The original dataset only has 1,464, 1,449 and 1,456 images with segmentation annotations for training, validation, and testing, respectively. As is standard practice, we include the additional 9,118 images and annotations from , giving 10,582 training samples in total. We measure accuracy by the usual metric of mean intersection-over-union (IoU). We report our on the validation set. Architecture We choose deep layer aggregation (DLA) as a strong, representative fully convolutional network architecture. DLA exploits the feature pyramid inside the network via iterative and hierarchical aggregation across layers. We will release code and the reference models implemented in PyTorch . Training We train our model on the original scale of the dataset. We optimize via stochastic gradient descent (SGD) with batch size 64, initial learning rate 0.01, momentum 0.9, and weight decay 0.0001 for 500 epochs. We use the "poly" learning rate schedule with power 0.9. For the model with no data augmentation ("w/o aug"), the input images are padded to 512 × 512. As for the "w/ aug" model, data augmentation includes cropping to 512 × 512, scaling in [0.5, 2], rotation in [−10 •, 10 •], color distortion , and horizontal flipping. Testing We test our model on different scales of the dataset in the [1.5, 4.0] range. We optimize the model parameters for adaptation via Adam , batching all image pixels together, and setting the learning rate to 0.001. The model is optimized episodically to each input, and the parameters are reset between inputs. No data augmentation is used during inference to isolate the role of dynamic inference by the model. We compare the semantic segmentation accuracy of our optimization with a prediction baseline and optimization by oracle and adversary. The baseline is a one-step dynamic model using feedforward scale regression to adapt receptive fields following . We train on a narrow range of scales and test on a broader range of scales to measure refinement, the improvement for the training scales, and generalization, the improvement for the new scales. This baseline is the initialization for our iterative optimization approach: the output and scale predictions for the initial iteration are inferred by the one-step model. For analysis , the oracle and adversary optimize during inference to respectively minimize/maximize the cross-entropy loss of the output and the truth. As reported in Table 1, our method consistently improves on the baseline by ∼2 points for all scales, which indicates that our unsupervised optimization for iterative inference helps the model generalize better across scales. When the scale shift is larger, there is likewise a larger gap. To evaluate the effect of data augmentation, we experiment with ("w/ aug") and without ("w/o aug"). Data augmentation significantly improves generalization across scales. Note that our optimization during inference still improves the model with data augmentation by the same amount. Results are scored by intersection-over-union (higher is better). "w/o aug" excludes data augmentation, where "w/ aug" includes scaling, rotation, and other augmentation. Even though data augmentation reduces the effect of scale variation, our method further improves accuracy for all scales. Table 2: Ablation of the number of iterations: entropy minimization saturates after 32 steps. We ablate the choice of parameters to optimize and the number of updates to make. We optimize during inference to adapt the task parameters (score) of the classifier and structure parameters (scale) of the scale regressor. The task parameters map between the visual features and the classification outputs. Updates to the task parameters are the most direct way to alter the pixelwise output distributions. Updates to the structure parameters address scale differences by adjusting receptive fields past the limits of the feedforward scale regressor. From the experiments in Table 3, both are helpful for refining accuracy and reducing the generalization gap between different scales. Optimizing end-to-end, over all parameters, fails to achieve better than baseline . Iterative optimization gives a simple control over the amount of computation: the number of updates. This is a trade-off, because enough updates are needed for adaptation, but too many requires excessive computation. Table 2 shows that 32 steps are enough for improvement without too much computation. Therefore, we set the number of steps as 32 for all experiments in this paper. For our network, one step of inference optimization takes ∼ 1 10 the time of a full forward pass. We analyze the distribution of scales in Figure 4 and show qualitative segmentation in Figure 5. While better compensating for scale shift is our main goal, our method also refines inference on in-distribution data. The in Table 3 for 1× training and testing show improvement of ∼1 point. We analyze our approach from an adversarial perspective by maximizing the entropy instead of minimizing. To measure the importance of a parameter, we consider how much accuracy degrades when adversarially optimizing it. The more performance degrades, the more it matters. Dynamic Inference Dynamic inference adapts the model to each input . Many approaches, designed and learned (; ; ; ;), rely on bottom-up prediction from the input. Our method extends bottom-up prediction with top-down optimization to iteratively update the model from the output. Recurrent approaches to iterative inference require changing the architecture and training more parameters. Our optimization updates parameters without architectural alteration. Entropy Objective We minimize entropy during testing, not training, in effect tuning a different model to each input. The entropy objectives in existing work are optimized during training, especially for regularization. Entropy is maximized/minimized for domain adaptation (; ; ;) and semi-supervised learning . In reinforcement learning, maximum entropy regularization improves policy optimization . We optimize entropy locally for each input during testing, while existing use cases optimize globally for a dataset during training. We optimize an unsupervised objective on output statistics to update model parameters for each test input. Energy minimization models and probabilistic graphical models learn model parameters during training then optimize over outputs during inference. The parameters of deep energy models and graphical models are fixed during testing, while our model is further optimized on the test distribution. Alternative schemes for learning during testing, like transduction and meta-learning, differ in their requirements. Transductive learning optimizes jointly over the training and testing sets, which can be impractical at deep learning scale. We optimize over each test input independently, hence scalably, without sustained need for the (potentially massive) training set. Meta-learning by gradients updates model parameters during inference, but requires supervision during testing and more costly optimization during meta-training. Dynamic inference by optimization iteratively adapts the model to each input. Our show that optimization to minimize entropy with respect to score and scale parameters extends adaptivity for semantic segmentation beyond feedforward dynamic inference. Generalization improves when the training and testing scales differ substantially, and modest refinement is achieved even when the training and testing scales are the same. While we focus on entropy minimization and scale inference, more optimization for dynamic inference schemes are potentially possible through the choice of objective and variables. is the out-of-distribution prediction for our iterative optimization method. Our method corrects noisy, over-segmented fragments and false negatives in true segments.

Unsupervised optimization during inference gives top-down feedback to iteratively adjust feedforward prediction of scale variation for more equivariant recognition.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question:
represent the article for finding a single-sentence tldr

------

For instance you may be given 'Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizon, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the synthesized model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one. Our experiments also show that the proposed model imitation method outperforms the state-of-the-art in terms of sample complexity and average return. Reinforcement learning (RL) has become of great interest because plenty of real-world problems can be modeled as a sequential decision-making problem. Model-free reinforcement learning (MFRL) is favored by its capability of learning complex tasks when interactions with environments are cheap. However, in the majority of real-world problems, such as autonomous driving, interactions are extremely costly, thus MFRL becomes infeasible. One critique about MFRL is that it does not fully exploit past queries over the environment, and this motivates us to consider the model-based reinforcement learning (MBRL). In addition to learning an agent policy, MBRL also uses the queries to learn the dynamics of the environment that our agent is interacting with. If the learned dynamic is accurate enough, the agent can acquire the desired skill by simply interacting with the simulated environment, so that the number of samples to collect in the real world can be greatly reduced. As a , MBRL has become one of the possible solutions to reduce the number of samples required to learn an optimal policy. Most previous works of MBRL adopt supervised learning with 2 -based errors (; ; or maximum likelihood , to obtain an environment model that synthesizes real transitions. These non-trivial developments imply that optimizing a policy on a synthesized environment is a challenging task. Because the estimation error of the model accumulates as the trajectory grows, it is hard to train a policy on a long synthesized trajectory. On the other hand, training on short trajectories makes the policy short-sighted. This issue is known as the planning horizon dilemma . As a , despite having a strong intuition at first sight, MBRL has to be designed meticulously. Intuitively, we would like to learn a transition model in a way that it can reproduce the trajectories that have been generated in the real world. Since the attained trajectories are sampled according to a certain policy, directly employing supervised learning may not necessarily lead to the mentioned especially when the policy is stochastic. The resemblance in trajectories matters because we estimate policy gradient by generating rollouts; however, the one-step model learning adopted by many MBRL methods do not guarantee this. Some previous works propose multi-step training (; ;); however, experiments show that model learning fails to benefit much from the multi-step loss. We attribute this outcome to the essence of super- Supervised learning Rollout from real world Figure 1: Distribution matching enables the learned transition to generate similar rollouts to the real ones even when the policy is stochastic or the initial states are close. On the other hand, training with supervised learning does not ensure rollout similarity and the ing policy gradient may be inaccurate. This figure considers a fixed policy sampling in the real world and a transition model. vised learning, which elementally preserves only one-step transition and the similarity between real trajectories and the synthesized ones cannot be guaranteed. In this work, we propose to learn the transition model via distribution matching. Specifically, we use WGAN to match the distributions of state-action-next-state triple (s, a, s) in real/learned models so that the agent policy can generate similar trajectories when interacting with either the true transition or the learned transition. Figure 1 illustrates the difference between methods based on supervised learning and distribution matching. Different from the ensemble methods proposed in previous works, our method is capable of generalizing to unseen transitions with only one dynamic model because merely incorporating multiple models does not alter the essence that one-step (or few-step) supervised learning fails to imitate the distribution of multi-step rollouts. Concretely, we gather some transitions in the real world according to a policy. To learn the real transition T, we then sample fake transitions from our synthesized model T with the same policy. The synthesized model serves as the generator in the WGAN framework and there is a critic that discriminates the two transition data. We update the generator and the critic alternatively until the synthesized data cannot be distinguished from the real one, which we will show later that it gives T → T theoretically. Our contributions are summarized below: • We propose an MBRL method called model imitation (MI), which enforces the learned transition model to generate similar rollouts to the real one so that policy gradient is accurate; • We theoretically show that the transition can be learned by MI in the sense that T → T by consistency and the difference in cumulative rewards can be bounded by the training error of WGAN; • To stabilize model learning, we deduce guarantee for our sampling technique and investigate training across WGANs; • We experimentally show that MI is more sample efficient than state-of-the-art MBRL and MFRL methods and outperforms them on four standard tasks. In this section, we introduce our motivation inspired by learning from demonstration (LfD) and give a brief survey of MBRL methods. A straightforward approach to LfD is to leverage behavior cloning (BC), which reduces LfD to a supervised learning problem. Even though learning a policy via BC is time-efficient, it cannot imitate a policy without sufficient demonstration because the error may accumulate without the guidance of expert . Generative Adversarial Imitation Learning (GAIL) is another state-of-the-art LfD method that learns an optimal policy by utilizing generative adversarial training to match occupancy measure . GAIL learns an optimal policy by matching the distribution of the trajectories generated from an agent policy with the distribution of the given demonstration. shows that the two distributions match if and only if the agent has learned the optimal policy. One of the advantages of GAIL is that it only requires a small amount of demonstration data to obtain an optimal policy but it requires a considerable number of interactions with environments for the generative adversarial training to converge. Our intuition is that transition learning (TL) is similar to learning from demonstration (LfD) by exchanging the roles of transition and policy. In LfD, trajectories sampled from a fixed transition are given, and the goal is to learn a policy. On the other hand, in TL, trajectories sampled from a fixed policy are given, and we would like to imitate the underlying transition. That being said, from LfD to TL, we interchange the roles of the policy and the transition. It is therefore tempting to study the counterpart of GAIL in TL; i.e., learning the transition by distribution matching. Fortunately, by doing so, the pros of GAIL remain while the cons are insubstantial in MBRL because sampling with the learned model is considered to be much cheaper than sampling in the real one. That GAIL learns a better policy than what BC does suggests that distribution matching possesses the potential to learn a better transition than supervised learning. For deterministic transition, @ -based error is usually utilized to learn the transition model. , an approach that uses supervised learning with mean-squared error as its objective, is shown to perform well under fine-tuning. To alleviate model bias, some previous works adopt ensembles , where multiple transition models with different initialization are trained at the same time. In a slightly more complicated manner, utilizes meta-learning to gather information from multiple models. Lastly, on the theoretical side, SLBO is the first algorithm that develops from solid theoretical properties for model-based deep RL via a joint model-policy optimization framework. For the stochastic transition, maximum likelihood estimator or moment matching are natural ways to learn a synthesized transition, which is usually modeled by the Gaussian distribution. Following this idea, Gaussian process and Gaussian process with model predictive control are introduced as an uncertainty-aware version of MBRL. Similar to the deterministic case, to mitigate model bias and foster stability, an ensemble method for probabilistic networks is also studied. An important distinction between training a deterministic or stochastic transition is that although the stochastic transition can model the noise hidden within the real world, the stochastic model may also induce instability if the true transition is deterministic. This is a potential reason why an ensemble of models is adopted to reduce variance. We consider the standard Markov Decision Process (MDP) . MDP is represented by a tuple S, A, T, r, γ, where S is the state space, A is the action space, T (s t+1 |s t, a t) is the transition density of state s t+1 at time step t + 1 given action a t made under state s t, r(s, a) is the reward function, and γ ∈ is the discount factor. A stochastic policy π(a|s) is a density of action a given state s. Let the initial state distribution be α. The performance of the triple (α, π, T) is evaluated in the expectation of the cumulative reward in the γ-discounted infinite horizon setting: Equivalently, R(α, π, T) is the expected cumulative rewards in a length-H trajectory {s t, a t} H−1 t=0 generated by (α, π, T) with H ∼ Geometric(1 − γ). When α and T are fixed, R(·) becomes a function that only depends on π, and reinforcement learning algorithms aim to find a policy π to maximize R(π). Given initial state distribution α(s), policy π(a|s) and transition T (s |s, a), the normalized occupancy measure ρ where P(·) is the probability measure and will be replaced by a density function if S or A is continuous. Intuitively, ρ α,π T (s, a) is a distribution of (s, a) in a length-H trajectory {s t, a t} H−1 t=0 with H ∼ Geometric(1 − γ) following the laws of (α, π, T). , the relation between ρ α,π T and (α, π, T) is characterized by the Bellman flow constraint. Specifically, x = ρ α,π T as defined in Eq. 2 is the unique solution to: In addition, Theorem 2 of gives that π(a|s) and ρ α,π T (s, a) have an one-to-one correspondence with α(s) and T (s |s, a) fixed; i.e., π(a|s) is the only policy whose occupancy measure is ρ. With the occupancy measure, the cumulative reward Eq. 1 can be represented as The goal of maximizing the cumulative reward can then be achieved by adjusting ρ α,π T, and this motivates us to adopt distribution matching approaches like WGAN to learn a transition model. In this section, we present a consistency and error bounds for WGAN. All proofs of the following theorems and lemmas can be found in Appendix A. In the setting of MBRL, the training objective for WGAN is By Kantorovich-Rubinstein duality , the optimal value of the inner maximization is exactly is the discounted distribution of (s, a, s). Thus, by minimizing over the choice of T, we are essentially finding p that minimizes W 1 (p(s, a, s)||p (s, a, s)), which gives the consistency . Proposition 1 (Consistency for WGAN). Let T and T be the true and synthesized transitions respectively. If WGAN is trained to its optimal point, we have The support constraint is inevitable because the training data is sampled from ρ T and guaranteeing anything beyond it can be difficult. Still, we will empirically show that the support constraint is not an issue in our experiments because the performance boosts up in the beginning, indicating that Supp(ρ α,π T) may be large enough initially. Now that training with WGAN gives a consistent estimate of the true transition, it is sensible to train a synthesized transition upon it. However, the consistency is too restrictive as it only discusses the optimal case. The next step is to analyze the non-optimal situation and observe how the cumulative reward deviates w.r.t. the training error. Theorem 1 indicates that if WGAN is trained properly, i.e., having small, the cumulative reward on the synthesized trajectory will be close to that on the true trajectory. As MBRL aims to train a policy on the synthesized trajectory, the accuracy of the cumulative reward over the synthesized trajectory is thus the bottleneck. Theorem 1 also implies that WGAN's error is linear to the (expected) length of the trajectory (1 − γ) −1. This is a sharp contrast to the error bounds in most RL literature, as the dependency on the trajectory length is usually quadratic , or of an even higher order. Since WGAN gives us a better estimation of the cumulative reward in the learned model, the policy update becomes more accurate. In this section, we present a practical MBRL method called model imitation (MI) that incorporates the transition learning mentioned in Section 4. Due to the long-term digression, it is hard to train the WGAN directly from a long synthesized trajectory. To tackle this issue, we use the synthesized transition T to sample N short trajectories with initial states sampled from the true trajectory. To analyze this sampling technique, let β < γ be the discount factor of the short trajectories so that the expected length is E is upper bounded by the training error of WGAN on short trajectories, which can be small empirically because the short ones are easier to imitate. and Lemma 1, where d is the dimension of (s, a). , where diam(·) is the diameter. The second term encourages β to be large while the third term does the opposite. Besides, β need not be large if N is large enough; in practice we may sample N short trajectories to reduce the error from W 1 (ρ T ||ρ T) to W 1 (ρ β T ||ρ T). Finally, since ρ β T is the occupancy measure we train on, from the proof of Theorem 1 we deduce that. Thus, WGAN may perform better under this sampling technique. To learn the real transition based on the occupancy measure matching mentioned in Section 4, we employ a transition learning scheme by aligning the distribution of (s, a, s) between the real and the learned environments. Inspired by how GAIL learns to align (s, a) via solving an MDP with rewards extracted from a discriminator, we formulate an MDP with rewards from a discriminator over (s, a, s). Specifically, the WGAN critic f (s, a, s) in Eq. 5 is used as the (psuedo) rewards r(s, a, s) of our MDP. Interestingly, there is a duality between GAIL and our transition learning: for GAIL, the transition is fixed and the objective is to train a policy to maximize the cumulative pseudo rewards, while for our transition learning, the policy is fixed and the objective is to train a synthesized transition to maximize the cumulative pseudo rewards. In practice, since the policy is updated alternatively with the synthesized model, we are required to train a number of WGANs along with the change of the policy. Although the generators across WGANs correspond to the same transition and can be similar, we observe that WGAN may get stuck at a local optimum when we switch from one WGAN training to another. The reason is that unlike GAN that mimics the Jensen-Shannon divergence and hence its inner maximization is upper bounded by log, WGAN mimics the Wasserstein distance and the inner maximization is unbounded from above. Intuitively, such unboundedness makes the WGAN critic so strong that the WGAN generator (the synthesized transition) cannot find a way out and get stuck at a local optimum. Thereby, we have to modify the WGAN objective to alleviate such a situation. To ensure the boundedness, for a fixed δ > 0, we introduce cut-offs at the WGAN objective so that the inner maximization is upper bounded by 2δ: As δ → ∞, Eq. 6 recovers the WGAN objective, Eq. 5. Therefore, this is a truncated version of WGAN. To comprehend Eq. 6 further, notice that it is equivalent to which is a hinge loss version of the generative adversarial objective. Such WGAN is introduced in , where the consistency is provided and further experiments are evaluated in. According to , the inner minimization can be interpreted as the soft-margin SVM. Consequently, it provides a geometric intuition of maximizing margin, which potentially enhances robustness. Finally, because the objective of transition learning is to maximize the cumulative pseudo rewards on the MDP, T does not directly optimize Eq. 7. Note that the truncation only takes part in the inner minimization: which gives us a WGAN critic f (s, a, s). As mentioned, f will be the pseudo reward function. Later, we will introduce a transition learning version of PPO to optimize the cumulative pseudo reward. for N epochs do for n policy epochs do 10: update π θ by TRPO on the data generated by T φ 11: end for 12: end for 13: end for After modifying the WGAN objective, to include both the stochastic and (approximately) deterministic scenarios, the synthesized transition is modeled by a Gaussian distribution T (s |s, a) = T φ (s |s, a) ∼ N (µ φ (s, a), Σ φ (s, a) ). Although the underlying transitions of tasks like MuJoCo are deterministic, modeling by a Gaussian does not harm the transition learning empirically. Recall that the synthesized transition is trained on an MDP whose reward function is the critic of the truncated WGAN. To achieve this goal with proper stability, we employ PPO , which is an efficient approximation of TRPO . Note that although the PPO is originally designed for policy optimization, it can be adapted to transition learning with a fixed sampling policy and the PPO objective (Eq. 7 of) where r t (φ) = T φ (s t+1 |s t, a t) T φold (s t+1 |s t, a t),Â t: advantage func. derived from the pseudo reward f (s t, a t, s t+1). To enhance stability of the transition learning, in addition to PPO, we also optimize maximum likelihood, which can be regarded as a regularization. We empirically observe that jointly optimizing both maximum likelihood and the PPO objective attains better transition model for policy gradient. The overall loss of the transition learning becomes where L mle is the loss of MLE, which is policy-agnostic and can be estimated with all collected real transitions. For more implementation details, please see Appendix B.1. We consider a training procedure similar to SLBO , where they consider the fact that the value function is dependent on the varying transition model. As a , unlike most of the MBRL methods that have only one pair of model-policy update for each real environment sampling, SLBO proposes to take multiple update pairs for each real environment sampling. Our proposed model imitation (MI) method is summarized in Algorithm 1. In the experiment section, we would like to answer the following questions. Does the proposed model imitation outperforms the state-of-the-art in terms of sample complexity and average return? Does the proposed model imitation benefit from distribution matching and is superior to its model-free and model-based counterparts, TRPO and SLBO? To fairly compare algorithms and enhance reproducibility, we adopt open-sourced environments released along with a model-based benchmark paper , which is based on a physical simulation engine, MuJoCo . Specifically, we evaluate the proposed algorithm MI on four continuous control tasks including Hopper, HalfCheetah, Ant, and Reacher. For hyperparameters mentioned in Algorithm 1 and coefficients such as entropy regularization λ, please refer to Appendix B.2. Table 1: Proportion of bench-marked RL methods that are inferior to MI in terms of 5% t-test. x/y indicates that among y approaches, MI is significantly better than x approaches. The detailed performance can be found in Table 1 of. It should be noted that the reported in We compare to two model-free algorithms, TRPO and PPO , to assess the benefit of utilizing the proposed model imitation since our MI (Algorithm 1) uses TRPO for policy gradient to update the agent policy. We also compare MI to four model-based methods. SLBO gives theoretical guarantees of monotonic improvement for model-based deep RL and proposes to update a joint model-policy objective. PETS propose to employ uncertainty-aware dynamic models with sampling-based uncertainty to capture both aleatoric and epistemic uncertainty. METRPO shows that insufficient data may cause instability and propose to use an ensemble of models to regularize the learning process. STEVE dynamically interpolates among model rollouts of various horizon lengths and favors those whose estimates have lower error. Figure 2 shows the learning curves for all methods. In Hopper, HalfCheetah, and Ant, MI converges fairly fast and learns a policy significantly better than competitors'. In Ant, even though MI does not improve the performance too much from the initial one, the fact that it maintains the average return at around 1,000 indicates that MI can capture a better transition than other methods do with only 5,000 transition data. Even though we do not employ an ensemble of models, the curves show that our learning does not suffer from high variance. In fact, the performance shown in Figure 2 indicates that the variance of MI is lower than that of methods incorporating ensembles such as METRPO and PETS. The questions raised at the beginning of this section can now be answered. The learned model enables TRPO to explore the world without directly access real transitions and therefore TRPO equipped with MI needs much fewer interactions with the real world to learn a good policy. Even though MI is based on the training framework proposed in SLBO, the additional distribution matching component allows the synthesized model to generate similar rollouts to that of the real environments, which empirically gives superior performance because we rely on long rollouts to estimate policy gradient. To better understand the performance presented in Figure 2, we further compare MI with benchmarked RL algorithms recorded in including state-of-the-art MFRL methods such as TD3 and SAC . It should be noted that the reported of are the final performance after 200k time-steps but we only use up to 100k time-steps to train MI. Table 1 indicates that MI significantly outperforms most of the MBRL and MFRL methods with 50% fewer samples, which verifies that MI is more sample-efficient by incorporating distribution matching. We have pointed out that the state-of-the-art methods concentrate on learning synthesized models in a supervised fashion, which does not guarantee that the policy is able to reproduce a similar trajectory in the learned model and therefore the model may not be accurate enough to estimate long rollouts. We have proposed to incorporate WGAN to achieve occupancy measure matching between the real transition and the synthesized model and theoretically shown that matching indicates the closeness in cumulative rewards between the synthesized model and the real environment. To enable stable training across WGANs, we have suggested using a truncated version of WGAN to prevent training from getting stuck at local optimums. The empirical property of WGAN application such as imitation learning indicates its potential to learn the transition with fewer samples than supervised learning. We have confirmed it experimentally by further showing that MI converges much faster and obtains better policy than state-of-the-art model-based and model-free algorithms. A.1 PROOF FOR WGAN Proposition 1 (Consistency for WGAN). Let α(s), π(a|s), T (s |s, a) be initial state distribution, policy and synthesized transition. Let T be the true transition, p(s, a, s) = ρ α,π T (s, a)T (s |s, a) be the discounted distribution of the triple (s, a, s) under the true transition. If the WGAN is trained to its optimal point, we have Proof. Because the loss function of WGAN is the 1-Wasserstein distance, we know p(s, a, s) = p (s, a, s) at its optimal points. Plug in to the Bellman flow constraint Eq., where the first inequality holds because r(s, a)/L r is 1-Lipschitz and the last equality follows from Kantorovich-Rubinstein duality. Since W 1 distance is symmetric, the same holds if we interchange T and T, so we arrive at |R(π, T) − R(π, T)| ≤ L r /(1 − γ). To enhance state exploration, we sample real transitions according to policy β ∼ N (µ θ (s), σ), where µ(s) is the mean of our Gaussian parameterized policy π θ and σ is a fixed standard deviation. In addition, since model the transition as a Gaussian distribution, we found that matching ρ α,π θ T with ρ α,β T is empirically more stable and more sample-efficient than matching ρ α,β T with ρ α,β T. For policy update, it is shown that using the mean µ φ of the Gaussian-parameterized transition can accelerate policy optimization and better balance exploration and exploitation. In order to enforce the Lipschitz constraint to the WGAN critic f, we employ gradient penalty Table 2: List of hyper-parameters adopted in our experiments.' and it should match with 'Our method incorporates WGAN to achieve occupancy measure matching for transition learning.' but not with 'We identify and formalize the memorization problem in meta-learning and solve this problem with novel meta-regularization method, which greatly expand the domain that meta-learning can be  applicable to and effective on.'.

Question: The Deep Image Prior is a fascinating recent approach for recovering images which appear natural, yet is not fully understood. This work aims at shedding some further light on this approach by investigating the properties of the early outputs of the DIP. First, we show that these early iterations demonstrate invariance to adversarial perturbations by classifying progressive DIP outputs and using a novel saliency map approach. Next we explore using DIP as a defence against adversaries, showing good potential. Finally, we examine the adversarial invariancy of the early DIP outputs, and hypothesize that these outputs may remove non-robust image features. By comparing classification confidence values we show some evidence confirming this hypothesis. 1 surprisingly showed that just the structure of a convolutional neural network is capable of capturing a good portion of images' statistics. They demonstrated that starting with an untrained network, and then training to guide the output towards a specific target image for image restoration tasks such as denoising, super-resolution, and in-painting achieved performance which is comparable to state-of-the-art approaches. Their approach, termed the Deep Image Prior (DIP), shows that the architecture of a network can act as a powerful prior. The same network has been found to have excellent performance as a natural image prior . The ability to detect natural images poses great significance in recent years, especially with the increasing security concerns raised over natural-looking images that are not correctly classified, called adversarial examples . These adversarial examples can be thought of as incremental, non-natural perturbations. As such, using the Deep Image Prior as a recovery method can indicate its ability to work as a natural denoiser, a hypothesis that will initially be tested. Furthermore, we use the Deep Image Prior to develop an adversarial defence, thereby investigating its potential. Then, we investigate the early iterations of the network by producing saliency maps of the Deep Image Prior outputs (DIP outputs). Saliency maps show the pixels which are most salient (relevant) in reaching a target classification. We hope to show that the salient pixels gather to display more clear, distinct, and robust features of the images. showed that adversarial examples are a of the existence of nonrobust features in the images, which are highly predictive, yet incomprehensible to humans. The successful performance of the Deep Image Prior in recovering the original classes from adversarial examples , raises the argument that the Deep Image Prior produces images that have'dropped' their non-robust features and are left with the robust image features. To test this theory we directly use the dataset from consisting of robust and non-robust image features, and passing these through the Deep Image Prior. By comparing the DIP outputs of the robust and non-robust image datasets, we hope to see evidence towards the ability of the Deep Image Prior to select robust images. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. As a prerequisite we first show the ability of the Deep Image Prior to recover from adversarial perturbations. For this investigation, three methods for generating adversarial examples will be considered: the Fast Gradient-Sign Method (FGSM) , the Basic Iterative method (BI), and the Least-Likely Class Iterative method (LLCI) . Adversarial examples were generated for 20 images using the three methods for various adversarial strengths. The DIP output was collected and classified every 100 iterations and the values of the true class confidence were obtained. The classifier used was ResNet18 (a). The are shown in Figure 1. More accurate classifiers were also briefly considered, such as Inception-v3, but no qualitative differences were found. It is apparent that the DIP outputs in the earlier iterations allow the classifier to recover the true class, as evident from the peaks in the confidence values. Then, the network output converges back to the adversary observed through the decrease in confidence values. The number of iterations at which peak confidence occurs appears to be different among adversaries and also among adversarial strengths. Nevertheless the theme is similar; exhibiting peaks at the earlier DIP iterations, showing the ability of the network to recover from adversarial perturbations. We have shown that the DIP can be an interesting and effective tool for recovering from adversarial perturbations. However, details about the iterative process and the transformation of the input into the various DIP outputs are still unknown. To test the nature of these outputs we introduce a novel saliency map approach, termed MIG-SG. This method is a variation of the integrated gradients approach while using SmoothGrad . More information about this method can be found in the Appendix. Figure 2 shows, on a step-by-step basis, how the salient features of the image change for progressive DIP outputs. While the image is very blurry after 300-400 iterations, the saliency map already shows that all the key features of the knife have already been identified. This is confirmed by the confidence of the DIP output, which increased to > 0:5 after just 200 iterations. On the contrary, observing the outputs at 2000 and 4000 iterations shows that salient features have become more focused on the blade of the knife. Previously, the salient features focused on the handle and the butt of the blade as observed from the bottom row of images in Figure 2. Furthermore, it is no longer clear what the salient features represent, a fact also illustrated in the decreasing value of the true class confidence. Overall, the salient maps "lost" various salient features as the DIP output was converging towards the adversarial example. Overall, with the clearest saliency maps observed at low iteration numbers, we observe evidence that the early DIP outputs are invariant towards adversarial effects. To mount a defence using the Deep Image Prior we aim to transform the input of the classifier to a state where the adversarial perturbations are not detectable. The classifier used for this investigation was ResNet18 (a). By using a simple classifier to make this defence, we are able to evaluate the potential of the Deep Image Prior to recover the original class from adversarial perturbations individually. Our are compared against the defence from that uses randomisation to defend against adversarial perturbations, and which also uses a similar evaluation method. Understandably, using the Deep Image Prior decreases the accuracy of the network on clean images. From Table 1, there is a noticeable decrease in top-1 accuracy, especially when using fewer DIP iterations. As the number of iterations is increased, the top-1 accuracy increases with it, at a loss of computational speed. Since the computational costs are already very high, the defence was not tested for larger iteration numbers, as that would make it slow and impractical. The of using the Deep Image Prior on adversarial examples are shown in Table 2 and display a very competitive performance with the reference defence method, having a higher accuracy across all three adversaries used in that comparison. The average accuracy is highest after 1000 iterations, however, this is not best for all the adversaries as observed from the FGSM adversary with = 10. Overall, we see a decreased ability to correctly classify images, combined with an increased ability to defend against adversaries. This , is similar to the one described by , where the classifier trained on the robust image dataset, highlighting the ability of the Deep Image Prior to select these robust features. 5 Using the robust image dataset For this test, We used the pre-generated robust and non-robust image datasets from. The architecture used for the Deep Image Prior had to be altered since CIFAR-10 images were used. Details can be found in the Appendix. The outputs were evaluated through the classification confidence of the original class of the image. Both figures in 3 show the difference between the classification of robust and non-robust datasets to an external classifier, where robust images hold more information about the true class compared to the non-robust images. Regarding the response at the earlier iteration numbers, it is very subtle, yet we see some evidence to support our hypothesis. The non-robust image datasets show a trough before converging to their final classification confidence, while the robust image datasets shows a peak in confidence, indicating that the the convergence of the network towards the robust images was faster than the convergence on the non-robust ones. 6 Discussion on architecture & showed that the DIP achieved remarkable , apparently due to the prior encoded in the network architecture. To test this, we evaluated the sensitivity of DIP to changes in network architecture. Surprisingly, we found that while some sensitivity exists, it is not high, with various architecture changes showing little to no changes in performance. However, some changes showed great influence on the performance of the network. In particular, the network was found to fail when no skip connections were used, or when a very shallow network was used. Nevertheless, no evidence was found that can describe this sensitivity as a "resonance", as stated in the original paper. See Appendix for details. We observed the network's ability to successfully recover from adversarial perturbations, caused by the resistance of the early DIP outputs to adversarial perturbations. This was further observed from looking at appropriate saliency maps, where we introduced a new method. Consequently, the network was found to create a promising adversarial defence. Lastly, we provided evidence for the ability of the Deep Image Prior to select robust image features over non-robust features in its early iterations, as defined by. As the name suggests, this method performs integration between a baseline and our image, numerically, by calculating the forward derivative of the network. SmoothGrad calculates this forward derivative, by performing the differentiation on a noisy version of the input, and averaging the derivative over multiple samples . As a , combining the two methods appears to yield significantly improved saliency maps. Since we are performing integration, solely taking the absolute value of the of the grad function, failed to produce . However, a small modification was made to the algorithm in an attempt to stop the method from failing. By also taking the absolute value of the final , the method produced very promising . Using the absolute values of the gradients for coloured images, enables negative gradients to also be contribute to the salient features of the image. Mathematically our saliency method can be expressed as: where x is the input image, x0 is the baseline image and is only used for comparative purposes . Additionally, m is the number of integration steps and N is the number of samples used in the computation of SmoothGrad . Lastly, Ft(x) returns the classification of class t before the cost function is calculated. Common saliency maps have been generated for a panda image, shown in Figure 4. The MIG-SG saliency map is observed in Figure 4e and while it can definitely appear as a scary image, it provides very interesting information about the panda. This saliency map, instead of picking up all the panda in the image, has instead focused on its characteristic features, the eyes and the mouth. This makes it a very useful tool to visually interpret images. For the Deep Image Prior, the original architecture was used . The number of iterations was left at a low value, as the of this work suggested that the DIP output is less sensitive to adversarial perturbations at earlier iterations. Three iteration numbers were tested: 500, 750 and 1000. The tests were conducted on a dataset of images from 200 randomly selected classes from the ImageNet database. From this dataset, 500 images correctly classified using the ResNet18 classifier were then chosen to test the performance of our defence. The diagram of the defence is shown in Figure 5. Two architectures were considered, the first is the one used in the original paper of the Deep Image Prior but with the encoder depth changed from 5 to 3, to allow for the decreased dimensionality of the CIFAR-10 images. The second architecture, uses only 16 feature maps in each layer, compared with the original number which was 128, while also the encoder depth was kept at 3. Architecture-1 and architecture-2 can be found in Tables 3 and 4 respectively.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We investigate properties of the recently introduced Deep Image Prior (Ulyanov et al, 2017)

Represent the short summary for finding the corresponding scientific article (long)

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.