Question: Represent the article for finding a single-sentence TLDR

Question: We study the problem of designing provably optimal adversarial noise algorithms that induce misclassification in settings where a learner aggregates decisions from multiple classifiers. Given the demonstrated vulnerability of state-of-the-art models to adversarial examples, recent efforts within the field of robust machine learning have focused on the use of ensemble classifiers as a way of boosting the robustness of individual models. In this paper, we design provably optimal attacks against a set of classifiers. We demonstrate how this problem can be framed as finding strategies at equilibrium in a two player, zero sum game between a learner and an adversary and consequently illustrate the need for randomization in adversarial attacks. The main technical challenge we consider is the design of best response oracles that can be implemented in a Multiplicative Weight Updates framework to find equilibrium strategies in the zero-sum game. We develop a series of scalable noise generation algorithms for deep neural networks, and show that it outperforms state-of-the-art attacks on various image classification tasks. Although there are generally no guarantees for deep learning, we show this is a well-principled approach in that it is provably optimal for linear classifiers. The main insight is a geometric characterization of the decision space that reduces the problem of designing best response oracles to minimizing a quadratic function over a set of convex polytopes. In this paper, we study adversarial attacks that induce misclassification when a learner has access to multiple classifiers. One of the most pressing concerns within the field of AI has been the welldemonstrated sensitivity of machine learning algorithms to noise and their general instability. Seminal work by has shown that adversarial attacks that produce small perturbations can cause data points to be misclassified by state-of-the-art models, including neural networks. In order to evaluate classifiers' robustness and improve their training, adversarial attacks have become a central focus in machine learning and security BID21 BID17 BID23.Adversarial attacks induce misclassification by perturbing data points past the decision boundary of a particular class. In the case of binary linear classifiers, for example, the optimal perturbation is to push points in the direction perpendicular to the separating hyperplane. For non-linear models there is no general characterization of an optimal perturbation, though attacks designed for linear classifiers tend to generalize well to deep neural networks BID21.Since a learner may aggregate decisions using multiple classifiers, a recent line of work has focused on designing attacks on an ensemble of different classifiers BID31 BID0 BID13. In particular, this line of work shows that an entire set of state-of-the-art classifiers can be fooled by using an adversarial attack on an ensemble classifier that averages the decisions of the classifiers in that set. Given that attacking an entire set of classifiers is possible, the natural question is then:What is the most effective approach to design attacks on a set of multiple classifiers?The main challenge when considering attacks on multiple classifiers is that fooling a single model, or even the ensemble classifier (i.e. the model that classifies a data point by averaging individual predictions), provides no guarantees that the learner will fail to classify correctly. Models may have different decision boundaries, and perturbations that affect one may be ineffective on another. Furthermore, a learner can randomize over classifiers and avoid deterministic attacks (see Figure 1). c 2 c 1 Figure 1: Illustration of why randomization is necessary to compute optimal adversarial attacks. In this example using binary linear classifiers, there is a single point that is initially classified correctly by two classifiers c1, c2, and a fixed noise budget α in the ℓ2 norm. A naive adversary who chooses a noise perturbation deterministically will always fail to trick the learner since she can always select the remaining classifier. An optimal adversarial attack in this scenario consists of randomizing with equal probability amongst both noise vectors. In this paper, we present a principled approach for attacking a set of classifiers which proves to be highly effective. We show that constructing optimal adversarial attacks against multiple classifiers is equivalent to finding strategies at equilibrium in a zero sum game between a learner and an adversary. It is well known that strategies at equilibrium in a zero sum game can be obtained by applying the celebrated Multiplicative Weights Update framework, given an oracle that computes a best response to a randomized strategy. The main technical challenge we address pertains to the characterization and implementation of such oracles. Our main contributions can be summarized as follows:• We describe the Noise Synthesis FrameWork (henceforth NSFW) for generating adversarial attacks. This framework reduces the problem of designing optimal adversarial attacks for a general set of classifiers to constructing a best response oracle in a two player, zero sum game between a learner and an adversary; • We show that NSFW is an effective approach for designing adversarial noise that fools neural networks. In particular, applying projected gradient descent on an appropriately chosen loss function as a proxy for a best response oracle achieves performance that significantly improves upon current state-of-the-art attacks (see in Figure 2); • We show that applying projected gradient descent on an appropriately chosen loss function is a well-principled approach. We do so by proving that for linear classifiers such an approach yields an optimal adversarial attack if the equivalent game has a pure Nash equilibrium. This is shown via a geometric characterization of the decision boundary space which reduces the problem of designing optimal attacks to a convex program; • If the game does not have a pure Nash equilibrium, there is an algorithm for finding an optimal adversarial attack for linear classifiers whose runtime is exponential in the number of classifiers. We show that finding an optimal strategy in this case is NP-hard. Paper organization. Following a discussion on related work, in Section 2 we formulate the problem of designing optimal adversarial noise and show how it can be modeled as finding strategies at equilibrium in a two player, zero sum game. Afterwards, we discuss our approach for finding such strategies using MWU and proxies for best response oracles. In Section 2.1, we justify our approach by proving guarantees for linear classifiers. Lastly, in Section 3, we present our experiments. Additional related work. The field of adversarial attacks on machine learning classifiers has recently received widespread attention from a variety of perspectives BID1 BID9 BID25 BID3. In particular, a significant amount of effort has been devoted to computing adversarial examples that induce misclassification across multiple models BID22 BID21. There has been compelling evidence which empirically demonstrates the effectiveness of ensembles as way of both generating and defending against adversarial attacks. For example, BID31 establish the strengths of ensemble training as a defense against adversarial attacks. Conversely, provide the first set of experiments showing that attacking an ensemble classifier is an effective way of generating adversarial examples that transfer to the underlying models. Relative to their investigation, our work differs in certain key aspects. Rather than analyzing adversarial noise from a security perspective and developing methods for black-box attacks, we approach the problem from a theoretical point of view and introduce a formal characterization of the optimal attack against a set of classifiers. Furthermore, by analyzing noise in the linear setting, we design algorithms for this task that have strong guarantees of performance. Through our experiments, we demonstrate how these algorithms motivate a natural extension for noise in deep learning that achieves state-of-the-art . Given a set of point-label pairs {( DISPLAYFORM0 where DISPLAYFORM1, a deterministic adversarial attack is a totally ordered set of noise vectors, V = (v 1, . . ., v m) ∈ R d×m. We say that q is an adversarial attack if q is a distribution over sets of noise vectors. An adversarial attack q is α-bounded if for all sets V that have non-zero probability under q, each individual noise vector v i ∈ V has bounded norm, e.g ||v i || p ≤ α. We focus on the case where each vector v i is bounded to have ℓ 2 norm less than a fixed value α, however, our model can be easily extended to a variety of norms. For a given classifier c: DISPLAYFORM0, a realization of the adversarial attack, V = (v 1, . . ., v m), induces misclassification on (x j, y j) if c(x j + v j) ∕ = y j. Given a finite set of classifiers C and a data set S = {(x i, y i)} m i=1 of point-label pairs as above, an optimal adversarial attack is a distribution q over sets of noise vectors that maximizes the minimum 0-1 loss of the classifiers in C: DISPLAYFORM1 Optimal adversarial attacks are equilibrium strategies in a zero sum game. An equivalent interpretation of the optimization problem described in Equation FORMULA3 is that of a best response in a two player, zero sum game played between a learner and an adversary. When the learner plays classifier c ∈ C and the adversary plays an attack V, the payoff to the adversary is M (c, DISPLAYFORM2, which is the average 0-1 loss of the learner.2 The learner and the adversary can choose to play randomized strategies p, q over classifiers and noise vectors yielding expected payout E(c,V)∼(p,q) M (c, V). The (mixed) equilibrium strategy of the game is the pair of distributions p, q that maximize the minimum loss DISPLAYFORM3 Computing optimal adversarial attacks via MWU. As discussed above, the optimization problem of designing optimal adversarial attacks reduces to that of finding strategies at equilibrium in a zero sum game. It is well known that the celebrated Multiplicative Weight Updates algorithm can be used to efficiently compute equilibrium strategies of zero sum games when equipped with a best response oracle that finds an optimal set of perturbations for any strategy chosen by the learner: DISPLAYFORM4 Our framework for generating adversarial noise applies the Multiplicative Weight Updates algorithm as specified in Algorithm 1. The algorithm returns distributions p, q that are within δ of the equilibrium value of the game λ = min DISPLAYFORM5 ln n δ 2 ) calls to a best response oracle.3 In this work, we focus on developing attacks on neural networks and linear models. Yet, our framework is general enough to generate optimal attacks for any domain in which one can approximate a best response. We analyze the convergence of NSFW in Appendix G.Approximating a best response. Given the framework described above, the main challenge is in computing a best response strategy. To do so, at every iteration, as a proxy for a best response, we apply projected gradient descent (PGD) to an appropriately chosen surrogate loss function. In particular, given DISPLAYFORM6 we aim to solve: DISPLAYFORM7 ℓ is a loss function that depends on the type of attack (targeted vs. untargeted) and the type of classifiers in C (linear vs. deep). We introduce a series of alternatives for ℓ in the following section. As we will now show, maximizing the loss of the learner by applying PGD to a weighted sum of loss functions is a well-principled approach to computing best responses as it is guaranteed to converge to the optimal solution in the case where C is composed of linear classifiers. While there are generally no guarantees for solving non-convex optimization problems of this sort for deep neural networks, in Section 3, we demonstrate the effectiveness of our approach by showing that it experimentally improves upon current state-of-the-art attacks. Input: DISPLAYFORM0 The main theoretical insight that leads to provable guarantees for generating adversarial noise is a geometric characterization of the underlying structure of adversarial attacks. Regardless of the type of model, selecting a distribution over classifiers partitions the input space into disjoint regions, each of which is associated with a single loss value for the learner. Given a distribution over classifiers played by the learner, computing a best response strategy for the adversary then reduces to a search problem. In this problem, the search is for points in each region that lie within the noise budget and can be misclassified. The best response is to select the region which induces the maximal loss. In the case of linear classifiers, the key observation is that the regions are convex. As a , designing optimal adversarial attacks reduces to solving a series of quadratic programs. Lemma 1. Selecting a distribution p over a set C of n linear classifiers, partitions the input space R d into k n disjoint, convex sets T j such that: DISPLAYFORM0 2. There exists a finite set of numbers a 1,... a k n, not necessarily all unique, such that DISPLAYFORM1 Proof Sketch (see full proof in Appendix C). Each set T j is defined according to the predictions of the classifiers c i ∈ C on points x ∈ T j. In particular, each region T j is associated with a unique label vector DISPLAYFORM2 Since the prediction of each classifier is the same for all points in a particular region, the loss of the learner i∈[n] p[i]ℓ 0-1 (c i, x, y) is constant over the entire region. Convexity then follows by showing that each T j is an intersection of hyperplanes. This characterization of the underlying geometry now allows us to design best response oracles for linear classifiers via convex optimization. For our analysis, we focus on the case where C consists of "one-vs-all" classifiers. In the appendix, we show how our can be generalized to other methods for multilabel classification by reducing these other approaches to the "one-vs-all" case. Given k classes, a "one-vs-all" classifier c i consists of k linear functions c i, DISPLAYFORM3 On input x, predictions are made according to the rule c i (x) = arg max j c i,j (x). Lemma 2. For linear classifiers, implementing a best response oracle reduces to the problem of minimizing a quadratic function over a set of k n convex polytopes. Proof Sketch (see full proof in Appendix C). The main idea behind this lemma is that given a distribution over classifiers, the loss of the learner can be maximized individually for each point (x, y) ∈ S. Furthermore, by Lemma 1, the loss can assume only finitely many values, each of which is associated with a particular convex region T j of the input space. Therefore, to compute a best response, we can iterate over all regions and choose the one associated with the highest loss. To find points in each region T j, we can simply minimize the ℓ 2 norm of a perturbation v such that x + v ∈ T j, which can be framed as minimizing a quadratic function over a convex set. These give an important characterization, but it also shows that the number of polytopes is exponential in the number of classifiers. To overcome this difficulty, we demonstrate how when there exists a pure strategy Nash equilibrium (PSNE), that is a single set of noise vectors V where every vector is bounded by α and min ci∈C M (c i, V) = 1, PGD applied to the reverse hinge loss, ℓ r, is guaranteed to converge to a point that achieves this maximum for binary classifiers. More generally, given a label vector s j ∈ [k] n, PGD applied to the targeted reverse hinge loss, ℓ t, converges to a point within the noise budget that lies within the specified set T j. We define ℓ r and ℓ t as follows: DISPLAYFORM4 The proof follows standard arguments for convergence of convex and β-smooth functions. Theorem 1. Given any precision > 0 and noise budget α > 0:• For a finite set of linear binary classifiers C and a point (x, y), running PGD for T = 4α/ iterations on the objective DISPLAYFORM5 converges to a point that is within of the pure strategy Nash equilibrium f (x + v *), if such an equilibrium exists;• For a finite set of linear multilabel classifiers C, given a label vector s j ∈ [k] n and a distribution p over C, running PGD for T = 4α/ iterations on the objective DISPLAYFORM6 Proof Sketch. From the definition of the reverse hinge loss, we see that ℓ r (c i, x ′, y) = 0 if and only if ℓ 0-1 (c i, x ′, y) = 1. Similarly, the targeted loss ℓ t (c i, x ′, j) is 0 if and only if c i predicts x ′ to have label j. For linear classifiers, both of these functions are convex and β-smooth. Hence PGD converges to a global minimum, which is zero if there exists a pure equilibrium in the game. The requirement that there exist a feasible point x ′ within T j is not only sufficient, it is also necessary in order to avoid a brute force search. Designing an efficient algorithm to find the region associated with the highest loss is unlikely as the decision version of the problem is NP-hard even for binary linear classifiers. We state the theorem below and defer the proof to the appendix. Theorem 2. Given a set C of n binary, linear classifiers, a number B, a point (x, y), noise budget α, and a distribution p, finding v with ||v|| 2 ≤ α s.t. the loss of the learner is exactly B is NP-complete. As we show in the following section, this hardness does not limit our ability to compute optimal adversarial examples. Most of the problems that have been examined in the context of adversarial noise suppose that the learner has access only to a small number of classifiers (e.g less than 5) BID8 BID0 BID31 BID13. In such cases we can solve the convex program over all regions and find an optimal adversarial attack, even when a pure Nash equilibrium does not exist. We evaluate the performance of NSFW at fooling a set of classifiers by comparing against noise generated by using state-of-the-art attacks against an ensemble classifier. Recent work by and BID31, demonstrates how attacking an ensemble of a set of classifiers generates noise that improves upon all previous attempts at fooling multiple classifiers. We test our methods on deep neural networks on MNIST and ImageNet, as well as on linear classifiers where we know that NSFW is guaranteed to converge to the optimal adversarial attack. We use the insights derived from our theoretical analysis of linear models to approximate a best response oracle for this new setting. Specifically, at each iteration of NSFW we compute a best response as in Equation FORMULA9 by running PGD on a weighted sum of untargeted reverse hinge losses, ℓ ut, introduced in this domain by BID4. Given a network c i, we denote c i,j (x) to be the probability assigned by the model to input x belonging to class j (the jth output of the softmax layer of the model). DISPLAYFORM0 For MNIST, the set of classifiers C consists of 5 convolutional neural networks, each with a different architecture, that we train on the full training set of 55k images (see Appendix for details). All classifiers (models) were over 97% accurate on the MNIST test set. For ImageNet, C consists of the InceptionV3, DenseNet121, ResNet50, VGG16, and Xception models with pre-trained weights Figure 2: Visual comparison of misclassification using state-of-the-art adversarial attacks. We compare the level of noise necessary to induce similar levels of misclassification by attacking an ensemble classifier using the (from left to right) Fast Gradient Method (FGM), the Madry attack, and the Momentum Iterative Method (MIM) versus applying NSFW (rightmost column) on the same set of classifiers. To induce a maximum of 17% accuracy across all models, we only need to set α to be 300 for NSFW. For the MIM attack on the ensemble we need to set α = 2000. For FGM and the Madry attack, the noise budget must be further increased to 8000.downloaded from the Keras repository BID6 BID12 BID27 BID7 BID30 BID14. To evaluate the merits of our approach, we compare our against attacks on the ensemble composed of C as suggested by. More specifically, we create an ensemble by averaging the outputs of the softmax layers of the different networks using equal weights. We generate baseline attacks by attacking the ensemble using the Fast Gradient Method by BID11, the Projected Gradient Method by, and the Momentum Iterative Method by BID8 which we download from the Cleverhans library BID24. We select the noise budget α by comparing against the average ℓ 2 distortion reported by similar papers in the field. For MNIST, we base ourselves off the values reported by BID4 and choose a noise budget of 3.0. For ImageNet, we compare against. In their paper, they run similar untargeted experiments on ImageNet with 100 images and report a noise budget of 22 when measured as the root mean squared deviation. Converted to the ℓ 2 norm, this corresponds to α ≥ 8500.6 We found this noise budget to be excessive, yielding images comparable to those in the leftmost column in Figure 2. Therefore, we chose α = 300 (roughly 3.5% of the total distortion used in) which ensures that the perturbed images are visually indistinguishable from the originals to the human eye (see rightmost column in Figure 2 Table 1 : Accuracies of ImageNet models under different noise algorithms using a noise budget of 300.0 in the ℓ2 norm. Entry (i, j) indicates the accuracy of each model j when evaluated on noise from attack i. The last two columns report the mean and max accuracy of the classifiers on a particular attack. We see that NSFW significantly outperforms noise generated by an ensemble classifier for all choices of attack algorithms.4 Specific details regarding model architectures as well as the code for all our experiments can be found in our repository which will be made public after the review period in order to comply with anonymity guidelines. The test set accuracies of all ImageNet classifiers are displayed on the Keras website.5 Momentum Iterative Method won the 2017 NIPS adversarial attacks competition. 6 In their paper, define the root mean squared deviation of two points x, x as (xi − x i) 2 /N where N is the dimension of x. For ImageNet, our images are of dimension 224×224×3, while for MNIST they are of size 28 × 28 × 1. For further perspective, if we convert our noise budgets from the ℓ2 norm to RMSD, our budgets would correspond to.77 and.11 for ImageNet and MNIST respectively. For our experiments, we ran NSFW for 50 MWU iterations on MNIST models and for 10 iterations on ImageNet classifiers. We use far fewer iterations than the theoretical bound since we found that in practice NSFW converges to the equilibrium solution in only a small number of iterations (see Figure 5 in Appendix A). At each iteration of the MWU we approximate a best response as described in Equation 3 by running PGD using the Adam optimizer on a sum of untargeted reverse hinge losses. Specifically, we run the optimizer for 5k iterations with a learning rate of.01. At each iteration, we clip images to lie in the range Finally, for evaluation, for both MNIST and ImageNet we selected 100 images uniformly at random from the set of images in the test sets that were correctly classified by all models. In Table 1, we report the empirical accuracy of all classifiers in the set C when evaluated on NSFW as well as on the three baseline attacks. To compare their performance, we highlight the average and maximum accuracies of models in C when attacked using a particular noise solution. From Table 1, we see that on ImageNet our algorithm in solutions that robustly optimize over the entire set of models using only a small amount of noise. The maximum accuracy of any classifier is 17% under NSFW, while the best ensemble attack yields a max accuracy of only 68%. If we wish to generate a similar level of performance from the ensemble baselines, we would need to increase the noise budget to 8000 for FGM and the Madry attack and to 2000 for the Momentum Iterative Method. We present a visual comparison of the different attacks under these noise budgets required to achieve accuracy of 17% in Figure 2. On MNIST, we find similar . NSFW yields a max accuracy of 22.6% compared to the next best of 48% generated by the Madry attack on the ensemble. We summarize the for MNIST in Table 2 presented in Appendix A. As seen in the previous section, noise generated by directly attacking an ensemble of classifiers significantly underperforms NSFW at robustly fooling the underlying models. In this section, we aim to understand this phenomenon by analyzing how the decision boundary of the ensemble model compares to that of the different networks. In particular, we visualize the class boundaries of convolutional neural networks using the algorithm proposed by BID28 for generating saliency maps. 8 The class saliency map indicates which features (pixels) are most relevant in classifying an image to have a particular label.9 Therefore, they serve as one way of understanding the decision boundary of a particular model by highlighting which dimensions carry the highest weight. In FIG0, we see that the class saliency maps for individual models exhibit significant diversity. The ensemble of all 5 classifiers appears to contain information from all models, however, certain regions that are of central importance for individual models are relatively less prominent in the ensemble saliency map. Compared to our approach which calculates individual gradients for classifiers in C, creating an ensemble classifier obfuscates key information regarding the decision boundary of individual models. We make this discussion rigorous by analyzing the linear case in Appendix B. NSFW on linear multiclass models using different noise functions and varying the noise budget α. NSFWOracle corresponds to running Algorithm 1 using the best response oracle described in Lemma 2. Similarly, NSFW-Untargeted shows the of running NSFW and applying PGD to a weighted sum of untargeted losses as in Equation FORMULA9. The label iteration method is described below. Lastly, the ensemble attack corresponds to the optimal noise on an equal weights ensemble of models in C. On the right, we illustrate the convergence of NSFW on linear binary classifiers with maximally different decision boundaries to compare against the convergence rate observed for neural nets in Figure 5 and better understand when weight adaptivity is necessary. In addition to evaluating our approach on neural networks, we performed experiments with linear classifiers. Since we have a precise characterization of the optimal attack on a set of linear classifiers, we can rigorously analyze the performance of different methods in comparison to the optimum. We train two sets of 5 linear SVM classifiers on MNIST, one for binary classification (digits 4 and 9) and another for multiclass (first 4 classes, MNIST 0-3). To ensure a diversity of models, we randomly zero out up to 75% of the dimensions of the training set for each classifier. Hence, each model operates on a random subset of features. All models achieve test accuracies of above 90%. For our experiments, we select 1k points from each dataset that are correctly classified by all models. In order to better compare across different best response proxies, we further extend NSFW by incorporating the label iteration method as another heuristic to generate untargeted noise. Given a point (x, y), the iterative label method attempts to calculate a best response by running PGD on the targeted reverse hinge loss for every label j ∈ [k] \ {y} and choosing the attack associated with the minimal loss. Compared to the untargeted reverse hinge loss, it has the benefit of being convex. As for deep learning classifiers, we compare our to the noise generated by the optimal attack on an ensemble of models in C. Since the class of linear classifiers is convex, creating an equal weights ensemble by averaging the weight vectors in just another linear classifier. We can compute the optimal attack by running the best response oracle described in Section 2.1 for the special case where C consists of a single model and then scaling the noise to have norm equal to α. As seen in the leftmost plot in FIG3, even for linear models there is a significant difference between the optimal attack and other approaches. Specifically, we observe an empirical gap between NSFW equipped with the best response oracle as described in Lemma 2 vs. NSFW with proxy best response oracles, e.g. the oracle that runs PGD on appropriately chosen loss functions.10 This difference in performance is consistent across a variety of noise budgets. Our main takeaway is that in theory and in practice, there is a significant benefit in applying appropriately designed best response oracles. Lastly, on the right in FIG3, we illustrate how the adaptivity of MWU is in general necessary to compute optimal attacks. While for most cases, NSFW converges to the equilibrium solution almost immediately, if the set of classifiers is sufficiently diverse, running NSFW for a larger number of rounds drastically boosts the quality of the attack. (See Appendix A for details.) Designing adversarial attacks when a learner has access to multiple classifiers is a non-trivial problem. In this paper we introduced NSFW which is a principled approach that is provably optimal on linear classifiers and empirically effective on neural networks. The main technical crux is in designing best response oracles which we achieve through a geometrical characterization of the optimization landscape. We believe NSFW can generalize to domains beyond those in this paper. A ADDITIONAL EXPERIMENTS AND DETAILS ON EXPERIMENTAL SETUP Figure 5: Fast convergence of NSFW on MNIST and ImageNet deep learning models. NSFW-Untargeted corresponds to running NSFW and applying PGD on a sum of untargeted reverese hinge losses as described in Section 3.1. The dotted lines correspond to running the indicated attack on the ensemble of models in C. For both datasets, we find that NSFW converges almost immediately to the equilibrium noise solution. These misclassification can also be examined in Tables 1 and 2.We now discuss further details regarding the setup of the experiments presented in Section 3. In the case of deep learning, we set hyperparameters for of all the baseline attacks (Fast Gradient Method, Madry attack, and the Momentum Iterative Method) by analyzing the values reported in the original papers. When running the Projected Gradient Method by Madry et al., given a noise budget α, we run the algorithm for 40 iterations with a step size of α/40 × 1.25 so as to mimic the setup of the authors. In the case of the Momentum Iterative Method, we run the attack for 5 iterations with a decay factor µ = 1.0 and a step size of α/5 as specified in BID8. FGM has no hyperparameters other than the noise budget. For all methods, we clip solutions to lie within the desired pixel range and noise budget. When comparing different algorithms to compute best responses for linear multiclass classifiers as described in Section 3.3, we run the NSFW algorithm with α =.2k for k ∈. In the case of binary classifiers FIG4 ), we find that the margins are smaller, and hence run NSFW with α =.05 +.1k for k ∈. For each value of α and choice of noise function, we run NSFW for 50 iterations. The one exception is that, for the multiclass experiments with α equal to.2 or.4, we ran the best response oracle for only 20 iterations due to computational constraints. When optimizing the loss of the learner through gradient descent (e.g when using PGD on appropriately chosen loses), we set the number of iterations to 3k and the learning rate to.01.We set up the weight adaptivity experiment described at the end of Section 3.3 (rightmost plot of FIG3) as follows. We train 5 linear binary SVM classifiers on our binary version of the MNIST dataset. For each classifier, we zero out 80% of the input dimensions so that each model has nonzero weights for a strictly different subset of features, thereby ensuring maximum diversity in the decision Table 2: Classification accuracies for deep learning MNIST models under different noise algorithms. As in the ImageNet case, we find that the NSFW algorithm improves upon the performance of state-of-the-art attacks and robustly optimizes over the entire set of classifiers. Moreover, we find that, for all attacks, there is a significant difference between the average and maximum accuracy of classifiers in C, further highlighting the need to design noise algorithms that are guaranteed to inhibit the performance of the best possible model. FIG3 NSFW equipped with the best response outperforms other approaches at generating noise for linear models. Furthermore, we see there is a performance gap between gradient based approaches and the theoretically optimal one that leverages convex programming.boundaries across models. In order to generate noise, we select 500 points uniformly at random from the test set that were correctly classified by all modes. We then run NSFW equipped with the best response oracle described in Lemma 2 for 50 iterations with a noise budget of.4. In this section, we provide a brief theoretical justification as to why methods designed to attack an ensemble constructed by averaging individual models in a set of classifiers (e.g. Attacks on ensemble classifiers, as seen in, typically consist of applying gradient based optimization to an ensemble model E(C, p) made up of classifiers C and ensemble weights p. For concreteness, consider the simplest case where C is composed of linear binary classifiers. To find adversarial examples, we run gradient descent on a loss function such as the reverse hinge loss that is 0 if and only if the perturbed example x ′ = x + v with true label y is misclassified by c i.Assuming x ′ is not yet misclassified by the ensemble, running SGD on the ensemble classifier with the reverse hinge loss function in a gradient update step of ∇ℓ r (E(C, p), x ′, y) = i p[i]w i. This is undesirable for two main reasons:• First, the ensemble obscures valuable information about the underlying objective. If x ′ is misclassified by a particular model c i but not the ensemble, c i still contributes p[i]w i to the ing gradient and biases exploration away from promising regions of the search space;• Second, fooling the ensemble does not guarantee that the noise will transfer across the underlying models. Assuming the true label y is -1 and that x ′ is correctly classified by all models, ℓ r (E(C, p), x ′, y) = 0 if and only if there exists a subset of classifiers DISPLAYFORM0 Hence, the strength of an ensemble classifier is only as good as its weakest weighted majority. Lemma 1. Selecting a distribution p over a set C of n linear classifiers, partitions the input space R d into k n disjoint, convex sets T j such that:1. For each T j, there exists a unique label vector s j ∈ [k] n such that for all x ∈ T j and c i ∈ C, c i (x) = s j,i, where s j,i is a particular label in [k].2. There exists a finite set of numbers a 1,... a k n, not necessarily all unique, such that n i=1 p[i]ℓ 0-1 (c i, x, y) = a j for a fixed y and all x ∈ T j 3. R d \ j T j is a set of measure zero. Proof. Given a label vector s j, we define each T j as the set of points x where c i (x) = s j,i for all i ∈ [n]. This establishes a bijection between the elements of [k] n and the sets T j. All the T j are pairwise disjoint since their corresponding label vectors in [k] n must differ in at least one index and by construction each classifier can only predict a single label for x ∈ T j.To see that these sets are convex, consider points x 1, x 2 ∈ T j and an arbitrary classifier c i ∈ C s.t. c i (x) = z for all x ∈ T j. If we let x ′ = γx 1 + (1 − γ)x 2 where γ ∈ then the following holds for all j ∈ [k] where j ∕ = z: DISPLAYFORM0 Furthermore, for each T j, there exists a number a j ∈ R ≥0 such that the expected loss of the learner DISPLAYFORM1 Since the distribution p is fixed, the loss of the learner is uniquely determined by the correctness of the predictions of all the individual classifiers c i. Since these are the same for all points in T j, the loss of the learner remains constant. Lastly, the set R d \ i T i is equal to the set of points x where there are ties for the maximum valued classifier. This set is a subset of the set of points K that lie at the intersection of two hyperplanes: DISPLAYFORM2 Finally, we argue that K has measure zero. For all ε > 0, x ∈ K, there exists an x ′ such that ||x − x ′ || 2 < ε and x ′ / ∈ K since the intersection of two distinct hyperplanes is of dimension two less than the overall space. Therefore, R d \ i T i must also have measure zero. Lemma 2. For linear classifiers, implementing a best response oracle reduces to the problem of minimizing a quadratic function over a set of k n convex polytopes. Proof. We outline the proof as follows. Given a distribution p over C, the loss of the learner DISPLAYFORM3 can be optimized individually for each v t since the terms in the sum are independent from one another. We leverage our from Lemma 1 to demonstrate how we can frame the problem of finding the minimum perturbation v j such that x + v j ∈ T j as the minimization of a convex function over a convex set. Since the loss of the learner is constant for points that lie in a particular set T j, we can find the optimal solution by iterating over all sets T j and selecting the perturbation with ℓ 2 norm less than α that is associated with the highest loss. The best response oracle then follows by repeating the same process for each point (x, y).Given a point (x, y) solving for the minimal perturbation v such that x + v ∈ T j can be expressed as the minimization of a quadratic function subject to n(k − 1) linear inequalities. DISPLAYFORM4 Each constraint in can be expressed as k − 1 linear inequalities. For a particular z ∈ [k], c i ∈ C we write c i (x + v) = z as c i,z (x + v) > c i,l (x + v) for all l ∕ = z. Lastly, squaring the norm of the vector is a monotonic transformation and hence does not alter the underlying minimum. Here we extend the from our analysis of linear classifiers to other methods for multilabel classification. In particular, we show that any "all-pairs" or multivector model can be converted to an equivalent "one-vs-all" classifier and hence all of our also apply to these other approaches. All-Pairs. In the "all-pairs" approach, each linear classifier c consists of k 2 linear predictors c i,j trained to predict between labels i, j ∈ [k]. As per convention, we let c i,j (x) = −c j,i (x). Labels are chosen according to the rule: DISPLAYFORM0 Given an "all-pairs" classifier c, we show how it can be transformed into a "one-vs-all" classifier c DISPLAYFORM1 Multivector. Lastly, we extend our to multilabel classification done via class-sensitive feature mappings and the multivector construction by again reducing to the "one-vs-all" case. Given a function Ψ: DISPLAYFORM2 n, labels are predicted according to the rule: DISPLAYFORM3 While there are several choices for the Ψ, we focus on the most common, the multivector construction:Ψ(x, y) = 0,..., 0 DISPLAYFORM4, 0,..., 0 DISPLAYFORM5 DISPLAYFORM6 This in effect ensures that becomes equivalent to that of the "one-vs-all" approach: DISPLAYFORM7 E CONVERGENCE ANALYSIS OF PROJECTED GRADIENT DESCENT Theorem 1. Given any precision > 0 and noise budget α > 0:• For a finite set of linear binary classifiers C and a point (x, y), running PGD for T = 4α/ iterations on the objective f (v) = n i=i p[i]ℓ r (c i, x + v, y) converges to a point that is within of the pure strategy Nash equilibrium f (x + v *), if such an equilibrium exists;• For a finite set of linear multilabel classifiers C, given a label vector s j ∈ [k] n and a distribution p over C, running PGD for T = 4α/ iterations on the objective f (v) = n i=i p[i]ℓ t (c i, x+v, s j,i) converges to a point x+v (T) such that f (x+v (T) )−f (x+v *) ≤ where x + v * ∈ T j and ||v * || 2 ≤ α, if such a point exists. Proof. We know that if a function f is convex and β-smooth, then running projected gradient descent over a convex set, in the following rate of convergence, where v is the optimal solution and v is the initial starting point (See Theorem 3.7 in BID2). DISPLAYFORM8 Given n classifiers, the objective is Furthermore, since v * is a pure strategy Nash equilibrium, f (v) = 0 and the maximum difference between f (v) − f (v), for any v, is bounded by: DISPLAYFORM9 Since ||v (T) − v || 2 ≤ α, we have that: DISPLAYFORM10 Lastly, we can normalize all the w i such that ||w i || 2 = 1 without changing the predictions of the c i and arrive at our desired . For the multiclass case, we have that: Using the fact that all weight vectors w i,j can be transformed to have ℓ 2 norm equal to 1, we have that DISPLAYFORM11 DISPLAYFORM12. Lastly, we can check that ℓ t is β-smooth with β = α n i=1 p[i], which yields the same bound as in the binary case. Theorem 2. Given a set C of n binary, linear classifiers, a number B, a point (x, y), noise budget α, and a distribution p, finding v with ||v|| 2 ≤ α s.t. the loss of the learner is exactly B is NP-complete. Proof. We can certainly verify in polynomial time that a vector v induces a loss of B simply by calculating the 0-1 loss of each classifier. Therefore the problem is in NP.To show hardness, we reduce from Subset Sum. Given n numbers p 1,... p n and a target number B, 11 we determine our input space to be R n, the point x to be the origin, the label y = −1, and the noise budget α = 1. Next, we create n binary classifiers of the form c i (x) = 〈e i, x〉 where e i is the ith standard basis vector. We let p i be the probability with which the learner selects classifier c i. We claim that there is a subset that sums to B if and only if there exists a region T j ⊂ R n on which the learner achieves loss B. Given the parameters of the reduction, the loss of the learner is determined by the sum of the probability weights of classifiers c i such that c i (x + v) = +1 for points x + v ∈ T j. If we again identify sets T j with sign vectors s j ∈ {±1} n as per Lemma 2, there is a bijection between the sets T j and the power set of {p 1, . . ., p n}. A number p i is in a subset U j if the ith entry of s j is equal to +1.Lastly, we can check that there are feasible points within each set T j and hence that all subsets within the original Subset Sum instance are valid. Each T j simply corresponds to a quadrant of R n. For any ε > 0 and for any T j, there exists a v j with ℓ 2 norm less than ε such that x + v j ∈ T j. Therefore, there is a subset U j that sums to B if and only if there is a region T j in which the learner achieves loss B.11 Without loss of generality, we can assume that instances of Subset Sum only have values in the range. We can reduce from the more general case by simply normalizing inputs to lie in this range.12 We can again normalize values so that they form a valid probability distribution.

Represent the TLDR for finding the corresponding scientific article!

Paper analyzes the problem of designing adversarial attacks against multiple classifiers, introducing algorithms that are optimal for linear classifiers and which provide state-of-the-art results for deep learning.

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the article for finding a single-sentence TLDR

Question: Multiagent systems where the agents interact among themselves and with an stochastic environment can be formalized as stochastic games. We study a subclass of these games, named Markov potential games (MPGs), that appear often in economic and engineering applications when the agents share some common resource. We consider MPGs with continuous state-action variables, coupled constraints and nonconvex rewards. Previous analysis followed a variational approach that is only valid for very simple cases (convex rewards, invertible dynamics, and no coupled constraints); or considered deterministic dynamics and provided open-loop (OL) analysis, studying strategies that consist in predefined action sequences, which are not optimal for stochastic environments. We present a closed-loop (CL) analysis for MPGs and consider parametric policies that depend on the current state and where agents adapt to stochastic transitions. We provide easily verifiable, sufficient and necessary conditions for a stochastic game to be an MPG, even for complex parametric functions (e.g., deep neural networks); and show that a closed-loop Nash equilibrium (NE) can be found (or at least approximated) by solving a related optimal control problem (OCP). This is useful since solving an OCP---which is a single-objective problem---is usually much simpler than solving the original set of coupled OCPs that form the game---which is a multiobjective control problem. This is a considerable improvement over the previously standard approach for the CL analysis of MPGs, which gives no approximate solution if no NE belongs to the chosen parametric family, and which is practical only for simple parametric forms. We illustrate the theoretical contributions with an example by applying our approach to a noncooperative communications engineering game. We then solve the game with a deep reinforcement learning algorithm that learns policies that closely approximates an exact variational NE of the game. In a noncooperative stochastic dynamic game, the agents compete in a time-varying environment, which is characterized by a discrete-time dynamical system equipped with a set of states and a state-transition probability distribution. Each agent has an instantaneous reward function, which can be stochastic and depends on agents' actions and current system state. We consider that both the state and action sets are subsets of real vector spaces and subject to coupled constraints, as usually required by engineering applications. A dynamic game starts at some initial state. Then, the agents take some action and the game moves to another state and gives some reward values to the agents. This process is repeated at every time step over a (possibly) infinite time horizon. The aim of each agent is to find the policy that maximizes its expected long term return given other agents' policies. Thus, a game can be represented as a set of coupled optimal-control-problems (OCPs), which are difficult to solve in general. OCPs are usually analyzed for two cases namely open-loop (OL) or closed-loop (CL), depending on the information that is available to the agents when making their decisions. In the OL analysis, the action is a function of time, so that we find an optimal sequence of actions that will be executed in order, without feedback after any action. In the CL setting, the action is a mapping from the state, usually referred as feedback policy or simply policy, so the agent can adapt its actions based on feedback from the environment (the state transition) at every time step. For deterministic systems, both OL and CL solutions can be optimal and coincide in value. But for stochastic system, an OL strategy consisting in a precomputed sequence of actions cannot adapt to the stochastic dynamics so that it is unlikely to be optimal. Thus, CL are usually preferred over OL solutions. For dynamic games, the situation is more involved than for OCPs, see, e.g., BID1. In an OL dynamic game, agents' actions are functions of time, so that an OL equilibrium can be visualized as a set of state-action trajectories. In a CL dynamic game, agents' actions depend on the current state variable, so that, at every time step, they have to consider how their opponents would react to deviations from the equilibrium trajectory that they have followed so far, i.e., a CL equilibrium might be visualized as a set of trees of state-action trajectories. The sets of OL and CL equilibria are generally different even for deterministic dynamic games BID10 BID5.The CL analysis of dynamic games with continuous variables is challenging and has only be addressed for simple cases. The situation is even more complicated when we consider coupled constraints, since each agent's actions must belong to a set that depends on the other agents' actions. These games, where the agents interact strategically not only with their rewards but also at the level of the feasible sets, are known as generalized Nash equilibrium problems BID3.There is a class of games, named Markov potential games (MPGs), for which the OL analysis shows that NE can be found by solving a single OCP; see BID6 BID25 for recent surveys on MPGs. Thus, the benefit of MPGs is that solving a single OCP is generally simpler than solving a set of coupled OCPs. MPGs appear often in economics and engineering applications, where multiple agents share a common resource (a raw material, a communication link, a transportation link, an electrical transmission line) or limitations (a common limit on the total pollution in some area). Nevertheless, to our knowledge, none previous study has provided a practical method for finding CL Nash equilibrium (CL-NE) for continuous MPGs. Indeed, to our knowledge, no previous work has proposed a practical method for finding or approximating CL-NE for any class of Markov games with continuous variables and coupled constraints. State-of-the-art works on learning CL-NE for general-sum Markov games did not consider coupled constraints and assumed finite state-action sets BID18 BID16.In this work, we extend previous OL analysis due to BID26 BID23 and tackle the CL analysis of MPGs with coupled constraints. We assume that the agents' policies lie in a parametric set. This assumption makes derivations simpler, allowing us to prove that, under some potentiality conditions on the reward functions, a game is an MPG. We also show that, similar to the OL case, the Nash equilibrium (NE) for the approximate game can be found as an optimal policy of a related OCP. This is a practical approach for finding or at least approximating NE, since if the parametric family is expressive enough to represent the complexities of the problem under study, we can expect that the parametric solution will approximate an equilibrium of the original MPG well (under mild continuity assumptions, small deviations in the parametric policies should translate to small perturbations in the value functions). We remark that this parametric policy assumption has been widely used for learning the solution of single-agent OCPs with continuous state-action sets; see, e.g., BID9; BID17 BID24 BID20. Here, we show that the same idea can be extended to MPGs in a principled manner. Moreover, once we have formulated the related OCP, we can apply reinforcement learning techniques to find an optimal solution. Some recent works have applied deep reinforcement learning (DRL) to cooperative Markov games BID4 BID22, which are a particular case of MPGs. Our show that similar approaches can be used for more general MPGs. We provide sufficient and necessary conditions on the agents' reward function for a stochastic game to be an MPG. Then, we show that a closed-loop Nash equilibrium can be found (or at least approximated) by solving a related optimal control problem (OCP) that is similar to the MPG but with a single-objective reward function. We provide two ways to obtain the reward function of this OCP: i) computing the line integral of a vector field composed of the partial derivatives of the agents' reward, which is theoretically appealing since it has the form of a potential function but difficult to obtain for complex parametric policies; ii) and as a separable term in the agents' reward function, which can be obtained easily by inspection for any arbitrary parametric policy. We illustrate the proposed approach by applying DRL to a noncoooperative Markov game that models a communications engineering application (in addition, we illustrate the differences with the previous standard approach by solving a classic resource sharing game analytically in the appendix). Let N {1, . . ., N} denote the set of agents. Let a k,i be the real vector of length A k that represents the action taken by agent k ∈ N at time i, where A k ⊆ R A k is the set of actions of agent k ∈ N. Let A k∈N A k denote the set of actions of all agents that is the Cartesian product of every agent's action space, such that A ⊆ R A, where A = k∈N A k. The vector that contains the actions of all agents at time i is denoted a i ∈ A. Let X ⊆ R S denote the set of states of the game, such that x i is a real vector of length S that represents the state of the game at time i, with components x i (s): DISPLAYFORM0Note that the dimensionality of the state set can be different from the number of agents (i.e., S = N). State transitions are determined by a probability distribution over the future state, conditioned on the current state-action pair: DISPLAYFORM1 where we use boldface notation for denoting random variables. State transitions can be equivalently expressed as a function, f: X × A × Θ → X, that depends on some random variable θ i ∈ Θ, with distribution p θ (·|x i, a i), such that DISPLAYFORM2 We include a vector of C constraint functions, DISPLAYFORM3, where g c: X × A → R; and define the constraint sets for i = 0: C 0 A ∩ {a 0 : g(x 0, a 0) ≤ 0}; and for i = 0,..., ∞: C i {X ∩ {x i : DISPLAYFORM4, which determine the feasible states and actions. The instantaneous reward of each agent, r k,i, is also a random variable conditioned on the current state-action pair: DISPLAYFORM5 We assume that θ i and σ k,i are independent of each other and of any other θ j and σ k,j, at every time step j = i, given x i and a i .Let π k : X → A k and π : X → A denote the policy for agent k and all agents, respectively, such that: DISPLAYFORM6 Let Ω k and Ω = k∈N Ω k denote the policy spaces for agent k and for all agents, respectively, such that π k ∈ Ω k and π ∈ Ω. Note that Ω(X) = A. Introduce also π −k: X → A −k as the policy of all agents except that of agent k. Then, by slightly abusing notation, we write: DISPLAYFORM7 The general (i.e., nonparametric) stochastic game with Markov dynamics consists in a multiobjective variational problem with design space Ω and objective space R N, where each agent aims to find a stationary policy that maximizes its expected discounted cumulative reward, for which the vector of constraints, g, is satisfied almost surely: DISPLAYFORM8 Similar to static games, since there might not exist a policy that maximizes every agent's objective, we will rely on Nash equilibrium (NE) as solution concept. But rather than trying to find a variational NE solution for, we propose a more tractable approximate game by constraining the policies to belong to some finite-dimensional parametric family. Introduce the set of parametric policies, Ω w, as a finite-dimensional function space with parameter w ∈ W ⊆ R W: Ω w {π(·, w): w ∈ W}. Note that for a given w, the parametric policy is still a mapping from states to actions: π(·, w): X → A. Let w k ∈ W k ⊆ R W k denote the parameter vector of length W k for the parametrized policy π k, so that it lies in the finite-dimensional space Ω w k DISPLAYFORM9 Let w −k denote the parameters of all agents except that of agent k, so that we can also write: DISPLAYFORM10 In addition, we use w k to denote the -th component of DISPLAYFORM11. By constraining the policy of G 1 to lie in Ω w k, we obtain a multiobjective optimization problem with design space W: DISPLAYFORM12 The solution concept in which we are interested is the parametric closed-loop Nash equilibrium (PCL-NE), which consists in a parametric policy for which no agent has incentive to deviate unilaterally. DISPLAYFORM13 Since G 2 is similar to G 1 but with an extra constraint on the policy set, loosely speaking, we can see a PCL-NE as a projection of some NE of G 1 onto the manifold spanned by parametric family of choice. Hence, if the parametric family has arbitrary expressive capacity (e.g., a neural network with enough neurons in the hidden layers), we can expect that the ing PCL-NE evaluated on G 1 will approximate arbitrarily close the performance of an exact variational equilibrium. We consider the following general assumptions. The state and parameter sets, X and W, are nonempty and convex. Assumption 2 The reward functions r k are twice continuously differentiable in X × W, ∀k ∈ N. The state-transition function, f, and constraints, g, are continuously differentiable in X × W, and satisfy some regularity conditions (e.g., Mangasarian-Fromovitz).Assumption 4 The reward functions r k are proper, and there exists a scalar B such that the level DISPLAYFORM0 are nonempty and bounded ∀k ∈ N.Assumptions 1-2 usually hold in engineering applications. Assumption 3 ensures the existence of feasible dual variables, which is required for establishing the optimality conditions. Assumption 4 will allow us to ensure the existence of PCL-NE. We say that r k is proper if: DISPLAYFORM1 In this section, we review the standard approach for tackling CL dynamic games (González-Sánchez and Hernández-). For simplicity, we consider deterministic game and no constraints: DISPLAYFORM0 First, it inverts f to express the policy in reduced form, i.e., as a function of current and future states: DISPLAYFORM1 This implicitly assumes that such function h: X × X → A exists, which might not be the case if f is not invertible. Next, π k is replaced with in each r k: DISPLAYFORM2 where r k: X×X → R is the reward in reduced-form. Then, the Euler equation (EE) and transversality condition (TC) are obtained from r k for all k ∈ N and used as necessary optimality conditions: DISPLAYFORM3 When r k are concave for all agents, and X ⊆ R + (i.e., X = {x i : DISPLAYFORM4, these optimality conditions become sufficient for Nash equilibrium (González-Sánchez and Hernández-, Theorem 4.1). Thus, the standard approach consists in guessing parametric policies from the space of functions Ω, and check whether any of these functions satisfies the optimality conditions. We illustrate this procedure with a well known resource-sharing game named "the great fish war" due to BID11, with Example 1 in Appendix A.Although the standard approach sketched above (see also Appendix A) has been the state-of-the-art for the analysis of CL dynamic games, it has some drawbacks: i) The reduced form might not exist; ii) constraints are not handled easily and we have to rely in ad hoc arguments for ensuring feasibility; iii) finding a specific parametric form that satisfies the optimality conditions can be extremely difficult since the space of functions is too large; and iv) the rewards have to be concave for all agents in order to guarantee that any policy that satisfies the conditions is an equilibrium. In order to overcome these issues, we propose to first constrain the set of policies to some parametric family, and then derive the optimality conditions for this parametric problem; as opposed to the standard approach that first derives the optimality conditions of G 1, and then guesses a parametric form that satisfies them. Based on this insight, we will introduce MPG with parametric policies as a class of games that can be solved with standard DRL techniques by finding the solution of a related (single-objective) OCP. We explain the details in the following section. In this section, we extend the OL analysis of BID25 to the CL case. We define MPGs with CL information structure; introduce a parametric OCP; provide verifiable conditions for a parametric approximate game to be an MPG in the CL setting; show that when the game is an MPG, we can find a PCL-NE by solving the parametric OCP with a specific objective function; and provide a practical method for obtaining such objective function. First, we define MPGs with CL information structure and parametric policies as follows. Definition 2 Given a policy family π(·, w) ∈ Ω w, game is an MPG if and only if there is a function J: X × W × Σ → R, named the potential, that satisfies the following condition ∀k ∈ N: DISPLAYFORM0 Definition 2 means that there exists some potential function, J, shared by all agents, such that if some agent k changes its policy unilaterally, the change in its reward, r k, equals the change in J.The main contribution of this paper is to show that when is a MPG, we can find one PCL-NE by solving a related parametric OCP. The generic form of such parametric OCP is as follows: DISPLAYFORM1 where we replaced the multiple objectives (one per agent) with the potential J as single objective. This is convenient since solving a single objective OCP is generally much easier than solving the Markov game. However, we still have to find out how to obtain J. The following Theorem formalizes the relationship between G 2 and P 1 and shows one way to obtain J (proof in Appendix C).Theorem 1 Let Assumptions 1-4 hold. Let the reward functions satisfy the following ∀k, j ∈ N: DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 where the expected value is taken component-wise. Then, game is an MPG that has a PCL-NE equal to the solution of OCP. The potential J that is the instantaneous reward for the OCP is given by line integral: DISPLAYFORM5 where η(z) (η k (z)) S m=1 and ξ(z) (ξ k (z)) k∈N are piecewise smooth paths in X and W, respectively, with components ξ k (z) (ξ k, (z)) W k =1, such that the initial and final state-action conditions are given by (η, ξ) and (η = x i, ξ = w).From FORMULA2, we can see that J is obtained through the line integral of a vector field with components the partial derivatives of the agents' rewards (see Appendix C), and so the name potential function. Note also that Theorem 1 proves that any solution to P 1 is also a PCL-NE of G 2, but we remark that there may be more equilibria of the game that are not solutions to P 1 (see Appendix C).The usefulness of Theorem 1 is that, once we have the potential function, we can formulate and solve the related OCP for any specific parametric policy family. This is a considerable improvement over the standard approach. On one hand, if the chosen parametric policy contains the optimal solution, then we will obtain the same equilibrium as the standard approach. On the other hand, if the chosen parametric family does not have the optimal solution, the standard approach will fail, while our approach will always provide a solution that is an approximation (a projection over Ω w) of an exact variational equilibrium. Moreover, as mentioned above, we can expect that the more expressive the parametric family, the more accurate the approximation to the variational equilibrium. In Appendix B, we show how to to solve "the great fish war" game with the proposed framework, yielding the same solution as with the standard approach, with no loss of accuracy. Although expressing J as a line integral of a field is theoretically appealing, if the parametric family is involved-as it is usually the case for expressive policies like deep neural-networks-then might be difficult to evaluate. The following show how to obtain J easily by visual inspection. First, the following corollary follows trivially from FORMULA10 - FORMULA25 and shows that cooperative games, where all agents have the same reward, are MPGs, and the potential equals the reward:Corollary 1 Cooperative games, where all agents have a common reward, such that DISPLAYFORM6 are MPGs; and the potential function equals the common reward function in.Second, we address noncooperative games, and show that the potential can be found by inspection as a separable term that is common to all agents' reward functions. Interestingly, we will also show that a game is an MPG in the CL setting if and only if all agents' policies depend on disjoint subsets of components of the state vector. More formally, introduce X π k as the set of state vector components that influence the policy of agent k and introduce a new state vector, x π k, and let x π −k,i be the vector of components that do not influence the policy of agent k: DISPLAYFORM7 In addition, introduce X r k as the set of components of the state vector that influence the reward of agent k directly (not indirectly through any other agent's policy), and define the state vectors: DISPLAYFORM8 Introduce also the union of these two subsets, DISPLAYFORM9, and its corresponding vectors: DISPLAYFORM10 Then, the following theorem allows us to obtain the potential function (proof in Appendix D).Theorem 2 Let Assumptions 1-4 hold. Then, game is an MPG if and only if: i) the reward function of every agent can be expressed as the sum of a term common to all agents plus another term that depends neither on its own state-component vector, nor on its policy parameter: DISPLAYFORM11 25) and ii) the following condition on the non-common term holds: DISPLAYFORM12 Moreover, if holds, then the common term in, J, equals the potential function.Note that holds in the following cases: i) when Θ k = 0, as the cooperative case described in Corollary 1; ii) when Θ k does not depend on the state but only on the parameter vector, i.e., Θ k: j∈N,j =k W j → R, as in "the great fish war" example described in Appendix B; or iii) when all agents have disjoint state-component subsets, i.e., X Θ k ∩ X Θ j = ∅, ∀(k, j) ∈ {N × N : k = j}. An interesting insight from Theorem 2 is that a dynamic game that is potential when it is analyzed in the OL case (i.e., the policy is a predefined sequence of actions), might not be potential when analyzed in the CL parametric setting. This is straightforward since the potentiality condition in the OL case provided by (, Cor. 1 In order to apply Theorems 1 and 2, we are implicitly assuming that there exists solution to the OCP. We finish this section, by showing that this is actually the case in our setting (proof in Appendix E). In other words, Prop. 1 shows that there exists a deterministic policy that achieves the optimal value of P 1, which is also an NE of G 2 if conditions- or equivalently- hold. We remark that there might be many other-possibly stochastic-policies that are also NE of the game. In this section, we show how to use the proposed MPGs framework to learn an equilibrium of a communications engineering application. We extend the Medium Access Control (MAC) game presented in BID25 to stochastic dynamics and rewards (where previous OL solutions would fail), and use the Trust Region Policy Optimization (TRPO) algorithm BID20, which is a reliable reinforcement learning method policy search method that approximates the policy with a deep-neural network, to learn a policy that is a PCL-NE of the game. We consider a MAC uplink scenario with N = 4 agents, where each agent is a user that sets its transmitter power aiming to maximize its data rate and battery lifespan. If multiple users transmit at the same time, they will interfere with each other and decrease their rate, using their batteries inefficiently, so that they have to find an equilibrium. Let x k,i ∈ [0, B k,max] X k denote the battery level for each agent k ∈ N, which is discharged proportionally to the transmitted power, Let a k,i ∈ [0, P k,max] A k be the transmitted power for the k-th user, where constants P k,max and B k,max stand for the maximum allowed transmitter power and battery level, respectively. The system state is the vector with all user's battery levels: x i = (x k,i) k∈N ∈ X; such that S = N and all state vector components are unshared, i.e., X = k∈N X k ⊂ R N, and X k = {k}. We remark that although each agent's battery depletion level depends directly on its action and its previous battery level only, it also depends indirectly on the strategies and battery levels of the rest of agents. The game can be formalized as follows: DISPLAYFORM0 where h k is the random fading channel coefficient for user k, α is the weight for the battery reward term, and δ is the discharging factor. First of all, note that each agent's policy and reward depend only on its own battery level, x k,i. Therefore, we can apply Theorem 2 and establish that the game is a MPG, with potential function: DISPLAYFORM1 Thus, we can formulate OCP with single objective given by.Since the battery level is a positive term in the reward, the optimal policy will make the battery deplete in finite time (formal argument can be derived from transversality condition). Moreover, since δ k,i ≥ 0, the episode gets into a stationary (i.e., terminal) state once the battery has been depleted. We have chosen the reward to be convex. The reason is that in order to compute a benchmark solution, we can solve the finite time-horizon convex OCP exactly with a convex optimization solver, e.g., CVX BID7, and use the as a baseline for comparing with the solution learned by a DRL algorithm. Nevertheless, standard solvers do not allow to include random variables. To surmount this issue, we generated 100 independent sequences of samples of h k,i and δ k,i for all k ∈ N and length T = 100 time steps each, and obtain two solutions with them. We set |h DISPLAYFORM2 where v k,i is uniform in [0.5, 1], |h 1 | 2 = 2.019, |h 2 | 2 = 1.002, |h 3 | 2 = 0.514 and |h 4 | 2 = 0.308; and δ k,i is uniform in [0.7, 1.3]. The first solution is obtained by averaging the sequences, and building a deterministic convex problem with the average sequence, which yielded an optimal value V cvx = 33.19. We consider V cvx to be an estimator of the optimal value of the stochastic OCP. The second solution is obtained by building 100 deterministic problems, solving them, and averaging their optimal values, which yielded an optimal value V avg,cvx = 34.90. We consider V avg,cvx to be an upper bound estimate of the optimal value of the stochastic OCP (Jensen's inequality). The batteries depleted at a level x T < 10 −6 in all cases, concluding that time horizon of T = 100 steps is valid. We remark that these benchmark solutions required complete knowledge of the game. When we have no prior knowledge of the dynamics and rewards, the proposed approach allows as to learn a PCL-NE of by using any DRL method that is suitable for continuous state and actions, like TRPO BID20, DDPG or A3C BID15. DRL methods learn by interacting with a black-box simulator, such that at every time step i, agents observe state x i, take action a i = π w (x i) and observe the new stochastic battery levels and reward values, with no prior knowledge of the reward or state-dynamic functions. As a proof of concept, we perform simulations with TRPO, approximating the policy with a neural network with 3 hidden layers of size 32 neurons per layer and RELU activation function, and an output layer that is the mean of a Gaussian distribution. Each iteration of TRPO uses a batch of size 4000 simulation steps (i.e., tuples of state transition, action and rewards). The step-size is 0.01. FIG0 shows the . After 400 iterations, TRPO achieves an optimal value V trpo = 32.34, which is 97.44% of V cvx, and 92.7% of the upper bound V avg,cvx. We have extended previous on MPGs with constrained continuous state-action spaces providing practical conditions and a detailed analysis of Nash equilibrium with parametric policies, showing that a PCL-NE can be found by solving a related OCP. Having established a relationship between a MPG and an OCP is a significant step for finding an NE, since we can apply standard optimal control and reinforcement learning techniques. We illustrated the theoretical by applying TRPO (a well known DRL method) to an example engineering application, obtaining a PCL-NE that yields near optimal , very close to an exact variational equilibrium. A EXAMPLE: THE "GREAT FISH WAR" GAME -STANDARD APPROACH Let us illustrate the standard approach described in Section 3 with a well known resource-sharing game named "the great fish war" due to BID11. We follow (González-Sánchez and Hernández-, Sec. 4.2). Example 1. Let x i be the stock of fish at time i, in some fishing area. Suppose there are N countries obtaining reward from fish consumption, so that they aim to solve the following game: DISPLAYFORM0 where x 0 ≥ 0 and 0 < α < 1 are given. In order to solve G fish, let us express each agent's action as: DISPLAYFORM1 so that the rewards can be also expressed in reduced form, as required by the standard-approach: DISPLAYFORM2 Thus, the Euler equations for every agent k ∈ N and all t = 0,..., ∞ become: DISPLAYFORM3 Now, the standard method consists in guessing a family of parametric functions that replaces the policy, and checking whether such parametric policy satisfies for some parameter vector. Let us try with policies that are linear mappings of the state: DISPLAYFORM4 By replacing in, we obtain the following set of equations: DISPLAYFORM5 Fortunately, it turns out that has solution (which might not be the case for other policy parametrization), with parameters given by: DISPLAYFORM6 Since 0 < α < 1 and 0 ≤ γ < 1, it is apparent that w k > 0 and the constraint π k (x i) ≥ 0 holds for all x i ≥ 0. Moreover, since k∈N w k < 1, we have that x i+1 ≥ 0 for any x 0 ≥ 0. In addition, since x i is a resource and the actions must be nonnegative, it follows that lim i→∞ x i = 0 (there is no reason to save some resource). Therefore, the transversality condition holds. Since the rewards are concave, the states are non-negative and the linear policies with these coefficients satisfy the Euler and transversality equations, we conclude that they constitute an equilibrium (González-Sánchez and Hernández-, Theorem 4.1).B EXAMPLE: "GREAT FISH WAR" GAME -PROPOSED APPROACHIn this section, we illustrate how to apply the proposed approach with the same "the great fish war" example, obtaining the same as with the standard approach. Example 2. Consider "the great fish war" game described in Example 1. In order to use our approach, we replace the generic policy with the specific policy mapping of our preference. We choose the linear mapping, π k (x i) = w k x i, to be able to compare the with those obtained with the standard approach. Thus, we have the following game: DISPLAYFORM7 Let us verify conditions FORMULA9 - FORMULA9. For all k, j ∈ N we have: DISPLAYFORM8 DISPLAYFORM9 Since conditions FORMULA9 - FORMULA9 hold, we conclude that FORMULA5 is an MPG. By applying the line integral FORMULA2, we obtain: DISPLAYFORM10 Now, we can solve OCP with potential function. For this particular problem, it is easy to solve the KKT system in closed form. Introduce a shorthand: DISPLAYFORM11 The Euler-Lagrange equation for this problem becomes: DISPLAYFORM12 The optimality condition with respect to the policy parameter becomes: DISPLAYFORM13 Let us solve for β i in: DISPLAYFORM14 Replacing FORMULA6 and the state-transition dynamics in FORMULA6, we obtain the following set of equations: DISPLAYFORM15 Hence, the parameters can be obtained as: DISPLAYFORM16 This is exactly the same solution that we obtained in Example 1 with the standard approach. We remark that for the standard approach, we were able to obtain the policy parameters since we put the correct parametric form of the policy in the Euler equation. If we had used another parametric family without a linear term, the Euler equations might have no solution and we would have got stuck. In contrast, with our approach, we could freely choose any other form of the parametric policy, and always solve the KKT system of the approximate game. Broadly speaking, we can say that the more expressive the parametric family, the more likely that the optimal policy of the original game will be accurately approximated by the optimal solution of the approximate game. Proof: The proof mimics the OL analysis from BID25. Let us build the KKT systems for the game and the OCP with parametric policies. For game, each agent's Lagrangian is given ∀k ∈ N by DISPLAYFORM0 where DISPLAYFORM1 ∈ R C are the vectors of multipliers at time i (which are random since they depend on θ i and x i), and we introduced: DISPLAYFORM2 Introduce a shorthand for the instantaneous Lagrangian of agent k: DISPLAYFORM3 The discrete time stochastic Euler-Lagrange equations applied to each agent's Lagrangian are different from the OL case studied in BID25 (see also (, Sec. 6 .1)), since we only take into account the variation with respect to the state: DISPLAYFORM4 where 0 S denotes the vector of length S. The transversality condition is given by DISPLAYFORM5 In addition, we have an optimality condition for the policy parameter w k: DISPLAYFORM6 From these first-order optimality conditions, we obtain the KKT system for every agent k ∈ N and all time steps i = 1,..., ∞: DISPLAYFORM7 DISPLAYFORM8 DISPLAYFORM9 DISPLAYFORM10 where λ k,i−1 is considered deterministic since it is known at time i. Now, we derive the KKT system of optimality conditions for the OCP. The Lagrangian for is given by: DISPLAYFORM11 where DISPLAYFORM12 ∈ R C are the corresponding multipliers, which are random variables since they depend on θ i and x i. By taking the discrete time stochastic EulerLagrange equations and the optimality condition with respect to the policy parameter for the OCP, we obtain are a KKT system for the OCP: i = 1,..., ∞: DISPLAYFORM13 DISPLAYFORM14 DISPLAYFORM15 DISPLAYFORM16 DISPLAYFORM17 where β i−1 is known at time i and includes the multipliers related to x i−1.By comparing FORMULA8 - FORMULA9 and FORMULA2 - FORMULA9, we conclude that both KKT systems are equal if the following holds ∀k ∈ N and i = 1,..., ∞: DISPLAYFORM18 DISPLAYFORM19 DISPLAYFORM20 Since Assumption 4 ensures existence of primal variable for the OCP, Assumption 3 guarantee the existence of dual variables that satisfy its KKT system. By applying and replacing the dual variables of the KKT of the game with the OCP dual variables for every agent, we obtain a system of equations where the only unknowns are the user strategies. This system is similar to the OCP in the primal variables. Therefore, the OCP primal solution also satisfies the KKT necessary conditions of the game. Moreover, from the potentiality condition, it is straightforward to show that this primal solution of the OCP is also a PCL-NE of the MPG (see also BID25, Theorem 1)).Introduce the following vector field:F (x i, w, σ i) ∇ (xi,w) J (x i, π(x i, w), σ i ).Since F is conservative by construction (, Theorems 10.4, 10.5 and 10.9), conditions- are equivalent to FORMULA10 - FORMULA25 and we can calculate a potential J through line integral. Proof: We can rewrite game by making explicit that the actions from the policy mapping, which yields an expression that reminds the OL problem but with extra constraints: DISPLAYFORM0 where it is clear that: a i (a k,i, a −k,i) = π(x i, w) Rewrite also OCP with explicit dependence on the actions: DISPLAYFORM1 By following the Euler-Lagrange approach described in Theorem 1, we have that the KKT systems for game and OCP are equal if the dual variables are equal (including new extra dual variables for the equality constraints that relate the action and the policy) and the following first-order conditions hold ∀k ∈ N and i = 1,..., ∞: DISPLAYFORM2 E ∇ a k,i r k x r k,i, a k,i, a −k,i, σ k,i = E ∇ a k,i J (x i, a i, σ i).The benefit of this reformulation is that the gradient in is taken with respect to the components in X r k only (instead of the whole set X), at the cost of replacing with the sequence of conditions. We have to realize that a k,i is indeed a function of variables x π k,i and w k. In order to understand the influence of this variable change, we use the identity a k,i = π w k (x π k,i) and apply the chain rule to both sides of, obtaining: DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 In addition, since J is proper, it must be upper bounded, i.e., ∃U ∈ R, such that J ≤ U. Then, we have: DISPLAYFORM7 Since B ≤ U, we have that DISPLAYFORM8 Therefore, the level sets are bounded. From Assumption 2 the fact that J can be obtained from line integral FORMULA2, and fundamental theorem of calculus, we deduce that J is continuous. Therefore, we conclude that these level sets are also compact. Thus, we can use (, Prop. 3.1.7, see also Sections 1.2 and 3.6) to ensure existence of an optimal policy.

Represent the TLDR for finding the corresponding scientific article

We present general closed loop analysis for Markov potential games and show that deep reinforcement learning can be used for learning approximate closed-loop Nash equilibrium.

represent the tldr for finding the corresponding scientific article

We introduce two approaches for efficient and scalable inference in stochastic simulators for which the density cannot be evaluated directly due to, for example, rejection sampling loops.