Question:
represent the article for finding a single-sentence tldr

Given We study SGD and Adam for estimating a rank one signal planted in matrix or tensor noise. The extreme simplicity of the problem setup allows us to isolate the effects of various factors: signal to noise ratio, density of critical points, stochasticity and initialization. We observe a surprising phenomenon: Adam seems to get stuck in local minima as soon as polynomially many critical points appear (matrix case), while SGD escapes those. However, when the number of critical points degenerates to exponentials (tensor case), then both algorithms get trapped. Theory tells us that at fixed SNR the problem becomes intractable for large $d$ and in our experiments SGD does not escape this. We exhibit the benefits of warm starting in those situations. We conclude that in this class of problems, warm starting cannot be replaced by stochasticity in gradients to find the basin of attraction. Reductionism consists of breaking down the study of complex systems and phenomena into their atomic components. While the use of stochastic gradient based algorithms has shown tremendous success at minimizing complicated loss functions arising in deep learning, our understanding of why, when and how this happens is still limited. Statements such as stochastic gradients escape from isolated critical points along the road to the best basin of attraction, or SGD generalizes better because it does not get stuck in steep local minima still need to be better understood. Can we prove or replicate these phenomena in the simplest instances of the problem? We study the behavior of stochastic gradient descent (SGD) BID11 and an adaptive variant (Adam) BID8 under a class of well studied non-convex problems. The single spiked models were originally designed for studying principal component analysis on matrices BID12 BID3 BID5 and have also been extended to higher order tensors BID10. Adaptive stochastic optimization methods have been gaining popularity in the deep learning community thanks to fast training on some benchmarks. However, it has been observed that despite reaching a low value of the loss function, the solutions found by Adam do not generalize as well as SGD solutions do. An assumption, widely spread and adopted in the community, has been that SGD's randomness helps escaping local critical points [WRS + 17]. While the problem has been thoroughly studied theoretically [MR14, HSS15, HSSS16, BAGJ18], our contribution is to propose experimenting with this simple model to challenge claims such as those on randomized gradient algorithms in this very simple setup. It is noteworthy that the landscape of non-global critical points of these toy datasets are studied BID0 BID2 BID1 and formally linked to the neural nets empirical loss functions BID2 BID9. For this problem, the statistical properties of the optimizers are well understood, and in the more challenging tensor situation, also the impact of (spectral) warm start has been discussed BID10. We will examine the solutions found by SGD and Adam and compare them with spectral and power methods. This allows to empirically elucidate the existence of multiple regimes: the strong signal regime where all first order methods seem to find good solutions when polynomially many critical points appear, in the matrix case, SGD converges while Adam gets trapped, unless if initialized in the basin of attraction in the presence of exponentially many critical points (the tensor case), all algorithms fail, unless if d is moderately small and the SNR large enough to allow for proper initialization.2 Single spiked models, and stochastic gradients Even though proving strong about non-convex loss functions is in general challenging, a class of nonconvex statistical problems is very well studied and relatively well understood. Principal component analysis (PCA) or finding the leading eigenvector of a covariance matrix is a problem of interest in statistics and machine learning. The proof of convergence of power method to the leading principal component and the geometry of critical points of the maximum likelihood problem maximize u, Au s.t. u 2 = 1, Rayleigh quotient for matrix PCA are well established using eigenvalue decomposition. In addition, more recently, a class of extremely simplified models have shed light on the phase transitions of the problem difficulty as a function of the signal-to-noise ratio in the model. The so-called single spiked models consist of considering a symmetric normalized noise matrix to which a rank one signal is added. DISPLAYFORM0 It is known BID3 that the spectrum of the noise matrix asymptotically forms a semi-circle situated between −2 and 2. When the signal to noise ratio is weak λ ∈ then the signal dilutes in noise, while the leading principal component pops out of the semi-circle as soon as the signal to noise ratio λ is above the critical value λ > λ c = 1, in which case the solution of the problem forms asymptotically a cosine value of 1 − λ −2 1/2 with the signal and the optimal value of the Rayleigh quotient is λ + λ −1 BID12 BID3 BID5. It is proven that the power method allows to obtain the solution after logarithmically many steps, as a function of the problem dimension d. We will minimize the unconstrained objective function (DISPLAYFORM1 We set the value of γ to the theoretical asymptotic value of the leading eigenvalue, γ = 2 for λ < 1 or λ + λ −1 for larger λ, and will add random normal noise to the gradient for stochasticity: ∇ σ (u) = −Au + γu + σz where z i ∼ N (0, 1/d). This function has a constant Hessian H (u) = −λu 0 u 0 T − Z + γI d which is positive semi-definite as soon as γ is equal or larger than the value of the leading eigenvalue of A. The tensor version of the problem (see BID10 for notations and more discussion on problem setting) DISPLAYFORM0 u 2 = 1 Rayleigh quotient tensor PCA under the tensor single spiked model defined for a symmetric (π is a permutation of 3 elements) How does data abundance explain the success of first order methods? On the positive side, we discuss that in this model and considering a large dataset of i.id. samples, weak signals in individual observations accumulate and allow to solve the problem if n √ d. The counter part to the strong requirement λ d 1/4 (conjectured in BID10 and proven in BID6 BID7) is that accumulation of observations compensate low signal to noise ratio in each individual sample. Formally, Remark 2.1. Assume n sample of data according to model "Single spiked tensor", with the same signal u 0 and different i.i.d. noises Z q are observed: DISPLAYFORM1 DISPLAYFORM2 There exists constants c 0, c 1 such that if n ≥ c 0 √ d, then, warm started power iteration produces a vector u, such that with high probability u 0, u > 1 − c 1 /λ. This is established using Theorems 5 in BID10 and 6.3 in BID7 and considering the average tensorĀ DISPLAYFORM3 SinceZ is symmetric andZ i,j,k ∼ N (0, 1/d), the tensorĀ is sampled from a similar distribution as Single spiked tensor with a SNR λ n = √ nλ. This means that the requirement λ d 1/4 BID7, in the average tensor case, relaxes to n √ d. In words, this means that if we are solving a problem with a tensor PCA complexity, and if the number of i.i.d. observations grows quadratically as a function of the problem dimension, we can compute the solution reliably using spectral warm start, even though the original problem looks intractable. Our numerical report performance of different algorithms at solving simulations of matrix and tensor PCA problems. Under various problem generation parameter choices, we report values of• cosine or u, u 0. This is measures the quality of planted (hidden) signal recovery from the noisy observation. Higher values are preferred, and it cannot exceed 1, since both u 0 and u are normalized. This quantity is to be qualitatively compared with the test error in standard learning problems where the true value of the parameter is not available.• Rayleigh is the value of the log-likelihood objective function that we are maximizing. Higher values are preferred. The theoretical maximum value of this objective is the operator norm of the observed tensor A. This is comparable with (minus) the training loss. The signal to noise ratio λ is to be compared with the number of observations in a supervised learning problem. The stochasticity of the gradients σ is to be compared with the number of sample points in each minibatch of data: large stochasticity mimics small minibatch situations. In Figure 1 we plot the values of the objective function or Rayleigh quotient and the cosine of the ground truth with the solution as a function of the iterates. We replicated these plots at values of the SNR parameter λ < λ c = 1, at the critical value λ = 1 and above it for λ = 2 where the problems is considered to be easy. The learning rate and stochasticity σ were set by generating instances of the problem with different noise matrices. These plots allow to compare the optimization power of different algorithms and also keep track of the quality of the solution found. We can see in these plots that Adam gets stuck around the wrong region very fast. Note that with the value (or larger) of γ = 2 for λ < 1 and γ = λ + λ −1 for λ ≥ 1 that we can set given the true value of λ, and knowing the concentration around the asymptotics BID12 BID3 BID5, the objective function is strongly convex so we expect first order methods to show the same convergence rates as the power method. One can also observe that gradient descent corresponds to performing power iteration on a shifted matrix A + αI d. Figure 2 shows the value of the objective and the correlation with the ground truth as a function of SNR λ. We can see that SGD is superior to Adam, uniformly along λ, while power method (also a first order method) rivals with SGD. These plots exhibit the instability around and below the critical value λ c = 1 while above λ c the behavior is more stable. In the tensor setting we experimented with spectral initialization. Spectral initialization consists of flattening the tensor to a d×d 2 matrix and initializing tensor algorithms with the left singular vector of the flattened tensor. We observe benefits of spectral initialization at locating the initial point in a basin of attraction that leads to better solutions when λ is large enough. For λ = 1.0 spectral initialization does not in better estimates in Figure 3, while for larger values of λ we can see the benefit of warm start. In FIG2 we also plot values of the gradient and the number of positive eigenvalues of the Hessian along the optimization iterates for d = 100, λ = 2, γ = 2. We observe that spectral initialization located the initial point of the iterations in the basin of attraction where the problem is convex (all eigenvalues of the Hessian are positive). Adam, while starting in this region, fails at finding a solution as good as SGD's. We experimented with the amount of noise added to each gradient evaluation and mapped median values of the estimate and optimization problems for values of λ and σ over 100 instances of the problem generated with different noise matrices and stochastic gradients. Stochasticity does not seem to remedy to the problem difficulty. Numerical experiments suggest that irrespective of the magnitude of the stochastic component added to the gradient, the first order methods, initialized at random, fail at finding the best basin of attraction. In the same setup, spectral initialized first order methods successfully find the solutions. We propose to study algorithms used for minimizing deep learning loss functions, at optimizing a non-convex objective on simple synthetic datasets. Studying simplified problems has the advantage that the problem's properties, and the behavior of the optimizer and the solution, can be studied rigorously. The use of such datasets can help to perform sanity checks on improvement ideas to the algorithms, or to mathematically prove or disprove intuitions. The properties of the toy data sets align with some properties of deep learning loss functions. From the optimization standpoint, the ing tensor problems may appear to be even harder than deep learning problems. We observe that finding good solutions is hard unless if proper initialization is performed, while the value of stochasticity in gradient estimates seems too narrow and does not appear to compensate for poor initialization heuristics. Each column represents the values of those quantities along iterations of the algorithm. The prefix sp. refers to spectral initialization and l. refers to a decreasing learning weight scheduled in 1/ √ t. We observe the value of warm starting as soon as λ is large enough. Even at high SNR λ = 6, randomly initialized SGD fails while spectrally initialized SGD succeeds. Adam drifts to a non optimal critical point in that regime, even with spectral warm start., a positive would be SGD and Adam under single spiked model for tensor PCA & a negative would be  In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.

Question: Adversarial perturbations cause a shift in the salient features of an image, which may in a misclassification. We demonstrate that gradient-based saliency approaches are unable to capture this shift, and develop a new defense which detects adversarial examples based on learnt saliency models instead. We study two approaches: a CNN trained to distinguish between natural and adversarial images using the saliency masks produced by our learnt saliency model, and a CNN trained on the salient pixels themselves as its input. On MNIST, CIFAR-10 and ASSIRA, our defenses are able to detect various adversarial attacks, including strong attacks such as C&W and DeepFool, contrary to gradient-based saliency and detectors which rely on the input image. The latter are unable to detect adversarial images when the L_2- and L_infinity- norms of the perturbations are too small. Lastly, we find that the salient pixel based detector improves on saliency map based detectors as it is more robust to white-box attacks. Adversarial examples highlight a crucial difference between human vision and computer image processing. Often computers fail to understand the relevant characteristics of an image for classification or fail to generalize locally, i.e., misclassify examples close to the training data . Attacks exploit this property by altering pixels the classifier heavily relies on -pixels which are irrelevant to humans for object recognition. As a consequence, adversarial perturbations fool classifiers while the correct class remains clear to humans. Saliency maps identify the pixels an image classifier uses for its prediction; as such, they can be used as a tool to understand why a classifier is fooled. Building on this concept, researchers have shown qualitatively that adversarial perturbations cause a shift in the saliency of classifiers . Figure 1 shows examples of a natural image and corresponding adversarial images, each above their respective saliency maps. The saliency maps corresponding to adversarial images show perceptible differences to that of the original image, even though adversarial images themselves often seem unperturbed. For the original image, the saliency map shows that the classifier focuses on the four (and a couple of random pixels on the left). We observe that for the adversarial images, the classifier starts focusing more on irrelevant aspects of the left side of the image. There is ample research into different techniques for finding saliency maps (see e.g. ; ; ; ; ; ; ;). However, not all saliency maps are equally informative . For example, the Jacobian 1 can be used to determine the saliency of a pixel in the classification of the image (b;). As the Jacobian is often used to generate adversarial examples, intuitively, we expect that it can be used effectively to detect adversarial perturbations. propose a defense to this effect: they determine whether an input is adversarial, given the Jacobian-based The top is the input image and the bottom shows the corresponding saliency map. In the second row, lighter colours correspond to higher saliency (black corresponds to a saliency of 0, the lowest possible value). The classifier predicts (from left to right) the images as: 4, 9, 9, 8, 9, 9. Note the stark difference between the saliency masks of the original image and those of the adversarial examples. saliency map concatenated with the image. However, as shown qualitatively by , gradients are not always able to capture differences between adversarial images and natural images (for an example see Figures 7 and 8 in Appendix D). 2 Here we inspect the proposed Jacobian-based approach and show that only the concatenated input affects the technique's performance in detecting adversarial examples, with the Jacobian having no effect. While gradients may not be informative for detection, saliency should be an effective tool for detecting adversarial images. In our analysis, we use more powerful model-based saliency techniques and show that the magnitude of the shift of the saliency map due to adversarial perturbations often exceeds the L 2 distance between the saliency maps of different natural images. Building on this , we consider two different possible effects adversarial perturbations might have on the classifier: 1. They might cause the classifier to focus on the wrong pixel locations 2. They might change the pixel values of salient pixels Based on these hypotheses, we employ two CNN classifier architectures to detect adversarial images. Claim can be captured by shifts in saliency maps, as previously considered by. In this work, we extend on their analysis 3 by proving the defensive capability of our model-based saliency against difficult black-box attacks, such as C&W and DeepFool 4, as well as white-box adversarial attacks. By considering claim, we demonstrate that incorporating pixel values improves the performance of the classifier when shifts in saliency maps do not suffice to capture adversarial perturbations. We also show that our salient-pixel based defense generalizes well (detecting stronger attacks when trained on weaker attacks) and is more robust than the saliency map defense against white-box attacks. Lastly, we demonstrate that saliency can be used to detect adversarial examples generated by small perturbations, contrary to other defenses, which exhibit threshold behavior: i.e., when the adversarial perturbation is too small, other defenses (specifically ;) are unable to detect the adversarial images. Saliency maps and adversarial perturbations have similar mathematical formulations and derivations. Both are computed by investigating the relation between the values of pixels and the classification score. Adversarial examples are found by deriving the minimal perturbations required to change the classification of an image. Saliency is computed by finding the pixels used by the model 2 show that gradient-based heat maps are less effective than other saliency methods in detecting adversarial perturbations generated using BIM . 3 Their main contribution is that saliency maps generated by different techniques are not equally effective in capturing changes due to adversarial perturbations (produced using BIM . 4 These attacks generate smaller L2 perturbations, making them more difficult to detect. The perturbation size used by can likely still be detected by a simple classifier that trains on images. to determine the class of an object . Saliency maps can be found by considering the smallest part of an image that is sufficient for a correct classification, known as the smallest sufficient region (SSR), or whose removal is sufficient for an incorrect classification, known as the smallest destroying region (SDR) . Observe that the latter definition of saliency is very close to that of adversarial examples. Mathematically, both saliency maps and adversarial perturbations can be derived in a similar fashion. Consider adversarial examples. The general formulation of an adversarial attack can be summarized as follows: where x is the natural image, r is the adversarial perturbation, y is the correct class, and y is an incorrect class. Due to the non-linearity of NNs, solving the above problem requires non-linear optimization. Therefore, in practice several different approaches to solving the above formulation have been implemented. For example, set r = εsign(δx). Similarly, saliency can be computed using the forward derivative δx (b;). Previous research has already started investigating the relation between saliency and adversarial examples. This includes: Using saliency to attack Researchers have devised adversarial attacks that use saliency (b;). The key idea is to use saliency to determine the pixel that is most sensitive to perturbation iteratively. The main benefit is that fewer pixels are perturbed -often perturbing as few as 4% of the pixels suffices to change the classification of the image (b). Using saliency to defend introduce a method that detects adversarial perturbations by using heat-map visualizations of the predicted class. However, in their analysis, they only use BIM , which is easily detected. hypothesize that there is a mismatch between the saliency of a classification model and the adversarial example. They propose a defense against adversarial attacks by training a classifier on images concatenated with their saliency map, which is computed by calculating the Jacobian of the classifier with respect to the image x, i.e., s x = ∇ x f (x). find that their method obtains a high accuracy (often near 100%) when detecting adversarial images generated by FGSM, MIM, and C&W attacks on MNIST, CIFAR-10, and 10-ImageNet. contradict these , and demonstrate that the gradients show imperceptible differences due to adversarial perturbations (see Figures 7 and 8 in Appendix D). Adversarial robustness and interpretability of models and 5 show that saliency maps can be used to explain adversary classifications. Both highlight an important trend: not all techniques used to compute saliency maps show shifts in saliency maps due to adversarial perturbations. shows that more robust models have more interpretable saliency masks. quantify the relation by investigating the alignment between the saliency map and the input image. In this section, we explain how we construct and evaluate our saliency-based adversarial example detectors. We train a convolutional neural network image classifier, which we target with black-box attacks; the architectures are summarized in Appendix A. We use cross-entropy loss and optimize the parameters using Adam ;;; b; b; , respectively). We use the implementation as provided in cleverhans (a). The hyper-parameters are summarized in Appendix B. To generate saliency masks, we adapt the method used by. Our reason is twofold: the technique computes high-quality saliency masks at a low computational cost. employ a U-Net with a novel loss function that targets SDR, SSR, mask sparsity, and mask smoothness. We adapt the original loss function to omit the total variational term, as mask smoothness is not required in our analysis. ) denote the generated map. First, the map average AV (f s) is used to ensure that the area of the map is small. Second, log(f c (Φ(x, f s))) is included to ensure that the salient pixels suffice to identify the correct class. Finally, f c (Φ(x, 1 − f s)) is included to ensure that the classifier can no longer recognize the class if the saliency map is removed. Therefore, our saliency loss function is: where f c is the softmax probability of the class c, Φ(x, f s) applies mask f s to image x, and λ i ≥ 0 are hyper-parameters. We adapt the PyTorch implementation provided by 6 and train the saliency model on standard, non-adversarial images only. For evaluation, we use the same saliency model for both natural and adversarial images. When generating the saliency maps for our images, we use the predicted classification for feature selection to prevent an information leak (which would occur if we use the true label). Our hypothesis is that if an image is adversarial, the classifier likely focuses on the wrong aspects or the pixels on which it focuses are misleading (due to the perturbed color or intensity) when classifying an image as adversarial. We consider two different cases by building classifiers for saliency maps and salient pixels. For both classifiers, we use the same architecture (and hyperparameters) as for the black-box image classifiers (as summarized in Appendix A). We build a detector based on the saliency maps of images as follows. First, we train a classifier and generate adversarial images for every natural image in the training dataset. Then we generate the saliency maps for the clean data {f s (X)} and adversarial images {f s (X adv)}. We build a binary detector for the saliency maps, which predicts whether the corresponding image is adversarial or natural. We abbreviate this defense as SMD (Saliency Map Defense). We do not concatenate the saliency maps to the input image. We construct a second classifier for the salient pixels. We follow the same steps as outlined in the previous section, aside from the final step. We define the salient pixels as f s (x) · x, where x is the image, f s (x) is the saliency map corresponding to x and · denotes the element-wise product. We abbreviate this defense as SPD (Salient Pixel Defense). Similarly to SMD, we do not concatenate the saliency maps to the input image. To benchmark our , we consider two baselines. First, we train a baseline classifier that classifies input as adversarial or natural based on the images alone. This allows us to evaluate the added benefit of using saliency maps. This method was implemented by. We abbreviate this defense as ID (Image Defense). Second, we compare our defense method with the saliency-based defense of (see Section 2). We abbreviate this defense as JSD, for Jacobian-based Saliency map Defense. In our implementation, we adapt the method of; we find that if we use f s (x) = ∇ x f (x) as the saliency map it leads to underflow, ing in a zero matrix. Therefore, instead we take the derivative with respect to the logits, i.e. f s (x) = ∇ x z(x). JSD is mathematically related to the other defenses. First, it is more general compared to ID: the filters of JSD can learn to ignore the Jacobian-based saliency, in which case the two methods are equivalent. Further, JSD is similar to SMD, as the filters can learn to ignore the image input. In this case, the only difference between JSD and SMD is that they use different techniques to derive saliency. However, JSD differs from SPD, as CNN filters cannot multiply one channel by another. We follow the evaluation protocol of and train each defense to detect adversarial images generated by a specific attack, thereby generating six different detection models (one for each black-box attack). To generate the training data, we generate one adversarial example for every clean image. The training data becomes [X, X adv], where X denotes the clean data and X adv denotes the adversarial data, and the labels are [1 n, 0 n], 1 n and 0 n are one-and zero-vectors of length n, respectively. We use the same training procedure and models, as summarized in Appendix A, and report the accuracy of the classifiers on the test dataset. We compare the performance of the models on MNIST, CIFAR-10 and ASSIRA (see ; ; , respectively). In addition to the two frequently used benchmarks, we consider the ASSIRA cats and dogs dataset 7 as it contains highquality images but is less computationally expensive than ImageNet. 8 Further details on the datasets can be found in Appendix A. Many defenses hold up against black-box attacks but often are unable to defend against white-box attacks (a). For this reason, we generate white-box attacks tailored to the defense strategy. Our white-box attacks are iterative gradient-based attacks, which target both the classifier and the defense. Inspired by FGSM, we can target the classifier f as and the defense d as where Clip clips the pixels to the pre-defined maximum range. Using the above idea, we iterate between Equations 3 and 4 to generate the white-box attack for ID (the defense based on image classification). We propose similar white-box attacks for the other defenses, as shown in Appendix C. We limit the number of iterations T to 5, as we find it to be adequate to generate a sufficiently strong attack and further increasing T does not improve the performance. Our method is similar to that of. They propose finding adversarial examples as: where in our case α = 0.5. The key difference is that we iterate between Equations 3 and 4, rather than applying 3 and 4 simultaneously. We find that this is more effective at targeting the defense, which is more difficult to fool than the original classifier. We start by assessing the shift in saliency maps generated by adversarial perturbations and then present the efficacy of the detector against different adversarial attacks. Details, such as attack success rate, can be found in Appendix B. We start by quantifying the shift in saliency maps due to adversarial perturbations; we compute the L 2 distance between saliency maps of a natural image and its corresponding adversarial image. As a baseline, we compare these values with the L 2 distance between two different natural images. These statistics are summarized in Table 5. For CIFAR-10 and ASSIRA, the L 2 -norm between the saliency maps of a natural image and its corresponding adversarial image is comparable to or larger than the L 2 distance between two different natural images. Using a Mann-Whitney U-test, we prove quantitatively that the shift is significant for most adversarial attacks on CIFAR-10 and ASSIRA images. This suggests that our saliency-based method is an effective way of capturing adversarial perturbations. Table 1: L 2 distance between saliency maps of different images (row labelled Different Images) and the saliency maps of natural images and the adversarial image (generated by the type of attack specified in the row). The entries correspond to MNIST/CIFAR-10/ASSIRA. The p-value is derived using the Mann Whitney U-test, where we test whether the sample of L 2 distances between a natural and adversarial image is from the same distribution as different images. We use a nonparametric test to avoid assuming normality of the data. Figure 2 summarizes the performance of the defense models trained on a single adversarial attack on different adversarial attacks; the values and standard deviations can found in Appendix G. The overall performance of the model-based saliency defense suggests that saliency can be used to determine whether an image is adversarial. Salient Pixel Defense outperforms Saliency Map Defense Overall, SPD (shown in blue) outperforms the other defenses, suggesting that the salient pixels provide useful information to the detector. Further, our defense generalizes well: even when trained on a weaker attack, SPD is able to detect stronger attacks. Both baseline methods, ID and JSD, only generalize well when trained on a stronger attack. When trained on a weaker attack, they are not able to detect stronger adversarial attacks. Worse generalization on JSMA We observe a drop in performance of the models when detecting JSMA, likely because JSMA is an L 0 -norm attack, which generates a different type of adversarial examples. This may suggest that defenses trained on a specific norm, only generalize well to other attacks generated by a norm that produces similar perturbations. FGSM, BIM, and MIM are L ∞ -norm attacks, and C &W and DF are L 2 -norm attacks. Both generate perturbations that are spread out over the entire image, contrary to L 0 norm attacks, which changes a few pixels using larger perturbations. Threshold Behavior Both ID and JSD exhibit threshold behavior: they are unable to detect adversarial examples if the perturbation size is below a given threshold. For example, see the performance of both defenses on the ASSIRA dataset. There is a strong correlation between detection accuracy and perturbation size, as measured by L ∞ and L 2 (see Table 2). ID is able to detect all adversarial images for which the perturbation size is either L 2 > 0.027 or L ∞ > 0.50, such as FGSM and JSMA. 10 However, the perturbations are much smaller for DF and CW, making these attacks harder to detect. The threshold appears to occur around L 2 = 0.025, as ID can sometimes detect the FGSM perturbations, generated with this size. This observation is in line with the of , who find that ID is highly efficient at detecting adversarial images with perturbations of ε ≥ 0.03 but unable to detect adversarial perturbations generated using ε = 0.01 (using FGSM for images scaled between 0 and 1), obtaining an accuracy of 50.0% in the latter case. 11 9 We observe that JSD performs similarly, although sometimes worse, compared to ID. Theoretically, the parameter space of ID is a subset of the parameters of JSD. The additional input (the Jacobian) makes the model more difficult to train. Therefore, the difference in can be attributed to training: the model is more difficult to train due to the increased number of parameters and does not learn to ignore the additional input. 10 FGSM is known to generate large perturbations. The perturbations for JSMA are relatively large as the attack minimizes the L0 norm, thereby perturbing as few pixels as possible, but by a large amount. 11 Our perturbations for FGSM are larger than 0.01 to ensure that FGSM is sufficiently strong (see Appendix B for a summary of the attack success rates). Table 3 summarizes the performance of different defenses against our white-box attack. Our whitebox methods are highly effective in fooling the classifier as well as the defenses for MNIST and ASSIRA, as shown by the before adversarial training . The white-box attack is unable to fool the detector for CIFAR-10 successfully. Next, we perform adversarial training: we iteratively train the detectors against the white-box attack and allow the white-box attack access to the new defense. The white-box attack no longer successfully defeats SPD, which becomes more robust against the attack, whereas SMD is not able to become robust against the white-box attack. In our analysis, we ascertain that the saliency maps of adversarial images differ from those of natural images. Further, we show that salient pixel based defenses perform better than a saliency map defense. When trained on a single black-box attack, our method is able to detect adversarial perturbations generated by different and stronger attacks. We show that gradients are unable to capture shifts in saliency due to adversarial perturbations and present an alternative adversarial defense using learnt saliency models that is effective against both black-box and white-box attacks. Building on the work of , we further establish the notion of threshold behavior, showing that the trend depends on the L 2 and L ∞ -norms of the perturbations and therefore also prevails when using other methods (JSD) and across different attacks. Future work could further investigate the performance of the defense in different applications. For example, as our method runs in real-time, it could be used to detect adversarial perturbations in video to counter recent attacks ). A ARCHITECTURES, HYPER-PARAMETERS AND DATA Figure 3: ASSIRA, CIFAR-10, and MNIST image classifier architecture and hyper-parameters. The first entry corresponds to the first layer, and the table proceeds chronologically until the last layer. Parameters f, k, p, s and n represent the number of filters, kernel size, pooling size, stride, number of filters, respectively. If stride is omitted, it is set to 1. All classifiers have a final softmax activation. We apply drop-out before every dense layer. Using a validation set, we experimented with different drop-out rates between 0.3 and 0.7 and found that the rate δ = 0.6 was optimal. We use a ReLu activation for the penultimates layers and a softmax activation for the final layer. We train the model for 10 epochs on batches of size 50. We compare the performance of the models on MNIST, CIFAR-10 and ASSIRA (see ; ; , respectively). For MNIST and CIFAR-10, we use the standard train and test splits, and for ASSIRA, we use 3, 000 images. We use 10% of the training data for the validation set, and re-train on the full training dataset once hyper-parameters were selected. Further experimentation of ID and JSD architecture We further experiment with the architectures of ID and JSD to determine whether the observed performance was the of the architecture. In particular, we considered the adjustments as summarized in Table A; however, we found that the changes did not improve performance. In this section, we present the black-box adversarial attack hyper-parameters (see Figure 4), the success rates of the different adversarial attacks (see Table 4) and an example of an adversarial image generated by the various black-box attacks (see Figure 5). et al., 2016a) are used. ε is the maximum perturbation allowed and ε i is the maximum perturbation allowed in an iteration. We use different hyperparameters for the MNIST and CIFAR-10 to ensure the attack is sufficiently strong. MNIST and CIFAR-10 Figure 5: Example of an Adversarial Image for the MNIST dataset. From top to bottom: the top row is the set of images; the bottom row shows the size of the noise added. Gray indicates no change, whereas white indicates that the image has been made lighter, and black indicates that the image has been made darker. As MNIST images are gray-scale low-resolution images, the adversarial perturbations are perceptible to the human eye. Nevertheless, the correct classification of the image is still clearly 4. However, the classifier predicts (from left to right) the images as: 4, 9, 9, 8, 9, 9. Further, we observe that the perturbations of FGSM, BIM, and MIM are more visible than those of C & W and DF. Algorithm 1 White-box attack for JSD 1: x adv ← x 2: for t = 0: T do 3: for j = 1: n do if x adv does not fool the classifier then 5: x adv ← x adv + εsign(∇f (x adv, y)) Algorithm 1 provides the white-box attack for JSD. As mentioned in Section 2, JSD concatenates the image with its saliency map (computed as the Jacobian) and uses this as an input to the classifier. Algorithms 2 and 3 provide the white-box attacks for our defenses: SPD and SMD. The function f s corresponds to generating the saliency map using the method introduced by. Their method returns a two-dimensional saliency map. However, as the image is three dimensions, we expand the last dimension and stack the map to match the number of channels (n c) of the image. In doing so, we assume that the saliency is constant along depth. Algorithm 2 White-box attack for SPD 1: x adv ← x 2: for j = 1: n do 3: for t = 1: T do if x adv does not fool the classifier then 5: end if 7: if sp(x adv) does not fool the detector then 10: if n c > 1, repeat r along the last dimension until it matches n c 12: x adv ← Clip(x adv + r) if s adv does not fool the detector then 9: r ← εsign(∇d(s adv, y)) 10: if n c > 1, repeat r along the last dimension until it matches n c 11: The first row shows the natural and adversarial images, and the second row shows their respective saliency maps. There are no perceptible differences between the saliency map of the original image and adversarial images generated using MIM, C&W, and DF. Further, we observe that for FGSM and JSMA, the gradients are all zero-valued. This is a second drawback of using gradients-they are unstable and generate uninformative saliency maps due to underflow. E L 2 DISTANCES BETWEEN SALIENCY MAPS CORRESPONDING TO ADVERSARIAL IMAGES GENERATED BY DIFFERENT ATTACKS F SINGLE BLACK-BOX ADVERSARIAL ATTACK DETECTOR Table 6 summarizes the accuracies of the different defenses when training a single detector against a combination of different types of black-box attacks. All methods perform relatively similarly as when trained against a single defense, obtaining slightly worse performances than when trained against a specific adversarial attack. This is useful in practice when it is unclear which adversarial attack is used.

We show that gradients are unable to capture shifts in saliency due to adversarial perturbations and present an alternative adversarial defense using learnt saliency models that is effective against both black-box and white-box attacks.

represent the tldr for finding the corresponding scientific article


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Disentangled encoding is an important step towards a better representation learning. However, despite the numerous efforts, there still is no clear winner that captures the independent features of the data in an unsupervised fashion. In this work we empirically evaluate the performance of six unsupervised disentanglement approaches on the mpi3d toy dataset curated and released for the NeurIPS 2019 Disentanglement Challenge. The methods investigated in this work are Beta-VAE, Factor-VAE, DIP-I-VAE, DIP-II-VAE, Info-VAE, and Beta-TCVAE. The capacities of all models were progressively increased throughout the training and the hyper-parameters were kept intact across experiments. The methods were evaluated based on five disentanglement metrics, namely, DCI, Factor-VAE, IRS, MIG, and SAP-Score. Within the limitations of this study, the Beta-TCVAE approach was found to outperform its alternatives with respect to the normalized sum of metrics. However, a qualitative study of the encoded latents reveal that there is not a consistent correlation between the reported metrics and the disentanglement potential of the model. Unsupervised disentanglement is an open problem in the realm of representation learning, incentivized around interpretability BID8 BID1. A disentangled representation is a powerful tool in transfer learning, few shot learning, reinforcement learning, and semi-supervised learning of downstream tasks (; BID9 BID1 .Here, we investigate the performance of some of the promising disentanglement methods from the family of variational autoencoders (VAE). The methods are evaluated based on five relatively established disentanglement metrics on the simplistic rendered images of the mpi3d toy dataset curated and released for the NeurIPS 2019 Disentanglement Challenge. To mitigate the sensitivity of the models to the initial state, as suggested by the findings of , an autoencoder model was pre-trained with the conventional VAE objective BID6 on the mpi3d toy dataset. This approach guaranteed that models did not collapse into a local minima with little to no reconstruction. It also facilitated the training process given the constraints on the length of training by the challenge. In this preliminary study, we implemented the variational objective functions proposed by the following methods: β-VAE BID4 ), β-TCVAE (, Factor-VAE BID5, Info-VAE BID11, DIP-I-VAE, and DIP-II-VAE BID7 .In β-TCVAE, the mutual information between the data variables and latent variables are maximized, while the mutual information between the latent variables are minimized. Defining x n as the nth sample of the dataset, the evidence lower bound (ELBO) of this objective can be simplified as follows 1 DISPLAYFORM0 where z j denotes the jth dimension of the latents. In the above equation, the first term is the reconstruction loss. The second term is the distance between the assumed prior distribution of the latent space and the empirical posterior latent distribution. The last term is an indication of the total correlation (TC) between the latent variables which is a generalization of the mutual information for more than two variables BID10. A total capacity constraint which limits the KL divergence between the posterior latent distribution and the factorized prior can encourage the latent representation to be more factorised. However, this will act as an information bottleneck for the reconstruction task and in a blurry reconstruction. Thus, progressively increasing the information capacity of VAE during training can help facilitate the robust learning of the factorized latents BID2. This is achieved by introducing the capacity term C and defining the distance between distributions as the absolute deviation from C: DISPLAYFORM0 Gradually increasing C has an annealing effect on the constraint and increases the reconstruction capacity of the model. For each learning algorithm, the hyper-parameter sub-spaces were independently searched. However, in order for the reported here to be comparable, the hyper-parameters were kept intact in between the following experiments. The input images were 64 × 64 pixels and the latent space was of size 20. The model capacity parameter, C, was initiated at zero and gradually increased up to 25 over 2000 iterations. Learning rate was initiated at 0.001 and was reduced by a factor of 0.95 when the loss function (Equation) did not decrease after two consecutive epochs, down to a minimum of 0.0001. Batch size was set to 64. Optimization was carried out using the Adam optimizer with the default parameters β1 = 0.9 and β2 = 0.999. The network architectures and other hyper-parameters are detailed in Appendix A.The trained models were evaluated based on five evaluation metrics, namely, DCI, FactorVAE metric, IRS, MIG, and SAP-Score. Results of these evaluations are presented in TAB0. The non-ignored latent variables of each method are traversed and the are visualized in Appendix B. Moreover, the evaluation logs during model training are visualized in Appendix C.All the models and experiments were implemented using the PyTorch deep learning library and packaged under the Disentanglement-PyTorch repository https://github.com/ amir-abdi/disentanglement-pytorch 2. In this work we compared the degree of disentanglement in latent encodings of six variational learning algorithms, namely, β-VAE, Factor-VAE, DIP-I-VAE, DIP-II-VAE, Info-VAE, and β-TCVAE. The empirical TAB0 point to β-TCVAE being marginally the superior option and, consequently, chosen as the best performing approach. However, a qualitative study of the traversed latent spaces (Appendix B) reveals that none of the models encoded a true disentangled representation. Lastly, although the DIP-VAE-II model is under performing according to the quantitative , it has the least number of ignored latent variables with a promising latent traversal compared to other higher performing methods (Appendix B). As a of these inconsistencies, we find the five metrics utilized in this study inadequate for the purpose of disentanglement evaluation. Among the limitations of this study is the insufficient search of the hyper-parameters space for all the six learning algorithms. Moreover, the NeurIPS 2019 Disentanglement Challenge imposed an 8-hour limit on the training time of the models which we found to be insufficient. This, while the maximum number of iterations was set to 200k in our experiments, this value was limited to 100k in the submissions made to the challenge portal.2. The repository will be publicly released upon the completion of the competition. The encoder neural network in all experiments consisted of 5 convolutional layers with strides of 2, kernel sizes of 3 × 3, and number of kernels gradually increasing from 32 to 256. The encoder ended with a dense linear layer which estimated the posterior latent distribution as a parametric Gaussian. The decoder network consisted of one convolutional followed with 6 deconvolutional (transposed convolutional) layers, with kernel sizes of 4, strides of 2, and the number of kernels gradually decreasing from 256 down to the number of channels of the image space. ReLU activations were used throughout the architecture, except for the last layers of the encoder and decoder networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Represent the short summary for finding the corresponding scientific article (long)

Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.


Question: Represent the article for finding a single-sentence TLDR

Question: Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly learning a policy for each agent to maximize a common reward. In this paper, we aim to address this from a different angle. In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not want to do. For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by first inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together. The objective of the manager is to maximize the overall productivity as well as minimize payments made to the workers for ad-hoc worker teaming. To train the manager, we propose Mind-aware Multi-agent Management Reinforcement Learning (M^3RL), which consists of agent modeling and policy learning. We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents. The experimental have validated the effectiveness of our approach in modeling worker agents' minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation. As the main assumption and building block in economics, self-interested agents play a central roles in our daily life. Selfish agents, with their private beliefs, preferences, intentions, and skills, could collaborate (ad-hoc teaming) effectively to make great achievement with proper incentives and contracts, an amazing phenomenon that happens every day in every corner of the world. However, most existing multi-agent reinforcement learning (MARL) methods focus on collaboration when agents selflessly share a common goal, expose its complete states and are willing to be trained towards the goal. While this is plausible in certain games, few papers address the more practical situations, in which agents are self-interested and inclined to show off, and only get motivated to work with proper incentives. In this paper, we try to model such behaviors. We have multiple workers and a manager, together to work on a set of tasks. The manager gets an external reward upon the completion of some tasks, or one specific task. Each worker has a skill set and preference over the tasks. Note that their skills and preferences may not align with each other (Fig. 1(a) ), and are not known to the manager (Fig. 1(b) ). Furthermore, manager may not get any external reward until a specific task is complete, which depends on other tasks. By default, the self-interested workers simply choose the most preferred tasks, which is often unproductive from the perspective of the entire project. Therefore, the manager gives additional incentives in the form of contracts. Each contract assigns a goal and a bonus for achieving the goal to a worker. Figure 1: Illustration of our problem setup. Workers have different skills (abilities for completing tasks) and preferences (which tasks they like) indicated by the bar charts. They are self-interested and perform the tasks they prefer the most. To achieve optimal collaboration, a manager has to first infer workers' minds, and assigns right bonuses to workers for finishing specified tasks in the form of contracts. Consequently, workers will adjust their intentions and work together accordingly. E.g., workers in the figure initially all want to do task B. To finish all tasks, the manager has to pay more bonus to worker 1 and 2 so that they will perform A and C respectively. With the external incentives, workers may choose different goals than their preferences. Upon completion of assigned goals, the manager receives the rewards associated with those goals and makes the promised payments to the workers. To generate optimal contracts, the manager must infer the workers' minds and learn a good policy of goal and reward assignment. Conventional approaches of mechanism design tackle similar problems by imposing strong assumptions (e.g., skill/preference distributions, task dependencies, etc) to find an analytic solution. In contrast, we aim to train a manager using reinforcement learning to i) assess minds of workers (skills, preferences, intentions, etc.) on the fly, ii) to optimally assign contracts to maximize a collaborative reward, and iii) is adapted to diverse and even evolving workers and environments. For this, we propose a novel framework -Mind-aware Multi-agent Management Reinforcement Learning (M 3 RL), which entails both agent modeling for estimating workers' minds and policy learning for contract generation. For agent modeling, we infer workers' identities by their performance history, and track their internal states with a mind tracker trained by imitation learning (IL). For contract generation, we apply deep reinforcement learning (RL) to learn goal and bonus assignment policies. To improve the learning efficiency and adaptation, we also propose high-level successor representation (SR) learning BID17 and agent-wise -greedy exploration. As a proof of concept, we evaluate our approach in two environments: Resource Collection and Crafting in 2D Minecraft, to simulate multi-agent management problems. The setup and underlying assumptions are designed to mimic real world problems, where workers are not compelled to reveal their true preferences and skills, and there may be dependency between tasks ing in delayed and sparse reward signals. Workers may also be deceitful (e.g., accepting a contract even when the assigned goal is unreachable). Our experiments demonstrate that the manager trained by our approach can i) estimate the mind of each worker from the recent behaviors, ii) motivate the workers to finish less preferable or intermediate tasks by assigning the right bonuses, iii) is adaptive to changing teams, e.g., change of members and/or change of workers' skills and preferences, iv) and has good generalization in different team sizes and novel environments. We have conducted substantial ablation studies by removing the key components, including IL, SR, agent-wise -greedy exploration, and performance history. Our approach shows a consistent performance in standard settings as well as in more challenging ones where workers' policies are stochastic and sub-optimal, or there are multiple levels of bonuses required to motivate workers. Multi-agent reinforcement learning. For collaboration problems, common multi-agent reinforcement learning BID19 BID7 usually trains agents BID26 BID11 BID28 BID27 BID20 so that they will jointly maximize a shared reward. There also have been work on contributing different credits to agents by factorized value functions BID16 BID12 BID35 BID30, but the spontaneous collaboration assumption is still required. In contrast, we instead train a manager to manage multiple self-interested workers for an optimal collaboration. Principal-agent problems. Our problem setup is closely related to principal-agent problems BID18 ) (or moral hazard problems BID14) in economics. Our manager and workers can be considered as the principal and agents respectively, where agents and principal have different objectives, and the principal needs to provide the right incentives to ensure that the agents make the best choices for what the principal delegates. These problems face similar technical challenges as our problem setup, e.g., information asymmetry between principals and agents, how to setup incentive cost, how to infer agents types, how to monitor their behaviors, etc. Traditional approaches in economics BID25 BID15 BID32 build mathematical models to address these issues separately in stateless games, often with the assumption that the utility functions and the behavior patterns of the agents are known, leading to complicated models with many tunable parameters. In comparison, our paper provides a practical end-to-end computational framework to address this problem in a data-driven way without any assumption about the agents' utilities and their decision making processes. Moreover, this framework is adaptive to changes of agents preferences and capabilities, which very few papers in economics have addressed. We also evaluate our approach in more complex game settings than the ones in the current economics literature. Mechanism design. Similar to our problem setting, mechanism design also tackles problems where agents have different and private preferences BID24 BID8. Its core idea is to set up rules so that the agents will truthfully reveal their preferences for their own interests, and ultimately an optimal collective outcome can be achieved. Our work differs from mechanism design in several ways. First, in addition to preferences, we also acknowledge the fact that agents may have different skills. Second, mechanism design does not consider sequential decision problems, whereas we have to dynamically change the contracts over time. Optimal reward design. The contract generation in our work can be seen as reward design. Some prior work has proposed optimal reward design approaches BID41 BID40 BID33 BID31, where a teacher designs the best reward so that the student will learn faster or alter its policy towards the target policy. In contrast, we try to use deep RL to train optimal reward design policies to manage multi-agents in more complex tasks. Meta-learning. Our work also resembles meta-learning BID37 BID10, which typically aims at learning a meta strategy for multiple tasks BID22 BID9 BID13 BID38 BID39 BID3 with good sample efficiency, or for a fast adaptation BID0. The meta-learning in this paper is for addressing the problem of ad-hoc teaming BID6 BID34 by training from a limited set of worker population. Theory of Mind. Our agent modeling is inspired by the prior work on computational theory of mind, where both Bayesian inference BID4 ) and end-to-end training BID29 have been applied to understand a single agent's decision making by inferring their minds. In this work, we extend this to optimal multi-agent management by understanding agents' minds. In an environment, there is a set of goals G corresponding to several tasks, N self-interested workers with different minds, and a manager which can observe workers' behaviors but is agnostic of their true minds. Different from the common Markov game setting for MARL in prior work BID19, we use an independent Markov Decision Process (MDP), i.e., S i, A i, R i, T i, ∀i ∈ N, to model each worker, where S i and A i are the state space and action space, R i: S i × G i → R is the reward function, and T i: S i × A i → S i is the state transition probabilities. For achieving goals, a worker has its own policy π i: S i × G i → A i. We define the key concepts in this work as follows. A contract is a combination of goal and bonus assignment initiated by the manager to a specific worker. For simplicity, we consider discrete bonuses sampled from a finite set B. Thus, for worker i at time t, it will receive a contract defined as (g Worker's mind. We model a worker's mind by its preferences, intentions, and skills. We do not study worker agents' beliefs in this paper, which we leave as future work. Preference. A worker's preference is formally defined as its bounded internal utilities of achieving different goals, u i = (u ig : g ∈ G), where 0 ≤ u ig ≤ u max. Combined with received contract, the worker agent's reward function can be defined as DISPLAYFORM0 where s g is the goal state. Intention. The intention of a worker is the goal it is pursuing at any time, i.e., I t i ∈ G, which is not fully revealed to the manager. Based on the reward defined in Eq., there are multiple ways to choose the goal. For a rational worker who is clear about its skills, it will choose the goal by maximizing expected return. I.e., I DISPLAYFORM1 ], where 0 < γ i ≤ 1 is its discount factor. However, this requires a worker to have a good estimate of its skills and to be honest, which is not always true. E.g., a worker may want to pursue some valuable goal that it can not reach. So an alternative way is to maximize the utility instead: DISPLAYFORM2. This will make a worker's behavior more deceptive as it may agree to pursue a goal but will rarely produce a fruitful . In this work, we focus on the second way to achieve a more realistic simulation. After determine which goal to pursue, a worker will decide whether to sign the assigned contact. We denote this by d t i ∈ {0, 1}, where d t i = 1 means that worker i signs the contract given at time t. Skill. The skill of a worker is jointly determined by its state transition probabilities T i and its policy conditioned on its intention, i.e., π i (·|s DISPLAYFORM3 Manager's objective. The manager in our setting has its own utility v = (v g : g ∈ G), where v g ≥ 0 is the utility of achieving goal g. To maximize its gain, the manager needs to assign contracts to workers optimally. For the sake of realism, we do not assume that the manager knows for sure if a worker agent is really committed to the assignment. The only way to confirm this is to check whether the goal achieved by the worker is consistent with its last assigned goal. If so, then the manager will gain certain reward based on its utility of that goal and pay the promised bonus to the worker. Thus, we may define the manager's reward function as: DISPLAYFORM4 where S t+1 = {s t+1 i: i = 1, · · ·, N} is the collective states of all present worker agents at time t + 1. The objective of the manager is to find optimal contract generation to maximize its expected return E[∞ t=0 γ t r t], where 0 < γ ≤ 1 is the discount factor for the manager. Note that the manager may get the reward of a goal for multiple times if workers reach the goal respectively. Population of worker agents. The trained manager should be able to manage an arbitrary composition of workers rather than only specific teams of workers. For this, we maintain a population of worker agents during training, and sample several ones from that population in each episode as the present workers in each episode. The identities of these workers are tracked across episodes. In testing, we will sample workers from a new population that has not been seen in training. Our approach has three main components as shown in Figure 2: i) performance history module for identification, ii) mind tracker module for agent modeling, and iii) manager module for learning goal and bonus assignment policies. We introduce the details of these three components as follows. To model a worker's mind, we first need to infer its identity so that the manager can distinguish it from other agents. Previous work BID29 typically identifies agents via their trajectories in recent episodes. This only works when diverse past trajectories of agents are available beforehand. However, this is impractical in our problem as the past trajectories of a worker depends on the manager's policy, and thus are highly correlated and can hardly cover all aspects of that agent. DISPLAYFORM0 Figure 2: Overview of our network architecture. In this work, we propose performance history for agent identification, which is inspired by the upper confidence bound (UCB) algorithm BID2 for multi-bandit arm (MAB) problems. Formally, the performance history of worker i is a set of matrices DISPLAYFORM0 is an empirical estimation of the probability of worker i finishing goal g within t steps after signing the contract if promised with a bonus of b. We discuss how to update this estimate in Algorithm 1. These matrices are then flatten into a vector and we encode it to a history representation, h i, for worker i. With identification, the manager uses an independent mind tracker module with shared weights to update its belief of a worker's current mental state online by encoding both current and past information: DISPLAYFORM1 t} is a trajectory of the worker's behavior and the contracts it has received upon current time t in the current episode. For contract generation, the manager has to consider all present workers as a context. Thus, we encode each worker's information and pool them over to obtain a context representation, i.e., In addition to learning policies for individual workers, we also want the manager to estimate the overall productivity of a team. A common choice in previous literature (e.g., BID20) is to directly learn a centralized value function based on the context. However, this is not informative in our case, as the final return depends on achieving multiple goals and paying different bonuses. It is necessary to disentangle goal achievements, bonus payments, and the final net gain. To this end, we adopt the idea of successor representation (SR) BID17; BID5 BID21, but use it to estimate the expectation of accumulated goal achievement and bonus payment in the future instead of expected state visitation. By defining two vectors φ g (c t) and φ b (c t) indicating goal achievement and bonus payment at time t respectively, we may define our high-level SR, Φ g and DISPLAYFORM0 We discuss the details in Appendix A.1. For a joint training of these three modules, we use advantage actor-critic (A2C) BID23 to conduct on-policy updates, and learn SR similar to BID17. In addition, we also use imitation learning (IL) to improve the mind tracker. In particular, we predict a worker's policy based on its mental state representation, i.e.,π(·|s), which is learned by an additional cross-entropy loss for action prediction. Section A.2 summarizes the details. As our experimental in Section 5 and Appendix C show, in difficult settings such as random preferences and multiple bonus levels, the policies based on the mental state representation trained with IL have a much better performance than the ones without it. As the manager is agnostic of workers' minds, it is important to equip the manager with a good exploration strategy to fully understand each worker's skills and preferences. A common exploration strategy in RL is -greedy, where an agent has a chance of to take random actions. However, this may cause premature ending of contracts where a worker does not have sufficient amount of time to accomplish anything. Therefore, we adopt an agent-wise -greedy exploration, where a worker has as a chance of to be assigned with a random goal at the beginning of an episode and the manager will never change that goal assignment throughout the whole episode. In this way, it is easier for a manager to understand why or why not a worker is able to reach an assigned goal. The details can be seen from the rollout procedure (Algorithm 1) in Appendix B. We introduce the general task settings as follows. Note that without additional specification, workers are implemented as rule-based agents (detailed in Appendix D.2). In Resource Collection, the goals are defined as collecting certain type of resources. There are 4 types of resources on a map FIG4 ) and the total quantity is 10. A worker can find any resources but only has the skills to dig out certain types of resources. Note that it may not be skilled at collecting its preferred resources. We consider three different settings:• S1: Each agent can collect up to three types of resources including its preferred type.• S2: Each agent can only collect one type of resource which may or may not be its preferred one.• S3: Similar to S2, except that an agent has a different random preference in each episode and thus its preference can not be inferred from history. A worker can take five actions: "move forward", "turn left", "turn right", "collect", and "stop", and its skill is reflected by the effect of taking the "collect' action. For workers, the internal utility of a resource is 1 if it is preferred; otherwise it is 0. The manager receives a reward of 3 for every resource collected under the contracts, and can choose to pay a worker with a bonus of 1 or 2. Different from previous work BID1 where all items can be directly crafted from raw materials, we consider three-level recipes FIG4 ): crafting a top-level item requires crafting certain intermediate item first. There are four work stations (colored blocks) for crafting the four types of items respectively. For the manager, each top-level item is worth a reward of 10, but collecting raw materials and crafting intermediate items do not have any reward. Note that certain materials are needed for crafting both top-level items, so the manager must strategically choose which one to craft. In each episode, there are raw materials sufficient for crafting one to two toplevel items. All collected materials and crafted items are shared in a common inventory. We define 8 goals including collecting raw materials and crafting items. Each worker prefers one of the collecting goals (the internal utility is 1), and is only capable of crafting one type of items. We expands the action space in Section 5.1.1 to include "craft", which will only take effect if it has the ability of crafting the intended item and there are sufficient materials and/or intermediate items. The manager can choose a bonus from 0 to 2 for the contracts, where 0 means no employment. For comparison, we have evaluated the following baselines:• Ours w/o SR: Learning a value function directly w/o successor representations.• Ours w/o IL: Removing action prediction loss.• Temporal -greedy: Replacing the agent-wise exploration with conventional -greedy exploration.• Agent identification using recent trajectories: Encoding an agent's trajectories in the most recent 20 episodes instead of its performance history, which is adopted from Rabinowitz et al. • UCB: Applying UCB BID2 by defining the management problem as N multi-armed bandit sub-problems, each of which is for a worker agent. In each MAB sub-problem, pulling an arm is equivalent to assigning a specific goal and payment combination to a worker agent (i.e., there are |G| · |B| arms for each worker agent).• GT types known: Revealing the ground-truth skill and preference of each worker and removing the performance history module, which serves as an estimation of the upper bound performance. During training, we maintain a population of 40 worker agents. In each episode, we sample a few of them (4 workers in Resource Collection and 8 workers in Crafting). All approaches we have evaluated follow the same training protocol. The learning curves shown in FIG5 demonstrate that ours consistently performs the best in all settings, and its converged rewards are comparable to the one trained using ground-truth agent types as part of the observations. Moreover, in more difficult settings, e.g., S3 of Resource Collection and Crafting, the benefits of IL, SR, agent-wise -greedy exploration, and the history representations based on the performance history are more significant. In particular, when there are tasks that do not have any reward themselves such as in Crafting, SR and IL appear to offer the most critical contributions. Without them, the network hardly gets any training signals. In all cases, the agent identification by encoding recent trajectories learns extremely slowly in Resource Collection and fails to learn anything at all in Crafting. In real world scenarios, the population of worker agents and their skills may evolve over time, which requires the manager to continuously and quickly adapt its policy to the unforeseeable changes through a good exploration. Thus we compare our agent-wise -greedy exploration with the temporal -greedy exploration in two cases: i) training with a population where workers' skills change drastically after 100,000 episodes (the manager does not know when and which workers' skill sets have been updated), and ii) testing with a team where 75% of the workers will be replaced with new ones after every 2,000 episodes. Both strategies keep the same constant exploration coefficient, i.e., = 0.1. To have a better sense of the upper bound in the testing case, we also show the performance of the baseline that knows ground-truth agent information where no exploration is need. The of the two cases are demonstrated in Figure 5 and in Figure 6 respectively. In the first case, there are moments when the significant change in a population's skill distribution (i.e., how many workers can reach a specific goal) will need the manager to greatly change its policy. E.g., the first two changes in Figure 5a in new types of resources being collected; the changes in Figure 5b force the team to craft a different type of top-level item. In such cases, our agentwise -greedy exploration significantly improves the learning efficiency and increases the converged rewards. When the change is moderate, the policy learned by ours is fairly stable. In the second case, the managers trained by the three methods achieve similar converged rewards in training. While the converged reward of our approach is slightly lower than the upper bound due to exploration, it allows the manager to quickly adapt itself to a new team where it has never seen the most team members. The temporal -greedy on the other hand never achieves a comparable reward even though its performance is comparable to ours when managing a fixed population. We also want the manager's policy to have good generalization in novel scenarios unseen in training, which, in our problems, has two aspects: i) generalization in different numbers of present worker agents, and ii) generalization in new environments. It can be seen from FIG7 that as the number of workers increases, the manager achieves higher reward until it hits a plateau. Our approach consistently performs better in all settings. It even gains higher rewards than the one with groundtruth does when there are fewer workers. We also add a few walls to create novel environments unseen in training. With the additional obstacles, workers' paths become more complex, which increases the difficulty of inferring their true minds. As suggested by Figure 8, the performance indeed decreases the most in S3 of Resource Collection where online intention inference is critical as the workers do not have fixed preferences. So far, we have only considered rule-based worker agents with deterministic plans. To see if our approach can handle stochastic and sub-optimal worker policies, we may randomize certain amount of actions taken by the workers (Figure 9) and train a manager with these random policies. When the randomness is moderate (e.g., ≤ 20%), the performance is still comparable to the one without random actions. As randomness increases, we start to see larger decrease in reward. In Crafting specifically, random policies make the workers unlikely to achieve assigned goals within the time limit, thus the manager may never get top-level items if the policies are too random. More . In addition to the main experimental discussed above, we further test our approach from different perspectives: i) showing the effect of the minimum valid period of a contract (i.e., constraints for the manager's commitment), ii) multiple bonus levels, and iii) training RL agents as workers. We summarize these in Appendix C. In this paper, we propose Mind-aware Multi-agent Management Reinforcement Learning (M 3 RL) for solving the collaboration problems among self-interested workers with different skills and preferences. We train a manager to simultaneously infer workers' minds and optimally assign contracts to workers for maximizing the overall productivity, for which we combine imitation learning and reinforcement learning for a joint training of agent modeling and management policy optimization. We also improve the model performance by a few techniques including learning high-level successor representation, agent-wise -greedy exploration, and agent identification based on performance history. Results from extensive experiments demonstrate that our approach learns effectively, generalizes well, and has a fast and continuous adaptation. We define two vectors indicating the goal achievement and bonus payment at time t: DISPLAYFORM0. Let w = (b : b ∈ B) be the weights for different bonus payments, then the reward for the manager at the current moment can be written as DISPLAYFORM1. Following the typical SR definition, we define our high-level SR as DISPLAYFORM2 and DISPLAYFORM3 Thus, the value function can be written as DISPLAYFORM4 A.2 DETAILS OF LEARNINGThe policy gradient for the goal assignment is: DISPLAYFORM5 where A(c t) is the advantage estimation defined as A(c DISPLAYFORM6 The successor representations may be updated by the following gradient: DISPLAYFORM7 For imitation learning, we use the following cross-entropy loss: DISPLAYFORM8 Note that the gradient from IL will be combined with Eq. 6, Eq. 7, and Eq. 8 to update corresponding parameters (see Algorithm 2 in Appendix B for details).In tasks where there the unknown dependency may introduce a large cost in the beginning of training, the manager's exploration may be restricted as the policy becomes too conservative in spending, which is common in many real world scenarios. To encourage exploration in these tasks, we adopt a two-phase learning curriculum. Namely, we optionally conduct a warm-up phase before the standard learning described above. In this warm-up phase, we give loans to the manager to cover its cost (i.e., setting the total payments to be zero when optimizing the networks). In practice, we apply this only to Crafting, where we set a fixed number of episodes at the beginning of training to be the warm-up phase. Note that this only apply to the optimization; we still need to deduct the payments from the rewards as the actually outcomes (this is equivalent to paying back the loans). We summarize the rollout algorithm and the learning algorithm in Algorithm 1 and Algorithm 2 respectively. The manager's commitment is defined as the shortest time a contract must remain unchanged, which essentially constrains how frequent the manager can change its goal and bonus assignment. While short-term commitment allows the manager to quickly update contracts once it has a better mind estimation or once a goal has been reached, long-term commitment often leads to a more accurate skill assessment when the tasks are difficult (e.g., crafting high-level items depends on the of other tasks and thus needs a longer time). This is supported by the in FIG9: shorter commitment works better in Resource Collection while Crafting needs a longer commitment. Note that the commitment constraint is 1 step and 10 steps for Resource Collection and Crafting respectively in all other experiments. In previous experiments, the internal utility of goals for a worker agent is either 0 or 1. Here, we sample the internal utility from 0 to 3. Consequently, the manager needs to select the right bonus from multiple choices to pay each worker (i.e., a bonus from 1 to 4 for Resource Collection and a bonus from 0 to 4 for Crafting). In Resource Collection, the manager will get a reward of 5 for every collected resource; in crafting, the reward for a top-level item is still 10. As shown in FIG10, the advantage of our approach is even more significant compared to the ones in single payment level. Finally, we train a population of 40 RL worker agents for Resource Collection, where each one is trained with only one goal, and for each goal we train 10 agents using different random seeds. This creates a population with similar skill distributions as in S2, but with very different policies. FIG11 suggests that training to manage RL agents is slower as their policies are less predictable and less rational, but our approach can still gradually learn a good policy whose performance is comparable to the one using rule-based worker agents. Performance History Module. We flatten the matrices in worker's performance history P i and concatenate them together to get a single vector. We then encode this vector into a 128-dim history representation h i.Mind Tracker Module. We represent the state of a worker by multiple channels corresponding to different types of items. We also use four channels to indicate its orientation. We augment the state with additional |A||G||B| channels, where a channel is either all ones or all zeros for indicating the action it takes, and the goal and bonus it receives. We then encode the state into a 128-dim hidden state by a convolutional layer with 64 channels and kernels of 1 × 1, a fully connected (FC) layer (128-dim), and an LSTM with 128 hidden units. We fuse this vector with the history representation h i. Specifically, we adopt an attention-based mechanism for the fusion, where we first get an attention vector (128-dim) from the history representation by an FC layer with sigmoid activation, and then do element-wise product between the attention vector and the hidden state from the LSTM. The fused vector becomes m t i. This can be formally written as m is element-wise product. We fuse it with the state using the same mechanism: f (φ(s) is the state encoding. By feeding the fused vector to an FC layer with softmax activation, we may get the predicted worker policy. Manager Module. For each worker, we concatenate its mind representation and history representation together and fuse it with the worker's state using the attention-based mechanism where the attention vector comes from the concatenated vector. By pooling over these fused vectors of individual workers, we can get the context vector, from which we construct the two successor representations by two separate FC layers. Here, we use average pooling, but one may also use other pooling mechanisms. Finally, for each worker, we concatenate the context vector with its fused vector we obtained before pooling, and consequently get the goal policy and bonus policy by two FC layers with softmax activation. All modules are trained with RMSProp BID36 ) using a learning rate of 0.0004. Each rule-based worker finds a shortest path to the nearest location related to a goal, and its skill is defined as the post effect of its "collect" and "craft" actions. In particular, for collecting certain resource/material, it will go to the closest one that has the same type as the goal indicates and is currently not being collected by other agents, whereas for crafting an item, it will go to the corresponding work station if it is currently unoccupied. If a worker can perform collecting tasks, then after it takes "collect" action, the item will be collected from the map and appears in the inventory; otherwise no real effect will appear. This applies to crafting tasks as well, except in crafting, task dependencies must also be satisfied before "craft" action can take real effect. When considering random actions, for each step, we sample a random action with the specified chance to replace the action from the rule-based plan. We implement all RL worker agents using the same network architecture, where an agent's state is augmented by additional channels to include the reward for each goal (i.e., |G||B| channels). We use a convolution layer with 64 channels and kernels of 1 × 1 to encode the state, and feed it to an 128-dim FC layer and then an LSTM with a 128-dim hidden state. We then predict the policy using an FC layer with softmax activation based on the hidden state from the LSTM. For each goal, we train 10 RL worker agents using 10 random seeds. For each episode, we randomly assign a reward from b ∈ B to an agent as the hypothetical reward it may receive from a manager. We then set the corresponding channel to be all ones and set the remaining |G||B| − 1 channels to be all zeros. Note that we assume all RL workers have the ability to perform "collect" and "craft" actions.

Represent the short summary for finding the corresponding scientific article (long)

We propose Mind-aware Multi-agent Management Reinforcement Learning (M^3RL) for training a manager to motivate self-interested workers to achieve optimal collaboration by assigning suitable contracts to them.

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems