Question: Represent the scientific article to find a short TLDR

Question: We present a new algorithm to train a robust neural network against adversarial attacks. Our algorithm is motivated by the following two ideas. First, although recent work has demonstrated that fusing randomness can improve the robustness of neural networks (Liu 2017), we noticed that adding noise blindly to all the layers is not the optimal way to incorporate randomness. Instead, we model randomness under the framework of Bayesian Neural Network (BNN) to formally learn the posterior distribution of models in a scalable way. Second, we formulate the mini-max problem in BNN to learn the best model distribution under adversarial attacks, leading to an adversarial-trained Bayesian neural net. Experiment demonstrate that the proposed algorithm achieves state-of-the-art performance under strong attacks. On CIFAR-10 with VGG network, our model leads to 14% accuracy improvement compared with adversarial training (Madry 2017) and random self-ensemble under PGD attack with 0.035 distortion, and the gap becomes even larger on a subset of ImageNet. Deep neural networks have demonstrated state-of-the-art performances on many difficult machine learning tasks. Despite the fundamental breakthroughs in various tasks, deep neural networks have been shown to be utterly vulnerable to adversarial attacks BID32 BID11. Carefully crafted perturbations can be added to the inputs of the targeted model to drive the performances of deep neural networks to chance-level. In the context of image classification, these perturbations are imperceptible to human eyes but can change the prediction of the classification model to the wrong class. Algorithms seek to find such perturbations are denoted as adversarial attacks BID5 BID4 BID28, and some attacks are still effective in the physical world BID17 BID9. The inherent weakness of lacking robustness to adversarial examples for deep neural networks brings out security concerns, especially for security-sensitive applications which require strong reliability. To defend from adversarial examples and improve the robustness of neural networks, many algorithms have been recently proposed BID27 BID37 BID17 BID12. Among them, there are two lines of work showing effective on medium-sized data (e.g., CIFAR-10). The first line of work uses adversarial training to improve robustness, and the recent algorithm proposed in BID25 has been recognized as one of the most successful defenses, as shown in. The second line of work adds stochastic components in the neural network to hide gradient information from attackers. In the black-box setting, stochastic outputs can significantly increase query counts for attacks using finite-difference techniques BID5, and even in the white-box setting the recent Random Self-Ensemble (RSE) approach proposed by BID23 achieves similar performance to Madry's adversarial training algorithm. In this paper, we propose a new defense algorithm called Adv-BNN. The idea is to combine adversarial training and Bayesian network, although trying BNNs in adversarial attacks is not new (e.g. BID20 BID10 BID30), and very recently BID36 also tried to combine Bayesian learning with adversarial training, this is the first time we scale the problem to complex data and our approach achieves better robustness than previous defense methods. The contributions of this paper can be summarized below:• Instead of adding randomness to the input of each layer (as what has been done in RSE), we directly assume all the weights in the network are stochastic and conduct training with techniques commonly used in Bayesian Neural Network (BNN).• We propose a new mini-max formulation to combine adversarial training with BNN, and show the problem can be solved by alternating between projected gradient descent and SGD.• We test the proposed Adv-BNN approach on CIFAR10, STL10 and ImageNet143 datasets, and show significant improvement over previous approaches including RSE and adversarial training. Notations A neural network parameterized by weights w ∈ R d is denoted by f (x; w), where x ∈ R p is an input example and y is the corresponding label, the training/testing dataset is D tr/te with size N tr/te respectively. When necessary, we abuse D tr/te to define the empirical distribu- DISPLAYFORM0 δ(x i)δ(y i), where δ(·) is the Dirac delta function. x o represents the original input and x adv denotes the adversarial example. The loss function is represented as f (x i ; w), y i, where i is the index of the data point. Our approach works for any loss but we consider the cross-entropy loss in all the experiments. The adversarial perturbation is denoted as ξ ∈ R p, and adversarial example is generated by x adv = x o + ξ. In this paper, we focus on the attack under norm constraint BID25, so that ξ ≤ γ. In order to align with the previous works, in the experiments we set the norm to · ∞. The Hadamard product is denoted as. In this section, we summarize related works on adversarial attack and defense. Attack: Most algorithms generate adversarial examples based on the gradient of loss function with respect to the inputs. For example, FGSM BID11 perturbs an example by the sign of gradient, and use a step size to control the ∞ norm of perturbation. BID17 proposes to run multiple iterations of FGSM. More recently, C&W attack BID3 formally poses attack as an optimization problem, and applies a gradient-based iterative solver to get an adversarial example. Both C&W attack and PGD attack BID25 have been frequently used to benchmark the defense algorithms due to their effectiveness. Throughout, we take the PGD attack as an example, largely following BID25.The goal of PGD attack is to find adversarial examples in a γ-ball, which can be naturally formulated as the following objective function: DISPLAYFORM0 Starting from x 0 = x o, PGD attack conducts projected gradient descent iteratively to update the adversarial example: DISPLAYFORM1 where Π γ is the projection to the set {x| x−x o ∞ ≤ γ}. Although multi-step PGD iterations may not necessarily return the optimal adversarial examples, we decided to apply it in our experiments, following the previous work of BID25 ). An advantage of PGD attack over C&W attack is that it gives us a direct control of distortion by changing γ, while in C&W attack we can only do this indirectly via tuning the regularizer. Since we are dealing with networks with random weights, we elaborate more on which strategy should attackers take to increase their success rate, and the details can be found in. In random neural networks, an attacker seeks a universal distortion ξ that cheats a majority of realizations of the random weights. This can be achieved by maximizing the loss expectation DISPLAYFORM2 Here the model weights w are considered as random vector following certain distributions. In fact, solving to a saddle point can be done easily by performing multi-step (projected) SGD updates. This is done inherently in some iterative attacks such as C&W or PGD discussed above, where the only difference is that we sample new weights w at each iteration. Defense: There are a large variety of defense methods proposed in recent years, e.g. denoiser based HGD BID21 and randomized image preprocessing BID34. Readers can find more from BID18. Below we select two representative ones that turn out to be effective to white box attacks. They are the major baselines in our experiments. The first example is the adversarial training BID32 BID11. It is essentially a data augmentation method, which trains the deep neural networks on adversarial examples until the loss converges. Instead of searching for adversarial examples and adding them into the training data, BID25 proposed to incorporate the adversarial search inside the training process, by solving the following robust optimization problem: DISPLAYFORM3 where D tr is the training data distribution. The above problem is approximately solved by generating adversarial examples using PGD attack and then minimizing the classification loss of the adversarial example. In this paper, we propose to incorporate adversarial training in Bayesian neural network to achieve better robustness. The other example is RSE BID23, in this algorithm the authors proposed a "noise layer", which fuses input features with Gaussian noise. They show empirically that an ensemble of models can increase the robustness of deep neural networks. Besides, their method can generate an infinite number of models on-the-fly without any additional memory cost. The noise layer is applied in both training and testing phases, so the prediction accuracy will not be largely affected. Our algorithm is different from RSE in two folds: 1) We add noise to each weight instead of input or hidden feature, and formally model it as a BNN. 2) We incorporate adversarial training to further improve the performance. The idea of BNN is illustrated in FIG0. Given the observable random variables (x, y), we aim to estimate the distributions of hidden variables w. In our case, the observable random variables correspond to the features x and labels y, and we are interested in the posterior over the weights p(w|x, y) given the prior p(w). However, the exact solution of posterior is often intractable: notice that p(w|x, y) = -but the denominator involves a high dimensional integral BID1, hence the conditional probabilities are hard to compute. To speedup inference, we generally have two approaches-we can either sample w ∼ p(w|x, y) efficiently without knowing the closed-form formula through, for example, Stochastic Gradient Langevin Dynamics (SGLD) BID33 ), or we can approximate the true posterior p(w|x, y) by a parametric distribution q θ (w), where the unknown parameter θ is estimated by minimizing KL q θ (w) p(w|x, y) over θ. For neural network, the exact form of KL-divergence can be unobtainable, but we can easily find an unbiased gradient estimator of it using backward propagation, namely Bayes by Backprop BID2.Despite that both methods are widely used and analyzed in-depth, they have some obvious shortcomings, making high dimensional Bayesian inference remain to be an open problem. For SGLD and its extension (e.g. BID19), since the algorithms are essentially SGD updates with extra Gaussian noise, they are very easy to implement. However, they can only get one sample w ∼ p(w|x, y) in each minibatch iteration at the cost of one forward-backward propagation, thus not efficient enough for fast inference. In addition, as the step size η t in SGLD decreases, the samples become more and more correlated so that one needs to generate many samples in order to control the variance. Conversely, the variational inference method is efficient to generate samples since we know the approximated posterior q θ (w) once we minimized the KL-divergence. The problem is that for simplicity we often assume the approximation q θ to be a fully factorized Gaussian distribution: DISPLAYFORM0 Although our assumption has a simple form, it inherits the main drawback from mean-field approximation. When the ground truth posterior has significant correlation between variables, the approximation in will have a large deviation from true posterior p(w|x, y). This is especially true for convolutional neural networks, where the values in the same convolutional kernel seem to be highly correlated. However, we still choose this family of distribution in our design as the simplicity and efficiency are mostly concerned. In fact, there are many techniques in deep learning area borrowing the idea of Bayesian inference without mentioning explicitly. For example, Dropout BID31 ) is regarded as a powerful regularization tool for deep neural networks, which applies an element-wise product of the feature maps and i.i.d. Bernoulli or Gaussian r.v. B(1, α) (or N (1, α)). If we allow each dimension to have an independent dropout rate and take them as model parameters to be learned, then we can extend it to the variational dropout method BID16. Notably, learning the optimal dropout rates for data relieves us from manually tuning hyper-parameter on hold-out data. Similar idea is also used in RSE BID23, except that it was used to improve the robustness under adversarial attacks. As we discussed in the previous section, RSE incorporates Gaussian noise ∼ N (0, σ 2) in an additive manner, where the variance σ 2 is user predefined in order to maximize the performance. Different from RSE, our Adv-BNN has two degrees of freedom (mean and variance) and the network is trained on adversarial examples. In our method, we combine the idea of adversarial training BID25 with Bayesian neural network, hoping that the randomness in the weights w provides stronger protection for our model. To build our Bayesian neural network, we assume the joint distribution q µ,s (w) is fully factorizable (see FORMULA6), and each posterior q µi,si (w i) follows normal distribution with mean µ i and standard deviation exp(s i) > 0. The prior distribution is simply isometric Gaussian N (0 d, s 2 0 I d×d). We choose the Gaussian prior and posterior for its simplicity and closed-form KL-divergence, that is, for any two Gaussian distributions s and t, DISPLAYFORM0 Note that it is also possible to choose more complex priors such as "spike-and-slab" BID14 or Gaussian mixture, although in these cases the KL-divergence of prior and posterior is hard to compute and practically we replace it with the Monte-Carlo estimator, which has higher variance, ing in slower convergence rate BID15.Following the recipe of variational inference, we adapt the robust optimization to the evidence lower bound (ELBO) w.r.t. the variational parameters during training. First of all, recall the ELBO on the original dataset (the unperturbed data) can be written as DISPLAYFORM1 rather than directly maximizing the ELBO in FORMULA8, we consider the following alternative objective, DISPLAYFORM2 This is essentially finding the minima for each data point (x i, y i) ∈ D tr inside the γ-norm ball, we can also interpret as an even looser lower bound of evidence. So the robust optimization procedure is to maximize, i.e. DISPLAYFORM3 To make the objective more specific, we combine with FORMULA10 and get arg max DISPLAYFORM4 In our case, p(y|x DISPLAYFORM5] is the network output on the adversarial sample (x adv i, y i). More generally, we can reformulate our model as y = f (x; w)+ζ and assume the residual ζ follows either Logistic or Gaussian distribution depending on the specific problem, so that our framework includes both classification and regression tasks. We can see that the only difference between our Adv-BNN and the standard BNN training is that the expectation is now taken over the adversarial examples (x adv, y), rather than natural examples (x, y). Therefore, at each iteration we first apply a randomized PGD attack (as introduced in eq) for T iterations to find x adv, and then fix the x adv to update µ, s. When updating µ and s, the KL term in can be calculated exactly by, whereas the second term is very complex (for neural networks) and can only be approximated by sampling. Besides, in order to fit into the back-propagation framework, we adopt the Bayes by Backprop algorithm BID2. Notice that we can reparameterize w = µ + exp(s), where ∼ N (0 d, I d×d) is a parameter free random vector, then for any differentiable function h(w, µ, s), we can show that DISPLAYFORM6 Now the randomness is decoupled from model parameters, and thus we can generate multiple to form a unbiased gradient estimator. To integrate into deep learning framework more easily, we also designed a new layer called RandLayer, which is summarized in appendix. It is worth noting that once we assume the simple form of variational distribution, we can also adopt the local reparameterization trick BID16. That is, rather than sampling the weights w, we directly sample the activations and enjoy the lower variance during the sampling process. Although in our experiments we find the simple Bayes by Backprop method efficient enough. For ease of doing SGD iterations, we rewrite into a finite sum problem by dividing both sides by the number of training samples N tr µ *, s * = arg min DISPLAYFORM7 here we define g(µ, s) KL(q µ,s (w) p(w)) by the closed form solution, so there is no randomness in it. We sample new weights by w = µ + exp (s) in each forward propagation, so that the stochastic gradient is unbiased. In practice, however, we need a weaker regularization for small dataset or large model, since the original regularization in can be too large. We fix this problem by adding a factor 0 < α ≤ 1 to the regularization term, so the new loss becomes DISPLAYFORM8 In our experiments, we found little to no performance degradation compared with the same network without randomness, if we choose a suitable hyper-parameter α, as well as the prior distribution N (0, s 2 0 I). The overall training algorithm is shown in Alg. 1. To sum up, our Adv-BNN method trains an arbitrary Bayesian neural network with the min-max robust optimization, which is similar to BID25. As we mentioned earlier, even though our model contains noise and eventually the gradient information is also noisy, by doing multiple forward-backward iterations, the noise will be cancelled out due to the law of large numbers. This is also the suggested way to bypass some stochastic defenses in.Algorithm 1 Code snippet for training Adv-BNN 1: procedure pgd attack(x, y, w) 2:Perform the PGD-attack, omitted for brevity 3: procedure train(data, w) 4:Input: dataset and network weights w 5:for (x, y) in data do 6:x adv ← pgd attack(x, y, w) Generate adversarial images 7: DISPLAYFORM9 loss ce ← cross entropy(ŷ, y) Cross-entropy loss 10:loss kl ← kl divergence(w) KL-divergence following FORMULA7 Will it be beneficial to have randomness in adversarial training? After all, both randomized network and adversarial training can be viewed as different ways for controlling local Lipschitz constants of the loss surface around the image manifold, and thus it is non-trivial to see whether combining those two techniques can lead to better robustness. The connection between randomized network (in particular, RSE) and local Lipschitz regularization has been derived in BID23. Adversarial training can also be connected to local Lipschitz regularization with the following arguments. Recall that the loss function given data (x i, y i) is denoted as f (x i ; w), y i, and similarly the loss on perturbed data (x i + ξ, y i) is f (x i + ξ; w), y i ). Then if we expand the loss to the first order DISPLAYFORM10 we can see that the robustness of a deep model is closely related to the gradient of the loss over the input, i.e. ∇ xi f (x i), y i. If ∇ xi f (x i), y i is large, then we can find a suitable ξ such that ∆ is large. Under such condition, the perturbed image x i + ξ is very likely to be an adversarial example. It turns out that adversarial training directly controls the local Lipschitz value on the training set, this can be seen if we combine FORMULA1 with DISPLAYFORM11 Moreover, if we ignore the higher order term O(ξ 2) then becomes DISPLAYFORM12 In other words, the adversarial training can be simplified to Lipschitz regularization, and if the model generalizes, the local Lipschitz value will also be small on the test set. Yet, as BID22 indicates, for complex dataset like CIFAR-10, the local Lipschitz is still very large on test set, even though it is controlled on training set. The drawback of adversarial training motivates us to combine the randomness model with adversarial training, and we observe a significant improvement over adversarial training or RSE alone (see the experiment section below). In this section, we test the performance of our robust Bayesian neural networks (Adv-BNN) with strong baselines on a wide variety of datasets. In essence, our method is inspired by adversarial training BID25 and BNN BID2, so these two methods are natural baselines. If we see a significant improvement in adversarial robustness, then it means that randomness and robust optimization have independent contributions to defense. Additionally, we would like to compare our method with RSE BID23, another strong defense algorithm relying on randomization. Lastly, we include the models without any defense as references. For ease of reproduction, we list the hyper-parameters in the appendix. Readers can also refer to the source code on github. It is known that adversarial training becomes increasingly hard for high dimensional data BID29. In addition to standard low dimensional dataset such as CIFAR-10, we also did experiments on two more challenging datasets: 1) STL-10 (BID6, which has 5,000 training images and 8,000 testing images. Both of them are 96 × 96 pixels; 2) ImageNet-143, which is a subset of ImageNet BID7, and widely used in conditional GAN training BID26. The dataset has 18,073 training and 7,105 testing images, and all images are 64×64 pixels. It is a good benchmark because it has much more classes than CIFAR-10, but is still manageable for adversarial training. In the first experiment, we compare the accuracy under the white box ∞ -PGD attack. We set the maximum ∞ distortion to γ ∈ [0:0.07:0.005] and report the accuracy on test set. The are shown in Fig. 2. Note that when attacking models with stochastic components, we adjust PGD accordingly as mentioned in Section 2.1. To demonstrate the relative performance more clearly, we show some numerical in Tab. 1. No defense BNN Adv. training AdvBNN RSE Figure 2: Accuracy under ∞ -PGD attack on three different datasets: CIFAR-10, STL-10 and ImageNet-143. In particular, we adopt a smaller network for STL-10 namely "Model A" 1, while the other two datasets are trained on VGG.From Fig. 2 and Tab. 1 we can observe that although BNN itself does not increase the robustness of the model, when combined with the adversarial training method, it dramatically increase the testing accuracy for ∼10% on a variety of datasets. Moreover, the overhead of Adv-BNN over adversarial training is small: it will only double the parameter space (for storing mean and variance), and the total training time does not increase much. Finally, similar to RSE, modifying existing network Table 1: Comparing the testing accuracy under different levels of PGD attacks. We include our method, Adv-BNN, and the state of the art defense method, the multi-step adversarial training proposed in BID25. The better accuracy is marked in bold. Notice that although our Adv-BNN incurs larger accuracy drop in the original test set (where ξ ∞ = 0), we can choose a smaller α in so that the regularization effect is weakened, in order to match the accuracy.architectures into BNN is fairly simple, we only need to replace Conv/BatchNorm/Linear layers by their variational version. Hence we can easily build robust models based on existing ones. Is our Adv-BNN model susceptible to transfer attack? we answer this question by studying the affinity between models, because if two models are similar (e.g. in loss landscape) then we can easily attack one model using the adversarial examples crafted through the other. In this section, we measure the adversarial sample transferability between different models namely None (no defense), BNN, Adv. Train, RSE and Adv-BNN. This is done by the method called "transfer attack" BID24. Initially it was proposed as a black box attack algorithm: when the attacker has no access to the target model, one can instead train a similar model from scratch (called source model), and then generate adversarial samples with source model. As we can imagine, the success rate of transfer attack is directly linked with how similar the source/target models are. In this experiment, we are interested in the following question: how easily can we transfer the adversarial examples between these five models? We study the affinity between those models, where the affinity is defined by DISPLAYFORM0 where ρ However, ρ A →B = ρ B →A is not necessarily true, so the affinity matrix is not likely to be symmetric. We illustrate the in FIG3.We can observe that {None, BNN} are similar models, their affinity is strong (ρ ≈ 0.85) for both direction: ρ BNN →None and ρ None →BNN. Likewise, {RSE, Adv-BNN, Adv. Train} constitute the other group, yet the affinity is not very strong (ρ ≈ 0.5∼0.6), meaning these three methods are all robust to the black box attack to some extent. Following experiments are not crucial in showing the success of our method, however, we still include them to help clarifying some doubts of careful readers. The first question is about sample efficiency, recall in prediction stage we sample weights from the approximated posterior and generate the label bŷ DISPLAYFORM0 In practice, we do not want to average over lots of forward propagation to control the variance, which will be much slower than other models during the prediction stage. Here we take ImageNet-143 data + VGG network as an example, to show that only 10∼20 forward operations are sufficient for robust and accurate prediction. Furthermore, the number seems to be independent on the adversarial distortion, as we can see in Fig. 4 (left). So our algorithm is especially suitable to large scale scenario. One might also be concerned about whether 20 steps of PGD iterations are sufficient to find adversarial examples. It has been known that for certain adversarial defense method, the effectiveness appears to be worse than claimed, if we increase the PGD-steps from 20 to 100. In Fig. 4 (right), we show that even if we increase the number of iteration to 1000, the accuracy does not change very much. This means that even the adversary invests more resources to attack our model, its marginal benefit is negligible. Figure 4: Left: we tried different number of forward propagation and averaged the to make prediction. We see that for different scales of perturbation γ ∈ {0, 0.01, 0.02}, choosing number of ensemble n = 10∼20 is good enough. Right: testing accuracy stabilizes quickly as #PGD-steps goes greater than 20, so there is no necessity to further increase the number of PGD steps. To conclude, we find that although the Bayesian neural network has no defense functionality, when combined with adversarial training, its robustness against adversarial attack increases significantly. So this method can be regarded as a non-trivial combination of BNN and the adversarial training: robust classification relies on the controlled local Lipschitz value, while adversarial training does not generalize this property well enough to the test set; if we train the BNN with adversarial examples, the robustness increases by a large margin. Admittedly, our method is still far from the ideal case, and it is still an open problem on what the optimal defense solution will be. We largely follow the guidelines of attacking networks with "obfuscated gradients" in. Specifically, we derive the algorithm for white box attack to random networks denoted as f (w;), where w is the (fixed) network parameters and is the random vector. Many random neural networks can be reparameterized to this form, where each forward propagation returns different . In particular, this framework includes our Adv-BNN model by setting w = (µ, s). Recall the prediction is made through "majority voting": DISPLAYFORM0 So the optimal white-box attack should maximize the loss on the ground truth label y *. That is, ξ * = arg max ξ E f (x + ξ; w,), y *,and then x adv x + ξ *. To do that we apply SGD optimizer and sampling at each iteration, DISPLAYFORM1 one can see the iteration approximately solves. It is very easy to implement the forward & backward propagation in BNN. Here we introduce the RandLayer that can seamlessly integrate into major deep learning frameworks. We take PyTorch as an example, the code snippet is shown in Alg. 1.

Represent the short summary for finding the corresponding scientific article (long)

We design an adversarial training method to Bayesian neural networks, showing a much stronger defense to white-box adversarial attacks

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: represent the article for finding a single-sentence tldr

Question: Federated learning distributes model training among a multitude of agents, who, guided by privacy concerns, perform training using their local data but share only model parameter updates, for iterative aggregation at the server. In this work, we explore the threat of model poisoning attacks on federated learning initiated by a single, non-colluding malicious agent where the adversarial objective is to cause the model to misclassify a set of chosen inputs with high confidence. We explore a number of strategies to carry out this attack, starting with simple boosting of the malicious agent's update to overcome the effects of other agents' updates. To increase attack stealth, we propose an alternating minimization strategy, which alternately optimizes for the training loss and the adversarial objective. We follow up by using parameter estimation for the benign agents' updates to improve on attack success. Finally, we use a suite of interpretability techniques to generate visual explanations of model decisions for both benign and malicious models and show that the explanations are nearly visually indistinguishable. Our indicate that even a highly constrained adversary can carry out model poisoning attacks while simultaneously maintaining stealth, thus highlighting the vulnerability of the federated learning setting and the need to develop effective defense strategies. Federated learning introduced by BID11 has recently emerged as a popular implementation of distributed stochastic optimization for large-scale deep neural network training. It is formulated as a multi-round strategy in which the training of a neural network model is distributed between multiple agents. In each round, a random subset of agents, with local data and computational resources, is selected for training. The selected agents perform model training and share only the parameter updates with a centralized parameter server, that facilitates aggregation of the updates. Motivated by privacy concerns, the server is designed to have no visibility into an agents' local data and training process. The aggregation algorithm is agnostic to the data distribution at the agents. In this work, we exploit this lack of transparency in the agent updates, and explore the possibility of a single malicious agent performing a model poisoning attack. The malicious agent's objective is to cause the jointly trained global model to misclassify a set of chosen inputs with high confidence, i.e., it seeks to introduce a targeted backdoor in the global model. In each round, the malicious agent generates its update by optimizing for a malicious objective different than the training loss for federated learning. It aims to achieve this by generating its update by directly optimizing for the malicious objective. However, the presence of a multitude of other agents which are simultaneously providing updates makes this challenging. Further, the malicious agent must ensure that its update is undetectable as aberrant. Contributions: To this end, we propose a sequence of model poisoning attacks, with the aim of achieving the malicious objective while maintaining attack stealth. For each strategy, we consider both attack strength as well as stealth. We start with malicious update boosting, designed to negate the combined effect of the benign agents, which enables the adversary to achieve its malicious objective with 100% confidence. However, we show that boosted updates can be detected as aberrant using two measures of stealth, accuracy checking on the benign objective and parameter update statistics. Observing that the only parameter updates that need to be boosted are those that con-tribute to the malicious objective, we design an alternating minimization strategy that improves attack stealth. This strategy alternates between training loss minimization and the boosting of updates for the malicious objective and is able to achieve high success rate on both the benign and malicious objectives. In addition, we show that estimating the other agents' updates improves attack success rates. Finally, we use a suite of interpretability techniques to generate visual explanations of the decisions made by a global model with and without a targeted backdoor. Interestingly, we observe that the explanations are nearly visually indistinguishable. This establishes the attack stealth along yet another axis of measurement and indicates that backdoors can be inserted without drastic changes in model focus at the input. In our experiments, we consider adversaries which only control a single malicious agent and at a given time step, have no visibility into the updates that will be provided by the other agents. We demonstrate that these adversaries can influence the global model to misclassify particular examples with high confidence. We work with both the and Adult Census 1, datasets and for settings with both 10 and 100 agents, our attacks are able to ensure the global model misclassifies a particular example in a target class with 100% confidence. Our alternating minimization attack further ensures that the global model converges to the same test set accuracy as the case with no adversaries present. We also show that a simple estimation of the benign agents' updates as being identical over two consecutive rounds aids in improving attack success. Related Work: While data poisoning attacks BID1 BID16 BID12 BID22 BID12 BID9 BID4 BID7 have been widely studied, model poisoning attacks are largely unexplored. A number of works on defending against Byzantine adversaries consider a threat model where Byzantine agents send arbitrary gradient updates BID2 BID5 BID13 BID23. However, the adversarial goal in these cases is to ensure a distributed implementation of the Stochastic Gradient Descent (SGD) algorithm converges to'suboptimal to utterly ineffective models', quoting from BID13. In complete constrast, our goal is to ensure convergence to models that are effective on the test set but misclassify certain examples. In fact, we show that the Byzantine-resilient aggregation mechanism'Krum BID2 is not resilient to our attack strategies (Appendix C). Concurrent work by BID0 considers multiple colluding agents performing poisoning via model replacement at convergence time. In contrast, our goal is to induce targeted misclassification in the global model by a single malicious agent even when it is far from convergence while maintaining its accuracy for most tasks. In fact, we show that updates generated by their strategy fail to achieve either malicious or benign objectives in the settings we consider. In this section, we formulate both the learning paradigm and the threat model that we consider throughout the paper. Operating in the federated learning paradigm, where model weights are shared instead of data, gives rise to the model poisoning attacks that we investigate. The federated learning setup consists of K agents, each with access to data D i, where DISPLAYFORM0 The total number of samples is i l i = l. Each agent keeps its share of the data (referred to as a shard) private, i.e. DISPLAYFORM1 li } is not shared with the server S. The objective of the server is to learn a global parameter vector w G ∈ R n, where n is the dimensionality of the parameter space. This parameter vector minimizes the loss 2 over D = ∪ i D i and the aim is to generalize well over D test, the test data. Federated learning is designed to handle non-i.i.d partitioning of training data among the different agents. Traditional poisoning attacks deal with a malicious agent who poisons some fraction of the data in order to ensure that the learned model satisfies some adversarial goal. We consider instead an agent who poisons the model updates it sends back to the server. This attack is a plausible threat in the federated learning setting as the model updates from the agents can (i) directly influence the parameters of the global model via the aggregation algorithm; and (ii) display high variability, due to the non-i.i.d local data at the agents, making it harder to isolate the benign updates from the malicious ones. Adversary Model: We make the following assumptions regarding the adversary: (i) there is exactly one non-colluding, malicious agent with index m (limited effect of malicious updates on the global model); (ii) the data is distributed among the agents in an i.i.d fashion (making it easier to discriminate between benign and possible malicious updates and harder to achieve attack stealth); (iii) the malicious agent has access to a subset of the training data D m as well as to auxiliary data D aux drawn from the same distribution as the training and test data that are part of its adversarial objective. Our aim is to explore the possibility of a successful model poisoning attack even for a highly constrained adversary. A malicious agent can have one of two objectives with regard to the loss and/or classification of a data subset at any time step t in the model poisoning setting: 1. Increase the overall loss: In this case, the malicious agent wishes to increase the overall loss on a subset D aux = {x i, y i} r i=1 of the data. The adversarial objective is in this setting is DISPLAYFORM0 DISPLAYFORM1. This corresponds to a targeted misclassification attempt by the malicious agent. In this paper, we will focus on malicious agents trying to attain the second objective, i.e. targeted misclassification. At first glance, the problem seems like a simple one for the malicious agent to solve. However, it does not have access to the global parameter vector w t G for the current iteration as is the case in standard poisoning attacks BID15 BID9 and can only influence it though the weight update δ t m it provides to the server S. The simplest formulation of the optimization problem the malicious agent has to solve such that her objective is achieved on the t th iteration is then DISPLAYFORM2 In order to illustrate how our attack strategies work with actual data and models, we use two qualitatively different datasets. The first is an image dataset, Fashion-MNIST 3 BID21 ) which consists of 28 × 28 grayscale images of clothing and footwear items and has 10 output classes. The training set contains 60,000 data samples while the test set has 10,000 samples. We use a Convolutional Neural Network achieving 91.7% accuracy on the test set for the model architecture. The second dataset is the UCI Adult dataset 4, which has over 40,000 samples containing information about adults from the 1994 US Census. The classification problem is to determine if the income for a particular individual is greater (class '0') or less (class '1') than $50, 000 a year. For this dataset, we use a fully connected neural network achieving 84.8% accuracy on the test set BID6 for the model architecture. Owing to space constraints, all for this dataset are in the Appendix. For both datasets, we study the case with the number of agents k set to 10 and 100. When k = 10, all the agents are chosen at every iteration, while with k = 100, a tenth of the agents are chosen at random every iteration. We run federated learning till a pre-specified test accuracy (91% for Fashion MNIST and 84% for the Adult Census data) is reached or the maximum number of time steps have elapsed (40 for k = 10 and 50 for k = 100). For most of our experiments, we consider the case when r = 1, which implies that the malicious agent aims to misclassify a single example in a desired target class. For both datasets, a random sample from the test set is chosen as the example to be misclassified. For the Fashion-MNIST dataset, the sample belongs to class'5' (sandal) with the aim of misclassifying it in class'7' (sneaker) and for the Adult dataset it belongs to class'0' with the aim of misclassifying it in class'1'. We begin by investigating baseline attacks which do not conform to any notion of stealth. We then show how simple detection methods at the server may expose the malicious agent and explore the extent to which modifications to the baseline attack can bypass these methods. In order to solve the exact optimization problem needed to achieve their objective, the malicious agent needs access to the current value of the overall parameter vector w t G, which is inaccessible. This occurs due to the nature of the federated learning algorithm, where S computes w t G once it has received updates from all agents. In this case, they have to optimize over an estimate of the value of w t G: DISPLAYFORM0 where f (·) is an estimator forŵ t G based on all the information I t m available to the adversary. We refer to this as the limited information poisoning objective. The problem of choosing a good estimator is deferred to Section 4 and the strategies discussed in the remainder of this section make the assumption thatŵ DISPLAYFORM1 In other words, the malicious agent ignores the effects of other agents. As we shall see, this assumption is often enough to ensure the attack works in practice. Using the approximation thatŵ DISPLAYFORM0 Depending on the exact structure of the loss, an appropriate optimizer can be chosen. For our experiments, we will rely on gradient-based optimizers such as SGD which work well for neural networks. In order to overcome the effect of scaling by α m at the server, the final updateδ t m that is returned, has to be boosted. Explicit Boosting: Mimicking a benign agent, the malicious agent can run E m steps of a gradientbased optimizer starting from w Implicit Boosting: While the loss is a function of a weight vector w, we can use the chain rule to obtain the gradient of the loss with respect to the weight update δ, i.e. ∇ δ L = α m ∇ w L. Then, initializing δ to some appropriate δ ini, the malicious agent can directly minimize with respect to δ. FIG1. The attack is clearly successful at causing the global model to classify the chosen example in the target class. In fact, after t = 3, the global model is highly confident in its (incorrect) prediction. The baseline attack using implicit boosting FIG3 ) is much less successful than the explicit boosting baseline, with the adversarial objective only being achieved in 4 of 10 iterations. Further, it is computationally more expensive, taking an average of 2000 steps to converge at each time step, which is about 4× longer than a benign agent. Since consistently delayed updates from the malicious agent might lead to it being dropped from the system in practice, we focus on explicit boosting attacks for the remainder of the paper as they do not add as much overhead. While the baseline attack is successful at meeting the malicious agent's objective, there are methods the server can employ in order to detect if an agent's update is malicious. We now discuss two possible methods and their implication for the baseline attack. We note that neither of these methods are part of the standard federated learning algorithm nor do they constitute a full defense at the server. They are merely metrics that may be utilized in a secure system. Accuracy checking: When any agent sends a weight update to the server, it can check the validation accuracy of w t i = w t−1 G + δ t i, the model obtained by adding that update to the current state of the global model. If the ing model has a validation accuracy much lower than that of the other agents, the server may be able to detect that model as coming from a malicious agent. This would be particularly effective in the case where the agents have i.i.d. data. In FIG1, the left plot shows the accuracy of the malicious model on the validation data (Acc. Mal) at each iteration. This is much lower than the accuracy of the global model (Acc. Global) and is no better than random for the first few iterations. Figure 3: Minimum and maximum L 2 distances between weight updates. For each strategy, we show the spread of L 2 distances between all the benign agents and between the malicious agent and the benign agents. Going from the baseline attack to the alternating minimization attack with and without distance constraints, we see that the gap in the spread of distances reduces, making the attack stealthier. The benign agents behave almost identically across strategies, indicating that the malicious agent does not interfere much with their training. Weight update statistics: There are both qualitative and quantitative methods the server can apply in order to detect weight updates which are malicious, or at the least, different from a majority of the other agents. We investigate the effectiveness of two such methods. The first, qualitative method, is the visualization of weight update distributions for each agent. Since the adversarial objective function is different from the training loss objective used by all the benign agents, we expect the distribution of weight updates to be very different. This is borne out by the representative weight update distribution at t = 4 observed for the baseline attack in FIG1. Compared to the weight update from a benign agent, the update from the malicious agent is much sparser and has a smaller range. This difference is more pronounced for later time steps (see FIG13 in Appendix B).The second, quantitative method uses the spread of pairwise L p distances between weight update vectors to identify outliers. At each time step, the server computes the pairwise distances between all the weight updates it receives, and flags those weight updates which are either much closer or much farther away than the others. In Figure 3, the spread of L 2 distances between all benign updates and between the malicious update and the benign updates is plotted. For the baseline attack, both the minimum and maximum distance away from any of the benign updates keeps decreasing over time steps, while it remains relatively constant for the other agents. This can enable detection of the malicious agent. To bypass the two detection methods discussed in the previous section, the malicious agent can try to simultaneously optimize over the adversarial objective and training loss for its local data shard Results: In practice, we optimize over batches of D m and concatenate each batch with the single instance {x, τ} to be misclassified, ensuring that the adversarial objective is satisfied. In fact, as seen in FIG1 in the plot on the right, the adversarial objective is satisfied with high confidence from the first time step t = 1. Since the entire weight update corresponding to both adversarial and training objectives is boosted, the accuracy of w t m on the validation is low throughout the federated learning process. Thus, this attack can easily be detected using the accuracy checking method. Further, while the weight update distribution for this attack FIG1 is visually similar to that of benign agents, its range differs, again enabling detection. The malicious agent only needs to boost the part of the weight update that corresponds to the adversarial objective. In the baseline attack, in spite of this being the entire update, the ing distribution is sparse and of low magnitude compared to a benign agent's updates. This indicates that the weights update needed to meet the adversarial objective could be hidden in an update that resembled that of a benign agent. However, as we saw in the previous section, boosting the entire weight update when the training loss is included leads to low validation accuracy. Further, the concatenation strategy does not allow for parts of the update corresponding to the two different objectives to be decoupled. To overcome this, we propose an alternating minimization attack strategy which works as follows for iteration t. For each epoch i, the adversarial objective is first minimized starting from w Results: In FIG6, the plot on the left shows the evolution of the metrics of interest over iterations. The alternating minimization attack is able to achieve its goals as the accuracy of the malicious model closely matches that of the global model even as the adversarial objective is met with high confidence for all time steps starting from t = 3. This attack can bypass the accuracy checking method as the accuracy on test data of the malicious model is close to that of the global model. Qualitatively, the distribution of the malicious weight update FIG6 ) is much more similar to that of the benign weights as compared to the baseline attack. Further, in Figure 3, we can see that the spread in distances between the malicious updates and benign updates much closer to that between benign agents compared to the baseline attack. Thus, this attack is stealthier than the baseline. To increase the attack stealth, the malicious agent can also add a distance-based constraint onw i,t m, which is the intermediate weight vector generated in the alternating minimization strategy. There could be multiple local minima which lead to low training loss, but the malicious agent needs to send back a weight update that is as close as possible (in an appropriate distance metric) to the update they would have sent had they been benign. So, w Constraints based on the empirical distribution of weights such as the Wasserstein or total variation distances may also be used. The adversarial objective is achieved at the global model with high confidence starting from time step t = 2 and the success of the malicious model on the benign objective closely tracks that of the global model throughout. The weight update distribution for this attack FIG6 ) is again similar to that of a benign agent. Further, in Figure 3, we can see that the distance spread for this attack closely follows and even overlaps that of benign updates throughout, making it hard to detect using the L 2 distance metric. In this section, we look at how the malicious agent can choose a better estimate for the effect of the other agents' updates at each time step that it is chosen. In the case when the malicious agent is not chosen at every time step, this estimation is made challenging by the fact that it may not have been chosen for many iterations. The malicious agent's goal is to choose an appropriate estimate for δ DISPLAYFORM0 Eq. 1. At a time step t when the malicious agent is chosen, the following information is available to them from the previous time steps they were chosen: i) Global parameter vectors w αm, this will negate the effects of the other agents. However, due to estimation inaccuracy and the fact that the optimizer has not accounted for this correction, this method leads to poor empirical performance. Pre-optimization correction: Here, the malicious agent assumes thatŵ DISPLAYFORM1 m. In other words, the malicious agent optimizes for δ t m assuming it has an accurate estimate of the other agents' updates. For attacks which use explicit boosting, this involves starting from w When the malicious agent is chosen at time step t 5, information regarding the probable updates from the other agents can be obtained from the previous time steps at which the malicious agent was chosen. Previous step estimate: In this method, the malicious agent's estimateδ t [k]\m assumes that the other agents' cumulative updates were the same at each step since t (the last time step at which at the malicious agent was chosen), i.e.δ DISPLAYFORM0 In the case when the malicious agent is chosen at every time step, this reduces toδ DISPLAYFORM1 Results: Attacks using previous step estimation with the pre-optimization correction are more effective at achieving the adversarial objective for both the baseline and alternating minimization attacks. In FIG10, the global model misclassifies the desired sample with a higher confidence for both the baseline and alternating minimization attacks at t = 2. Neural networks are often treated as black boxes with little transparency into their internal representation or understanding of the underlying basis for their decisions. Interpretability techniques are designed to alleviate these problems by analyzing various aspects of the network. These include (i) identifying the relevant features in the input pixel space for a particular decision via Layerwise Relevance Propagation (LRP) techniques BID14 ); (ii) visualizing the association between neuron activations and image features (Guided Backprop , DeConvNet ); (iii) using gradients for attributing prediction scores to input features (e.g., Integrated Gradients BID20), or generating sensitivity and saliency maps (SmoothGrad BID18), Gradient Saliency Maps BID17 )) and so on. The semantic relevance of the generated visualization, relative to the input, is then used to explain the model decision. These interpretability techniques, in many ways, provide insights into the internal feature representations and working of a neural network. Therefore, we used a suite of these techniques to try and discriminate between the behavior of a benign global model and one that has been trained to satisfy the adversarial objective of misclassifying a single example. FIG11 compares the output of the various techniques for both the benign and malicious models on a random auxiliary data sample. Targeted perturbation of the model parameters coupled with tightly bounded noise ensures that the internal representations, and relevant input features used by the two models, for the same input, are almost visually imperceptible. This reinforces the stealth achieved by our attacks along with respect to another measure of stealth, namely various interpretability-based detection techniques. In this paper, we have started an exploration of the vulnerability of multi-party machine learning algorithms such as federated learning to model poisoning adversaries, who can take advantage of the very privacy these models are designed to provide. In future work, we plan to explore more sophisticated detection strategies at the server, which can provide guarantees against the type of attacker we have considered here. In particular, notions of distances between weight distributions are promising defensive tools. Our attacks in this paper demonstrate that federated learning in its basic form is very vulnerable to model poisoning adversaries, as are recently proposed Byzantine resilient aggregation mechanisms. While detection mechanisms can make these attacks more challenging, they can be overcome, demonstrating that multi-party machine learning algorithms robust to attackers of the type considered here must be developed. When the number of agents increases to k = 100, the malicious agent is not selected in every step. Further, the size of |D m | decreases, which makes the benign training step in the alternating minimization attack more challenging. The challenges posed in this setting are reflected in Figure 8, where although the baseline attack is able to introduce a targeted backdoor, it cannot ensure it for every step due to steps where only benign agents provide updates. The alternating minimization attack is also able to introduce the backdoor, as well as increase the classification accuracy of the malicious model on test data. However, the improvement in performance is limited by the paucity of data for the malicious agent. It is an open question if data augmentation could help improve this accuracy. Figure B shows the evolution of weight update distributions for the 4 different attack strategies on the CNN trained on the Faishon MNIST dataset. Time slices of this evolution were shown in the main text of the paper. The baseline and concatenated training attacks lead to weight update distributions that differ widely for benign and malicious agents. The alternating minimization attack without distance constraints reduces this qualitative difference somewhat but the closest weight update distributions are obtained with the alternating minimization attack with distance constraints. C BYPASSING BYZANTINE-RESILIENT AGGREGATION MECHANISMS BID2 recently proposed a gradient aggregation mechanism known as'Krum' that is provably resilient to Byzantine adversaries. We choose to evaluate Krum as it is efficient, provably resilient and can be used a building block for better mechanisms BID13. As stated in the introduction, the aim of Byzantine adversaries considered in this work and others BID5 BID13;; BID23 ) is to ensure convergence to ineffective models. The goals of the adversary in this paper are to ensure convergence to effective models with targeted backdoors. This difference in objectives leads to'Krum' being ineffective against our attacks. We now briefly describe Krum. Given n agents of which f are Byzantine, Krum requires that n ≥ 2f + 3. At any time step t, updates (δ t 1, . . ., δ t n) are received at the server. For each δ t i, the n − f − 2 closest (in terms of L p norm) other updates are chosen to form a set C i and their distances added up to give a score S(δ In FIG1, we see the effect of our attack strategies on Krum with a boosting factor of λ = 2 for a federated learning setup with 10 agents. Since there is no need to overcome the constant scaling factor α m, the attacks can use a much smaller boosting factor λ to ensure the global model has the targeted backdoor. Even with the baseline attack, the malicious agent's update is the one chosen by Krum for 34 of 40 time steps but the global model is unable to attain high test accuracy. The alternating minimization attack ensures that the global model maintains relatively high test accuracy while the malicious agent is chosen for 26 of 40 time steps. These conclusively demonstrate the effectiveness of model poisoning attacks against Krum.

Effective model poisoning attacks on federated learning able to cause high-confidence targeted misclassification of desired inputs

Represent the TLDR for finding the corresponding scientific article

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: Represent the scientific article to find a short TLDR

Question: Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs. Small amounts of noise can destroy the performance of an otherwise state-of-the-art model. To harden models against noise, practitioners often perform data augmentation, adding artificially-noised examples to the training set, carrying over the original label. In this paper, we hypothesize that a clean example and its superficially perturbed counterparts shouldn't merely map to the same class--- they should map to the same representation. We propose invariant-representation-learning (IRL): At each training iteration, for each training example, we sample a noisy counterpart. We then apply a penalty term to coerce matched representations at each layer (above some chosen layer). Our key , demonstrated on the LibriSpeech dataset are the following: (i) IRL significantly reduces character error rates (CER)on both `clean' (3.3% vs 6.5%) and `other' (11.0% vs 18.1%) test sets; (ii) on several out-of-domain noise settings (different from those seen during training), IRL's benefits are even more pronounced. Careful ablations confirm that our are not simply due to shrinking activations at the chosen layers. The IRL algorithm is simple: First, during training, for each example x, we produce a noisy version 90 by sampling from x ∼ ν(x), where ν is a stochastic function. In our experiments, this function takes 91 a random snippet from a noise database, sets its amplitude by drawing from a normal distribution, and 92 adds it to the original (in sample space), before converting to spectral features. We then incorporate a 93 penalty term in our loss function to penalize the distance between the encodings of the original data 94 point φ e (x) and the noisy data point φ e (x), where φ l is representation at layer l. In our experiments, 95 we choose φ e to be the output of the encoder in our Seq2Seq model. We illustrate the learning setup 96 graphically in FIG0. In short, our loss function consists of three terms, one to maximize the 97 probability assigned to the the clean example's label, another to maximize the probability our model 98 assigned to the noisy example's (identical) label scaled by hyper-parameter α, and a penalty term to 99 induce noise-invariant representations L d. In the following equations, we express the loss calculated 100 on a single example x and its noisy counterpart x, omitting sums over the dataset for brevity. DISPLAYFORM0 where θ denotes our model parameters. Because our experiments address multiclass classification, 102 our primary loss L c is cross-entropy: DISPLAYFORM1 where C denotes the vocabulary size andŷ is our model's softmax output. To induce similar 104 representations for clean and noised data, we apply a penalty consisting of two terms, the first 105 penalizes the L 2 distance between φ e (x) and φ e (x), the second penalizes their negative cosine DISPLAYFORM2 We jointly penalize the L 2 and cosine distance for the following reason. It is possible to lower the 108 L 2 distance between the two (clean and noisy) hidden representations simply by shrinking the scale 109 of all encoded representations. Trivially, these could then be dilated again simply by setting large 110 weights in the following layer. On the other hand, it is possible to assign high cosine similarity to layers will also be identical and thus those penalties will go to 0. We can express this loss as a sum 122 over successive representations φ l of the clean φ l (x) and noisy φ l (x) data: DISPLAYFORM3 In our experiments, we find that IRL-C consistently gives a small improvement over achieved 124 with IRL-E. These approaches are identical for the L 2 penalty but not for the cosine distance penalty, owing to 133 the normalizing factor which may be different at each time step. In this work we take approach (i) concatenating the representations across time steps and then calculating the penalty. All of our models are based off of the sequence-to-sequence due to. The input to the encoder is a In our experiments with IRL-E (penalty applied on a single layers), we use the output of the encoder 141 to calculate the penalty. Note that there is one output per step in the input sequence and thus we are 142 concatenating across the T 1 steps. To calculate IRL-C, we also start with the encoder output concatenating across all T 1 sequence steps first add MUSAN noise to the training data point at a signal-to-noise ratio drawn from a Gaussian 162 with a mean of 12dB and a variance of 8dB. This aligns roughly with the scale of noise employed in other papers using multi-condition training. Before presenting our main , we briefly describe the model architectures, training details, and 166 the various baselines that we compare against. We also present details on our pipeline for synthesizing 167 noisy speech and explain the experimental setup for evaluating on out-of-domain noise. Instead we decode predictions from all models via beam search with width 10. To ensure fair comparisons, we perform hyper-parameter searches separately for each model and We train all models with the Adam optimizer with an initial learning rate of 0.001. We employ a The primary loss function for each model is cross-entropy loss and our primary evaluation metric to 193 evaluate all models is the character error rate. As described above, the additional loss terms for our IRL models are L2 loss and cosine distance between representations of clean and noisy audio. We train all models on the LibriSpeech corpus, generating noisy data by adding randomly selected 216 noise tracks from the MUSAN dataset with a signal to noise ratio drawn from a Gaussian distribution 217 (12dB mean, 8dB standard deviation) and temporal shift drawn from a uniform distribution (with Our final experiments test the effects of various out-of-domain noise on our models. The are 245 shown in TAB2 . We found that our models trained with the IRL procedure had stronger (and significantly less degradation) across all tasks compared to the baseline and the purely data 247 overlapping speech compared to 91.5% for the baseline and 32.0% on the data augmented model. We 251 found that decreasing the signal-to-noise ratio also effected the baseline models more than the models 252 trained on the IRL algorithm: our IRL-C model received a character error rate of 5.7% compared to 253 27.8% for baseline and 10.8% for the purely data augmented model. We found that modifying the 254 volume of the speaker did not effect the accuracy of any of the networks. Finally, we found that our 255 models trained with the IRL algorithm performed better for re-sampled telephony data, achieving In this paper, we demonstrated that enforcing noise-invariant representations by penalizing differences 267 between pairs of clean and noisy data can increase model accuracy on the ASR task, produce models 268 that are robust to out-of-domain noise, and improve convergence speed. The performance gains 269 achieved by IRL come without any impact to inference throughput. We note that our core ideas 270 here can be applied broadly to deep networks for any supervised task. While the speech setting is 271 particularly interesting to us, our methods are equally applicable to other machine learning fields,

represent the short summary for finding the corresponding scientific article (long)

In this paper, we hypothesize that superficially perturbed data points shouldn’t merely map to the same class---they should map to the same representation.

Represent the short summary for finding the corresponding scientific article (long).

An experimental paper that proves the amount of redundant weights that can be freezed from the third epoch only, with only a very slight drop in accuracy.


Question: Represent the article for finding a single-sentence TLDR

Question: In this paper we design a harmonic acoustic model for pitch detection. This model arranges conventional convolution and sparse convolution in a way such that the global harmonic patterns captured by sparse convolution are composed of the enough number of local patterns captured by layers of conventional convolution. When trained on the MAPS dataset, the harmonic model outperforms all existing pitch detection systems trained on the same dataset. Most impressively, when trained on MAPS with simple data augmentation, the harmonic model with an LSTM layer on top surpasses an up-to-date, more complex pitch detection system trained on the MAESTRO dataset to which complicated data augmentation is applied and whose training split is an order-of-magnitude larger than the training split of MAPS. The harmonic model has demonstrated potential to be used for advanced automatic music transcription (AMT) systems. In recent years, deep learning has emerged as a promising approach to pitch detection. Pitch detection is the process of detecting the pitches present in a frame, i.e., a short snippet of musical waveform. The of pitch detection can be post-processed to extract note contours, i.e., note onsets and offsets. The whole process of pitch detection and note extraction is called automatic music transcription (AMT). This paper is devoted to pitch detection for solo piano music. For AMT, of first importance is an acoustic model that can predict the active pitches in a frame. Using acoustic models as building blocks, more advanced architectures can be constructed for various purposes. designed an acoustic model for pitch detection, which is referred to as the Kelz model hereafter. This model was modified from the one developed in Schlüter & Böck for onset detection and resembles the LeNet-5 model . The Kelz model treats pitch detection simply as image-related tasks by using convolution layers to capture local frequency patterns. It does not explicitly capture the harmonic patterns of pitched music. Instead, it relies on fully connected layers to this end. The above handling of harmonic patterns weakens the model's generalisation capability. This problem has been partially studied in. designed an AMT system. This system consists of an onset detector and a frame detector that detects the pitches in each frame. The two detectors have similar structures by topping the Kelz model with a bi-directional LSTM. Skip connections are featured by feeding the onset detector's output into the other detector to serve as additional intermediate features. used a similar AMT system as a sub-system to build a bigger system for piano sound generation. The AMT system in uses more features and introduces a separate detector for offset detection. The dataset used in is far larger than the one used in. designed an AMT system consisting of three separate detectors for pitch, onset and offset detection. The pitch detector is a Kelz model. Some intermediate features of the pitch detector are used as the sole inputs to the other two detectors. Both the onset and offset detectors only have a convolution and a fully connected layer. Finally, note contours are extracted by fusing the predictions from the three detectors with a hidden Markov chain model (HMM). designed a fully convolutional acoustic model named the harmonic constant-q transform (HCQT) model. It uses HCQTs as input representation. HCQTs are constructed from CQTs, which like any spectrograms have a time and a frequency dimension. Besides the above two dimensions, HCQTs have a newly added dimension called the harmonic dimension. The harmonic dimension consists of the fundamental frequency/pitch (denoted by f 0), a sub-harmonic at 1 2 f 0, and four harmonics ranging from 2f 0 to 5f 0. Rearranging CQTs into HCQTs enables the capture of harmonic patterns/structures specific to pitched music, whereas the convolutional architecture enables the capture of local patterns as in most image-related problems. designed an AMT system consisting of six cascaded networks that are trained one after another. Skip connections are used among these networks in a similar fashion as residual networks do. Except the first network, all other networks are multilayer perceptrons (MLPs) that have two to three layers. The first network N1 takes variable-q transforms (VQTs) as inputs and detects tentative pitches by linearly combing 50 frequency bins for each pitch that include harmonics and non-harmonics. N1 is essentially a convolution layer with a single sparse kernel. A second network N2 performs more accurate pitch estimation from the output of N1. The overall effect of N1 and N2 is equivalently an acoustic model that resembles the HCQT model. The difference is that N1 and N2 take into account more frequency bins when capturing harmonic patterns, whereas the HCQT model pays more attention to local patterns. The remaining four networks estimate onsets, offsets and tentative notes, and compute probabilities for each note, respectively. Loosely speaking, the waveform of a music note is composed of a number of monotones, among which the lowest frequency is called the fundamental frequency or the pitch, and all the other are integer multiples of the pitch referred to as the harmonics. For polyphonic music, there can be multiple notes active at the same time; and their pitches and harmonics often overlap. This leads to the challenge for pitch detection. Figure 1 shows an example VQT spectrogram. This spectrogram features a grid structure composed of frequency stripes. These stripes are a of non-perfect pitches and the windowing effect (also known as power leakage) in the calculation of the discrete Fourier transform (DFT). We call these stripes the local patterns, as they only involve a small number of local neighbouring frequencies. Next let us talk about harmonic patterns. To tell if a specific note is active, intuitively we will first check if its pitch (denoted by f 0) and harmonics are light (i.e., have energy). If it is the case, then it is likely the note is active. On the other hand, if it is also light at 1 2 f 0 or 1 3 f 0, then the likelihood drops, because now f 0 could be the second or third harmonic of another note. When the degree of polyphony increases, this logical reasoning is definitely over our heads and deep learning comes in handy. We call the interaction patterns among the pitches and harmonics of different notes the harmonic patterns. Since the constituent frequencies of harmonic patterns are sparsely distributed, it is inappropriate to capture harmonic patterns with the conventional convolution whose receptive field can cover only contiguous frequencies. Instead, we need a type of convolution whose receptive field in the frequency dimension corresponds to the above sparsely distributed frequencies. We term this type of convolution sparse convolution. For pitch detection, it is critical as well as challenging to capture both of the above two types of frequency patterns. The existing acoustic models focused only on either of them. The existing AMT systems concentrated on more advanced, complex network structures. The contribution of this paper is as follows. 1. We design a harmonic acoustic model for pitch detection. This model takes VQTs as inputs. In the first part of this model, the stripe-shaped local frequency patterns are captured with layers of conventional convolution. Then in the second part, the global harmonic patterns are captured with sparse convolution. 2. When all the systems participating in the comparison are trained on MAPS, the harmonic model achieves the best performance, leading the runner-up by 3.5%. 3. When trained on MAPS with simple data augmentation, the harmonic model enhanced by an LSTM layer outperforms an up-to-date, more complex system trained on the MAESTRO dataset to which complicated data augmentation is applied and whose training split is 15 times as large as the training split of MAPS, demonstrating the potential of the harmonic model to be used for building advanced AMT systems. 2.1 MAPS MAPS is a piano dataset generated by a software synthesizer and a real piano, namely, a Yamaha Disklavier upright piano, from standard MIDI files. Disklavier can be controlled by computer, and accept MIDI files as inputs and output MIDI files that are strictly synchronized with the sound generated. These output MIDI files can be used to generate ground-truth labels. The sound from Disklavier was recorded under two settings, namely, a close and an ambient setting. The synthesizer has 7 settings that differ in soundfont and reverberation configuration. Thus, there are 9 settings in total. Each setting has 30 recordings, ing in a total of 270 recordings. Since there are only 160 musical compositions, these 9 settings have overlap in composition. Next we need to partition the dataset into three splits, namely, a training, a validation and a test split. In this process, the general criterion is that the training and test splits should have no instrument and composition overlap so as to fairly compare the generalisation capability of different models. We choose the 60 recordings from Disklavier as the test split. We exclude the 71 recordings whose compositions appear in the test split from the 210 synthesized recordings and use the remaining 139 recordings as the training split. We use the above 71 recordings as the validation split. MAESTRO is a piano dataset generated by Yamaha Disklavier grand pianos from 9 years of an international piano competition. It has 1184 recordings in total that have a total duration of about 172 hours and a total size of about 103 GB. By contrast, MAPS only has a total duration of about 18 hours and a total size of about 11 GB. When partitioning MAESTRO into training, validation and test splits, we simply follow the recommendation given in. In particular, the training split has 954 recordings, the validation split has 105 recordings, and the test split has 125 recordings. The training split of MAESTRO is about 15 times as large as the training split of MAPS. 3 MODEL CONFIGURATION The harmonic model uses VQT as input representation. The procedure for computing VQT is given in appendix A.1. The setting for VQT computation is as follows. The sampling rate is 44.1 kHz. The number of frequency bins per octave (denoted by B) is 36. The minimum frequency is the frequency of MIDI note 21 multiplied by a factor of 2 −1/B. The maximum frequency is the frequency of MIDI note 132 multiplied by a factor of 2 1/B. Thus, we have 336 frequency bins in total. The bandwidth for each frequency bin is set to where f k is the centre frequency. The hop size (denoted by h) is 64 samples. For pitch detection, we do not like the hop size to be too small, because in this case adjacent frames in the spectrogram will be highly correlated. So we down-sample the ing spectrogram by a factor of 22. We formulate frame-wise pitch detection as a multi-label classification problem. Frame labels are determined from the MIDI file. First, we translate the use of the sustain pedal into extended note duration as in. Next, we express the onset and offset for a specific note in terms of samples as where t on and t off are the onset and offset times in seconds, respectively. Finally, the start and end frames of this note can be expresses as We propose a harmonic acoustic model for frame-wise pitch detection. The structure of this model is given in Table 1. In this table, we use -to mean that the output shape of a layer is the same as the output shape of the above layer. Layer 0 is the input with shape (none × none × 336 × 1) where the dimension order is batch, frame, frequency and channel. Here we use none to denote a dynamic dimension. The abbreviations used in this table are defined as follows. 1) conv(32 × 3 × 3) stands for a convolution layer with 32 kernels of receptive field 3 × 3. 2) dropout(0.8) is a dropout layer with keeping probability 0.8. 3) sparse-conv(256 × 1 × 79) is a sparse convolution layer with 256 kernels of receptive field 1 × 79. 4) maxpool(1 × 3) is a max-pooling layer with receptive 1 × 3, frame stride 1, and frequency stride 3. 5) FC(n) is a fully connected layer with n features. This structure can be divided into three parts. Part 1 consists of layers 1 through 7. This part uses four consecutive convolution layers to capture local frequency patterns. The overall receptive field of these layers in the frequency domain is 9 frequency bins. Part 2 consists of layers 8, 9 and 10. Layer 8 is a sparse convolution layer. This layer does the same job as network N1 of. In particular, for each pitch f 0 we select 50 frequency bins relative to f 0 as the input features for detecting the presence of f 0. These 50 frequency bins include bins over and under f 0, Figure 2: The receptive field of the sparse convolution in the frequency dimension where the solid vertical line is the pitch and the dotted vertical lines are the input features for the pitch . and harmonics and non-harmonics. Please refer to for a complete list of these bins. Figure 2 shows the distribution of these bins. The original 50 bins were given when the number of bins per octave is 240. And when converted to 36 bins per octave used by the harmonic model, some bin indices become non-integers. For these non-integers, we take both their floors and ceils. Thus, for each f 0 we have a total of 79 input features. For an f 0, some of its input features could be out of the VQT's bin range. In this case, we assume that the out-of-range input features are zeros. Since a typical piano only has 88 keys ranging from MIDI notes 21 to 108, among the VQT's 336 frequency bins only the first 264 are pitches to detect. Therefore, after the sparse convolution layer, we get an output of shape (none × none × 264 × 256) where 256 is the number of output features for each pitch. This output is at pitch level. However, the datasets only allow labels at note level. So we use max-pooling of receptive field 1 × 3 to down-sample the output to note level, ing in an output of shape (none × none × 88 × 256). Part 3 consists of layers 11, 12 and 13, after which we get predictions for the 88 notes. In the above structure except the last layer, when it is applicable, ReLU is used as activation function and the output is batch-normalized. The last layer uses sigmoid as activation function. We use dropout at different places to control overfitting. The difference between the harmonic model and the Kelz model is that the former explicitly captures harmonic frequency patterns with sparse convolution, whereas the latter does this implicitly by letting the network to learn them on itself. This implicit handling of harmonic patterns makes the trained model overfit the timbres and composition styles of the training split. However, the training and test splits often have different timbres and composition styles so that this implicit handling will impact the generalisation capability of the trained model. The harmonic model differs from network N1 of in two aspects. First, the harmonic model captures enough number of local frequency patterns with part 1, whereas N1 did not consider Compared with the HCQT model, the harmonic model is able to capture more complex frequency patterns by using more features for each pitch and placing sparse convolution after conventional convolution. The overall receptive field of the convolution layers in the HCQT model is over one octave and thus could lead to overfitting of the composition styles. This problem has been advertently avoided in both the Kelz and the harmonic model by using a relatively small overall receptive filed when capturing local frequency patterns. We use the binary cross entropy loss. Denote by p ∈ {0, 1} the ground-truth label for a note in a frame and byp ∈ the predicted probability for this note. The loss for this note is formulated as As a convention, the performance of pitch detection is solely measured by the f-measure defined as In the above equation P and R are the precision and recall defined, respectively, as where TPs is the number of true positives, FPs is the number of false positives, and FNs is the number of false negatives. When there is more than one recording, the above metrics can be calculated in two ways. We can first calculate them for individual recordings and then average these . We call metrics obtained this way the average . We can alternatively treat all the recordings as an ensemble and directly calculate the metrics. We refer to metrics obtained this way the ensemble . In this section, we will compare performance of the harmonic model, the Kelz model and other more complex pitch detection systems built upon acoustic models. To enhance the performance, the existing systems exploited various techniques/tricks, such as data augmentation , RNN (;, HMM , joint-task training , and larger training split . Therefore, for a fair comparison we will apply two tricks to the harmonic model, namely, data augmentation and RNN. The first type of data augmentation is pitch shift whose procedure is detailed in appendix A.2. The second type is to change the powers of different frequencies by multiplying the amplitude of each frequency bin by a random number ranging from 0.9 to 1.1. The third type is to use the two sound channels of each recording in the training split as independent training examples. In this case, for testing we will average the logits of the two sound channels. To enhance the harmonic model, we can top the harmonic model with an LSTM layer. Specifically, in this case we first train the harmonic model. Next, we remove layers 11, 12 and 13 and keep the parameters of the remaining layers fixed and untrainable. Then we replace these removed layers with an LSTM of 64 hidden units, which is in turn followed by an FC layer (i.e., a fully connected layer that converts the number of features for each note from 64 to 1). We use Tensorflow 1.13.1 as the neural network framework and implement sparse convolution ourselves. The optimizer is the Adam optimizer. For the harmonic model without LSTM on top, each example has 300 frames; the batch size is 2; and the learning rate is 10 −4. For the harmonic model with LSTM on top, each example has 600 frames; the batch size is 1; and the learning rate is 10 −3. Table 2 compares the performance of different pitch detection systems. The system in was trained on a self-made training dataset that was purely synthesized and does not overlap in musical composition with the test split of MAPS. The system in was trained on MAESTRO. also augmented the training split by pitch shift, compressing, equalising, and adding reverberation and pink noise. The of the Kelz model are cited from which implemented this model and tested it on MAPS by following the training-test split partition given in section 2.1. Note that MAESTRO cannot be used both for training and for testing, because this has the problem of instrument overlap. Therefore, to objectively access a system's generalisation capability, we only compare the test performance of different systems on the test split of MAPS. Among the existing systems, some used the ensemble and some used the average . For the system of trained without data augmentation, only the f-measure is available. In its pure form without data augmentation and LSTM, the harmonic model defeats all the existing systems except the system in. The system in trained without data augmentation leads the pure harmonic model by 0.24. This lead is attributed to the MAESTRO's far larger training split and joint-task training with more complex network structure, as evidenced and contrasted by the of that were obtained on MAPS with a less complex network structure. The system of with data augmentation surpassed the harmonic model with data augmentation but without LSTM by 1.49. However, when data augmentation and LSTM are both applied, the harmonic model outperforms the system of with data augmentation by 0.91 and reaches a record high of 85.82. Note that, even when LSTM is applied to the harmonic model, none of the existing systems except the Kelz model is simpler than the harmonic model. In this paper we designed a harmonic acoustic model for pitch detection. This model effectively captures the complex frequency interactions characterizing polyphonic pitched music through conventional convolution and sparse convolution inspired by the harmonic structure of pitched music. In its pure form without RNN and data augmentation, the harmonic model outperformed most of the existing pitch detection systems. Most noticeably, when trained on MAPS and data augmentation is done, the harmonic model with an LSTM layer on top outdid the complex system in trained on MAESTRO whose training split 15 times as large as the training split of MAPS. Thus, the harmonic model has shown great potential to be used for building advanced AMT systems. A possible future direction is to make more potential of complex spectrograms, instead of using only amplitude spectrograms. A mixture of signal can be inseparable in the real number domain but could be separable in the complex number domain. has done some preliminary study in this direction. However, our own study showed that the technique of deep complex network proposed in did not yield a performance comparable with that of real networks. Therefore, definitely more can be done. Since there is no clear, standard procedure for computing VQT, here we will sketch such a procedure by ignoring the underlying details. CQT is arguably the only choice for characterizing harmonic frequency patterns due to the following advantages. First, it was born for processing musical signal of equal tempered scale, because its frequency bins are strictly log-linearly spaced and the bandwidth of each filter is proportional to the centre frequency . Second, it is preferable over the traditional wavelet transform, because the latter cannot provide sufficient frequency resolution for musical signal processing. Third, there exists a fast algorithm for CQT (Schörkhuber &). The drawback with CQT is the low time resolution for lower frequencies. For example, when the number of frequency bins per octave is 36, the frame length for MIDI note 21 is 1.87 seconds. Therefore, CQT is not ideal for detecting lower notes. Hence there comes VQT . In VQT the bandwidth of a filter within the filter bank can be expresses as where f k is the filter's centre frequency, B is the number of frequency bins per octave, and γ is a non-negative constant. When γ is zero, VQT reduces to CQT. The rationale behind VQT is that by enlarging the bandwidth we can shrink the frame length and thus get better time resolution. In order to properly apply VQT, we need to understand what the bandwidth means. In VQT, filters have strictly finite bandwidth therefore infinite length in the time domain. That is, the frame length for each frequency is infinite. We can only consider the frame length from an engineering perspective. Let us define the frame length as the length of the zone within the infinite frame that contains a major proportion of the frame's energy. If this proportion is to be over 99%, then it can be proved that the frame length is 2.88/Ω k seconds, or 2.88 × sr/Ω k samples where sr is the sampling rate. Thus, if we like a maximum frame length of 20000 samples when sr is 44.1 kHz, we should set the minimum bandwidth to be 2.88 × 44100/20000 = 6.35 Hz. Next, let us talk about zero padding. In VQT, when computing the spectral coefficients for each frequency at different times, the signals involved are cyclic shifts of the original recording. Thus, when computing the coefficient at sample zero, half the data comes from the end of the recording. We can get around this undesirable effect by zero padding. To be specific, we can pad zeros at each end of the recording, where h is the hop size, and then compute the VQT. After that we throw away 1.44/Ω min × sr/h coefficients at each end of the spectrogram. Finally, let us talk about the hop size in terms of samples. In VQT, the hop size is any number such that sr/h ≥ Ω max where Ω max is the maximum bandwidth. After getting the spectrogram, we convert it to dB scale according to 20 × log 10 |VQT| + 10 −10 + 200 where | · | is the absolute operator to get the amplitude. Adding 10 −10 is to make the computation numerically stable. When the amplitude is zero, we get the minimum power of −200 dB. When padding values for the inputs to the first neural network layer, it is desirable that the padded values are physically meaningful. In our case, we would like these values to stand for a zero power level. Thus, these values should be -200. For convenience we alternatively shift the dB scaled spectrogram by 200 so that we can keep on using zeros for padding. Step 1 Setting the number of frequency bins per octave (B) to a higher value of 84, calculate the VQT. Step 2 As per the procedure given in Schörkhuber et al., shift the pitch of the above VQT by 0, ±1 and ±2 frequency bins. Step 3 For each pitch-shifted VQT, recover the discrete Fourier transform (DFT) of the signal. Step 4 From the above DFT, calculate the VQT for B = 36.

Represent the TLDR for finding the corresponding scientific article!

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.