Question: Represent the scientific article to find a short TLDR

Question: Predicting structured outputs such as semantic segmentation relies on expensive per-pixel annotations to learn strong supervised models like convolutional neural networks. However, these models trained on one data domain may not generalize well to other domains unequipped with annotations for model finetuning. To avoid the labor-intensive process of annotation, we develop a domain adaptation method to adapt the source data to the unlabeled target domain. To this end, we propose to learn discriminative feature representations of patches based on label histograms in the source domain, through the construction of a disentangled space. With such representations as guidance, we then use an adversarial learning scheme to push the feature representations in target patches to the closer distributions in source ones. In addition, we show that our framework can integrate a global alignment process with the proposed patch-level alignment and achieve state-of-the-art performance on semantic segmentation. Extensive ablation studies and experiments are conducted on numerous benchmark datasets with various settings, such as synthetic-to-real and cross-city scenarios. Recent deep learning-based methods have made significant progress on vision tasks, such as object recognition BID17 and semantic segmentation BID19, relying on large-scale annotations to supervise the learning process. However, for a test domain different from the annotated training data, learned models usually do not generalize well. In such cases, domain adaptation methods have been developed to close the gap between a source domain with annotations and a target domain without labels. Along this line of research, numerous methods have been developed for image classification BID29 BID8, but despite recent works on domain adaptation for pixel-level prediction tasks such as semantic segmentation BID14, there still remains significant room for improvement. Yet domain adaptation is a crucial need for pixel-level predictions, as the cost to annotate ground truth is prohibitively expensive. For instance, road-scene images in different cities may have various appearance distributions, while conditions even within the same city may vary significantly over time or weather. Existing state-of-the-art methods use feature-level BID14 or output space adaptation BID31 to align the distributions between the source and target domains using adversarial learning BID11 BID37. These approaches usually exploit the global distribution alignment, such as spatial layout, but such global statistics may already differ significantly between two domains due to differences in camera pose or field of view. Figure 1 illustrates one example, where two images share a similar layout, but the corresponding grids do not match well. Such misalignment may introduce an incorrect bias during adaptation. Instead, we consider to match patches that are more likely to be shared across domains regardless of where they are located. One way to utilize patch-level information is to align their distributions through adversarial learning. However, this is not straightforward since patches may have high variation among each other and there is no guidance for the model to know which patch distributions are close. Motivated by recent advances in learning disentangled representations BID18 BID24, we adopt a similar approach by considering label histograms of patches as a factor and learn discriminative Figure 1: Illustration of the proposed patch-level alignment against the global alignment that considers the spatial relationship between grids. We first learn discriminative representations for source patches (solid symbols) and push a target representation (unfilled symbol) close to the distribution of source ones, regardless of where these patches are located in the image.representations for patches to relax the high-variation problem among them. Then, we use the learned representations as a bridge to better align patches between source and target domains. Specifically, we utilize two adversarial modules to align both the global and patch-level distributions between two domains, where the global one is based on the output space adaptation BID31, and the patch-based one is achieved through the proposed alignment by learning discriminative representations. To guide the learning process, we first use the pixel-level annotations provided in the source domain and extract the label histogram as a patch-level representation. We then apply K-means clustering to group extracted patch representations into K clusters, whose cluster assignments are then used as the ground truth to train a classifier shared across two domains for transferring a learned discriminative representation of patches from the source to the target domain. Ideally, given the patches in the target domain, they would be classified into one of K categories. However, since there is a domain gap, we further use an adversarial loss to push the feature representations of target patches close to the distribution of the source patches in this clustered space (see Figure 1). Note that our representation learning can be viewed as a kind of disentanglement guided by the label histogram, but is different from existing methods that use pre-defined factors such as object pose BID18.In experiments, we follow the domain adaptation setting in BID14 and perform pixellevel road-scene image segmentation. We conduct experiments under various settings, including the synthetic-to-real, i.e., GTA5 BID27 )/SYNTHIA BID28 to Cityscapes BID5 ) and cross-city, i.e., Cityscapes to Oxford RobotCar BID23 scenarios. In addition, we provide extensive ablation studies to validate each component in the proposed framework. By combining global and patch-level alignments, we show that our approach performs favorably against state-of-the-art methods in terms of accuracy and visual quality. We note that the proposed framework is general and could be applicable to other forms of structured outputs such as depth, which will be studied in our future work. The contributions of this work are as follows. First, we propose a domain adaptation framework for structured output prediction by utilizing global and patch-level adversarial learning modules. Second, we develop a method to learn discriminative representations guided by the label histogram of patches via clustering and show that these representations help the patch-level alignment. Third, we demonstrate that the proposed adaptation method performs favorably against various baselines and state-of-the-art methods on semantic segmentation. Within the context of this work, we discuss the domain adaptation methods, including image classification and pixel-level prediction tasks. In addition, algorithms that are relevant to learning disentangled representations are discussed in this section. Domain Adaptation. Domain adaptation approaches have been developed for the image classification task via aligning the feature distributions between the source and target domains. Conventional methods use hand-crafted features BID10 BID7 to minimize the discrep-ancy across domains, while recent algorithms utilize deep architectures BID8 BID32 to learn domain-invariant features. One common practice is to adopt the adversarial learning scheme BID9 and minimize the Maximum Mean Discrepancy BID20. A number of variants have been developed via designing different classifiers BID21 and loss functions BID33. In addition, other recent work aims to enhance feature representations by pixel-level transfer BID1 and domain separation BID0.Compared to the image classification task, domain adaptation for structured pixel-level predictions has not been widely studied. BID14 first introduce to tackle the domain adaptation problem on semantic segmentation for road-scene images, e.g., synthetic-to-real images. Similar to the image classification case, they propose to use adversarial networks and align global feature representations across two domains. In addition, a category-specific prior is extracted from the source domain and is transferred to the target distribution as a constraint. However, these priors, e.g., object size and class distribution, may be already inconsistent between two domains. Instead of designing such constraints, the CDA method BID36 applies the SVM classifier to capture label distributions on superpixels as the property to train the adapted model on the target domain. Similarly, as proposed in BID4, a class-wise domain adversarial alignment is performed by assigning pseudo labels to the target data. Moreover, an object prior is extracted from Google Street View to help alignment for static objects. The above-mentioned domain adaptation methods on structured output all use a global distribution alignment and some class-specific priors to match statistics between two domains. However, such class-level alignment does not preserve the structured information like the patches. In contrast, we propose to learn discriminative representations for patches and use these learned representations to help patch-level alignment. Moreover, our framework does not require additional priors/annotations and the entire network can be trained in an end-to-end fashion. Compared to the recently proposed output space adaptation method BID31 ) that also enables end-to-end training, our algorithm focuses on learning patch-level representations that aid the alignment process. Learning Disentangled Representation. Learning a latent disentangled space has led to a better understanding for numerous tasks such as facial recognition BID26, image generation BID3 BID24, and view synthesis BID18 BID35. These approaches use pre-defined factors to learn interpretable representations of the image. BID18 propose to learn graphic codes that are disentangled with respect to various image transformations, e.g., pose and lighting, for rendering 3D images. Similarly, BID35 synthesize 3D objects from a single image via an encoder-decoder architecture that learns latent representations based on the rotation factor. Recently, AC-GAN BID24 ) develops a generative adversarial network (GAN) with an auxiliary classifier conditioned on the given factors such as image labels and attributes. Although these methods present promising on using the specified factors and learning a disentangled space to help the target task, they focus on handling the data in a single domain. Motivated by this line of research, we propose to learn discriminative representations for patches to help the domain adaptation task. To this end, we take advantages of the available label distributions and naturally utilize them as a disentangled factor, in which our framework does not require to pre-define any factors like conventional methods. In this section, we describe our proposed domain adaptation framework for predicting structured outputs, our adversarial learning scheme to align distributions across domains, and the use of discriminative representations for patches to help the alignment. Given the source and target images I s, I t ∈ R H×W ×3 and the source labels Y s, our goal is to align the predicted output distribution O t of the target data with the source distribution O s. As shown in FIG0 (a), we use a loss function for supervised learning on the source data to predict the structured output, and an adversarial loss is adopted to align the global distribution. Based on this baseline model, we further incorporate a classification loss in a clustered space to learn patch-level discriminative representations F s from the source output distribution O s, shown in FIG0 (b). For target data, we employ another adversarial loss to align the patch-level distributions between F s and F t, where the goal is to push F t to be closer to the distribution of F s.Objective Function. As described in FIG0 (b), we formulate the adaptation task as composed of the following loss functions: DISPLAYFORM0 where L s and L d are supervised loss functions for learning the structured prediction and the discriminative representation on source data, respectively, while Γ denotes the clustering process on the ground truth label distribution. To align the target distribution, we utilize global and patch-level adversarial loss functions, which are denoted as L g adv and L l adv, respectively. Here, λ's are the weights for different loss functions. The following sections describe details of the baseline model and the proposed framework. Figure 3 shows the main components and loss functions of our method. We first adopt a baseline model that consists of a supervised cross-entropy loss L s and an output space adaptation module using L g adv for global alignment as shown in FIG0 (a). The loss L s can be optimized by a fully-convolutional network G that predicts the structured output with the loss summed over the spatial map indexed with h, w and the number of categories C: DISPLAYFORM0 where O s = G(I s) ∈ is the predicted output distribution through the softmax function and is up-sampled to the size of the input image. Here, we will use the same h and w as the index for all the formulations. For the adversarial loss L g adv, we follow the practice of GAN training by optimizing G and a discriminator D g that performs the binary classification to distinguish whether the output prediction is from the source image or the target one. DISPLAYFORM1 Then we optimize the following min-max problem for G and D g, with inputs to the functions dropped for simplicity: min DISPLAYFORM2 3.3 PATCH-LEVEL ALIGNMENT WITH DISCRIMINATIVE REPRESENTATIONS Figure 1 shows that we may find transferable structured output representations shared across source and target images from smaller patches rather than from the entire image or larger grids. Based on this observation, we propose to perform a patch-level domain alignment. Specifically, rather than naively aligning the distributions of all patches between two domains, we first perform clustering Figure 3: The proposed network architecture that consists of a generator G and a categorization module H for learning discriminative patch representations. In the clustered space, solid symbols denote source representations and unfilled ones are target representations pulled to the source distribution.on patches from the source-domain examples using ground truth segmentation labels to construct a set of prototypical patch patterns. Then, we let patches from the target domain adapt to this disentangled (clustered) space of source patch representations by guiding them to select the closest cluster regardless of the spatial location via adversarial objective. In the following, we describe details of the proposed patch-level alignment. Learning Discriminative Representations. In order to learn a disentangled space, class labels BID30 or pre-defined factors BID24 are usually provided as supervisory signals. However, it is non-trivial to assign some sort of class membership to individual patches of an image. One may apply unsupervised clustering of image patches using pixel representations, but it is unclear whether the constructed clustering would separate patches in a semantically meaningful way. In this work, we take advantage of already available per-pixel annotations in the source domain to construct semantically disentangled space of patch representations. To achieve this, we use label histograms for patches as the disentangled factor. We first randomly sample patches from source images, use a 2-by-2 grid on patches to extract spatial label histograms, and concatenate them into a vector, where each histogram is a 2 · 2 · C dimensional vector. Second, we apply K-means clustering on these histograms, whereby the label for any patch can be assigned as the cluster center with the closest distance on the histogram. To incorporate this clustered space during training the network G on source data, we add a classification module H after the predicted output O s, to simulate the procedure of constructing the label histogram and learn a discriminative representation. We denote the learned representation as F s = H(G(I s)) ∈ U ×V ×K through the softmax function, where K is the number of clusters. Here, each data point on the spatial map F s corresponds to a patch of the input image, and we obtain the group label Γ(Y s) for each patch accordingly. Then the learning process to construct the clustered space can be formulated as a cross-entropy loss: DISPLAYFORM3 Patch-level Adversarial Alignment. The ensuing task is to align the representations of target patches to the clustered space constructed in the source domain. To this end, we utilize another adversarial loss between F s and F t, where F t is generated in the same way as described above. Our goal is to align patches regardless of where they are located in the image, that is, without the spatial and neighborhood supports. Thus, we reshape F by concatenating the K-dimensional vectors along the spatial map, which in U · V independent data points. We note that a similar effect can be achieved by using a convolution layer with a proper stride and kernel size. We denote this reshaped data asF and formulate the adversarial objective: DISPLAYFORM4 where D l is the discriminator to classify whether the feature representationF is from the source or the target domain. Finally, we integrate and into the min-max problem in: DISPLAYFORM5 3.4 NETWORK OPTIMIZATION Following the standard procedure for training a GAN BID11, we alternate the optimization between three steps: 1) update the discriminator D g, 2) update the discriminator D l, and 3) update the network G and H while fixing the discriminators. Update the Discriminator D g. We train the discriminator D g to distinguish between the source output distribution (labeled as 1) and the target distribution (labeled as 0). The maximization problem on D g in FORMULA6 is equivalent to minimizing the binary cross-entropy loss: DISPLAYFORM6 Update the Discriminator D l. Similarly, we train the discriminator D l to classify whether the feature representationF is from the source or the target domain: DISPLAYFORM7 Update the Network G and H. The goal of this step is to push the target distribution closer to the source distribution using the optimized D g and D l, while maintaining good performance on the main tasks using G and H. As a , the minimization problem in FORMULA6 is the combination of two supervised loss functions, namely, FORMULA1 and FORMULA4, with two adversarial loss functions, where the adversarial ones can be expressed as binary cross-entropy loss functions that assign the source label to the target distribution: DISPLAYFORM8 We note that updating H would also update G through back-propagation, and thus the feature representations are enhanced in G. In addition, we only require G during the testing phase, so that runtime is unaffected compared to the baseline approach. Discriminator. For the discriminator D g using a spatial map O as the input, we adopt an architecture similar to but use fully-convolutional layers. It contains 5 convolution layers with kernel size 4 × 4, stride 2 and channel numbers {64, 128, 256, 512, 1}. In addition, a leaky ReLU activation BID22 ) is added after each convolution layer, except the last layer. For the discriminator D l, input data is a K-dimensional vector and we utilize 3 fully-connected layers similar to BID33, with leaky ReLU activation and channel numbers {256, 512, 1}. Generator. The generator consists of the network G with a categorization module H. For a fair comparison, we follow the framework used in BID31 ) that adopts DeepLab-v2 BID2 with the ResNet-101 architecture BID13 pre-trained on ImageNet BID6 ) as our baseline network G. To add the module H on the output prediction O, we first use an adaptive average pooling layer to generate a spatial map, where each data point on the map has a desired receptive field corresponding to the size of extracted patches. Then this pooled map is fed into two convolution layers and a feature map F is produced with the channel number K. Figure 3 illustrates the main components of the proposed architecture. Implementation Details. We implement the proposed framework using the PyTorch toolbox on a single Titan X GPU with 12 GB memory. To train the discriminators, we use the Adam optimizer BID16 with initial learning rate of 10 −4 and momentums set as 0.9 and 0.99. For learning the generator, we use the Stochastic Gradient Descent (SGD) solver where the momentum is 0.9, the weight decay is 5 × 10 −4 and the initial learning rate is 2.5 × 10 −4. For all Table 1: Ablation study on GTA5-to-Cityscapes using the ResNet-101 network. We also show the corresponding loss functions used in each setting. the networks, we decrease the learning rates using the polynomial decay with a power of 0.9, as described in BID2. During training, we use λ d = 0.01, λ g adv = λ l adv = 0.0005 and K = 50 for all the experiments. Note that we first train the model only using the loss L s for 10K iterations to avoid initially noisy predictions and then train the network using all the loss functions for 100K iterations. More details of the hyper-parameters such as image and patch sizes are provided in the appendix. DISPLAYFORM0 We evaluate the proposed framework for domain adaptation on semantic segmentation. We first conduct an extensive ablation study to validate each component in the algorithm on the GTA5-toCityscapes (synthetic-to-real) scenario. Second, we show that our method performs favorably against state-of-the-art approaches on numerous benchmark datasets and settings. We evaluate our domain adaptation method on semantic segmentation under various settings, including synthetic-to-real and cross-city scenarios. First, we adapt the synthetic GTA5 BID27 dataset to the Cityscapes BID5 dataset that contains real road-scene images. Similarly, we use the SYNTHIA BID28 dataset with a larger domain gap compared to Cityscapes images. For the above experiments, we follow BID14 to split the training and test sets. To overcome the realistic case when two domains are in different cities under various weather conditions, we adapt Cityscapes with sunny images to the Oxford RobotCar BID23 dataset that contains rainy scenes. We manually select 10 sequences in the Oxford RobotCar dataset annotated with the rainy tag, in which we randomly split them into 7 sequences for training and 3 for testing. We sequentially sample 895 images as training images and annotate 271 images with per-pixel semantic segmentation ground truth as the test set for evaluation. The annotated ground truth will be made publicly available. For all the experiments, intersection-over-union (IoU) ratio is used as the metric to evaluate different methods. In Table 1, we conduct an ablation study on the GTA5-to-Cityscapes scenario to understand the impact of different loss functions and design choices in the proposed framework. Loss Functions. In the first row of Table 1, we show different steps of the proposed method, including disentanglement, global alignment, and patch-level alignment. Interestingly, we find that adding disentanglement without any alignments (L s + L d) also improves the performance (from 36.6% to 38.8%), which demonstrates that the learned feature representation enhances the discrimination and generalization ability. Finally, as shown in the last of the second row, our method that combines both the global and patch-level alignments achieve the highest IoU as 43.2%.Impact on L d and L l adv. In the first two of the second row, we conduct experiments to validate the effectiveness of our patch-level alignment. We show that both losses, L d and L l adv, are necessary to assist this alignment process. Removing either of them will in performance loss, i.e., 1.9% and 1.5% lower than our final . The reason behind this is that, L d is to construct a clustered space so that L l adv can then effectively perform patch-level alignment in this space. Without ReshapedF. In the module H that transforms the output distribution to the clustered space, the features are reshaped as independent data pointsF to remove the spatial relationship and are then used as the input to the discriminator D l. To validate the usefulness, we show that without the reshaping process, the performance drops 2.4% in IoU. This matches our assumption that patches with similar representations should be aligned regardless of their locations. Visualization of Feature Representations. In FIG1, we show the t-SNE visualization BID34 of the patch-level features in the clustered space of our method and compare with the one without patch-level adaptation. The shows that with adaptation in the clustered space, the features are embedded into groups and the source/target representations overlap to each other well. Example patch visualizations are provided in the appendix. In this section, we compare the proposed method with state-of-the-art algorithms under various scenarios, including synthetic-to-real and cross-city cases. Synthetic-to-real Case. We first present experimental for adapting GTA5 to Cityscapes in TAB1. The methods in the upper group adopt the VGG-16 architecture as the base network and we show that our approach performs favorably against state-of-the-art adaptations via feature BID14 BID36, pixel-level BID15, and output space BID31 alignments. In the bottom group, we further utilize the stronger ResNet-101 base network and compare our with BID31 under two settings, i.e., feature and output space adaptations. We show that the proposed method improves the IoU with a gain of 1.8% and achieves the best IoU on 14 out of the 19 categories. In TAB2, we show for adapting SYNTHIA to Cityscapes and similar improvements are observed comparing with state-of-the-art methods. In addition, we shows visual comparisons in Figure 5 and more are presented in the appendix. Cross-city Case. Adapting between real images across different cities and conditions is an important scenario for practical applications. We choose a challenge case where the weather condition is different (i.e., sunny v.s rainy) in two cities by adapting Cityscapes to Oxford RobotCar. The proposed Target Image Ground Truth Before Adaptation Global Alignment Ours Figure 5: Example for GTA5-to-Cityscapes. Our method often generates the segmentation with more details (e.g., sidewalk and pole) while producing less noisy regions. BID31, we run the authors' released code and obtain a mean IoU of 63.6%, which is 1.4% lower than the proposed method. Further and comparisons are provided in the appendix. In this paper, we present a domain adaptation method for structured output via a general framework that combines global and patch-level alignments. The global alignment is achieved by the output space adaptation, while the patch-level one is performed via learning discriminative representations of patches across domains. To learn such patch-level representations, we propose to construct a clustered space of the source patches and adopt an adversarial learning scheme to push the target patch distributions closer to the source ones. We conduct extensive ablation study and experiments to validate the effectiveness of the proposed method under numerous challenges on semantic segmentation, including synthetic-to-real and cross-city scenarios, and show that our approach performs favorably against existing algorithms. To train the model in an end-to-end manner, we randomly sample one image from each of the source and target domain (i.e., batch size as 1) in a training iteration. Then we follow the optimization strategy as described in Section 3.4 of the paper. TAB3 shows the image and patch sizes during training and testing. Note that, the aspect ratio of the image is always maintained (i.e., no cropping) and then the image is down-sampled to the size as in the table. BID12 can be used as a loss in our model to push the target feature representation F t to one of the source clusters. To add this regularization, we replace the adversarial loss on the patch level with an entropy loss as in BID21 (u,v,k), H is the information entropy function, σ is the softmax function, and τ is the temperature of the softmax. The model with adding this entropy regularization achieves the IoU as 41.9%, that is lower than the proposed patchlevel adversarial alignment as 43.2%. The reason is that, different from the entropy minimization approach that does not use the source distribution as the guidance, our model learns discriminative representations for the target patches by pushing them closer to the source distribution in the clustered space guided by the label histogram. DISPLAYFORM0 In FIG3, we show example patches from the source and target domains corresponding to the t-SNE visualization. For each group in the clustered space via t-SNE, we show that source and target patches share high similarity between each other, which demonstrates the effectiveness of the proposed patch-level alignment. In TAB4, we present the complete for adapting Cityscapes (sunny condition) to Oxford RobotCar (rainy scene). We compare the proposed method with the model without adaptation and the output space adaptation approach BID31. More qualitative are provided in FIG4 and 8. We provide more visual comparisons for GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes scenarios from Figure 9 to Figure 11. In each row, we present the of the model without adaptation, output space adaptation BID31, and the proposed method. We show that our approach often yields better segmentation outputs with more details and produces less noisy regions. We sequentially show images in a video and their adapted segmentations generated by our method. Target Image Ground Truth Before Adaptation Global Alignment Ours Figure 9: Example of adapted segmentation for the GTA5-to-Cityscapes setting. For each target image, we show before adaptation, output space adaptation BID31, and the proposed method. Target Image Ground Truth Before Adaptation Global Alignment Ours Figure 10: Example of adapted segmentation for the GTA5-to-Cityscapes setting. For each target image, we show before adaptation, output space adaptation BID31, and the proposed method. Target Image Ground Truth Before Adaptation Global Alignment Ours Figure 11: Example of adapted segmentation for the SYNTHIA-to-Cityscapes setting. For each target image, we show before adaptation, output space adaptation BID31, and the proposed method.

A domain adaptation method for structured output via learning patch-level discriminative feature representations

This paper proposes a deep generative classifier which is effective to detect out-of-distribution samples as well as classify in-distribution samples, by integrating the concept of Gaussian discriminant analysis into deep neural networks.


Question: Represent the scientific article to find a short TLDR

Question: Many machine learning algorithms are vulnerable to almost imperceptible perturbations of their inputs. So far it was unclear how much risk adversarial perturbations carry for the safety of real-world machine learning applications because most methods used to generate such perturbations rely either on detailed model information (gradient-based attacks) or on confidence scores such as class probabilities (score-based attacks), neither of which are available in most real-world scenarios. In many such cases one currently needs to retreat to transfer-based attacks which rely on cumbersome substitute models, need access to the training data and can be defended against. Here we emphasise the importance of attacks which solely rely on the final model decision. Such decision-based attacks are applicable to real-world black-box models such as autonomous cars, need less knowledge and are easier to apply than transfer-based attacks and are more robust to simple defences than gradient- or score-based attacks. Previous attacks in this category were limited to simple models or simple datasets. Here we introduce the Boundary Attack, a decision-based attack that starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. The attack is conceptually simple, requires close to no hyperparameter tuning, does not rely on substitute models and is competitive with the best gradient-based attacks in standard computer vision tasks like ImageNet. We apply the attack on two black-box algorithms from Clarifai.com. The Boundary Attack in particular and the class of decision-based attacks in general open new avenues to study the robustness of machine learning models and raise new questions regarding the safety of deployed machine learning systems. An implementation of the attack is available as part of Foolbox (https://github.com/bethgelab/foolbox). Figure 1: (Left) Taxonomy of adversarial attack methods. The Boundary Attack is applicable to realworld ML algorithms because it only needs access to the final decision of a model (e.g. class-label or transcribed sentence) and does not rely on model information like the gradient or the confidence scores. (Right) Application to the Clarifai Brand Recognition Model. Many high-performance machine learning algorithms used in computer vision, speech recognition and other areas are susceptible to minimal changes of their inputs BID26. As a concrete example, a modern deep neural network like VGG-19 trained on object recognition might perfectly recognize the main object in an image as a tiger cat, but if the pixel values are only slightly perturbed in a specific way then the prediction of the very same network is drastically altered (e.g. to bus). These so-called adversarial perturbations are ubiquitous in many machine learning models and are often imperceptible to humans. Algorithms that seek to find such adversarial perturbations are generally denoted as adversarial attacks. Adversarial perturbations have drawn interest from two different sides. On the one side, they are worrisome for the integrity and security of deployed machine learning algorithms such as autonomous cars or face recognition systems. Minimal perturbations on street signs (e.g. turning a stop-sign into a 200 km/h speed limit) or street lights (e.g. turning a red into a green light) can have severe consequences. On the other hand, adversarial perturbations provide an exciting spotlight on the gap between the sensory information processing in humans and machines and thus provide guidance towards more robust, human-like architectures. Adversarial attacks can be roughly divided into three categories: gradient-based, score-based and transfer-based attacks (cp. Figure 1). Gradient-based and score-based attacks are often denoted as white-box and oracle attacks respectively, but we try to be as explicit as possible as to what information is being used in each category 1. A severe problem affecting attacks in all of these categories is that they are surprisingly straight-forward to defend against:• Gradient-based attacks. Most existing attacks rely on detailed model information including the gradient of the loss w.r.t. the input. Examples are the Fast-Gradient Sign Method (FGSM), the Basic Iterative Method (BIM) BID11, DeepFool , the Jacobian-based Saliency Map Attack (JSMA) BID20, Houdini BID5 and the Carlini & Wagner attack BID2. Defence: A simple way to defend against gradient-based attacks is to mask the gradients, for example by adding non-differentiable elements either implicitly through means like defensive distillation BID21 or saturated non-linearities BID18, or explicitly through means like non-differentiable classifiers BID15 ).• Score-based attacks. A few attacks are more agnostic and only rely on the predicted scores (e.g. class probabilities or logits) of the model. On a conceptual level these attacks use the predictions to numerically estimate the gradient. This includes black-box variants of JSMA BID17 and of the Carlini & Wagner attack BID4 as well as generator networks that predict adversarials BID8. Defence: It is straight-forward to severely impede the numerical gradient estimate by adding stochastic elements like dropout into the model. Also, many robust training methods introduce a sharp-edged plateau around samples BID28 which not only masks gradients themselves but also their numerical estimate.• Transfer-based attacks. Transfer-based attacks do not rely on model information but need information about the training data. This data is used to train a fully observable substitute model from which adversarial perturbations can be synthesized BID22. They rely on the empirical observation that adversarial examples often transfer between models. If adversarial examples are created on an ensemble of substitute models the success rate on the attacked model can reach up to 100% in certain scenarios BID13. Defence: A recent defence method against transfer attacks BID28, which is based on robust training on a dataset augmented by adversarial examples from an ensemble of substitute models, has proven highly successful against basically all attacks in the 2017 Kaggle Competition on Adversarial Attacks 2.The fact that many attacks can be easily averted makes it often extremely difficult to assess whether a model is truly robust or whether the attacks are just too weak, which has lead to premature claims of robustness for DNNs BID3.This motivates us to focus on a category of adversarial attacks that has so far received fairly little attention:• Decision-based attacks. Direct attacks that solely rely on the final decision of the model (such as the top-1 class label or the transcribed sentence).The delineation of this category is justified for the following reasons: First, compared to score-based attacks decision-based attacks are much more relevant in real-world machine learning applications where confidence scores or logits are rarely accessible. At the same time decision-based attacks have the potential to be much more robust to standard defences like gradient masking, intrinsic stochasticity or robust training than attacks from the other categories. Finally, compared to transferbased attacks they need much less information about the model (neither architecture nor training data) and are much simpler to apply. There currently exists no effective decision-based attack that scales to natural datasets such as ImageNet and is applicable to deep neural networks (DNNs). The most relevant prior work is a variant of transfer attacks in which the training set needed to learn the substitute model is replaced by a synthetic dataset (b). This synthetic dataset is generated by the adversary alongside the training of the substitute; the labels for each synthetic sample are drawn from the black-box model. While this approach works well on datasets for which the intra-class variability is low (such as MNIST) it has yet to be shown that it scales to more complex natural datasets such as CIFAR or ImageNet. Other decision-based attacks are specific to linear or convex-inducing classifiers BID6 BID14 BID19 and are not applicable to other machine learning models. The work by BID0 basically stands between transfer attacks and decision-based attacks in that the substitute model is trained on a dataset for which the labels have been observed from the black-box model. This attack still requires knowledge about the data distribution on which the black-box models was trained on and so we don't consider it a pure decision-based attack. Finally, some naive attacks such as a line-search along a random direction away from the original sample can qualify as decision-based attacks but they induce large and very visible perturbations that are orders of magnitude larger than typical gradient-based, score-based or transfer-based attacks. Throughout the paper we focus on the threat scenario in which the adversary aims to change the decision of a model (either targeted or untargeted) for a particular input sample by inducing a minimal perturbation to the sample. The adversary can observe the final decision of the model for arbitrary inputs and it knows at least one perturbation, however large, for which the perturbed sample is adversarial. The contributions of this paper are as follows:• We emphasise decision-based attacks as an important category of adversarial attacks that are highly relevant for real-world applications and important to gauge model robustness.• We introduce the first effective decision-based attack that scales to complex machine learning models and natural datasets. The Boundary Attack is conceptually surprisingly simple, extremely flexible, requires little hyperparameter tuning and FORMULA6 is competitive with the best gradient-based attacks in both targeted and untargeted computer vision scenarios.• We show that the Boundary Attack is able to break previously suggested defence mechanisms like defensive distillation.• We demonstrate the practical applicability of the Boundary Attack on two black-box machine learning models for brand and celebrity recognition available on Clarifai.com. Throughout the paper we use the following notation: o refers to the original input (e.g. an image), y = F (o) refers to the full prediction of the model F (·) (e.g. logits or probabilities), y max is the predicted label (e.g. class-label). Similarly,õ refers to the adversarially perturbed image,õ k refers to the perturbed image at the k-th step of an attack algorithm. Vectors are denoted in bold. The basic intuition behind the boundary attack algorithm is depicted in Figure 2: the algorithm is initialized from a point that is already adversarial and then performs a random walk along the boundary between the adversarial and the non-adversarial region such that it stays in the adversarial region and the distance towards the target image is reduced. In other words we perform rejection sampling with a suitable proposal distribution P to find progressively smaller adversarial perturbations according to a given adversarial criterion c. The basic logic of the algorithm is described in Algorithm 1, each individual building block is detailed in the next subsections. DISPLAYFORM0 Algorithm 1: Minimal version of the Boundary Attack. The Boundary Attack needs to be initialized with a sample that is already adversarial 3. In an untargeted scenario we simply sample from a maximum entropy distribution given the valid domain of the input. In the computer vision applications below, where the input is constrained to a range of per pixel, we sample each pixel in the initial imageõ 0 from a uniform distribution U. We reject samples that are not adversarial. In a targeted scenario we start from any sample that is classified by the model as being from the target class. The efficiency of the algorithm crucially depends on the proposal distribution P, i.e. which random directions are explored in each step of the algorithm. The optimal proposal distribution will generally depend on the domain and / or model to be attacked, but for all vision-related problems tested here a very simple proposal distribution worked surprisingly well. The basic idea behind this proposal distribution is as follows: in the k-th step we want to draw perturbations η k from a maximum entropy distribution subject to the following constraints:1. The perturbed sample lies within the input domain, DISPLAYFORM0 2. The perturbation has a relative size of δ, DISPLAYFORM1 3. The perturbation reduces the distance of the perturbed image towards the original input by a relative amount, DISPLAYFORM2 Adjusting step-size of #1 50% of orthogonal perturbations should be within adversarial region Success rate of total perturbation should be higher then threshold (e.g. 25%).classified incorrectly (adversarial) Figure 2: (Left) In essence the Boundary Attack performs rejection sampling along the boundary between adversarial and non-adversarial images. (Center) In each step we draw a new random direction by (#1) drawing from an iid Gaussian and projecting on a sphere, and by (#2) making a small move towards the target image. (Right) The two step-sizes (orthogonal and towards the original input) are dynamically adjusted according to the local geometry of the boundary. In practice it is difficult to sample from this distribution, and so we resort to a simpler heuristic: first, we sample from an iid Gaussian distribution η k i ∼ N and then rescale and clip the sample such that FORMULA1 and FORMULA2 hold. In a second step we project η k onto a sphere around the original image o such DISPLAYFORM0 and FORMULA1 hold. We denote this as the orthogonal perturbation and use it later for hyperparameter tuning. In the last step we make a small movement towards the original image such that and hold. For high-dimensional inputs and small δ, the constraint will also hold approximately. A typical criterion by which an input is classified as adversarial is misclassification, i.e. whether the model assigns the perturbed input to some class different from the class label of the original input. Another common choice is targeted misclassification for which the perturbed input has to be classified in a given target class. Other choices include top-k misclassification (the top-k classes predicted for the perturbed input do not contain the original class label) or thresholds on certain confidence scores. Outside of computer vision many other choices exist such as criteria on the worderror rates. In comparison to most other attacks, the Boundary Attack is extremely flexible with regards to the adversarial criterion. It basically allows any criterion (including non-differentiable ones) as long as for that criterion an initial adversarial can be found (which is trivial in most cases). The Boundary Attack has only two relevant parameters: the length of the total perturbation δ and the length of the step towards the original input (see Fig. 2). We adjust both parameters dynamically according to the local geometry of the boundary. The adjustment is inspired by Trust Region methods. In essence, we first test whether the orthogonal perturbation is still adversarial. If this is true, then we make a small movement towards the target and test again. The orthogonal step tests whether the step-size is small enough so that we can treat the decision boundary between the adversarial and the non-adversarial region as being approximately linear. If this is the case, then we expect around 50% of the orthogonal perturbations to still be adversarial. If this ratio is much lower, we reduce the step-size δ, if it is close to 50% or higher we increase it. If the orthogonal perturbation is still adversarial we add a small step towards the original input. The maximum size of this step depends on the angle of the decision boundary in the local neighbourhood (see also Figure 2). If the success rate is too small we decrease, if it is too large we increase it. Typically, the closer we get to the original image, the flatter the decision boundary becomes and the smaller has to be to still make progress. The attack is converged whenever converges to zero. We quantify the performance of the Boundary Attack on three different standard datasets: MNIST , CIFAR-10 (BID10) and ImageNet-1000 BID7. To make the comparison with previous as easy and transparent as possible, we here use the same MNIST and CIFAR networks as BID2 4. In a nutshell, both the MNIST and CIFAR model feature nine layers with four convolutional layers, two max-pooling layers and two fully-connected layers. For all details, including training parameters, we refer the reader to BID2. On ImageNet we use the pretrained networks VGG-19 , ResNet-50 BID9 and Inception-v3 BID27 provided by Keras 5.We evaluate the Boundary Attack in two settings: an untargeted setting in which the adversarial perturbation flips the label of the original sample to any other label, and a targeted setting in which the adversarial flips the label to a specific target class. In the untargeted setting we compare the Boundary Attack against three gradient-based attack algorithms:• Fast-Gradient Sign Method (FGSM). FGSM is among the simplest and most widely used untargeted adversarial attack methods. In a nutshell, FGSM computes the gradient g = ∇ o L(o, c) that maximizes the loss L for the true class-label c and then seeks the smallest for which o+ ·g is still adversarial. We use the implementation in Foolbox 0.10.0 BID24 ).• DeepFool. DeepFool is a simple yet very effective attack. In each iteration it computes for each class = 0 the minimum distance d(, 0) that it takes to reach the class boundary by approximating the model classifier with a linear classifier. It then makes a corresponding step in the direction of the class with the smallest distance. We use the implementation in Foolbox 0.10.0 BID24. DISPLAYFORM0 The attack by Carlini & Wagner BID2 ) is essentially a refined iterative gradient attack that uses the Adam optimizer, multiple starting points, a tanh-nonlinearity to respect box-constraints and a max-based adversarial constraint function. We use the original implementation provided by the authors with all hyperparameters left at their default values 4.To evaluate the success of each attack we use the following metric: let η A,M (o i) ∈ R N be the adversarial perturbation that the attack A finds on model M for the i-th sample o i. The total score S A for A is the median squared L2-distance across all samples, DISPLAYFORM1 For MNIST and CIFAR we evaluate 1000 randomly drawn samples from the validation set, for ImageNet we use 250 images. In the untargeted setting an adversarial is any image for which the predicted label is different from the label of the original image. We show adversarial samples synthesized by the Boundary Attack for each dataset in Figure 3. The score FORMULA6 Despite its simplicity the Boundary Attack is competitive with gradient-based attacks in terms of the minimal adversarial perturbations and very stable against the choice of the initial point (Figure 5). This finding is quite remarkable given that gradient-based attacks can fully observe the model whereas the Boundary Attack is severely restricted to the final class prediction. To compensate for this lack of information the Boundary Attack needs many more iterations to converge. As a rough measure for the run-time of an attack independent of the quality of its implementation we tracked the number of forward passes (predictions) and backward passes (gradients) through the network requested by each of the attacks to find an adversarial for ResNet-50: averaged over 20 samples and under the same conditions as before, DeepFool needs about 7 forward and 37 backward passes, the Carlini & Wagner attack requires 16.000 forward and the same number of backward passes, and the Boundary Attack uses 1.200.000 forward passes but zero backward passes. While that (unsurprisingly) makes the Boundary Attack more expensive to run it is important to note that the Boundary Attacks needs much fewer iterations if one is only interested in imperceptible perturbations, see figures 4 and 6. We can also apply the Boundary Attack in a targeted setting. In this case we initialize the attack from a sample of the target class that is correctly identified by the model. A sample trajectory from the starting point to the original sample is shown in FIG2. After around 10 4 calls to the model 4.4e-08 4.4e-08 4.5e-08 4.6e-08 4.8e-089.8e-08 9.9e-08 9.9e-08 1.0e-07 1.0e-07 As discussed in the introduction, many attack methods are straight-forward to defend against. One common nuisance is gradient masking in which a model is implicitely or explicitely modified to yield masked gradients. An interesting example is the saturated sigmoid network BID18 in which an additional regularization term leads the sigmoid activations to saturate, which in turn leads to vanishing gradients and failing gradient-based attacks.Another example is defensive distillation BID21. In a nutshell defensive distillation uses a temperature-augmented softmax of the type DISPLAYFORM0 and works as follows:1. Train a teacher network as usual but with temperature T.2. Train a distilled network-with the same architecture as the teacher-on the softmax outputs of the teacher. Both the distilled network and the teacher use temperature T.3. Evaluate the distilled network at temperature T = 1 at test time. Initial were promising: the success rate of gradient-based attacks dropped from close to 100% down to 0.5%. It later became clear that the distilled networks only appeared to be robust because they masked their gradients of the cross-entropy loss BID3: as the temperature of the softmax is decreased at test time, the input to the softmax increases by a factor of T and so the probabilities saturate at 0 and 1. This leads to vanishing gradients of the cross-entropy loss w.r.t. to the input on which gradient-based attacks rely. If the same attacks are instead applied to the logits the success rate recovers to almost 100% BID2.Decision-based attacks are immune to such defences. To demonstrate this we here apply the Boundary Attack to two distilled networks trained on MNIST and CIFAR. The architecture is the same as in section 3 and we use the implementation and training protocol by BID2 which is available at https://github.com/carlini/nn_robust_attacks. Most importantly, we do not operate on the logits but provide only the class label with maximum probability to the Boundary Attack. The size of the adversarial perturbations that the Boundary Attack finds is fairly similar for the distilled and the undistilled network. This demonstrates that defensive distillation does not significantly increase the robustness of network models and that the Boundary Attack is able to break defences based on gradient masking. In many real-world machine learning applications the attacker has no access to the architecture or the training data but can only observe the final decision. This is true for security systems (e.g. face identification), autonomous cars or speech recognition systems like Alexa or Cortana. In this section we apply the Boundary Attack to two models of the cloud-based computer vision API by Clarifai 6. The first model identifies brand names in natural images and recognizes over 500 brands. The second model identifies celebrities and can recognize over 10.000 individuals. Multiple identifications per image are possible but we only consider the one with the highest confidence score. It is important to note that Clarifai does provide confidence scores for each identified class (but not for all possible classes). However, in our experiments we do not provide this confidence score to the Boundary Attack. Instead, our attack only receives the name of the identified object (e.g. Pepsi or Verizon in the brand-name detection task).We selected several samples of natural images with clearly visible brand names or portraits of celebrities. We then make a square crop and resize the image to 100 × 100 pixels. For each sample we make sure that the brand or the celebrity is clearly visible and that the corresponding Clarifai We show five samples for each model alongside the adversarial image generated by the Boundary Attack in FIG3. We generally observed that the Clarifai models were more difficult to attack than ImageNet models like VGG-19: while for some samples we did succeed to find adversarial perturbations of the same order (1e −7) as in section 3 (e.g. for Shell or SAP), most adversarial perturbations were on the order of 1e −2 to 1e −3 ing in some slightly noticeable noise in some adversarial examples. Nonetheless, for most samples the original and the adversarial image are close to being perceptually indistinguishable. In this paper we emphasised the importance of a mostly neglected category of adversarial attacksdecision-based attacks-that can find adversarial examples in models for which only the final decision can be observed. We argue that this category is important for three reasons: first, attacks in this class are highly relevant for many real-world deployed machine learning systems like autonomous cars for which the internal decision making process is unobservable. Second, attacks in this class do not rely on substitute models that are trained on similar data as the model to be attacked, thus making real-world applications much more straight-forward. Third, attacks in this class have the potential to be much more robust against common deceptions like gradient masking, intrinsic stochasticity or robust training. We also introduced the first effective attack in this category that is applicable to general machine learning algorithms and complex natural datasets: the Boundary Attack. At its core the Boundary Attack follows the decision boundary between adversarial and non-adversarial samples using a very simple rejection sampling algorithm in conjunction with a simple proposal distribution and a dynamic step-size adjustment inspired by Trust Region methods. Its basic operating principlestarting from a large perturbation and successively reducing it-inverts the logic of essentially all previous adversarial attacks. Besides being surprisingly simple, the Boundary attack is also extremely flexible in terms of the possible adversarial criteria and performs on par with gradient-based attacks on standard computer vision tasks in terms of the size of minimal perturbations. The mere fact that a simple constrained iid Gaussian distribution can serve as an effective proposal perturbation for each step of the Boundary attack is surprising and sheds light on the brittle information processing of current computer vision architectures. Nonetheless, there are many ways in which the Boundary attack can be made even more effective, in particular by learning a suitable proposal distribution for a given model or by conditioning the proposal distribution on the recent history of successful and unsuccessful proposals. Decision-based attacks will be highly relevant to assess the robustness of machine learning models and to highlight the security risks of closed-source machine learning systems like autonomous cars. We hope that the Boundary attack will inspire future work in this area.

Represent the TLDR for finding the corresponding scientific article

A novel adversarial attack that can directly attack real-world black-box machine learning models without transfer.

Represent the short summary for finding the corresponding scientific article (long)

We can identify prototypical and outlier examples in machine learning that are quantifiably very different, and make use of them to improve many aspects of neural networks.


Question: Represent the scientific article to find a short TLDR

Question: We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy. Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent. In particular, we add user-level privacy protection to the federated averaging algorithm, which makes large step updates from user-level data. Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work. We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset. Deep recurrent models like long short-term memory (LSTM) recurrent neural networks (RNNs) have become a standard building block in modern approaches to language modeling, with applications in speech recognition, input decoding for mobile keyboards, and language translation. Because language usage varies widely by problem domain and dataset, training a language model on data from the right distribution is critical. For example, a model to aid typing on a mobile keyboard is better served by training data typed in mobile apps rather than from scanned books or transcribed utterances. However, language data can be uniquely privacy sensitive. In the case of text typed on a mobile phone, this sensitive information might include passwords, text messages, and search queries. In general, language data may identify a speaker-explicitly by name or implicitly, for example via a rare or unique phrase-and link that speaker to secret or sensitive information. Ideally, a language model's parameters would encode patterns of language use common to many users without memorizing any individual user's unique input sequences. However, we know convolutional NNs can memorize arbitrary labelings of the training data BID22 and recurrent language models are also capable of memorizing unique patterns in the training data BID5. Recent attacks on neural networks such as those of BID19 underscore the implicit risk. The main goal of our work is to provide a strong guarantee that the trained model protects the privacy of individuals' data without undue sacrifice in model quality. We are motivated by the problem of training models for next-word prediction in a mobile keyboard, and use this as a running example. This problem is well suited to the techniques we introduce, as differential privacy may allow for training on data from the true distribution (actual mobile usage) rather than on proxy data from some other source that would produce inferior models. However, to facilitate reproducibility and comparison to non-private models, our experiments are conducted on a public dataset as is standard in differential privacy research. The remainder of this paper is structured around the following contributions:1. We apply differential privacy to model training using the notion of user-adjacent datasets, leading to formal guarantees of user-level privacy, rather than privacy for single examples.4. In extensive experiments in §3, we offer guidelines for parameter tuning when training complex models with differential privacy guarantees. We show that a small number of experiments can narrow the parameter space into a regime where we pay for privacy not in terms of a loss in utility but in terms of an increased computational cost. We now introduce a few preliminaries. Differential privacy (DP) BID10 BID8 BID9 ) provides a well-tested formalization for the release of information derived from private data. Applied to machine learning, a differentially private training mechanism allows the public release of model parameters with a strong guarantee: adversaries are severely limited in what they can learn about the original training data based on analyzing the parameters, even when they have access to arbitrary side information. Formally, it says: Definition 1. Differential Privacy: A randomized mechanism M: D → R with a domain D (e.g., possible training datasets) and range R (e.g., all possible trained models) satisfies (, δ)-differential privacy if for any two adjacent datasets d, d ∈ D and for any subset of outputs S ⊆ R it holds that DISPLAYFORM0 The definition above leaves open the definition of adjacent datasets which will depend on the application. Most prior work on differentially private machine learning (e.g. BID7 BID4 ; BID0 BID21 BID16) deals with example-level privacy: two datasets d and d are defined to be adjacent if d can be formed by adding or removing a single training example from d. We remark that while the recent PATE approach of BID16 can be adapted to give user-level privacy, it is not suited for a language model where the number of classes (possible output words) is large. For problems like language modeling, protecting individual examples is insufficient-each typed word makes its own contribution to the RNN's training objective, so one user may contribute many thousands of examples to the training data. A sensitive word or phrase may be typed several times by an individual user, but it should still be protected.2 In this work, we therefore apply the definition of differential privacy to protect whole user histories in the training set. This user-level privacy is ensured by using an appropriate adjacency relation:Definition 2. User-adjacent datasets: Let d and d be two datasets of training examples, where each example is associated with a user. Then, d and d are adjacent if d can be formed by adding or removing all of the examples associated with a single user from d. Model training that satisfies differential privacy with respect to datasets that are user-adjacent satisfies the intuitive notion of privacy we aim to protect for language modeling: the presence or absence of any specific user's data in the training set has an imperceptible impact on the (distribution over) the parameters of the learned model. It follows that an adversary looking at the trained model cannot infer whether any specific user's data was used in the training, irrespective of what auxiliary information they may have. In particular, differential privacy rules out memorization of sensitive information in a strong information theoretic sense. Our private algorithm relies heavily on two prior works: the FederatedAveraging (or FedAvg) algorithm of BID14, which trains deep networks on user-partitioned data, and the moments accountant of BID0, which provides tight composition guarantees for the repeated application of the Gaussian mechanism combined with amplification-via-sampling. While we have attempted to make the current work as self-contained as possible, the above references provide useful . FedAvg was introduced by BID14 for federated learning, where the goal is to train a shared model while leaving the training data on each user's mobile device. Instead, devices download the current model and compute an update by performing local computation on their dataset. It is worthwhile to perform extra computation on each user's data to minimize the number of communication rounds required to train a model, due to the significantly limited bandwidth when training data remains decentralized on mobile devices. We observe, however, that FedAvg is of interest even in the datacenter when DP is applied: larger updates are more resistant to noise, and fewer rounds of training can imply less privacy cost. Most importantly, the algorithm naturally forms peruser updates based on a single user's data, and these updates are then averaged to compute the final update applied to the shared model on each round. As we will see, this structure makes it possible to extend the algorithm to provide a user-level differential privacy guarantee. We also evaluate the FederatedSGD algorithm, essentially large-batch SGD where each minibatch is composed of "microbatches" that include data from a single distinct user. In some datacenter applications FedSGD might be preferable to FedAvg, since fast networks make it more practical to run more iterations. However, those additional iterations come at a privacy cost. Further, the privacy benefits of federated learning are nicely complementary to those of differential privacy, and FedAvg can be applied in the datacenter as well, so we focus on this algorithm while showing that our also extend to FedSGD.Both FedAvg and FedSGD are iterative procedures, and in both cases we make the following modifications to the non-private versions in order to achieve differential privacy: A) We use random-sized batches where we select users independently with probability q, rather than always selecting a fixed number of users. B) We enforce clipping of per-user updates so the total update has bounded L 2 norm. C) We use different estimators for the average update (introduced next). D) We add Gaussian noise to the final average update. The pseudocode for DP-FedAvg and DP-FedSGD is given as Algorithm 1. In the remainder of this section, we introduce estimators for C) and then different clipping strategies for B). Adding the sampling procedure from A) and noise added in D) allows us to apply the moments accountant to bound the total privacy loss of the algorithm, given in Theorem 1. Finally, we consider the properties of the moments accountant that make training on large datasets particular attractive. Bounded-sensitivity estimators for weighted average queries Randomly sampling users (or training examples) by selecting each independently with probability q is crucial for proving low privacy loss through the use of the moments accountant BID0. However, this procedure produces variable-sized samples C, and when the quantity to be estimated f (C) is an average rather than a sum (as in computing the weighted average update in FedAvg or the average loss on a minibatch in SGD with example-level DP), this has ramifications for the sensitivity of the query f. Specifically, we consider weighted databases d where each row k ∈ d is associated with a particular user, and has an associated weight w k ∈. This weight captures the desired influence of the, 1 for all users k W = k∈d w k for each round t = 0, 1, 2,... do C t ← (sample users with probability q) DISPLAYFORM0 Algorithm 1: The main loop for DP-FedAvg and DP-FedSGD, the only difference being in the user update function (UserUpdateFedAvg or UserUpdateFedSGD). The calls on the moments accountant M refer to the API of BID1. row on the final outcome. For example, we might think of row k containing n k different training examples all generated by user k, with weight w k proportional to n k. We are then interested in a bounded-sensitivity estimate of f (C) = k∈C w k ∆ k k∈C w k for per-user vectors ∆ k, for example to estimate the weighted-average user update in FedAvg. Let W = k w k. We consider two such estimators: DISPLAYFORM1 Notef f is an unbiased estimator, since E[k∈C w k] = qW. On the other hand,f c matches f exactly as long as we have sufficient weight in the sample. For privacy protection, we need to control the sensitivity of our query functionf, defined as S(f) = max C,k f (C ∪ {k}) −f (C) 2, where the added user k can have arbitrary data. The lower-bound qW min on the denominator off c is necessary to control sensitivity. Assuming each w k ∆ k has bounded norm, we have: Lemma 1. If for all users k we have w k ∆ k 2 ≤ S, then the sensitivity of the two estimators is bounded as DISPLAYFORM2 A proof is given in Appendix §A.Clipping strategies for multi-layer models Unfortunately, when the user vectors ∆ k are gradients (or sums of gradients) from a neural network, we will generally have no a priori bound 3 S such that ∆ k ≤ S. Thus, we will need to "clip" our updates to enforce such a bound before applying f f orf c. For a single vector ∆, we can apply a simple L 2 projection when necessary: and report the value of for which (, δ)-differential privacy holds after 1 to 10 6 rounds. For large datasets, additional rounds of training incur only a minimal additional privacy loss. However, for deep networks it is more natural to treat the parameters of each layer as a separate vector. The updates to each of these layers could have vastly different L 2 norms, and so it can be preferable to clip each layer separately. DISPLAYFORM3 Formally, suppose each update DISPLAYFORM4 We consider the following clipping strategies, both of which ensure the total update has norm at most S:1. Flat clipping Given an overall clipping parameter S, we clip the concatenation of all the layers as ∆ k = π(∆ k, S). 2. Per-layer clipping Given a per-layer clipping parameter S j for each layer, we set DISPLAYFORM5 j. The simplest model-independent choice is to take DISPLAYFORM6 for all j, which we use in experiments. We remark here that clipping itself leads to additional bias, and ideally, we would choose the clipping parameter to be large enough that nearly all updates are smaller than the clip value. On the other hand, a larger S will require more noise in order to achieve privacy, potentially slowing training. We treat S as a hyper-parameter and tune it. A privacy guarantee Once the sensitivity of the chosen estimator is bounded, we may add Gaussian noise scaled to this sensitivity to obtain a privacy guarantee. A simple approach is to use an (, δ)-DP bound for this Gaussian mechanism, and apply the privacy amplification lemma and the advanced composition theorem to get a bound on the total privacy cost. We instead use the Moments Accountant of BID0 to achieve much tighter privacy bounds. The moments accountant for the sampled Gaussian mechanism upper bounds the total privacy cost of T steps of the Gaussian mechanism with noise N (0, σ 2) for σ = z · S, where z is a parameter, S is the sensitivity of the query, and each row is selected with probability q. Given a δ > 0, the accountant gives an for which this mechanism satisfies (, δ)-DP. The following theorem is a slight generalization of the in BID0; see §A for a proof sketch. Theorem 1. For the estimator (f f,f c), the moments accountant of the sampled Gaussian mechanism correctly computes the privacy loss with the noise scale of z = σ/S and steps T, where S = S/qW for (f f) and 2S/qW min for (f c).Differential privacy for large datasets We use the implementation of the moments accountant from BID1. The moments accountant makes strong use of amplification via sampling, which means increasing dataset size makes achieving high levels of privacy significantly easier. Table 1 summarizes the privacy guarantees offered as we vary some of the key parameters. The takeaway from this table is that as long as we can afford the cost in utility of adding noise proportional to z times the sensitivity of the updates, we can get reasonable privacy guarantees over a large range of parameters. The size of the dataset has a modest impact on the privacy cost of a single query (1 round column), but a large effect on the number of queries that can be run without significantly increasing the privacy cost (compare the 10 6 round column). For example, on a dataset with 10 users, the privacy upper bound is nearly constant between 1 and 10 6 calls to the mechanism (that is, rounds of the optimization algorithm).There is only a small cost in privacy for increasing the expected number of (equally weighted) users C = qW = qK selected on each round as long asC remains a small fraction of the size of the total dataset. Since the sensitivity of an average query decreases like 1/C (and hence the amount of noise we need to add decreases proportionally), we can increaseC until we arrive at a noise level that does not adversely effect the optimization process. We show empirically that such a level exists in the experiments. In this section, we evaluate DP-FedAvg while training an LSTM RNN tuned for language modeling in a mobile keyboard. We vary noise, clipping, and the number of users per round to develop an intuition of how privacy affects model quality in practice. We defer our experimental on FedSGD as well as on models with larger dictionaries to Appendix §D. To summarize, they show that FedAvg gives better privacy-utility trade-offs than FedSGD, and that our empirical extend to larger dictionaries with relatively little need for additional parameter tuning despite the significantly larger models. Some less important plots are deferred to §C.Model structure The goal of a language model is to predict the next word in a sequence s t from the preceding words s 0...s t−1. The neural language model architecture used here is a variant of the LSTM recurrent neural network BID13 trained to predict the next word (from a fixed dictionary) given the current word and a state vector passed from the previous time step. LSTM language models are competitive with traditional n-gram models BID20 and are a standard baseline for a variety of ever more advanced neural language model architectures BID12 BID15 BID11. Our model uses a few tricks to decrease the size for deployment on mobile devices (total size is 1.35M parameters), but is otherwise standard. We evaluate using AccuracyTop1, the probability that the word to which the model assigns highest probability is correct. Details on the model and evaluation metrics are given in §B. All training began from a common random initialization, though for real-world applications pre-training on public data is likely preferable (see §B for additional discussion).Dataset We use a large public dataset of Reddit posts, as described by BID2. Critically for our purposes, each post in the database is keyed by an author, so we can group the data by these keys in order to provide user-level privacy. We preprocessed the dataset to K = 763, 430 users each with 1600 tokens. Thus, we take w k = 1 for all users, so W = K. We writeC = qK = qW for the expected number of users sampled per round. See §B for details on the dataset and preprocessing. To allow for frequent evaluation, we use a relatively small test set of 75122 tokens formed from random held-out posts. We evaluate accuracy every 20 rounds and plot metrics smoothed over 5 evaluations (100 rounds).Building towards DP: sampling, estimators, clipping, and noise Recall achieving differential privacy for FedAvg required a number of changes (§2, items A-D). In this section, we examine the impact of each of these changes, both to understand the immediate effects and to enable the selection of reasonable parameters for our final DP experiments. This sequence of experiments also provides a general road-map for applying differentially private training to new models and datasets. For these experiments, we use the FedAvg algorithm with a fixed learning rate of 6.0, which we verified was a reasonable choice in preliminary experiments. 4 In all FedAvg experiments, we used a local batch size of B = 8, an unroll size of 10 tokens, and made E = 1 passes over the local dataset; thus FedAvg processes 80 tokens per batch, processing a user's 1600 tokens in 20 batches per round. First, we investigate the impact of changing the estimator used for the average per-round update, as well as replacing a fixed sample of C = 100 users per round to a variable-sized sample formed by selecting each user with probability q = 100/763430 for an expectation ofC = 100 users. None of these changes significantly impacted the convergence rate of the algorithm (see Figure 5 in §C). In particular, the fixed denominator estimatorf f works just as well as the higher-sensitivity clipped-denominator estimatorf c. Thus, in the remaining experiments we focus on estimatorf f. Next, we investigate the impact of flat and per-layer clipping on the convergence rate of FedAvg. The model has 11 parameter vectors, and for per-layer clipping we simply chose to distribute the clipping budget equally across layers with S j = S/ √ 11. Figure 2 shows that choosing S ∈ has at most a small effect on convergence rate. Finally, Figure 3 shows the impact of various levels of per-coordinate Gaussian noise N (0, σ 2) added to the average update. Early in training, we see almost no loss in convergence for a noise of σ = 0.024; later in training noise has a larger effect, and we see a small decrease in convergence past σ = 0.012. These experiments, where we sample only an expected 100 users per round, are not sufficient to provide a meaningful privacy guarantee. We have S = 20.0 andC = qW = 100, so the sensitivity of estimatorf f is 20/100.0 = 0.2. Thus, to use the moments accountant with z = 1, we would need to add noise σ = 0.2 (dashed red vertical line), which destroys accuracy. Estimating the accuracy of private models for large datasets Continuing the above example, if instead we choose q soC = 1250, set the L 2 norm bound S = 15.0, then we have sensitivity Table 3: Count histograms recording how many of a model's (row's) top 10 predictions are found in the n = 10, 50, or 100 most frequent words in the corpus. Models that predict corpus top-n more frequently have more mass to the right.15/1250 = 0.012, and so we add noise σ = 0.012 and can apply the moments account with noise scale z = 1. The computation is now significantly more computationally expensive, but will give a guarantee of (1.97, 10 −9)-differential privacy after 3000 rounds of training. Because running such experiments is so computationally expensive, for experimental purposes it is useful to ask: does using an expected 1250 users per round produce a model with different accuracy than a model trained with only 100 expected users per round? If the answer is no, we can train a model with C = 100 and a particular noise level σ, and use that model to estimate the utility of a model trained with a much larger q (and hence a much better privacy guarantee). We can then run the moments accountant (without actually training) to numerically upper bound the privacy loss. To test this, we trained two models, both with S = 15 and σ = 0.012, one withC = 100 and one withC = 1250; recall the first model achieves a vacuous privacy guarantee, while the second achieves (1.97, 10 −9)-differential privacy after 3000 rounds. Figure 7 in §C shows the two models produce almost identical accuracy curves during training. Using this observation, we can use the accuracy of models trained withC = 100 to estimate the utility of private models trained with much largerC. See also Figure 6 in §C, which also shows diminishing returns for larger C for the standard FedAvg algorithm. FIG1 compares the true-average fixed-sample baseline model (see Figure 5 in §C) with models that use varying levels of clipping S and noise σ atC = 100. Using the above approach, we can use these experiments to estimate the utility of LSTMs trained with differential privacy for different sized datasets and different values ofC. TAB2 shows representative values settingC so that z = 1. For example, the model with σ = 0.003 and S = 15 is only worse than the baseline by an additive −0.13% in AccuracyTop1 and achieves (4.6, 10 −9)-differential privacy when trained with C = 5000 expected users per round. As a point of comparison, we have observed that training on a different corpus can cost an additive −2.50% in AccuracyTop1. Adjusting noise and clipping as training progresses FIG1 shows that as training progresses, each level of noise eventually becomes detrimental (the line drops somewhat below the baseline). This suggests using a smaller σ and correspondingly smaller S (thus fixing z so the privacy cost of each round is unchanged) as training progresses. FIG3 (and Figure 8 in §C) shows this can be effective. We indeed observe that early in training (red), S in the 10 -12.6 range works well (σ = 0.006 -0.0076). However, if we adjust the clipping/noise tradeoff after 4885 rounds of training and continue for another 6000, switching to S = 7.9 and σ = 0.0048 performs better. Comparing DP and non-DP models While noised training with DP-FedAvg has only a small effect on predictive accuracy, it could still have a large qualitative effect on predictions. We hy-pothesized that noising updates might bias the model away from rarer words (whose embeddings get less frequent actual updates and hence are potentially more influenced by noise) and toward the common "head" words. To evaluate this hypothesis, we computed predictions on a sample of the test set using a variety of models. At each s t we intersect the top 10 predictions with the most frequent 10, 50, 100 words in the dictionary. So for example, an intersection of size two in the top 50 means two of the model's top 10 predictions are in the 50 most common words in the dictionary. Table 3 gives histograms of these counts. We find that better models (higher AccuracyTop1) tend to use fewer head words, but see little difference from changingC or the noise σ (until, that is, enough noise has been added to compromise model quality, at which point the degraded model's bias toward the head matches models of similar quality with less noise). In this work, we introduced an algorithm for user-level differentially private training of large neural networks, in particular a complex sequence model for next-word prediction. We empirically evaluated the algorithm on a realistic dataset and demonstrated that such training is possible at a negligible loss in utility, instead paying a cost in additional computation. Such private training, combined with federated learning (which leaves the sensitive training data on device rather than centralizing it), shows the possibility of training models with significant privacy guarantees for important real world applications. Much future work remains, for example designing private algorithms that automate and make adaptive the tuning of the clipping/noise tradeoff, and the application to a wider range of model families and architectures, for example GRUs and character-level models. Our work also highlights the open direction of reducing the computational overhead of differentially private training of non-convex models. Proof of Lemma 1. For the first bound, observe the numerator in the estimatorf f can change by at most S between neighboring databases, by assumption. The denominator is a constant. For the second bound, the estimatorf c can be thought of as the sum of the vectors w k ∆ k divided by max(qW min, k∈C ∆ k). Writing Num(C) for the numerator k∈C w k ∆ k, and Den(C) for the denominator max(qW min, k∈C w k), the following are immediate for any C and C def = C ∪ {k}: DISPLAYFORM0 Here in the last step, we used the fact that f c (C) ≤ S. The claim follows. Proof of Theorem 1. It suffices to verify that 1. the moments (of the privacy loss) at each step are correctly bounded; and, 2. the composability holds when accumulating the moments of multiple steps. At each step, users are selected randomly with probability q. If in addition the L 2 -norm of each user's update is upper-bounded by S, then the moments can be upper-bounded by that of the sampled Gaussian mechanism with sensitivity 1, noise scale σ/S, and sampling probability q. Our algorithm, as described in FIG1, uses a fixed noise variance and generates the i.i.d. noise independent of the private data. Hence we can apply the composability as in Theorem 2.1 in BID0.We obtain the theorem by combining the above and the sensitivity boundsf f andf c. Model The first step in training a word-level recurrent language model is selecting the vocabulary of words to model, with remaining words mapped to a special "UNK" (unknown) token. Training a fully differentially private language model from scratch requires a private mechanism to discover which words are frequent across the corpus, for example using techniques like distributed heavyhitter estimation BID6 BID3. For this work, we simplified the problem by pre-selecting a dictionary of the most frequent 10,000 words (after normalization) in a large corpus of mixed material from the web and message boards (but not our training or test dataset).Our recurrent language model works as follows: word s t is mapped to an embedding vector e t ∈ R by looking up the word in the model's vocabulary. The e t is composed with the state emitted by the model in the previous time step s t−1 ∈ R 256 to emit a new state vector s t and an "output embedding" o t ∈ R 96. The details of how the LSTM composes e t and s t−1 can be found in BID13. The output embedding is scored against the embedding of each item in the vocabulary via inner product, before being normalized via softmax to compute a probability distribution over the vocabulary. Like other standard language modeling applications, we treat every input sequence as beginning with an implicit "BOS" (beginning of sequence) token and ending with an implicit "EOS" (end of sequence) token. Unlike standard LSTM language models, our model uses the same learned embedding for the input tokens and for determining the predicted distribution on output tokens from the softmax. 6 This reduces the size of the model by about 40% for a small decrease in model quality, an advantageous tradeoff for mobile applications. Another change from many standard LSTM RNN approaches is that we train these models to restrict the word embeddings to have a fixed L 2 norm of 1.0, a modification found in earlier experiments to improve convergence time. In total the model has 1.35M trainable parameters. Initialization and personalization For many applications public proxy data is available, e.g., for next-word prediction one could use public domain books, Wikipedia articles, or other web content. In this case, an initial model trained with standard (non-private) algorithms on the public data (which is likely drawn from the wrong distribution) can then be further refined by continuing with differentially-private training on the private data for the precise problem at hand. Such pre-training is likely the best approach for practical applications. However, since training models purely on private data (starting from random initialization) is a strictly harder problem, we focus on this scenario for our experiments. Our focus is also on training a single model which is shared by all users. However, we note that our approach is fully compatible with further on-device personalization of these models to the particular data of each user. It is also possible to give the central model some ability to personalize simply by providing information about the user as a feature vector along with the raw text input. LSTMs are well-suited to incorporating such additional context. We evaluate using AccuracyTop1, the probability that the word to which the model assigns highest probability is correct (after some minimal normalization). We always count it as a mistake if the true next word is not in the dictionary, even if the model predicts UNK, in order to allow fair comparisons of models using different dictionaries. In our experiments, we found that our model architecture is competitive on AccuracyTop1 and related metrics (Top3, Top5, and perplexity) across a variety of tasks and corpora. Dataset The Reddit dataset can be accessed through Google BigQuery (Reddit Comments Dataset). Since our goal is to limit the contribution of any one author to the final model, it is not necessary to include all the data from users with a large number of posts. On the other hand, processing users with too little data slows experiments (due to constant per-user overhead). Thus, we use a training set where we have removed all users with fewer than 1600 tokens (words), and truncated the remaining K = 763, 430 users to have exactly 1600 tokens. We intentionally chose a public dataset for research purposes, but carefully chose one with a structure and contents similar to private datasets that arise in real-world language modeling task such as predicting the next-word in a mobile keyboard. This allows for reproducibility, comparisons to nonprivate models, and inspection of the data to understand the impact of differential privacy beyond coarse aggregate statistics (as in Table 3). Figure 5: Comparison of sampling strategies and estimators. Fixed sample is exactly C = 100 users per round, and variable sample selects uniformly with probability q forC = 100. The true average corresponds to f, fixed denominator isf f, and clipped denominator isf c. FIG5, a smaller value would actually be better when doing private training). FedSGD is more sensitive to noise than FedAvg, likely because the updates are smaller in magnitude. Experiments with SGD We ran experiments using FedSGD taking B = 1600, that is, computing the gradient on each user's full local dataset. To allow more iterations, we usedC = 50 rather than 100. Examining Figures 9 and 10, we see S = 2 and σ = 2 · 10 −3 are reasonable values, which suggests for private training we would need in expectation qW = S/σ = 1500 users per round, whereas for FedAvg we might choose S = 15 and σ = 10 −2 forC = qW = 1000 users per round. That is, the relative effect of the ratio of the clipping level to noise is similar between FedAvg and FedSGD. However, FedSGD takes a significantly larger number of iterations to reach equivalent accuracy. Fixing z = 1,C = 5000 (the value that produced the best accuracy for a private model in TAB2) and total of 763,430 users gives (3.81, 10 −9)-DP after 3000 rounds and (8.92, 10 −9)-DP after 20000 rounds, so there is indeed a significant cost in privacy to these additional iterations. Models with larger dictionaries We repeated experiments on the impact of clipping and noise on models with 20000 and 30000 token dictionaries, again using FedAvg training with η = 6, equally weighted users with 1600 tokens, andC = 100 expected users per round. The larger dictionaries give only a modest improvement in accuracy, and do not require changing the clipping and noise parameters despite having significantly more parameters. Results are given in FIG1.Other experiments We experimented with adding an explicit L 2 penalty on the model updates (not the full model) on each user, hoping this would decrease the need for clipping by preferring updates with a smaller L 2 norm. However, we saw no positive effect from this.

represent the short summary for finding the corresponding scientific article (long)

User-level differential privacy for recurrent neural network language models is possible with a sufficiently large dataset.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.