Question:
represent the text to find a very short summary (1-2 sentences max)

Examples:
'This paper studies \emph{model inversion attacks}, in which the access to a model is abused to infer information about the training data. Since its first introduction by~\citet{fredrikson2014privacy}, such attacks have raised serious concerns given that training data usually contain sensitive information. Thus far, successful model inversion attacks have only been demonstrated on simple models, such as linear regression and logistic regression. Previous attempts to invert neural networks, even the ones with simple architectures, have failed to produce convincing . We present a novel attack method, termed the \emph{generative model inversion attack}, which can invert deep neural networks with high success rates. Rather than reconstructing private training data from scratch, we leverage partial public information, which can be very generic, to learn a distributional prior via generative adversarial networks (GANs) and use it to guide the inversion process. Moreover, we theoretically prove that a model's predictive power and its vulnerability to inversion attacks are indeed two sides of the same coin---highly predictive models are able to establish a strong correlation between features and labels, which coincides exactly with what an adversary exploits to mount the attacks. Our experiments demonstrate that the proposed attack improves identification accuracy over the existing work by about $75\%$ for reconstructing face images from a state-of-the-art face recognition classifier. We also show that differential privacy, in its canonical form, is of little avail to protect against our attacks. Deep neural networks (DNNs) have been adopted in a wide range of applications, including computer vision, speech recognition, healthcare, among others. The fact that many compelling applications of DNNs involve processing sensitive and proprietary datasets raised great concerns about privacy. In particular, when machine learning (ML) algorithms are applied to private training data, the ing models may unintentionally leak information about training data through their output (i.e., black-box attack) or their parameters (i.e., white-box attack). A concrete example of privacy attacks is model inversion (MI) attacks, which aim to reconstruct sensitive features of training data by taking advantage of their correlation with the model output. Algorithmically, MI attacks are implemented as an optimization problem seeking for the sensitive feature value that achieves the maximum likelihood under the target model. The first MI attack was proposed in the context of genomic privacy , where the authors showed that adversarial access to a linear regression model for personalized medicine can be abused to infer private genomic attributes about individuals in the training dataset. Recent work extended MI attacks to other settings, e.g., recovering an image of a person from a face recognition model given just their name, and other target models, e.g., logistic regression and decision trees. Thus far, effective MI attacks have only been demonstrated on the aforementioned simple models. It remains an open question whether it is possible to launch the attacks against a DNN and reconstruct its private training data. The challenges of inverting DNNs arise from the intractability and ill-posedness of the underlying attack optimization problem. For neural networks, even the ones with one hidden layer, the corresponding attack optimization becomes a non-convex problem; solving it via gradient descent methods may easily stuck in local minima, which leads to poor attack performance. Moreover, in the attack scenarios where the target model is a DNN (e.g., attacking face recognition models), the sensitive features (face images) to be recovered often lie in a high-dimensional, continuous data space. Directly optimizing over the high-dimensional space without any constraints may generate unrealistic features lacking semantic information (See Figure 1). Figure 1: Reconstruction of the individual on the left by attacking three face recognition models (logistic regression, one-hidden-layer and twohidden-layer neural network) using the existing attack algorithm in In this paper, we focus on image data and propose a simple yet effective attack method, termed the generative model inversion (GMI) attack, which can invert DNNs and synthesize private training data with high fidelity. The key observation supporting our approach is that it is arguably easy to obtain information about the general data distribution, especially for the image case. For example, against a face recognition classifier, the adversary could randomly crawl facial images from the Internet without knowing the private training data. We find these datasets, although may not contain the target individuals, still provide rich knowledge about how a face image might be structured; extraction and proper formulation of such prior knowledge will help regularize the originally ill-posed inversion problem. We also move beyond specific attack algorithms and explore the fundamental reasons for a model's susceptibility to inversion attacks. We show that the vulnerability is unavoidable for highly predictive models, since these models are able to establish a strong correlation between features and labels, which coincides exactly with what an adversary exploits to mount MI attacks. Our contributions can be summarized as follows: We propose to use generative models to learn an informative prior from public datasets so as to regularize the ill-posed inversion problem. We propose an end-to-end GMI attack algorithm based on GANs, which can reveal private training data of DNNs with high fidelity. We present a theoretical that uncovers the fundamental connection between a model's predictive power and its susceptibility to general MI attacks and empirically validate it. We conduct extensive experiments to demonstrate the performance of the proposed attack. Experiment code is publicly available at https://tinyurl.com/yxbnjk4s. Related Work Privacy attacks against ML models consist of methods that aim to reveal some aspects of training data. Of particular interest are membership attacks and MI attacks. Membership attacks aim to determine whether a given individual's data is used in training the model . MI attacks, on the other hand, aim to reconstruct the features corresponding to specific target labels. In parallel to the emergence of various privacy attack methods, there is a line work that formalizes the privacy notion and develops defenses with formal and provable privacy guarantees. One dominate definition of privacy is differential privacy (DP), which carefully randomizes an algorithm so that its output does not to depend too much on any individuals' data . In the context of ML algorithms, DP guarantees protect against attempts to infer whether a data record is included in the training set from the trained model . By definition, DP limits the success rate of membership attacks. However, it does not explicitly protect attribute privacy, which is the target of MI attacks . The first MI attack was demonstrated in , where the authors presented an algorithm to recover genetic markers given the linear regression that uses them as input features, the response of the model, as well as other non-sensitive features of the input. proposed a algorithm that allows MI attacks to be carried out without the knowledge of non-sensitive features by poisoning training data properly. Despite the generality of the algorithmic frameworks proposed in the above two papers, the evaluation of the attacks is only limited to linear models. discussed the application of MI attacks to more complex models including some shallow neural networks in the context of face recognition. Although the attack can reconstruct face images with identification rates much higher than random guessing, the recovered faces are indeed blurry and hardly recognizable. Moreover, the quality of reconstruction tends to degrade for more complex architectures. Yang et al. (2019b) proposed to train a separate network that swaps the input and output of the target network to perform MI attacks. The inversion model can be trained with black-box accesses to the target model. However, their approach cannot directly be benefited from the white-box setting. Moreover, several recent papers started to formalize MI attacks and study the factors that affect a model's vulnerability from a theoretical viewpoint. For instance, characterized model invertibility for Boolean functions using the concept of influence from Boolean analysis; formalized the risk that the model poses specifically to individuals in the training data and shows that the risk increases with the degree of overfitting of the model. However, their theory assumed that the adversary has access to the join distribution of private feature and label, which is overly strong for many attack scenarios. Our theory does not rely on this assumption and better supports the experimental findings. An overview of our GMI attack is illustrated in Figure 2. In this section, we will first discuss the threat model and then present our attack method in details. In traditional MI attacks, an adversary, given a model trained to predict specific labels, uses it to make predictions of sensitive features used during training. Throughout the paper, we will refer to the model subject to attacks as the target network. We will use face recognition classifiers as a running example for the target network. Face recognition classifiers label an image containing a face with an identifier corresponding to the individual depicted in the image. We assume that the adversary employs an inference technique to discover the face image x for some specific identity y output by the classifier f. Following the canonical setup of MI attacks, we assume that the adversary has access to the target network f. In addition to f, the adversary may also have access to some auxiliary knowledge that facilitates his inference. Possible Auxiliary Knowledge Examples of auxiliary knowledge could be a blurred or corrupted image which only contains nonsenstive information, such as pixels in a face image. This auxiliary knowledge might be easy to obtain, as blurring and corruption are often applied to protect anonymity of individuals in public datasets . The setup of MI attacks on images resembles the widely studied image inpainting tasks in computer vision, which also try to fill missing pixels of an image. The difference is, however, in the goal of the two. MI attacks try to fill the sensitive features associated with a specific identity in the training set. In contrast, image inpainting tasks only aim to synthesize visually realistic and semantically plausible pixels for the missing regions; whether the synthesized pixels are consistent with a specific identity is beyond the scope. Despite the difference, our approach to MI attacks leverages some training strategies from the venerable line of work on image inpainting (; ; a) and significantly improves the recognizability of the reconstructed images over the existing attack methods. To realistically reconstruct missing sensitive regions in an image, our approach utilizes the generator G and the discriminator D, all of which are trained with public data. After training, we aim to find the latent vectorẑ that achieves highest likelihood under the target network while being constrained to the data manifold learned by G. However, if not properly designed, the generator may not allow the target network to easily distinguish between different latent vectors. For instance, in extreme cases, if the generated images of all latent vectors collapse to the same point in the feature space of the target network, then there is no hope to identify which one is more likely to appear in its private training set of the target network. To address this issue, we present a simple yet effective loss term to promote the diversity of the data manifold learned by G when projected to the target network's feature space. Specifically, our reconstruction process consists of two stages: Public knowledge distillation, in which we train the generator and the discriminators on public datasets in order to encourage the generator to generate realistic-looking images. The public datasets can be unlabeled and have no identity overlapping with the private dataset. Secret revelation, in which we make use of the generator obtained from the first stage and solve an optimization problem to recover the missing sensitive regions in an image. For the first stage, we leverage the canonical Wasserstein-GAN training loss. The loss function is adapted to the two discriminators for our case: In addition, inspired by Yang et al. (2019a), we introduce a diversity loss term that promotes the diversity of the images synthesized by G when projected to the target network's feature space. Let F denote the feature extractor of the target network. The diversity loss can thus be expressed as As discussed above, larger diversity will facilitate the targeted network to discern the generated image that is most likely to appear in its private training set. Our full objective for public knowledge distillation can be written as In the secret revelation stage, we solve the following optimization to find the latent vector that generates an image achieving the maximum likelihood under the target network while remaining where the prior loss L prior (z) penalizes unrealistic images and the identity loss L id (z) encourages the generated images to have likelihood under the targeted network. They are defined, respectively, by where C(G(z)) represent the probability of G(z) output by the target network. For a fixed data point (x, y), we can measure the performance of a model f for predicting the label y of feature x using the log likelihood log p f (y|x). It is known that maximizing the log likelihood is equivalent to minimizing the cross entropy loss-one of the most commonly used loss function for training DNNs. Thus, throughout the following analysis, we will focus on the log likelihood as a model performance measure. Now, suppose that (X, Y) is drawn from an unknown data distribution p(X, Y). Moreover, X = (X s, X ns), where X s and X ns denote the sensitive and non-sensitive part of the feature, respectively. We can define the predictive power of the sensitive feature X s under the model f (or equivalently, the predictive power of model f using X s) as the change of model performance when excluding it from the input, i.e.,. Similarly, we define the predictive power of the sensitive feature given a specific class y and nonsensitive feature x ns as We now consider the measure for the MI attack performance. Recall the goal of the adversary is to guess the value of x s given its corresponding label y, the model f, and some auxiliary knowledge x ns. The best attack outcome is the recovery of the entire posterior distribution of the sensitive feature, i.e., p(X s |y, x ns). However, due to the incompleteness of the information available to the adversary, the best possible attack that adversary can achieve under the attack model can be captured by p f (X s |y, x ns) ∝ p f (y|X s, x ns)p(X s |x ns), assuming that the adversary can have a fairly good estimate of p(X s |x ns). Such estimate can be obtained by, for example, learning from public datasets using the method in Section 2.2. Although MI attack algorithms often output a single feature vector as the attack , these algorithms can be adapted to output a feature distribution instead of a single point by randomizing the starting guess of the feature. Thus, it is natural to measure the MI attack performance in terms of the similarity between p(X s |y, x ns) and p f (X s |y, x ns). The next theorem indicates that the vulnerability to MI attacks is unavoidable if the sensitive features are highly predictive under the model. When stating the theorem, we use the negative KL-divergence S KL (·||·) to measure the similarity between two distributions. Theorem 1. Let f 1 and f 2 be two models such that for any fixed label We omit the proof of the theorem to the supplementary material. Intuitively, highly predictive models are able to build a strong correlation between features and labels, which coincides exactly with what an adversary exploits to launch MI attacks; hence, more predictive power inevitably leads to higher attack performance. , it is argued that a model is more vulnerable to MI attacks if it overfits data to a greater degree. Their is seemingly contradictory with ours, because fixing the training performance, more overfitting implies that the model has less predictive power. However, the assumption underlying their is fundamentally different from ours, which leads to the disparities. The in assumes that the adversary has access to the joint distribution p(X s, X ns, Y) that the private training data is drawn from and their setup of the goal of the MI attack is to learn the sensitive feature associated with a given label in a specific training dataset. By contrast, our formulation of MI attacks is to learn about private feature distribution p(X s |y, x ns) for a given label y from the model parameters. We do not assume that the adversary has the prior knowledge of p(X s, X ns, Y), as it is a overly strong assumption for our formulation-the adversary can easily obtain p(X s |y, x ns) for any labels and any values of non-sensitive features when having access to the joint distribution. Dataset We evalaute our method using three datasets: the MNIST handwritten digit data (MNIST), the Chest X-ray Database ) (ChestX-ray8), and the CelebFaces Attributes Dataset (CelebA) containing 202,599 face images of 10,177 identities with coarse alignment. We crop the images at the center and resize them to 64×64 so as to remove most . Protocol We split each dataset into two disjoint parts: one part used as the private dataset to train the target network and the other as a public dataset for prior knowledge distillation. The public data, throughout the experiments, do not have class intersection with the private training data of the target network. Therefore, the public dataset in our experiment only helps the adversary to gain knowledge about features generic to all classes and does not provide information about private, class-specific features for training the target network. This ensures the fairness of the comparison with the existing MI attack . Models We implement several different target networks with varied complexities. For all the adapted networks, we modify the FC-layer to fit in our task. For digit classification on MNIST, our target network consists of 3 convolutional layers and 2 pooling layers. For the disease prediction on ChestX-ray8, we use ResNet-18 adapted from as our target network. For the face recognition tasks on CelebA, we use the following networks: VGG16 adapted from ;ResNet-152 adapted from ; face.eoLVe adapted from the state-of-the-art face recognition network . Training We split the private dataset defined above into training set (90%) and test set (10%) and use the SGD optimizer with learning rate 10 −2, batch size 64, momentum 0.9 and weight decay 10 −4 to train these networks. To train the GAN in the first stage of our attack pipeline, we set λ d = 0.5 and use the Adam optimizer with the learning rate 0.004, batch size 64, β 1 = 0.5, and β 2 = 0.999 . In the second stage, we set λ i = 100 and use the SGD optimizer to optimize the latent vector z with the learning rate 0.01, batch size 64 and momentum 0.9. z is drawn from a zero-mean unit-variance Gaussian distribution. We randomly initialize z for 5 times and optimize each round for 1500 iterations. We choose the solution with the lowest identity loss as our final latent vector. Evaluating the success of MI attacks requires to assess whether the recovered image exposes the private information about a target individual. Previous works analyzed the attack performance mainly qualitatively by visual inspection. Herein, we introduce four metrics which allow to quantitatively judge the MI attack efficacy and perform evaluation at a large scale. Peak Signal-to-Noise Ratio (PSNR) PSNR is the ratio of an image's maximum squared pixel fluctuation over the mean squared error between the target image and the reconstructed image. PSNR measures the pixel-wise similarity between two images. The higher the PSNR, the better the quality of the reconstructed image. However, oftentimes, the reconstructed image may still reveal identity information even though it is not close to the target image pixel-wise. For instance, a recovered face with different translation, scale and rotation from the target image will still incur privacy loss. This necessitates the need for the following metrics that can evaluate the similarity between the reconstructed and the target image at a semantic level. Attack Accuracy (Attack Acc) We build an evaluation classifier that predicts the identity based on the input reconstructed image. If the evaluation classifier achieves high accuracy, the reconstructed image is considered to expose private information about the target individual. The evaluation classifier should be different from the target network because the reconstructed images may incorporate features that overfit the target network while being semantically meaningless. Moreover, the evaluation classifier should be highly performant. For the reasons above, we adopt the state-of-the-art architecture in each task as the evaluation classifier. For MNIST, our evaluation network consists of 5 convolutional layers and 2 pooling layers. For ChestX-ray8, we adapt VGG-19 from as our evaluation network. For CeleA, we use the model in for the evaluation classifier. We first pretrain it on the MS-Celeb-1M and then fine tune on the identities in the training set of the target network. The ing evaluation classifier can achieve 96% accuracy on these identities. Feature Distance (Feat Dist) Feat Dist measures the l 2 feature distance between the reconstructed image and the centroid of the target class. The feature space is taken to be the output of the penultimate layer of the evaluation network. K-Nearest Neighbor Distance (KNN Dist) KNN Dist looks at the shortest distance from the reconstructed image to the target class. We identify the closest data point to the reconstructed image in the training set and output their distance. The distance is measured by the l 2 distance between the two points in the feature space of the evaluation classifier. Figure 3: Qualitative comparison of the proposed GMI attack with the existing MI attack (EMI), the pure image inpainting method (PII). The ground truth target image is shown in 1st col. We compare our approach with two baselines: Existing model inversion attack (EMI), which implements the algorithm in . For this algorithm, the adversary only exploits the identity loss for image reconstruction and return the pixel values that minimize the the identity loss; Pure image inpainting (PII), which minimizes the W-GAN loss and performs image recovery based on the information completely from the public dataset. For CelebA, the private set comprises 21,152 images of 1000 identities and samples from the rest are used as a public dataset. We evaluate the attack performance in the three settings: the attacker does not have any auxiliary knowledge about the private image, in which case he will recover the image from scratch; the attacker has access to a blurred version of the private image and his goal is to deblur the image; the attacker has access to a corrupted version of the private image wherein the sensitive, identity-revealing features (e.g., nose, mouth, etc) are blocked. Table 1 compares the performance of our proposed GMI attack against EMI for different network architectures. We can see that the EMI works poorly on the deep nets and achieve around zero attack accuracy. GMI is much more effective than EMI. Particularly, our method improves the accuracy of the attack against the state-of-the-art face.evoLVe classifier over the existing MI attack by 75% in terms of Top-5 attack accuracy. Also, note that models that are more sophisticated and have more predictive power are more susceptible to attacks. We will examine this phenomenon in more details in Section 4.3.3. We now discuss the case where the attacker has access to some auxilliary knowledge in terms of blurred or partially blocked images. For the latter, we consider two types of masks-center and face "T", illustrated by the second column of Figure 3 (c) and (d), respectively. The center mask blocks the central part of the face and hides most of the identity-revealing features, such as eyes and nose, while the face T mask is designed to obstruct all private features in a face image. Table 2 shows that our method consistently outperforms the two baselines discussed above. Since the existing MI attack does not exploit any prior information, the inversion optimization problem is extremely ill-posed and performing gradient descent ends up at some visually meaningless local minimum, as illustrated by Figure 3. Interestingly, despite having the meaningless patterns, these images can all be classified correctly into the target label by the target network. Hence, the existing MI attack tends to generate "adversarial examples" that can fool the target network but does not exhibit any recognizable features of the private data. Figure 3 also compares our with PII, which is completely based on the information from the public dataset to recover the private image. We can see that although PII leads to realistic recoveries, the reconstructed images do not present the same identity features as the target images. This can be further corroborated by the quantitative in Table 2. Note that the attacks are more effective for the center mask than the face T mask. This is because the face T mask we designed completely hides the identity revealing features on the face while the center mask may still expose the mouth information. We have seen that distilling prior knowledge and properly incorporating it into the attack algorithm are important to the success of MI attacks. In our proposed method, the prior knowledge is gleaned from public datasets through GAN. We now evaluate the impact of public datasets on the attack performance. We first consider the case where the public data is from the same distribution as the private data and study how the size of the public data affects the attack performance. We change the size ratio (1:1, 1:4, 1:6, 1:10) of the public over the private data by varying the number of identities in the public dataset. As shown in Table 3, the attack performance varies by less than 7% when shrinking the public data size by 10 times. Moreover, we study the effect of the distribution shift between the public and private data on the attack performance. We train the GAN on the PubFig83 dataset, which contains 13,600 images with 83 identities, and attack the target network trained on CelebA. There are more faces with sunglasses in PubFig83 than CelebA, which makes it harder to distill generic face information. Without any pre-processing, the attack accuracy drops by more than 20% despite still outperforming the existing MI attack by a large margin. To further improve the reconstruction quality, we detect landmarks in the face images, rotate the images such that the eyes lie on a horizontal line, and crop the faces to remove the . These pre-processing steps make the public datasets better present the face information, thus improving the attack accuracy significantly. We perform experiments to validate the connection between predictive power and the vulnerability to MI attacks. We measure the predictive power of sensitive feature under a model using the difference of model testing accuracy based on all features and just non-sensitive features. We consider the following different ways to construct models with increasing feature predictive powers, namely, enlarging the training size per class, adding dropout regularization, and performing batch normalization. For the sake of efficiency, we slightly modify the proposed method in Section 2.2 in order to avert re-training GANs for different architectures. Specifically, we exclude the diversity loss from the attack pipeline so that multiple architectures can share the same GAN for prior knowledge distillation. Figure 4 shows that, in general, the attack performance will be better for models with higher feature predictive powers. Moreover, this trend is consistent across different architectures. Figure 5: Visualization of the recovered input images by the GMI and the EMI attack. For MNIST, we use all 34265 images with labels 5, 6, 7, 8, 9 as private set, and the rest of 35725 images with labels 0, 1, 2, 3, 4 as a public dataset. Note that the labels in the private and public data have no overlaps. We augment the public data by training an autoencoder and interpolating in the latent space. Our GMI attack is compared with the baseline in Table 4. We omit the PII baseline because the public and private set defined in this experiment are rather disparate and the PII essentially produce close to random guesses. We can see from the table that the performance of GMI is significantly better than the EMI. Examples of the recovered images with both attacks are compared in Figure 5. For ChestX-ray8, we use 10000 images of seven classes as the private data and the other 10000 with different labels as public data. The GMI and EMI attack are compared in Table 5. Again, the GMI attack outperforms the EMI attack by a large margin. We investigate the implications of DP for MI attacks. (, δ)-DP is ensured by adding Gaussian noise to clipped gradients in each training iteration. We find it challenging to produce useful face recognition models with DP guarantees due to the complexity of the task. Therefore, we turn to a simpler dataset, MNIST, which is commonly used in differential private ML studies. We set δ = 10 −5 and vary the noise scale to obtain target networks with different. The attack performance against these target networks and their utility are illustrated in Figure 4 (d). Since the attack accuracy of the GMI attack on differentially private models is higher than that of PII which fills missing regions completely based on the public data, it is clear that the GMI attack can expose private information from differentially private models, even with stringent privacy guarantees, like = 0.1. Moreover, varying differential privacy budgets helps little to protect against the GMI attack; sometimes, more privacy budgets even improve the attack performance (e.g., changing from 1 to 0.1). This is because DP, in its canonical form, only hides the presence of a single instance in the training set. Limiting the learning of specific individuals may facilitate the learning of generic features of a class, which, in turn, helps to stage MI attacks. In this paper, we present a generative approach to MI attacks, which can achieve the-state-of-the-art success rates for attacking the DNNs with high-dimensional input data. The idea of our approach is to extract generic knowledge from public datasets via GAN and use it to regularize the inversion problem. Our experimental show that our proposed attack is highly performant even when the public datasets do not include the identities that the adversary aims to recover, are unlabeled, have small sizes, come from a different distribution from the private data. We also provide theoretical analysis showing the fundamental connection between a model's predictive power and its vulnerability to inversion attacks. For future work, we are interested in extending the attack to the black-box setting and studying effective defenses against MI attacks. A PROOF OF THEOREM 1 Theorem 2. Let f 1 and f 2 are two models such that for any fixed label y ∈ Y, U f1 (x ns, y) ≥ U f2 (x ns, y). Then, S KL (p(X s |y, x ns)||p f1 (X s |y, x ns)) ≥ S KL (p(X s |y, x ns)||p f2 (X s |y, x ns)). Proof. We can expand the KL divergence D KL (p(X s |y, x ns)||p f1 (X s |y, x ns) as follows. Thus, B EXPERIMENTAL DETAILS B.1 NETWORK ARCHITECTURE The detailed architectures for the two encoders, the decoder of the generator, the local discriminator, and the global discriminator are presented in Table 6, Table 7, Table 8, Table 9, and Table 10, respectively. LeNet adapted from , which has three convolutional layers, two max pooling layers and one FC layer; SimpleCNN, which has five convolutional layers, each followed by a batch normalization layer and a leaky ReLU layer; SoftmaxNet, which has only one FC layer. We split the MNIST dataset into the private set used for training target networks with digits 0 ∼ 4 and the public set used for distilling prior knowledge with digits 5 ∼ 9. The target network is implemented as a Multilayer Perceptron with 2 hidden layers, which have 512 and 256 neurons, respectively. The evaluation classifier is a convulutional neural network with three convolution layers, followed by two fully-connected layers. It is trained on the entire MNIST training set and can achieve 99.2% accuracy on the MNIST test set. Differential privacy of target networks is guaranteed by adding Gaussian noise to each stochastic gradient descent step. We use the moment accounting technique to keep track of the privacy budget spent during training . During the training of the target networks, we set the batch size to be 256. We fix the number of epochs to be 40 and clip the L2 norm of per-sample gradient to be bounded by 1.5. We set the ratio between the noise scale and the gradient clipping threshold to be 0, 0.694, 0.92, 3, 28, respectively, to obtain the target networks with ε = ∞, 9.89, 4.94, 0.98, 0.10 when δ = 10 −5. For model with ε = 0.1, we use the SGD with a small learning rate 0.01 to ensure stable convergence; otherwise, we set the learning rate to be 0.1. The architecture of the generator in Section B.1 is tailored to the MNIST dataset. We reduce the number of input channels, change the size of kernels, and modify the layers of discriminators to be compatible with the shape of the MNIST data. To train the GAN in the first stage of our GMI attack, we set the batch size to be 64 and use the Adam optimizer with the learning rate 0.004, β 1 = 0.5, and β 2 = 0.999 . For the second stage, we set the batch size to be 64 and use the SGD with the Nesterov momentum that has the learning rate 0.01 and momentum 0.9. The optimization is performed for 1500 iterations. The center mask depicted in the main text is used to block the central part of digits. We report the attack accuracy averaged across 640 randomly sampled images from the private set and 5 random initializations of the latent vector for each sampled image.' == 'We develop a privacy attack that can recover the sensitive input data of a deep net from its output' != 'This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples. '

Question: In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations. We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views. Formally, multiple views of the same sentence are mapped to close representations. On the contrary, views from other sentences are mapped further. By contrasting different linguistic views, we aim at building embeddings which better capture semantic and which are less sensitive to the sentence outward form. We propose to learn sentence embeddings by contrasting multiple linguistic representations. The motivation is to benefit from linguistic structures diversity to discard noises inherent to each representation. We aim at encoding high-level representations by aligning the underlying shared information from multiple views. As illustrated in Figure 1, we train our model with a contrastive framework which aims at mapping close input sentences to close representations while separating unrelated sentences. In Natural Language Processing (NLP), this framework has been widely used to learn word representations (a; b) for example. This model relies on the distributional hypothesis which conjectures that words within similar context share similar meaning. Such framework has also been extended to sentences with the similar hypothesis that the meaning can be inferred from the context sentences (; . We propose to extend this framework by assuming that different views of the same sentence should lead to close representation. We considered the dependency trees, a linguistic framework that describes the compositional structure of a sentence. As illustrated in Figure 1, in this framework, the sentence is mathematically described as an oriented acyclic graph where the nodes are words and edges describe the relations between words. Such structure has benefited from an important attention in the NLP community and efficient parser tools for various languages are available, which makes it possible to obtain such information almost freely in the sense it does not require additional hand annotated data. Tree representations are then mapped in a shared embedding space using appropriate Tree LSTM networks introduced in . Model parameters are learned using a discriminating objective as proposed in . Representation learning has gained significant attention from the NLP community . As illustrated in Figure 3 it is structured in two-steps: (A) a representation is learned using a proxy objective on a usually very large corpora (B) the representation is then used to solve a variety of downstream tasks. Literature comes with a variety of proxy tasks to learn representations. Proxy objectives usually fall in two categories: (i) predicting a contextual information or reconstructing Figure 1: Contrastive multi-views framework. The model is trained to distinguish between different views of context sentences and negative examples. The two views are obtained using a standard LSTM and a Tree LSTM networks on top of a dependency tree structure. A discriminative objective is used to contrast between samples. an altered input or (ii) discriminating multiple representations of the same data among negative examples. Proxy (i) has been declined into multiple variations in the literature. Words embeddings methods Mikolov et al. (2013b) proposes that words with close distributions should lead to close representations. For images, proposes to learn representations by solving jigsaw puzzles. With a close taste to NLP tasks, aims at learning representations by trying to fill the missing part of an image. van den; methods aim at predicting the future to represent video or audio. The second proxy objective (ii) is motivated by the observation that end-to-end frameworks tend to learn semantic similarities among classes. For example reports that the hidden layers of a supervised neural networks assigns close representations to leopard and jaguar images. However an image from lifeboat and leopard will be apart from each other. This motivates for contrastive learning methods where the network aims at distinguishing individual samples to insure a global consistency in the dataset. Many contrastive learnings have been proved effective in a variety of domains: image processing , audio input (van den), sentence representation or word embedding (a; b;). Besides the nature of data, formalisms might differ on two main aspects: (i) How to measure the proximity in both the original and the representation space and (ii) What training objective should be used to achieve well distinction between samples. One possible answer on how to select samples that should lead to close representation is to consider different views of the same sample as proposed in; who propose to combine multi-views of images with contrastive learning.; van den propose contrastive frameworks for sentences but without multi-views setting. Multiple metrics to measure the proximity in the representation space are proposed in the literature: enumerates multiple so called critic functions. The most straight forward also used in beeing the scalar product: The objective function for contrastive learning has also been declined in multiple work: Many relate to Mutual Information (MI); or methods based on triplet loss or max-margin . Contrastive learning is a self-supervised learning method which aims at learning a semantic mapping. Data is embedded with compact representations such that close samples are mapped to nearby points while unrelated ones are affected to apart points. In practice different methods exist to learn such mapping. A straight forward method is to treat each data sample as a distinct class and train a classifier to distinguish between individual instance classes. Such approach is used for word embedding where the goal is to predict a word given his context. However this is computationally difficult and the outputs are not a finite set in our case since an infinite number of correct sentences might be expressed. A method to approximate a full instance discrimination is to use Negative sampling and Noise Contrastive estimation methods. For every data point x from a dataset D we have access to one paired sample x + which should be close. We then draw K negative samples x K which should be apart. We learn a scoring function h θ which assigns large scores to positive paired samples (x, x +) while low values for negative pairs (x, x − k). In practice the function is trained using a classifier which aims at identifying the correct pair. In our setup we use a softmax classifier as follow: The classifier is trained using the negative log likelihood loss: As in the used scoring function is the inner product h θ (u, v) = u T v in order to avoid the situation where the model learns poor representations but compensates with an excellent classifier on the proxy task. However some authors report excellent with other critic functions: for example uses a combination of absolute distance and angle measures. Given the scoring vectors for each pair representation we use a simple softmax classifier with a log negative likelihood loss. In our setup, the positive pair (x, x +) is built from different views of the input data. From the original sentences, we construct multiple representations from different views of the object. For every sentence we consider two encoders. x s which is the representation obtained with a bidirectional LSTM who assumes the underlying structure of the sentence to be a sequence, while allowing for long term dependencies. The final representation of the sentence is the concatenation of both direction last state. x d is obtained with the dependency tree representation combined with a ChildSum Tree LSTM . The final representation is the state from the root node of the sentence graph. Negative examples are obtained using the dependency Tree LSTM. The dataset D is therefore augmented with the positive examples and multi-views examples Similarly to , the model is trained to identify the sentence appearing in the context from the target sentence. The key difference and our contribution is to build different views of the data. The target sequence is encoded using the sequential Tree LSTM, while the positive and negative samples are encoded using the ChildSum Tree LSTM. As described in the previous section, the objective is to maximize the log probability of the correct sentence to appear in the context. 4.1 DATA Models are trained on the BookCorpus dataset. Since the dataset is no longer distributed, a similar dataset is generated using smashword open book data 1. The obtained dataset contains 78M sentences from 17.000 books. We define a 20k words vocabulary, containing the corpus most frequent tokens. Sentences are tokenized using Microsoft BlingFire tool 2. Models are trained on a single epoch on the entire corpus without any train-test split. The use of Tree LSTM models supposes to parse sentences in dependency which is operated using the publicly available Spacy parser. The contrastive learning method supposes to draw negative examples for each sentence pair. We follow the training procedure proposed in and build mini-batches respecting books sentences order. Therefore samples from the mini-batch are used as negative examples. Models are trained using the Adam optimizer with a 5e-3 learning rate. Gradient is clipped with a value of 5.0. The hidden layer for both Tree LSTM is fixed to 1000 and the embedding dimension to 300. The batch size is 100 and the model is trained on a 1080Ti Nvidia GPU. All model weights are initialized with a xavier distribution and biases to 0. We observe biases initialization to have a significant impact on the training. Indeed leaf nodes are initialized with zero vectors for state and hidden and initial biases with too significant values tend to slow the training convergence. A point is computed every hour during the training phase. Despite the batch computation implementation describes in Section A, the Contrastive Tree LSTM is much slower to train. The training was stoped after 33 hours and training on 5.8% of the available training data. The evaluation setup for the SICK tasks is describe in Section 5. The Tree LSTM network was implemented using a batch procedure described in Section A. However the training was significantly slower than vanilla LSTM. The training phase was stopped after 33 hours of training. The training phase was completed on only 4.6M sentences among the 78M available. We monitored the performance on downstream tasks during training as illustrated in Figure 2. As described in Figure 3, the process requires specific evaluation processes at each learning step. Although methods are developed to control the properties of the intermediate representation such as probing tasks for NLP, the intermediate representation is usually evaluated on his ability to solve the final task: downstream evaluation. To facilitate the comparison across multiple representations and limit the impact of the downstream algorithm, the linear evaluation protocol is usually used. It proposes to solve the downstream task with a minimal and simple model: a logistic regression. At test time, the concatenation of two encoders is used. Figure 3: (left) End-to-end framework where the algorithm learns to map directly the inputs to the target. The hidden sates of the neural network are only used for this task. (right) In contrast to end-to-end approaches the algorithm proceeds in two phases. First a representation is learned with a self-supervised proxy task. The same representation might then be used for different tasks. Sentence representations are evaluated on downstream tasks that require to capture the underlying sentence semantic. 7 classification tasks from the SentEval benchmark are applied: movie review sentiment (MR) , product reviews (CR) , subjectivity classification (SUBJ) , opinion polarity (MPQA) , question type classification (TREC) , semantic relatedness and entailment (SICK-R, SICK-E) and paraphrase identification (MRPC) . The MR, CR, SUBJ, MPQA tasks are binary classification tasks with no pre-defined train-test split. A 10-fold cross validation is used in reporting test performance for these tasks. For other tasks we use the proposed train/dev/test splits. The dev set is used for choosing the regularization parameter and are reported on the test set. We follow the linear evaluation protocol of where a logistic regression classifier is trained on top of sentence representations with the cross-validation procedure described earlier. This protocol facilitates the comparison across studies and avoids the case of a good classifier which compensates for bad sentence representations. The script from is used for downstream evaluation. The scores are not as good as the state of the art reported in Table 1. However the model was not trained on the entire available training set used in. The model shows encouraging properties and trends in Figure 2 suggest it might improve with a larger exposition to the training data. Table 1: Comparison of sentence representations on downstream tasks. For the SICK-R task, the pearson coefficient is indicated. For the MRPC task, this is the F1 score. The baseline is reported from and is obtained using a bag-of-words representation. LSTM, BidirLSTM and Tree LSTM are reported from. Fastsent is reported from , Dissent and Infersent from , Skip-thoughts from and Quick-thoughts from . The table is divided into different sections. The bold-face numbers indicate the best performance values among models in the current and all previous sections. Best overall values in each column are underlined. argues that downstream tasks require complex form of inference, which makes it difficult to assess the fine grained quality of a representation. Probing tasks are supposed to overcome this limitation and separately evaluate individual linguistic properties of representations. The probing benchmark includes surface information tasks: predicting the length of the sentence (SentLen) and the word content (WC). Syntactic tasks include predicting the depth of the sentence tree (TreeDepth), the root top constituents (TopConst) and the word order (BShift). Finally semantic tasks aim at predicting the verb tense (Tense), if the subject and complement are singular or plural (SubjNum, ObjNum) and semantic or coordination inversion (SOMO, CoordInv). The probing are presented in Table 2 for both Contrastive LSTM (CL) and Contrastive Tree LSTM (CTL). Table 2: Probing task scores for the Contrastive Tree LSTM (CTL) and the Contrastive LSTM (CL). Bold-face numbers indicate the best performance values among the two models. The proposed traintest split is used for each task. Parameters are fixed on the dev set and are presented on the test set. Results for the Contrastive LSTM (CL) are obtained using a model we trained on our version of the bookcorpus as described in Section 4. We observe the interaction between the standard Tree LSTM and the Tree LSTM is sensible to the sentence length which was also observed by . All other scores are bellow the obtained by the application of the sole LSTM, pointing that the introduction of linguistic knowledge does not compensate enough for the smaller training set. Our training setup and share a similar framework but with different models to embed sentences. We retrieved the nearest neighbor from several query examples with the two methodologies. The sentences are extracted from the SICK test set. The closest neighbors are determined using the cosine similarity and presented in Table 3. Examples are selected to illustrate the ability of models to capture desired linguistic properties such as passive form, concepts, counting faculties or gender identification. Table 3: Nearest neighbors retrieved by the Contrastive Tree LSTM (CTL) and the Contrastive LSTM (CL). For each sentence, the 5 closest neighbors from the test split of the SICK dataset are retrieved using the cosine similarity from their representations. Results are presented in decreasing order of cosine distance. Both models might be fooled with surface information and retrieve examples with similar words but different meanings. Sole LSTM encoder presents remarkable properties to identify passive form. The Tree LSTM presents interesting properties such as retrieving sentences with similar principal proposition but complement which do not alter the sense of the sentence. Tree LSTM also captures number properties and identifies concepts such as 4 people might be referred as a group. We exploit the diversity of linguistic structures to build sentence representations. Out method shows promising and does not require hand annotated data. More scalable implementations might be considered to explore more experimentation setups. Although are below state of the art performances, our model is trained on only a small proportion of the bookcorpus sentences as stated in Figure 2. A larger exposition to the training data and an extended training time might benefit to the downstream and probing scores. Other linguistic structures might also be tested such as constituency tree associated with N-ary Tree LSTM or Tree LSTM improved with an attention mechanism. A COMPUTING METHOD FOR TREE LSTM We implemented a batching procedure to fasten Tree LSTM computations. Group of nodes are computed sequentially to insure all node children have already been computed. Nodes are considered given their distance to the root node. First, Leaf nodes with highest depth are computed to progressively compute inner nodes. The Tree LSTM cell implementation is specifically designed to treat simultaneously all nodes in each the step. Figure 4: The batching procedure to optimize the graph computation. For each batch, the computation is decomposed in steps which insure that every node dependent have already been computed. At each step, node with the same depth to the root are computed in a single operation and the output fed to the next computational step.

Represent the TLDR for finding the corresponding scientific article

We aim to exploit the diversity of linguistic structures to build sentence representations.

represent the short summary for finding the corresponding scientific article (long)

Applying the softmax function in training leads to indirect and unexpected supervision on features. We propose a new training objective to explicitly induce dense feature regions for locally sufficient samples to benefit adversarial robustness.


Question: Represent the following document

Question: The peripheral nervous system represents the input/output system for the brain. Cuff electrodes implanted on the peripheral nervous system allow observation and control over this system, however, the data produced by these electrodes have a low signal-to-noise ratio and a complex signal content. In this paper, we consider the analysis of neural data recorded from the vagus nerve in animal models, and develop an unsupervised learner based on convolutional neural networks that is able to simultaneously de-noise and cluster regions of the data by signal content. Recent advances have made chronic observation of, and limited control over the peripheral nervous system possible. To characterise the dynamics of the signals passing to and from the brain, we wish to categorise patterns of activity within the peripheral nervous system. However, consistent detection of single neuron activity remains a challenge with current cuff electrode technology suitable for in vivo neural data acquisition. The relative position of an extracellular recording electrode and neuronal axons close enough to sit above the noise floor affects the polarity of presented signal components, and their summation at the electrode occludes the presence of individual action potentials during periods of neuronal activity. Instead, local field potentials (LFPs), the combination of many neuronal responses arriving concurrently at the electrode are observed. These population level responses are potentially informationally richer, but preclude the use of conventional spike-sorting methodologies on such data. Instead, we develop a method based on convolutional neural networks (CNN) that simultaneously de-noises the data and categorises the observed signals. We train this model on approximately one hour of data taken from a single subject approximately twelve hours post surgical implantation. We further show that it is applicable without further training to data from a second subject thirty days post surgical implantation, demonstrating cross-time, cross-subject applicability of the trained models. Neural data are collected from two nerve cuffs implanted on the vagus nerve, each recording LFP data at 30000Hz using a chronically implanted ITX PNS (peripheral nervous system) implant (BIOS, * To whom correspondence should be addressed. Figure 1: The architecture of the Coordinate-VAE. The input signal is encoded via a series of convolutional/pooling/leaky-ReLu/dropout blocks to a categorical latent vector representing the core process observed in the input signal. To allow the decoder to account for phase shifts, time warping, et cetara, a set of time coordinates for which the signal is closest to zero are sampled from each channel of the input signal. These pass through a 'coordinate encoder', before being concatenated with the latent vector. The decoder then upsamples with convolution to reconstruct the original signal. Cambridge, UK). We begin by standardising the mean and standard deviation of the data coming from each cuff, and applying a fifth-order Butterworth bandpass (50-1000Hz) filter, before rescaling such that the training data lie in the range (-1, 1). We then sample small time windows of equal size w from the data as input to the Coordinate-VAE. In the shown here, w is fixed at 256 samples, that is, at 256 30000 seconds. The basic model architecture is shown in Figure 1. For each window, the goal is to reduce the observed data to a one-hot latent vector of size L. We achieve this by training a variational auto-encoder (VAE) with a Gumbel-Softmax activation on the latent space. Encoding to the latent space is done through a series of convolutional blocks, where parameters for each block in the encoder are kept constant except for the number of filters in each convolutional layer, which doubles with each block. Pooling takes place in each block where this would not reduce the dimension of the data to less than the size of the convolutional layer. Decoding similarly follows a standard upsampling/convolutional scheme, with a hyperbolic tangent activation following the final convolutional layer. The temperature of the Gumbel layer is slowly annealed throughout training such that the temperature at epoch E is 2e −0.0003E. During inference, the temperature is fixed at 0.1. We define the loss as a weighted sum of the mean squared error on the reconstruction and the negative of the Kullback-Leibler divergence. Models were trained in Tensorflow (v 1.12.2) on a single Nvidia Tesla K80. Hyperparameter tuning was carried out over one thousand evaluations for each model using a Tree-structured Parzen Estimator as implemented in the hyperopt package. For the primary data set, the data were divided at random into training/validation/test sets comprising 70%, 20% and 10% of the data respectively. With a small (L = 20) one-hot latent space, a standard VAE is unable to reconstruct any signal (Fig. 2(a) ). Given a sufficiently large (L = 50)) latent space, there is sufficient information to reconstruct the signal, but at the cost of retaining much of the noise, signal artefacts, and increasing the complexity of the latent space (Fig. 2(b) ). We solve this by allowing the leakage of some information directly from the original signal to the decoder, bypassing the latent space. For each channel, we find the set of n time-coordinates at which the observed signal is closest to zero. To prevent memorisation of the signal based on these coordinates, we randomly sample a subset n of these coordinates. Since these data can be ordered over time, we then apply a 1-d convolutional network to this input in a similar fashion to the encoder, giving an encoding of the signal as defined by the sampling from the coordinates. This'coordinate encoding' is concatenated to the upsampled layers in each step of the decoder. This allows a small (L = 20) one-hot latent space to identify the signal present in the data while removing the noise (Fig. 2(c) ). For this analysis n = 5 and n = 1, that is, a single value taken from the time-axis for each data channel and passed to the encoder via a CNN is sufficient to allow reconstruction of the latent space. We are able to reduce the latent space further (L = 10 and L = 5) while maintaining identification of the signal (data not shown). The input data from the first (blue) and second (orange) cuffs is reduced to a single value in the latent space which evolves over time, and is used to reconstruct the original signal. Without the coordinate encoding, no reconstruction is possible using a small latent space (a). With a large latent space (b), reconstruction is possible but with a complex latent space and the reconstruction of noise in addition to signal. With a coordinate encoder (c) the latent space is relatively simple and the reconstruction is effectively de-noised. Figure 2 demonstrates the ability of a Coordinate-VAE model to effectively de-noise peripheral neural data and cluster the observed signals within a relatively simple latent space. Furthermore, we can apply the model trained on data from a single subject to other subjects. Figure 3 shows the latent space and reconstructed signal from vagus nerve recordings from a second subject taken sixty days post surgical implantation. Despite the increased noise levels in this data set, the trained model is able to de-noise the signal and characterise the signals within the data. Data from this subject were human-labelled as containing regions of neural activity corresponding to respiration modulation. There is a clear correlation between respiration modulation events and the amplitude of the reconstructed signal, suggesting that the latent space is able to capture meaningful physiological signals from neural data. Furthermore, the latent space shows strong differences between the latent values prevalent within regions of respiration modulation and those without, with latent values 0, 7, 10, 13, 16 and 19 being significantly (χ 2 -test, moderated for dependence of neighbouring values) over-represented within the respiration modulation events. This suggests that, in the absence of labels, the latent space representation may still give useful information with which to identify physiological events. We explore the de-noising ability of this technique further through simulation studies (Figure 4). We simulate noise in each channel by independently sampling from Morlet wavelets, whose parameters are further sampled from independent normal distributions, and whose location on the time series are uniformly distributed. We combine this'noise' with'signal', also sampled from Morlet wavelets, but now located within short'impulse' time periods and correlated between the two signal channels. We then reconstruct the combined waveform and estimate the ratio of the power of the reconstruction within the impulse regions to the power of the reconstruction outside the impulse regions. By varying the amplitude of the'noise' signal, we acquire different values for the true signal-to-noise ratio (SNR) and compare this to the SNR post-reconstruction. Particularly for low true SNR, the post-reconstruction data show a considerably improved SNR. The recent development of chronic neural interfacing implant systems that are able to record neural signals over period of months or years will create large sets of primarily unlabelled data, with numerous signals occurring over a range of time-scales. These data are currently un-characterisable with standard methods (e.g. spike-sorting). Previous work in this field has relied on mixing categorical and real-valued latent vectors. Westhuizen et al used an adversarial auto-encoder to project neural data to labels, incorporating an approximately one-hot encoding in the latent space but also including an approximately Gaussian vector to allow reconstruction. Since both vectors are trained simultaneously, the Gaussian component of the latent space may contain the relevant labelling information for one or more true classes. InfoGAN, a GAN implementation in which the discriminator identifies components of the latent space is similarly capable of one-hot latent representation of the data, but without constraints on the information carried within the one-hot encoding. The Coordinate-VAE approach, in restricting the information available to the encoder creating the non-categorical portion of the latent space, allows unsupervised characterisation of the signals in timeseries data, while simultaneously de-noising the signal. Models are transferable between individuals, suggesting that we may gain the ability to pre-train large models for the reduction to latent space representations. As shown in Figure 3, there is some evidence to suggest that these latent space representations are also informative for physiological features. We might then rapidly train a final classifier or agent for monitoring or control of individual patients, as in Pandarianth et al, in which an auto-encoder is used as a dimension reduction technique on collections of neural spiking data acquired from macaque motor and pre-motor cortices, following which a GLM is used to map the complex latent space to spiking activity.

Represent the short summary for finding the corresponding scientific article (long)

Unsupervised analysis of data recorded from the peripheral nervous system denoises and categorises signals.

Represent the TLDR for finding the corresponding scientific article

We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.