Question: represent the text to find a very short summary (1-2 sentences max)

Question: It is clear that users should own and control their data and privacy. Utility providers are also becoming more interested in guaranteeing data privacy. Therefore, users and providers can and should collaborate in privacy protecting challenges, and this paper addresses this new paradigm. We propose a framework where the user controls what characteristics of the data they want to share (utility) and what they want to keep private (secret), without necessarily asking the utility provider to change its existing machine learning algorithms. We first analyze the space of privacy-preserving representations and derive natural information-theoretic bounds on the utility-privacy trade-off when disclosing a sanitized version of the data X. We present explicit learning architectures to learn privacy-preserving representations that approach this bound in a data-driven fashion. We describe important use-case scenarios where the utility providers are willing to collaborate with the sanitization process. We study space-preserving transformations where the utility provider can use the same algorithm on original and sanitized data, a critical and novel attribute to help service providers accommodate varying privacy requirements with a single set of utility algorithms. We illustrate this framework through the implementation of three use cases; subject-within-subject, where we tackle the problem of having a face identity detector that works only on a consenting subset of users, an important application, for example, for mobile devices activated by face recognition; gender-and-subject, where we preserve facial verification while hiding the gender attribute for users who choose to do so; and emotion-and-gender, where we hide independent variables, as is the case of hiding gender while preserving emotion detection. will make such devices not to understand the data until the visual or sound trigger is detected, with the capability to do this at the sensor level and without modifying the existing recognition system. This new paradigm of collaborative privacy environment is critical since it has also been shown that algorithmic or data augmentation and unpredictable correlations can break privacy BID7; BID14;; BID18. The impossibility of universal privacy protection has been studied extensively in the domain of differential privacy BID4, where a number of authors have shown that assumptions about the data or the adversary must be made in order to be able to provide utility BID5; BID6; BID10. We can, however, minimize the amount of privacy we are willing to sacrifice for a given level of utility. Other recent data-driven privacy approaches like have also explored this notion, but do not integrate the additional collaborative constraints. Therefore, it is important to design collaborative systems where each user shares a sanitized version of their data with the service provider in such a way that user-defined non-sensitive tasks can be performed but user-defined sensitive ones cannot, without the service provider requiring to change any data processing pipeline otherwise. Contributions-We consider a scenario where a user wants to share a sanitized representation of data X in a way that a latent variable U can be inferred, but a sensitive latent variable S remains hidden. We formalize this notion using privacy and transparency definitions. We derive an informationtheoretic bound on privacy-preserving representations. The metrics induced by this bound are used to learn such a representation directly from data, without prior knowledge of the joint distribution of the observed data X and the latent variables U and S. This process can accommodate for several user-specific privacy requirements, and can be modified to incorporate constraints about the service provider's existing utility inference algorithms enabling several privacy constraints to be satisfied in parallel for a given utility task. We apply this framework to challenging use cases such as hiding gender information from a facial image (a relatively easy task) while preserving subject verification (a much harder task), or designing a sanitization function that preserves subject identification on a consenting subset of users, while disallowing it on the general population. Blocking a simpler task while preserving a harder one and blocking a device from constantly listening out-of-sample data are new applications in this work, here addressed with theoretical foundations and respecting the provider's existing algorithms, which can simultaneously handle sanitized and non-sanitized data. The problem statement is detailed in Section 2, and the information-theoretic bounds are derived in Section 3. Section 4 defines a trainable adversarial game that directly attempts to achieve this bound; the section also discusses how service-provider specific requirements can be incorporated. Examples of this framework are shown in Section 5. The paper is concluded in Section 6. Complementary information and proofs are presented in the Supplementary Material. We describe a scenario in which we have access to possibly high-dimensional data X ∈ X, this data depends on two special latent variables U and S. U is called the utility latent variable, and is a variable we want to communicate, while S is called the secret, and is a variable we want to protect. We consider two agents, a service provider that wants to estimate U from X, and an actor that wants to infer S from X.We define a third agent, the privatizer, that wants to learn a space-preserving stochastic mapping Q: X → Q ⊃ X in such a way that Q(X) provides information about the latent variable U, but provides relatively little information of S. In other words, we want to find a data representation that is private with respect to S and transparent with respect to U. 1 We first recall the definition of privacy presented in BID9 DISPLAYFORM0 Definition 2.1. Privacy: Let δ s be a measure of distance between probability distributions, b s ∈ R + a positive real number, and P (S) the marginal distribution of the sensitive attribute S. The stochastic mapping Q(X) is (δ s, b s)-private with respect to S if δ s (P (S), P (S|Q(X))) < b s.We can define transparency in the same fashion: Definition 2.2. Transparency: Let δ u be a measure of distance between probability distributions, b u ∈ R + a positive real number, and P (U |X) the posterior conditional distribution of the utility variable U after observing X. The stochastic mapping Q(X) is (δ u, b u)-transparent with respect to U if δ u (P (U |X), P (U |Q(X))) < b u.Both definitions depend on the learned mapping Q; in the following section, we derive an information-theoretic bound between privacy and transparency, and show that this bound infers a particular choice of metrics δ u, δ s. We then show that this inferred metric can be directly implemented as a loss function to learn privatization transformations from data using standard machine learning tools. A similar analysis of these bounds for the special case where we directly observe the utility variable U (X = U) was analyzed in BID1 in the context of the Privacy Funnel. Here, we extend this to the more general case where U is observed indirectly. More importantly, these bounds are used to design a data-driven implementation for learning privacy-preserving mappings. Consider the utility and secret variables U and S defined over discrete alphabets U, S,and the observed data variable X, defined over X, with joint distribution P X,U,S. FIG0 illustrates this set-up, and shows the fundamental relationship of their entropies H(·) and mutual information. We analyze the properties of any mapping Q: X → Q, and measure the ing mutual information between the transformed variable Q(X) and our quantities of interest. Our goal is to find Q such that the information leakage from our sanitized data I(S; Q(X)) is minimized, while maximizing the shared information of the utility variable I(U ; Q(X)). We will later relate these quantities and the bounds here developed with the privacy/utility definitions presented in the previous section. Maximizing I(U ; Q(X)) is equivalent to minimizing I(U ; X | Q(X)), since I(U ; X|Q(X)) = I(U ; X) − I(U ; Q(X)). The quantity I(U ; X | Q(X)) is the information X contains about U that is censured by the sanitization mapping Q. FIG0 illustrates I(S; Q(X)) and I(U ; X|Q(X)). One can see that there exists a trade-off area, I(U, S) − I(U, S|X), that is always included in the union of I(S; Q(X)) and I(U ; X|Q(X)). The lower we make I(S; Q(X)), the higher we make the censored information I(U ; X|Q(X)), and vice versa. This induces a lower bound over the performance of the best possible mappings Q(X) that is formalized in the following lemma. Lemma 3.1. Let X, U, S be three discrete random variables with joint probability distribution P X,U,S. For any stochastic mapping Q: X → Q we have DISPLAYFORM0 Proof of this lemma is shown in Supplementary Material. To show that this bound is reachable in some instances, consider the following example. Let U and S be independent discrete random variables, and X = (U, S). The sanitization mapping Q(X) = U satisfies this bound with equality. We can also prove, trivially, an upper bound for these quantities. Lemma 3.2. Let X, U, S be three discrete random variables with joint probability distribution P X,U,S. For any stochastic mapping Q: X → Q we have: DISPLAYFORM1 That simply states that the information leakage about the secret and the censured information on the utility variable cannot exceed the total information present in the original observed variable X. We relate the terms I(S; Q(X)) and I(U ; X | Q(X)) in Eq.1 back to our definitions of privacy and transparency. DISPLAYFORM0 Here we used the fact that U is conditionally independent of Q given X. We then observe that Eq. Similarly, we can analyze I(S; Q) to get, DISPLAYFORM1 We can see from Eq.4 that the natural induced metric for measuring privacy δ s in Def,2.1 is the reverse Kullback-Leibler divergence RD KL.We can thus rewrite our fundamental tradeoff equation as DISPLAYFORM2 We show next how this bound can be used to define a trainable loss metric, allowing the privatizer to select different points in the transparency-privacy trade-off space. Assume that for any given stochastic transformation mapping Q ∼ Q(X), we have access to the posterior conditional probability distributions P (S | Q), P (U | Q), and P (U | X). Assume we also have access to the prior distribution of P (S). Inspired by the bounds from the previous section, the proposed privatizer loss is DISPLAYFORM0 where α ∈ is a tradeoff constant. A low α value implies a high degree of transparency, while a high value of α implies a high degree of privacy. Using Eq.5 we have a lower bound on how private or transparent the privatizer can be for any given α value, as detailed next. Theorem 3.3. For any α ∈, and stochastic mapping Q: X → Q the solution to Eq.6 guarantees the following bounds, DISPLAYFORM1 The proof is shown in Supplementary Material. To recap, we proposed a privatizer loss Eq.6 with a controllable trade-off parameter α, and showed bounds on how transparent and private our data can be for any given value of α. Next we show how to optimize this utility-privacy formulation. Even if the joint distribution of P (U, S, X) is not known, the privatizer can attempt to directly implement Eq.6 in a data-driven architecture to find the optimal Q. Assume the privatizer has access to a dataset {(x, s, u)}, where s and u are the ground truth secret and utility values of observation x. Under these conditions, the privatizer searches for a parametric mapping q = Q θ (x, z), where z is an independent random variable, and attempts to predict the best possible attack by learning P η (s | q), an estimator of P (s | q). The privatizer also needs P ψ (u|q) and P φ (u|x), estimators of P (u | q) and P (u | x) respectively, to measure how much information about the utility variable is censored with the proposed mapping. Under this setup Q θ (x, z) is obtained by optimizing the following adversarial game: DISPLAYFORM0 Here the first three terms are crossentropy loss terms to ensure our estimators P η (s|q), P ψ (u|q), and P φ (u|x) are a good approximation to the true posterior distributions. The final loss term attempts to find the best possible sampling function Q θ (x, z) such that (1 − α)I 2 (U ; X | Q) + αI 2 (S; Q) is minimized. Details on the algorithmic implementation are given in Section 7.3.1. Performance on simulated datasets is shown in Section 7.2. The proposed framework naturally provides a means to achieve collaboration from the utility provider. In this scenario, the utility provider wishes to respect the user's desired privacy, but is unwilling to change their estimation algorithm Pφ(u | x), and expects the privatizer to find a mapping that minimally affects its current performance.2 This is a more challenging scenario, with worse tradeoff characteristics, in which Q θ (x, z) is obtained by optimizinĝ DISPLAYFORM0 2 Recall that the utility provider wants to use the same algorithm for sanitized and non-sanitized data, a unique aspect of the proposed framework and critical to accept its collaboration. A final scenario addressed by the proposed framework arises when the utility provider is the sole agent to access the sanitized data, and it has estimation algorithms for both the utility and the privacy variable Pφ(u | x), Pη(s | x), that it is unwilling to modify. The service provider wishes to reassure the users that they are unable to infer the secret attribute from the sanitized data, if and when the user decides so. Under these conditions, we optimize for DISPLAYFORM0 The following examples are based on the framework presented in FIG8. Here we have the three key agents mentioned before: the utility algorithm that is used by the provider to estimate the information of interest. This algorithm can take the raw data (X) or the mapped data (Q(X)) and be able to infer the utility; the secret algorithm that is able to operate on the mapped data to infer the secret; the privatizer that learns a space preserving mapping Q that allows the provider to learn the utility but prevents the secret algorithm to infer the secret. The utility algorithm is trained to perform well on raw data, the secret algorithm is adversarially trained to infer the secret variable after sanitization. In the next examples we show how the proposed framework performs under different scenarios, the privatizer architecture is kept unchanged across all experiments to show that the same architecture can achieve very different objectives using the proposed framework, the detailed architectures are shown in Section 7.3.2. Extra experiments under known conditions are shown in 7.2.Figure 2: Three components of the collaborative privacy framework. Raw data can be directly fed into the secret and utility inferring algorithm. Since the privatization mapping is space preserving, the privatized data can also be directly fed to both tasks without any need for further adaptations. We begin by analyzing the subject-within-subject problem. Imagine a subset of users wish to unlock their phone using facial identification, while others opt out of the feature; we wish the face identification service to work only on the consenting subset of users. We additionally assume that the utility provider wishes to comply with the user's wishes, so we can apply the framework described in Section 4.2. Note that in this problem, the utility and secrecy variables are mutually exclusive. We solve this problem by training a space-preserving stochastic mapping Q on facial image data X, where the utility and secret variable U and S are categorical variables over consenting and nonconsenting users respectively. We test this over the FaceScrub dataset BID8 ), using VGGFace2 Cao et al. (2017 as the utility and secrecy inferring algorithm. The stochastic mapping was implemented using a stochastic adaptation of the , architecture details are provided in Section 7.3.2. Table 1 shows the top-5 categorical accuracy of the utility network over the sanitized data at various α points in the privacy-utility trade-off. Figure 3 show some representantive images on how images are sanitized. It also shows that the sanitization function is able to preserve information about the utility variable while effectively censoring the secret variable, even for unobserved images. A phone equipped with this filter at the sensor level would be effectively incapable of collecting information on nonconsenting users. Figure 3 : Left and center figures show images of consenting and nonconsenting (private) users respectively, along with their sanitized counterparts. The identity of consenting users is still easily verified, while the identity of nonconsenting users is effectively censored. Table on the right shows Top-5 accuracy performance of the subject detector after sanitization across several sanitation levels α. Performance is shown across 3 subsets, consenting users (CU) are users that decided to be detected by the utility algorithm, observed private users (OPU) are those that explicitly decided to protect their privacy, while unobserved private users (UPU) are users that decided to protect their privacy but where not available during training. Consenting users are still recognized by the system, while nonconsenting users are not. For example, for α = 0.4, we significantly block OPU and UPU while preserving CU Here we continue to work on facial image data X, where utility variable U is gender recognition, and the secret variable S is emotion (smiling/non-smiling). In this scenario, variables U and S are independent. We implement this over the CelebA dataset BID13, using Xception networks as our utility and privacy estimators. Table. 2 shows the distribution of the utility and secrecy estimators over the sanitized data. FIG3 shows example sanitized images. It is visually possible to identify the gender of the subject but not their emotion. Most importantly, the existing gender detection algorithm still performs correctly over the sanitized images. Table 2: Gender and emotion detection on users on raw and sanitized data. In this setup, we want to find a mapping Q that hides the gender attribute but allows subject verification. The mapping Q should prevent a standard gender detection algorithm from performing its task, while allowing a standard subject detector algorithm to still perform subject verification. This is the only experiment in this section where the secret inference algorithm is fixed. The mapping that incorporates a pretrained FaderNet was chosen as the baseline for the stochastic mapping function since this network is already trained to defeat a gender discriminator in its encoding space. This proves a suitable baseline comparison and starting point for a mapping function that needs to fool a gender discriminator in image space while simultaneously preserving subject verification performance. We show the performance of using only the pretrained gender FaderNet and demonstrate how we can improve its performance by training a posterior processing mapping (UNET) using the loss proposed in Eq.10.We tested this framework on the FaceScrub dataset BID15. FIG5 shows how the output probabilities of the gender classification model approach the prior distribution of the dataset as α increases. We see that sanitized images produce output gender probabilities close to the dataset prior even for relatively low α values. Last column of FIG5 shows how the top-5 categorical accuracy of the subject verification task varies across different α values. These suggest that under these conditions we can achieve almost perfect privacy while maintaining reasonable utility performance. DISPLAYFORM0 Inspired by information-theory bounds on the privacy-utility trade-off, we introduced a new paradigm where users and entities collaborate to achieve both utility and privacy per a user's specific requirements. One salient feature of this paradigm is that it can be completely transparentinvolving only the use of a simple user-specific privacy filter applied to user data -in the sense that it requires otherwise no modifications to the system infrastructure, including the service provider algorithmic capability, in order to achieve both utility and privacy. Representative architectures and suggest that a collaborative user-controlled privacy approach can be achieved. While the here presented clearly show the potential of this approach, much has yet to be done, of particular note is extending this approach to continuous utility and privacy variables. While the underlying framework still holds, reliably measuring information between continuous variables is a more challenging task to perform and optimize for. The proposed framework provides privacy metrics and bounds in expectation; we are currently investigating how the privacy tails concentrate as data is acquired and if there is a need to use information theory metrics with worst-case scenario guarantees. Modifying the information theory metrics to match some of the theoretical in (local) differential privacy is also the subject of future research. Privacy is closely related to fairness, transparency, and explainability, both in goals and in some of the underlying mathematics. A unified theory of these topics will be a great contribution to the ML community. Consider the equality I(U ; S) − I(U ; S|X) = I(S; Q(X)) − I(S; Q(X)|U ) + I(U ; X|Q(X)) − I(U ; X|Q(X), S).We know that DISPLAYFORM0 so we can guarantee DISPLAYFORM1 Proof. Theorem 3.3 DISPLAYFORM2 and α ∈.Minimizing Eq.6 respecting Eq.5 and Eq.12 is equivalent to solving: DISPLAYFORM3 Consider the following relaxation of Eq.15 DISPLAYFORM4 where A and B are positive real values. Eq.16 is a relaxation of Eq.15 because the space of possible tuples (A Q, B Q) is included in the space of possible values of R Suppose Q * is the solution to Eq.15, with corresponding values (A Q *, B Q *), and suppose (A *, B *) is the solution to Eq. 16. We know DISPLAYFORM0 Assume DISPLAYFORM1 However, A Q * > 0 and A Q * + B * ≤ A * + B * ≤ K. Therefore, (A Q *, B *) is a valid solution to Eq.16, and is smaller than the lower bound (A *, B *).This contradiction arises from assuming A * > A Q *, we thus conclude that DISPLAYFORM2 Similarly for B * and B Q * we get B * ≤ B Q *.Additionally, Eq.16 is easily solvable and has solutions DISPLAYFORM3 Consequently, we proved DISPLAYFORM4 The following experiments attempt to show how close to the theoretical bound shown in Eq. 5 we can get by following Algorithm 1 under known conditions. Consider the following scenario: Utility variable U and secret variable S are two Bernoulli variables with the following joint distribution: DISPLAYFORM0 where the marginal probabilities are P (U = 1) = ρ, P (S = 1) = β, and k is a parameter that controls the dependence between and S, k ∈ [0, min{ρ DISPLAYFORM1 For these experiments, we make both marginals equal to 0.5 (ρ = β = 0.5). Note that when k = 1, U and S are independent (I(U ; S) = 0) and when k = 0 or k = 2 they reach maximum mutual information (I(U ; S) = H(U) = H(S) = H b (0.5) = ln nats).Our observations X will be taken in the extreme case, where X contains almost perfect information about the values of U and S. We do this by assuming that X ∈ R 2 is a Gaussian Mixture Model X with the following conditional distribution: DISPLAYFORM2 We choose a low σ = 0.05; this makes it so that every pair (u, s) is mapped to almost entirely disjoint regions, therefore knowing X gives nearly perfect information about (u, s) (I(U ; X) H(U), I(S; X) H(S), I(U ; S|X) 0). For added simplicity, the privacy filter is linear: Figure 6 shows how the raw and sanitized data are distributed for varying levels of codependence k and tradeoff α for linear sanitization functions. Figure 7 shows that privacy filters optimized using Algorithm 1 learn effective privacy-preserving mappings close to the theoretical bounds, even for a simple filtering architecture. They do so without any explicit modelling of the underlying data-generating distributions, and we can achieve different tradeoff points by simply modifing the parameter α. Note that when variables U and S are perfectly independent or codependent, the linear filter is perfectly able to reach any point in the optimal bound. For intermediate cases, the linear filter was capable of reaching the bound in the region where no utility is compromised, but was not capable of following the optimal tradeoff line for higher levels of privacy. Figure 7: Top row shows the best privacy-utility loss so far on the validation set for different levels of codependence I(U ; S) and trade-off parameter α. Middle row shows how the sum of the estimated informations approximate the information bound for the best privacy-utility loss so far on the validation set. Finally, bottom row illustrates the trajectory in information space of the privacy-utility filters as they are being trained. DISPLAYFORM3 Here, we explicitly elaborate on how we optimize the data-driven loss functions shown in equations 8, 9, and 10 using an adversarial approach. We first detail the exact adversarial training setup that was used to perform the experiments in Section 5, and then provide the concrete network architectures used for all shown . Minimizing the objective functions in equations 8, 9, and 10, is a challenging problem in general. By focusing our attention on Eq. 8, we see that each of the four loss terms have a distinct purpose: DISPLAYFORM0 The first three loss terms minimize a crossentropy objective for functions P η (s | q), P ψ (u | q), and P φ (u | x); this ensures that these functions are good estimators of the unknown true distributions of P (s | q), P (u | q), and P (u | x), where samples q are drawn from the learned sanitization mapping q = Q θ (x, z). The final loss term attempts to find the best possible sampling function DISPLAYFORM1 We can approximately solve this problem by applying iterative Stochastic Gradient Descent to each of the individual loss terms with respect to their relevant parameters, this is similar to the procedure used to train Generative Adversarial Networks. The algorithm we used to solve Eq. 8 is shown in Algorithm 1, similarly, algorithms to solve Eq. 9 and Eq. 10 are shown in Algorithm 2 and Algorithm 3 respectively. DISPLAYFORM2 Evaluate crossentropy loss on raw utility inference 5: DISPLAYFORM3 Evaluate crossentropy loss on filtered utility inference 7: DISPLAYFORM4 Evaluate crossentropy loss on secret inference 9: DISPLAYFORM5 10: DISPLAYFORM6 Evaluate sanitation loss 11: DISPLAYFORM7 Stochastic gradient descent step on Q θ (x, z)12: until convergence We now describe the exact architecture used to implement the privacy filter on all experiments shown in Section 5. Figure 8 shows the network diagram. The architecture presented in Figure 8 is fully convolutional, so the same network definition could be used across all three experiments by varying the input layer. To speed up convergence to a good filtering solution, filters were initially trained to copy the image (under RMSE loss), and optionally infer some meaningful attribute from the input (in subject-within-subject, this attribute was a simple class label on whether the subject wished their privacy preserved). We stress that this was only done for initialization, final training of the network was done exactly as described in Algorithm 1. η ← η − lr∇ η H(η) Stochastic gradient descent step on P η (s | q)6: DISPLAYFORM0 Evaluate sanitation loss 7: DISPLAYFORM1 Stochastic gradient descent step on Q θ (x, z) Θ(θ) = (1 − α) DISPLAYFORM2 Evaluate sanitation loss 5: DISPLAYFORM3 Stochastic gradient descent step on Q θ (x, z) 6: until convergence Figure 8: Architecture of privacy filter, based on UNET. There is a single noise layer (shown in yellow) where standard Gaussian noise is injected into the network to make the ing filtered image stochastic in nature. The other notable component is the auxiliary label softmax, used for the subject-within-subject experiment. This extra layer was trained only to initialize the network, but was not preserved during the final training stage. Input image sizes are shown for the subject-within-subject experiment. The architecture of the networks used to infer the utility and secret attribute in the emotion vs. gender experiment are identical, and are shown in Figure 9.Networks used for the experiments in Section 7.2 are shown in FIG0.All other networks used in the section are implemented as described in their respective papers.

represent this text to find a much much longer article about it with say hundreds-thousands of words E.g.:
We propose a novel combination of adversarial training and provable defenses which produces a model with state-of-the-art accuracy and certified robustness on CIFAR-10.  == We propose a new method to train neural networks based on a novel combination of adversarial training and provable defenses. The key idea is to model training as a procedure which includes both, the verifier and the adversary. In every iteration, the verifier aims to certify the network using convex relaxation while the adversary tries to find inputs inside that convex relaxation which cause verification to fail. We experimentally show that this training method is promising and achieves the best of both worlds – it produces a model with state-of-the-art accuracy (74.8%) and certified robustness (55.9%) on the challenging CIFAR-10 dataset with a 2/255 L-infinity perturbation. This is a significant improvement over the currently known best of 68.3% accuracy and 53.9% certified robustness, achieved using a 5 times larger network than our work. The discovery of adversarial examples in deep learning has increased the importance of creating new training methods which produce accurate and robust neural networks with provable guarantees. Existing work: adversarial and provable defenses Adversarial training provides a basis framework which augments the training procedure with adversarial inputs produced by an adversarial attack. instantiated adversarial training using a strong iterative adversary and showed that their approach can train models which are highly robust against the strongest known adversarial attacks . This method has also been used to train robust ImageNet models . While promising, the main drawback of the method is that when instantiated in practice, via an approximation of an otherwise intractable optimization problem, it provides no guarantees -it does not produce a certificate that there are no possible adversarial attacks which could potentially break the model. To address this lack of guarantees, recent line of work on provable defenses; has proposed to train neural networks which are certifiably robust under a specific attacker threat model. However, these guarantees come at the cost of a significantly lower standard accuracy than models trained using adversarial training. This setting raises a natural question -can we leverage ideas from both, adversarial training techniques and provable defense methods, so to obtain models with high accuracy and certified robustness? This work: combining adversarial and provable defenses In this work, we take a step towards addressing this challenge. We show that it is possible to train more accurate and provably robust neural networks using the same convex relaxations as those used in existing, state-of-the-art provable defense methods, but with a new, different optimization procedure inspired by adversarial training. Our optimization works as follows: (i) to certify a property (e.g., robustness) of the network, the verifier produces a convex relaxation of all possible intermediate vector outputs in the neural network, then (ii) an adversary now searches over this (intermediate) convex regions in order to find, what we refer to as a latent adversarial example -a concrete intermediate input contained in the relaxation that when propagated through the network causes a misclassification that prevents verification, and finally (iii) the ing latent adversarial examples are now incorporated into our training scheme using adversarial training. Overall, we can see this method as bridging the gap between adversarial training and provable defenses (it can conceptually be instantiated with any convex relaxation). We experimentally show that the method is promising and in a neural network with state-of-theart 78.8% accuracy and 58.1% certified robustness on the challenging CIFAR-10 dataset with 2/255 L ∞ perturbation (the best known existing are 68.3% accuracy and 53.9% certified robustness using 5 times larger network). • A new method we refer to as layerwise adversarial training which can train provably robust neural networks and conceptually bridges the gap between adversarial training and existing provable defense methods. • Instantiation of layerwise adversarial training using linear convex relaxations used in prior work (accomplished by introducing a projection operator). • Experimental showing layerwise adversarial training can train neural network models which achieve both, state-of-the-art accuracy and certified robustness on CIFAR-10 with 2/255 L ∞ perturbation. Overall, we believe the method presented in this work is a promising step towards training models that enjoy both, higher accuracy and higher certification guarantees. An interesting item for future work would be to explore instantiations of the method with other convex relaxations than the one considered here. We now discuss some of the closely related work on robustness of neural networks. Heuristic adversarial defenses After the first introduction of adversarial examples , defense mechanisms to train robust neural networks were built based on the inclusion of adversarial examples to the training set . Models trained using adversarial training with projected gradient descent (PGD) were shown to be robust against the strongest known attacks . This is in contrast to other defense mechanisms which have been broken by new attack techniques . While models trained using adversarial training achieve robustness against strong adversaries, there are no guarantees that model is robust against any kind of adversarial attack under the threat model considered. Provable adversarial defenses Another line of work proposes to learn classifiers which come with robustness guarantees. These approaches are based on linear or semidefinite (; a) relaxations, hybrid zonotope or interval bound propagation . While these approaches currently obtain robustness guarantees, accuracy of these networks is relatively small and limits practical use of these methods. There has also been recent work on certification of general neural networks, not necessarily trained in a special way. These methods are based on SMT solvers , mixed-integer linear programs, abstract interpretation, restricted polyhedra (b) or combinations of those (a). Another line of work proposes to replace neural networks with a randomized classifier (; ; a) which comes with probabilistic guarantees on its robustness. While these approaches scale to larger datasets such as ImageNet (although with probabilistc instead of exact guarantees), their bounds come from the relationship between L 2 robustness and Gaussian distribution. In this paper, we consider general verification problem where input is not necessarily limited to an L p ball, but arbitrary convex set. In this work we consider a threat model where an adversary is allowed to transform an input x ∈ R d0 into any point from a convex set S 0 (x) ⊆ R d0. For example, for a threat model based on L ∞ perturbations, the convex set will be defined as Figure 1: An iteration of layerwise adversarial training. Latent adversarial example x 1 is found in the convex region C 1 (x) and propagated through the rest of the layers in a forward pass which is shown with the blue line. During backward pass, gradients are propagated through the same layers, shown with the red line. Note that the first convolutional layer does not receive any gradients. We now describe our layerwise adversarial training approach which yields a provable defense that bridges the gap between standard adversarial training and existing provable defenses. Motivation: latent adversarial examples Consider an already trained neural network model h θ which we would like to certify using convex relaxations. A fundamental issue here is that certification methods based on convex relaxations can struggle to prove the target property (e.g., robustness) due to the iterative accumulation of errors introduced by the relaxation. More precisely, assume the neural network actually satisfies the property from Equation 1 for an input x, meaning that. Naturally, this also implies that the neural network behaves correctly in the latent space of its first hidden layer in the region S 1 (x). Formally, this means that c T h 2:k θ (x 1)+d < 0, ∀x 1 ∈ S 1 (x). However, if one would use a certification method which replaces the region S 1 (x) by its convex relaxation C 1 (x), then it is possible that we would fail to certify our desired property. This is due to the fact that there may exist an input Of course, we could repeat the above thought experiment and possibly find more violating latent inputs in the set C i (x) \ S i (x) of any hidden layer i. The existence of points found in the difference between a convex relaxation and the true region is a fundamental reason for the failure of certification methods based on convex approximations. For convenience, we refer to such points as latent adversarial examples. Next, we describe a method which trains the neural network in a way that aims to minimize the number of latent adversarial examples. Layerwise provable optimization via convex relaxations Our key observation is that the two families of defense methods described earlier are in fact different ends of the same spectrum: methods based on adversarial training maximize the cross-entropy loss in the first convex region C 0 (x) while provable defenses maximize the same loss, but in the last convex region C k (x). Both methods then backpropagate the loss through the network and update the parameters using SGD. However, as explained previously, certification methods may fail even before the last layer due to the presence of latent adversarial examples in the difference of the regions C i (x) and S i (x). A natural question then is -can we leverage adversarial training so to eliminate latent adversarial examples from hidden layers and obtain a provable network? To this end, we propose adversarial training in layerwise fashion. The initial phase of training is equivalent to adversarial training as used by. In this phase in the inner loop we repeatedly find an input in C 0 (x) which maximizes the cross-entropy loss and update the parameters of the neural network so to minimize this loss using SGD. Note that the outcome of this phase is a model which is highly robust against strong multi-step adversaries. However, certification of this fact often fails due to the previously mentioned accumulation of errors in the particular convex relaxation being used. The next step of our training method is visually illustrated in Figure 1. Here, we propagate the initial convex region through the first layer of the network and obtain the convex relaxation C 1 (x). We then solve the optimization problem to find a concrete point x 1 inside of C 1 (x) which produces Algorithm 1: Layerwise adversarial training via convex relaxations Data: k-layer neural network h θ, training set (X, Y), learning rate η, step size α, inner steps n Result: Update in parallel n times: Freeze parameters θ l+1 of layer l + 1; 12 end the maximum loss when this point is propagated further through the network (this forward pass is shown with the blue line). Finally, we backpropagate the final loss (red line) and update the parameters of the network so to minimize the loss. Critically, we do not backpropagate through the convex relaxation in the first layer as standard provable defenses do ). We instead freeze the first layer and stop backpropagation after the update of the second layer. Because of this, our optimization problem is significantly easier -the neural network only has to learn to behave well on the concrete points that were found in the convex region C l (x). This can be viewed as an extension of the robust optimization method that found to work well in practice. We then proceed with the above process for later layers. Formally, this training process amounts to (approximately) solving the following min-max optimization problem at the l-th step: Note that for l = 0 this formulation is equivalent to the standard min-max formulation in Equation 2 because C 0 (x) = S 0 (x). Our approach to solve this min-max optimization problem for every layer l is shown in Algorithm 1. We initialize every batch by random sampling from the corresponding convex region. Then, in every iteration we use projected gradient descent (PGD) to maximize the inner loss in 3. We first update x j in the direction of the gradient of the loss and then project it back to C l (x j) using the projection operator Π. Note that this approach assumes the existence of an efficient projection method to the particular convex relaxation the method is instantiated with. In the next section, we show how to instantiate the training algorithm described above to a particular convex relaxation which is generally tighter than a hyperrectangle and where we derive an efficient projection operation. So far we described the general approach of layerwise adversarial training. Now we show how to instantiate it for a particular convex relaxation based on linear approximations. If instead one would use interval approximation ) as the convex relaxation, then all regions C l (x) will be hyperrectangles and projection to these sets is fast and simple. However, the interval relaxation provides a coarse approximation which motivates the need to train with relaxations that provide tighter bounds. Thus, we consider linear relaxations which are generally tighter than those based on intervals. In particular we leverage the same relaxation which was previously proposed in;; as an effective way to certify neural networks. Here, each convex region is represented as a set Vector a l represents the center of the set and the matrix A l represents the affine transformation of the hypercube [−1, 1] m l. The initial convex region C 0 (x) is represented using a 0 = x and A 0 = I d0 is a diagonal matrix. Propagation of these convex regions through the network is out of the scope of this paper -a full description can be found in or. At a high level, the convolutional and fully connected layers are handled by multiplying A l and a l by appropriate matrices. To handle the ReLU activation, for ReLU units which cross 0, we apply a convex relaxation which amounts to multiplying A l and a l by appropriately chosen scalar values, depending whether the ReLU is activated or not. Using this relaxation of ReLU, we can recursively obtain all convex regions C l (x). In practice, A l e can be computed without explicitly constructing matrix A l because A l e = W l Λ l−1 W l−2 · · · M 0 e. Then we can perform matrix-vector multiplication right to left to obtain vector A l e. We provide more detailed description of this propagation in Appendix A. Projection to linear convex regions To use our training method we now need to instantiate Algorithm 1 with a suitable projection operator Π C l (x). The key insight here is that the vector x ∈ C l (x) is uniquely determined by auxiliary vector e ∈ [−1, 1] m l where x = a l + A l e. Then instead of directly solving for x which requires projecting to C l (x), we can solve for e instead which would uniquely determine x. Crucially, the domain of e is a hyperrectangle [−1, 1] m l which is easy to project to. To visualize this further we provide an example in Figure 2. The goal is to project the red point x in the right picture to the convex region C l (x). To project, we first perform change of variables to substitute x with e and then project e to the square [−1, 1] × [−1, 1] to obtain the blue point Π(e) on the left. Then, we again perform change of variables to obtain the blue point Π(x) on the right, the projection of x we were looking for. Based on these observations, we modify Line 7 of Algorithm 1 to first update the coefficients e j using the following update rule: e j ← clip(e j + αA Here clip is function which thresholds its argument between -1 and 1, formally clip(x, −1, 1) = min(max(x, −1), 1). This is followed by an update to x j via x j ← a l + A l e j, completing the update step. Sparse representation While our representation of convex regions with matrix A l and vector a l has clean mathematical properties, in practice, a possible issue is that the matrix A l can grow to be quite large. Because of this, propagating it through the network can be memory intensive and prohibit the use of larger batches. To overcome this difficulty, we first observe that A l is quite sparse. We start with a very sparse, diagonal matrix A 0 at the input. After each convolution, an element of matrix A l+1 is non-zero only if there is a non-zero element inside of its convolutional kernel in matrix A l. We can leverage this observation to precompute positions of all non-zero elements in matrix A l+1 and compute their values using matrix multiplication. This optimization is critical to enabling training to take place altogether. An interesting item for future work is further optimizing the current relaxation (via a specialized GPU implementation) or developing more memory friendly relaxations, so to scale the training to larger networks. After training a neural network via layerwise adversarial training, our goal is to certify the target property (e.g., robustness). Here we leverage certification techniques which are not fast enough to be incorporated into the training procedure, but which can significantly boost the certification performance. The linear relaxation of ReLU that we are using is parameterized by slopes λ of the linear relaxation. Prior work which employed this relaxation; ) chose these slopes in a greedy manner by minimizing the area of the relaxation. During training we also choose λ in the same way. However, during certification, we can also optimize for the values of λ that give rise to the convex region inside of which the maximum loss is minimized. This optimization problem can be written as: Solving this is computationally too expensive inside the training loop, but during certification it is feasible to approximate the solution. We solve for λ using the Adam optimizer and clipping the elements between 0 and 1 after each update. We remark that the idea of learning the slope is similar to Dvijotham et al. (2018b) who propose to optimize dual variables in a dual formulation, however here we stay in the primal formulation. Combining convex relaxations with exact bound propagation During layerwise adversarial training we essentially train the network to be certified on all regions C 0 (x),..., C k (x). While computing exact regions S l (x) ⊆ C l (x) is not feasible during training, we can afford it to some extent during certification. The idea is to first propagate the bounds using convex relaxations until one of the hidden layers l and obtain a region C l (x). If training was successful, there should not exist a concrete point x l ∈ C l (x) which, if propagated through the network, violates the correctness property in Equation 1. We can encode both, the property and the propagation of the exact bounds S l (x) using a Mixed-Integer Linear Programming (MILP) solver. Note that we can achieve this because we represent the region C l (x) using a set of linear constraints, however, for general convex shapes this may not be possible. We perform the MILP encoding using the formulation from. It is usually possible to encode only the last two layers using MILP due to the poor scalability of these solvers for realistic network sizes. One further improvement we also include is to tighten the convex regions C l (x) using refinement via linear programming as described in Singh et al. (2019a). We remark that this combination of convex relaxation and exact bound propagation does not fall under the recently introduced convex barrier to certification Salman et al. (2019b). We now present an evaluation of our training method on the challenging CIFAR-10 dataset. Experimental setup We evaluate on a desktop PC with 2 GeForce RTX 2080 Ti GPU-s and 16-core Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz. We use Gurobi as a MILP solver. Our method is implemented in PyTorch and we plan to release both, the code and the trained models. Neural network architecture All presented are on a 4-layer convolutional network with 49 402 neurons: first 3 layers are convolutional layers with filter sizes 32, 32 and 128, kernel sizes 3, 4, 4 and strides 1, 2, 2 respectively. These are followed by a fully connected layer with 250 hidden units. After each of these layers, there is a ReLU activation. Training We use batch size 50 and L1 regularization 0.00001 for training. We perform optimization using Adam with initial learning rate 0.001 which is decreased by 10× every 100 epochs. During layerwise training we start with perturbation which is 10% higher than the one we certify and we decrease it by 5% when the training progresses to the next layer. Certification After training completes, we perform certification as follows: for every image, we first try to certify it using only linear relaxations (with the improvement of learned slopes, Section 6). If this fails, we encode the last layer as MILP and try again. Finally, if this fails we encode the ReLU activation after the last convolution using additional 300 binary variables and the rest using the triangle formulation. We consider an image to be not certifiable if we fail to verify it using these methods. We always evaluate on the first 1 000 images from the test set. Comparison to prior work We first train and certify using our method for the L ∞ perturbation 2/255. Results are shown in Table 1. We always compare to the best reported and reproducible et al. 68.3 53.9 70.1 50.0 59.9 46.1 62.3 45.5 61.1 45.9, as this improvement is also orthogonal to the method here. Thus, we only consider their best single network architecture (inline with prior work which compares to a single architecture). We believe all methods listed in Table 1, including ours, would benefit from additional techniques such as cascades, pre-training and leveraging unlabeled data. Experimentally, we find that the neural network trained using our method substantially outperforms all existing approaches, both in terms of standard accuracy and certified robustness for 2/255. Note that here we are using the same linear relaxation as, but our optimization procedure is different and shows significant improvements over the one used in their work. We also run the same experiment for L ∞ perturbation 8/255. Here we do not include comparison with as their were found to be not reproducible (; ;). These are presented in Table 2. Here we substantially outperform all existing approaches in terms of standard accuracy. However, in terms of certified robustness we are not able to achieve similar to whose method is based on a combination of interval approximation and linear relaxation. The main issue is that our 4-layer network lacks capacity to solve this task -even if training only using standard adversarial training our empirical robustness does not go above ∼ 34%. We remark that capacity was found to be one of the key components necessary to obtain a robust classifier . Due to promising for 2/255, we believe achieving state-of-the-art for 8/255 is very likely an issue of instantiating our method with a convex relaxation that is more memory efficient, which we believe is an interesting item for future work. We presented a new method to train certified neural networks. The key concept was to combine techniques from provable defenses using convex relaxations with those of adversarial training. Our method achieves state-of-the-art 78.8% accuracy and 58.1% certified robustness on CIFAR-10 with a 2/255 L ∞ perturbation, significantly outperforming prior work when considering a single network (it also achieves competitive on 8/255 L ∞). The method is general and can be instantiated with any convex relaxation. A promising future work item is scaling to larger networks: this will require tight convex relaxations with a low memory footprint that allow for efficient projection. Here we provide additional details that were omitted in the main body of the paper. In this section, we describe how to propagate convex relaxations of the form C l (x) = {a l + A l e | e ∈ [−1, 1] m l } through the network, for a single input x. As explained before, these relaxations were previously proposed in;. For the sake of completeness we describe them here using our notation. Depending on the form of function h i θ representing operation applied at layer i we distinguish different cases. Here we assume we have obtained region C i−1 (x) and our goal is to compute the region C i (x) using convex relaxation g i θ of the function h i θ. Initial convex region Let be L ∞ radius that we are certifying. We then compute minimum and maximum pixel values for each pixel as x l = max(0, x −) and x u = min(1, x +). We define initial convex region as: Convolutional and fully connected layers For both convolutional and fully connected layers, the update is given by mi. We can then compute: Using this formula, we define convex region C i+1 (x) = {a i+1 + A i+1 e | e ∈ [−1, 1] mi+1 } where: We will explain the transformation of a single element x i,j = a i,j + A T i,j e. We first compute lower bound l i,j and upper bound u i,j of element x i in the set C i (x): In the other case where 0 is between l i,j and u i,j we define ReLU (x i,j) = λ i,j x i,j + µe mi+j where e mi+j ∈ [−1, 1] is a coefficient for a new error term. Formulas for λ i,j and µ i,j are the following: This computation can also be written in the matrix form as ReLU (x i) = Λ i+1 x i + M i+1 e new where Λ i+1 and M i+1 are diagonal matrices with elements computed as above. Finally, new convex region C i+1 (x) = {a i+1 + A i+1 e | e ∈ [−1, 1] mi+1 } is defined as: where [] denotes concatenation of matrices. Here we describe how we apply random projection approach from to estimate the bounds during training. While operate in dual framework, their method to statistically estimate the bounds during training can also be applied in primal framework which we use in this paper. Recall that lower and upper bound for each neuron are computed as Thus, we need to compute ||A i || 1, which is L 1 norm of each row in the matrix A i. Using the method from, based on the from , we estimate ||A i || 1 with the method of random projections. Here A i is a matrix with d i rows and m i columns, where d i is dimensionality of the output vector in the i-th layer and m i is number of unit terms in the definition of region C i (x). The method of random projections samples standard Cauchy random matrix R of dimensions m i × k (k is number of projections) and then estimates ||A i || 1 ≈ median(|A i R|). To avoid computing the entire matrix A i we substitute: In the formula above, we set M 0 = A 0. Now, Dvijotham et al. (2018a) 83.4 62.4 To calculate A i R we split R = [R 0, R 2, ..., R i−1] and compute: Crucially, each summand can now be efficiently computed due to the associativity of matrix multiplication by performing the multiplication backwards. In this section we present additional on SVHN and MNIST datasets. SVHN We evaluated on SVHN dataset. For this experiment, we used convolutional network with 2 convolutional layers of kernel size 4 and stride 1 with 32 and 128 filters respectively. These convolutional layers are followed by a fully connected layer with 200 neurons. Each of the layers is followed by a ReLU activation function. For our training, we started with perturbation 10% higher than the one we are certifying and decreased it by 5% when progressing to the next layer. We trained each layer for 50 epochs and used L 1 regularization factor 0.0001. Results of this experiment are shown in Table 3. We certified first 1 000 images in SVHN test dataset. Our network has both, higher accuracy and higher certified robustness, than networks trained using other techniques for provable defense. MNIST In the next experiment, we evaluted on MNIST dataset. For this experiment, we used convolutional network with 2 convolutional layers with kernel sizes 5 and 4, and strides 2 followed by 1 fully connected layer. Each of the layers is followed by a ReLU activation function. For our training, we started with perturbation 10% higher than the one we are certifying and decreased it by 5% when progressing to the next layer. We trained each layer for 50 epochs and used L 1 regularization factor 0.00001. We certified first 1 000 images in MNIST test dataset. For perturbation 0.1, convolutional layers have filter sizes 32 and 64, and fully connected layer has 150 neurons. Results are presented in Table 4. Here, our numbers are comparable to those of state-of-the-art approaches. For perturbation 0.3, convolutional layers have filter sizes 32 and 128, and fully connected layer has 400 neurons. Results are presented in Table 4. Here, our certified robustness is somewhat lower than state-of-the-art. We believe this is due to the imprecise estimates of lower and upper bund via random projections. This is also reflected in relatively poor performance of who also rely on the same statistical estimates. Thus, for MNIST dataset and perturbation 0.3 it is likely necessary to use exact propagation instead of the estimates. However, this also induces large cost to the runtime. During training, we use two regularization mechanisms to make our convex relaxation tighter, both previosuly proposed in. First, we use L 1 -norm regularization which is known to induce sparsity in the weights of the network. has shown that weight sparsity helps induce more stable ReLU units which in turn makes our convex relaxation tighter (as for stable ReLU units it is already precise). 98.9 97.7 99.0 94. 4 98.9 96.3 Dvijotham et al. (2018a) 98.8 95. 6 98.7 95.8 99.0 95.6 Table 5: Evaluation on MNIST dataset with L ∞ perturbation 0.3 Method Accuracy (%) Certified robustness (%) Our work 97.6 84. 6 98.3 91.9 98.5 91.5 85.1 56.9 96.6 89.3 97.3 80.7 Second, in the i-th phase of training, we explicitly introduce a loss based on the volume of convex region C i+1. To make the relaxation tighter and minimize the volume, for each neuron j in layer i + 1 we add a loss of the form max(0, −l i+1,j) max(0, u i+1,j). This loss corresponds to the area under ReLU relaxation, see e.g. for a derivation.

Learning privacy-preserving transformations from data. A collaborative approach

represent the short summary for finding the corresponding scientific article (long).

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.