Question:
represent the scientific article to find a short tldr!
Fewshots:
'We propose and study a method for learning interpretable representations for the task of regression. Features are represented as networks of multi-type expression trees comprised of activation functions common in neural networks in addition to other elementary functions. Differentiable features are trained via gradient descent, and the performance of features in a linear model is used to weight the rate of change among subcomponents of each representation. The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation. We compare several stochastic optimization approaches within this framework. We benchmark these variants on 100 open-source regression problems in comparison to state-of-the-art machine learning approaches. Our main finding is that this approach produces the highest average test scores across problems while producing representations that are orders of magnitude smaller than the next best performing method (gradient boosting). We also report a negative in which attempts to directly optimize the disentanglement of the representation in more highly correlated features. The performance of a machine learning (ML) model depends primarily on the data representation used in training BID3, and for this reason the representational capacity of neural networks (NN) is considered a central factor in their success in many applications BID19. To date, there does not seem to be a consensus on how the architecture should be designed. As problems grow in complexity, the networks proposed to tackle them grow as well, leading to an intractable design space. One design approach is to tune network architectures through network hyperparameters using grid search or randomized search BID4 with cross validation. Often some combination of hyperparameter tuning and manual design by expertise/intuition is done BID19. Many approaches to network architecture search exist, including weight sharing BID53 and reinforcement learning BID70. Another potential solution explored in this work (and others) is to use population-based stochastic optimization (SO) methods, also known as metaheuristics BID44. In SO, several candidate solutions are evaluated and varied over several iterations, and heuristics are used to select and update the candidate networks until the population produces a desirable architecture. The general approach has been studied at least since the late 80s in various forms BID45 BID69 BID60 for NN design, with several recent applications BID55 BID28 BID9 BID54.In practice, the adequacy of the architecture is often dependent on conflicting objectives. For example, interpretability may be a central concern, because many researchers in the scientific community rely on ML models not only to provide predictions that match data from various processes, but to provide insight into the nature of the processes themselves. Approaches to interpretability can be roughly grouped into semantic and syntactic approaches. Semantic approaches encompass methods that attempt to elucidate the behavior of a model under various input conditions as a way of explanation (e.g. BID56). Syntactic methods instead focus on the development of concise models that offer insight by virtue of their simplicity, in a similar vein to models built from first-principles (e.g. BID63 BID57). Akin to the latter group, our goal is to discover the simplest description of a process whose predictions generalize as well as possible. Good representations should also disentangle the factors of variation BID3 in the data, in order to ease model interpretation. Disentanglement implies functional modularity; i.e., sub-components of the network should encapsulate behaviors that model a sub-process of the task. In this sense, stochastic methods such as evolutionary computation (EC) appear well-motivated, as they are premised on the identification and propagation of building blocks of solutions BID23. Experiments with EC applied to networks suggest it pressures networks to be modular BID24 BID29. Although the identification functional building blocks of solutions sounds ideal, we have no way of knowing a priori whether a given problem will admit the identification of building blocks of solutions via heuristic search BID49. Our goal in this paper is thus to empirically assess the performance of several SO approaches in a system designed to produce intelligible representations from NN building blocks for regression. In Section 2, we introduce a new method for optimizing representations that we call the feature engineering automation tool (FEAT) 1. The purpose of this method is to optimize an archive of representations that characterize the trade-off between conciseness and accuracy among representations. Algorithmically, two aspects of the method distinguish FEAT from previous work. First, it represents the internal structure of each NN as a set of syntax trees, with the goal of improving the transparency of the ant representations. Second, it uses weights learned via gradient descent to provide feedback to the variation process at a more granular level. We compare several multi-objective variants of this approach using EC and non-EC methods with different sets of objectives. We discuss related work in more detail in Section 3. In section 4 and 5, we describe and conduct an experiment that benchmarks FEAT against state-of-the-art ML methods on 100 open-source regression problems. Future work based on this analysis is discussed in Section 6, and additional detailed are provided in the Appendix. We are interested in the task of regression, for which the goal is to build a predictive modelŷ(x) using DISPLAYFORM0. The regression modelŷ(x) associates the inputs x ∈ R d with a real-valued output y ∈ R. The goal of feature engineering / representation learning is to find a new representation of x via a m-dimensional feature mapping φ(x): R d → R m, such that a model y(φ(x)) outperforms the modelŷ(x). We will assume that each predictor in x is scaled to zero mean, unit-variance. When applying a NN to a traditional ML task like regression or classification, a fixed NN architecture φ(x, θ), parameterized by θ, is chosen and used to fit a model DISPLAYFORM1 T is a NN representation with m nodes in the final hidden layer and a linear output layer with estimated coefficientsβ = [β 1 . . .β m]T. Typically the problem is then cast as a parameter optimization problem that minimizes a loss function via gradient descent. In order to tune the structure of the representation, we instead wish to solve the joint optimization problem DISPLAYFORM2 whereφ(x,θ) is chosen to minimize a cost function L, with global optimum φ * (x, θ *). (L may depend on θ and β in the case of regularization.) S is the space of possible representations realizable by the search procedure, and φ * is the true structure of the process underlying the data. The assumption of SO approaches such as evolutionary computation (EC) and simulated annealing (SA) BID11. FEAT uses a typical µ + λ evolutionary updating scheme, where µ = λ = P. The method optimizes a population of potential representations, N = {n 1 . . . n P}, where n is an "individual" in the population, iterating through these steps:1. Fit a linear modelŷ = x Tβ. Create an initial population N consisting of this initial representation, φ = x, along with P − 1 randomly generated representations that sample x proportionally toβ. (a) Select parents P ⊆ N using a selection algorithm. (b) Apply variation operators to parents to generate P offspring O; N = N ∪ O (c) Reduce N to P individuals using a survival algorithm.3. Select and return n ∈ N with the lowest error on a hold-out validation set. Individuals are evaluated using an initial forward pass, after which each representation is used to fit a linear model (Eqn. 1) using ridge regression BID22. The weights of the differentiable features in the representation are then updated using stochastic gradient descent. The salient aspect of the proposed method is its use of syntax trees to represent the internal architecture of the network, as shown in FIG0. FEAT constructs these trees from elementary boolean-and continuousvalued functions and literals (see TAB0). This scheme is inspired by symbolic regression (SR) BID34. In contrast to typical SR, each individual n is a set of such trees, the output of which is interpreted as a candidate representation, i.e. φ(x) = [φ 1 . . . φ m] for an individual with m trees. The second difference from traditional SR is that the weights of differentiable nodes are encoded in the edges of the graph, rather than as independent nodes. We include instructions typically used as activation functions used in NN, e.g. tanh, sigmoid, logit and relu nodes, elementary arithmetic and boolean operators. Although a fully connected feedforward NN could be represented by this construction, representations in FEAT are biased to be thinly connected. Our hypothesis is that by doing so, we will improve the representation's legibility without sacrificing its capacity for modelling nonlinear relationships. During variation, the representations are perturbed using a set of mutation and crossover methods. FEAT chooses among 6 variation operators that are as follows. Point mutation changes a node type to a random one with matching output type and arity. Insert mutation replaces a node with a randomly generated depth 1 subtree. Delete mutation removes a feature or replaces a sub-program with an input node, with equal probability. Insert/Delete dimension adds or removes a new feature. Sub-tree crossover replaces a sub-tree from one parent with the sub-tree of another parent. Dimension crossover swaps two features between parents. The exact probabilities of each variation operator will affect the performance of the algorithm, and others have proposed methods for adjusting these probabilities, e.g. BID26. For the purposes of our study, we use each operator with uniform probability. Feedback The use of an ML model to assess the fitness of each representation can be used to provide information about the elements of the representation that should be changed. In particular, we assume that programs in the representation with small coefficients are the best candidates for mutation and crossover. With this in mind, let n be an m-dimensional candidate representation with associated coefficients DISPLAYFORM0 The probability of mutation for tree i in n is denoted P M i (n), and defined as follows: DISPLAYFORM1 The normalized coefficient magnitudesβ ∈ are used to define softmax-normalized probabilities, s in Eqn. 3. The smaller the coefficient, the higher the probability of mutation. The parameter f is used to control the amount of feedback used to weight the probabilities; 1 m in this case represents uniform probability. Among nodes in tree m, mutation occurs with uniform probability. This weighting could be extended for differentiable nodes by weighting the within-tree probabilities by the magnitude of the weights associated with each node. However we expect this would yield diminishing returns. The selection step selects among P parents those representations that will be used to generate offspring. Following variation, the population consists of 2P representations of parents and offspring. The survival step is used to reduce the population back to size P, at which point the generation is finished. In our initial study, we empirically compared five algorithms for selection and survival: 1) -lexicase selection (Lex) BID39, 2) non-dominated sorting genetic algorithm (NSGA2) BID11, 3) a novel hybrid algorithm using Lex for selection and NSGA2 for survival, 4) simulated annealing BID31, and 5) random search. These comparisons are described in Appendix Section A.2. We found that the hybrid algorithm performed the best; it is described below. Parents are selected using Lex. Lex was proposed for regression problems BID39 BID40 as an adaption of lexicase selection BID58 for continuous domains. Under -lexicase selection, parents are chosen by filtering the population according to randomized orderings of training samples with the threshold defined relative to the sample loss among the selection pool. This filtering strategy scales probability of selection for an individual based on the difficulty of the training cases the individual performs well on. Lex has shown strong performance among SR methods in recent tests, motivating our interest in studying it BID50. The survival step for Lex just preserves offspring plus the best individual in the population. Survival is conducted using the survival sub-routine of NSGA2, a popular strategy for multi-objective optimization BID11. NSGA2 applies preference for survival using Pareto dominance relations. An individual (n i) is said to dominate another (n j) if, for all objectives, n i performs at least as well as n j, and for at least one objective, n i strictly outperforms n j. The Pareto front is the set of individuals in N that are non-dominated in the population and thus represent optimal trade-offs between objectives found during search. Individuals are assigned a Pareto ranking that specifies the number of individuals that dominate them, thereby determining their proximity to the front. The survival step of NSGA2 begins by sorting the population according to their Pareto front ranking and choosing the lowest ranked individuals for survival. To break rank ties, NSGA2 assigns each individual a crowding distance measure, which quantifies an individual's distance to its two adjacent neighbors in objective space. If a rank level does not completely fit in the survivor pool, individuals of that rank are sorted by highest crowding distance and added in order until P individuals are chosen. We consider three objectives in our study corresponding to three goals: first, to reduce model error; second, to minimize complexity of the representation; and third, to minimize the entanglement of the representation. We test the third objective using two different metrics: the correlation of the transformation matrix φ(x) and its condition number. These metrics are defined below. The first objective always corresponds to the mean squared loss function for individual n, and the second corresponds to the complexity of the representation. There are many ways to define complexity of an expression; one could simply look at the number of operations in a representation, or look at the behavioral complexity of the representation using a polynomial order BID66 ). The one we use, which is similar to that used by BID33, is to assign a complexity weight to each operator (see TAB0), with higher weights assigned to operators considered more complex. If the weight of operator o is c o, then the complexity of an expression tree beginning at node o is defined recursively as DISPLAYFORM0 where o has k arguments, and C(a) is the complexity of argument a. The complexity of a representation is then defined as the sum of the complexities of its output nodes. The goal of defining complexity in such a way is to discourage deep sub-expressions within complex nodes, which are often hard to interpret. It's important to note that the choice of operator weights is bound to be subjective, since we lack an objective notion of interpretability. For this reason, although we use Eqn. 4 to drive search, our experimental comparisons with other algorithms rely on the node counts of the final models for benchmarking interpretability of different methods. Disentanglement is a term used to describe the notion of a representation's ability to separate factors of variation in the underlying process BID3. Although a thorough review is beyond the scope of this section, there is a growing body of literature addressing disentanglement, primarily with unsupervised learning and/or image analysis BID46 BID67 BID21 BID17 BID20 BID36. There are various ways to quantify disentanglement. For instance, Brahma et al. FORMULA2 proposed measuring disentanglement as the difference between geodesic and Euclidean distances among points on a manifold (i.e. training instances). If the latent structure is known, the information-theoretic metrics proposed by BID13 may be used. In the case of regression, a disentangled representation ideally contains a minimal set of features, each corresponding to a separate latent factor of variation, and each orthogonal to each other. In this regard, we attempt to minimize the collinearity between features in φ as a way to promote disentanglement. We tested two measurements of collinearity (a.k.a. multicollinearity) in the derived feature space. The first is the average squared Pearson's correlation among features of φ, i.e., DISPLAYFORM1 Eqn. 5 is relatively inexpensive to compute but only captures bivariate correlations in φ. As a we also test the condition number (CN). Consider the N × m representation matrix Φ. The CN of Φ is defined as DISPLAYFORM2 where µ max and µ min are the largest and smallest singular values of Φ. Unlike Corr, CN can capture higher-order dependencies in the representation. CN is also related directly to the sensitivity of Φ to perturbations in the training data BID2 BID8, and thus captures a notion of network invariance explored in previous work by BID18. We consider another common measure of multicollinearity, the variance inflation factor , to be too expensive for our purposes. The idea to evolve NN architectures is well established in literature, and is known as neuroevolution. Popular methods of neuroevolution include neuroevolution of augmenting topologies (NEAT BID60 and Hyper-NEAT BID61, and compositional pattern producing networks BID59 . The aforementioned approaches eschew the parameter learning step common in other NN paradigms, although others have developed integrations BID14 . In addition, they have been developed predominantly for other task domains such as robotics and control BID16, image classification BID55 BID54, and reinforcement learning BID25 BID9 . Reviews of these methods are available BID15 BID62 .Most neuroevolution strategies do not have interpretability as a core focus, and thus do not attempt to use multi-objective methods to update the networks. An exception is the work of BID68, in which a template NN was optimized using a multi-objective EC method with size as an objective. In this case, the goal was to reduce computational complexity in face detection. Neuroevolution is a part of a broader research field of neural architecture search (NAS) BID70 BID42 BID43. NAS methods vary in approach, including for example parameter sharing BID53, sequential model-based optimization BID43, reinforcement learning BID70, and greedy heuristic strategies BID10.FEAT is also related to SR approaches to feature engineering BID35 BID0 BID41 BID47 ) that use EC to search for possible representations and couple with an ML model to handle the parametrization of the representations. SR methods have been successful in developing intelligible models of physical systems BID57 BID38. FEAT differs from these methods in the following ways. A key challenge in SR is understanding functional modularity within representations/programs that can be exploited for search. FEAT is designed with the insight that ML weights can be leveraged during variation to promote functional building blocks, an exploit not used in previous methods. Second, FEAT uses multiple type representations, and thus can learn continuous and rule-based features within a single representation, unlike previous methods. This is made possible using a stack-based encoding with strongly-typed operators. Finally, FEAT incorporates two elements of NN learning to improve its representational capacity: activation functions commonly used in NN and edge-based encoding of weights. Traditionally, SR operates with standard mathematical operators, and treats constants as leaves in the expression trees rather than edge weights. An exception is MRGP BID0, which encodes weights at each node but updates them via Lasso instead of using gradient descent with backpropagation. SR methods have also been paired with various parameter learning strategies, including those based on backpropagation BID64 BID32 BID27. It should be noted that non-stochastic methods for SR exist, such as mixed integer non-linear programming, which has been demonstrated for small search spaces BID1. Our goals with the experiment are to 1) robustly compare FEAT to state-of-the-art regression methods, including hyperparameter optimization of feedforward NNs; 2) characterize the complexity of the models; and 3) assess whether disentanglement objectives lead to less correlated representations. For the regression datasets, we use 100 real-world and simulated datasets available from OpenML (BID65 . The datasets are characterized in terms of number of features and sample sizes in FIG3 . We use the standardized versions of the datasets available in the Penn Machine Learning Benchmark repository BID48 . We compare the FEAT configurations to multi-layer perceptron (MLP), random forest (RF) regression, kernel ridge (KernelRidge) regression, and elastic net (ElasticNet) regression, using implementations from scikit-learn BID52. In addition, we compare to XGBoost (XGB), a gradient boosting method that has performed well in recent competitions BID6. Code to reproduce these experiments is available online. Each method's hyperparameters are tuned according to TAB1 in Appendix A.1. For FEAT, we limit optimization to 200 iterations or 60 minutes, whichever comes first. We also stop sooner if the median validation fitness stops improving. For each method, we use grid search to tune the hyperparameters with 10-fold cross validation (CV). We use the R 2 CV score for assessing performance. In our we report the CV scores for each method using its best hyperparameters. The algorithms are ranked on each dataset using their median CV score over 5 randomized shuffles of the dataset. For comparing complexity, we count the number of nodes in the final model produced by each method for each trial on each dataset. To quantify the "entanglement" of the feature spaces, we report Eqn. 5 in the raw data and in the final hidden layer of FEAT and MLP models. We also test two additional versions of Feat, denoted FeatCorr and FeatCN, that include a third objective corresponding to Eqn. 5 and 6, respectively. Finally, we examine the FEAT in detail for one of the benchmark datasets. For this dataset we plot the final population of models, illustrate model selection and compare the ant features to from linear and ensemble tree-based . This gives practical insight into the method and provides a sense of the intelligibility of an example representation. The score statistics for each method are shown in Fig. 3 across all datasets. Full statistical comparisons are reported in Appendix A.3. Over all, FEAT and XGBoost produce the best predictive performance across datasets without significant differences between the two (p=1.0). FEAT significantly outperforms MLP, RF, KernelRidge and ElasticNet (p ≤1.18e-4), as does XGBoost (p ≤1.6e-3).As measured by the number of nodes in the final solutions, the models produced by FEAT are significantly less complex than XGBoost, RF, and MLP, as shown in FIG0 ). FEAT's final models tend to be within 1 order of magnitude of the linear models (ElasticNet), and 2-4 orders of magnitude smaller than the other non-linear methods. A comparison of wall-clock time is given in FIG5 in the appendix. FEAT and MLP take approximately the same time to run, followed by XGBoost, RF, KernelRidge, and ElasticNet, in that order. FIG4 shows the average pairwise correlations of the representations produced by Feat variants and MLP in comparison to the correlation structure of the original data. In general, MLP and FEAT tend to produce correlated feature spaces, and Feat's representations tend to contain more bivariate correlations than MLP. Furthermore, the suggest that explicitly minimizing collinearity DISPLAYFORM0 64.8 Figure 6: (Left) Representation archive for the visualizing galaxies dataset. (Right) Selected model and its weights. Internal weights omitted.(FeatCorr and FeatCN) tends to produce representations that exhibit equivalent or higher levels of correlation. This conflicts with our hypothesis, and is discussed more in Section 6. This paper proposes a feature engineering archive tool that optimizes neural network architectures by representing them as syntax trees. FEAT uses model weights as feedback to guide network variation in an EC optimization algorithm. We conduct a thorough analysis of this method applied to the task of regression in comparison to state-of-the-art methods. The suggest that FEAT achieves state-of-the-art performance on regression tasks while producing representations that are significantly less complex than those ing from similarly performing methods. This improvement comes at an additional computational cost, limited in this study to 60 minutes per training instance. We expect this limitation to be reasonable for many applications where intelligibility is the prime motivation. Future work should consider the issue of representation disentanglement in more depth. Our attempts to include additional search objectives that explicitly minimize multicollinearity were not successful. Although more analysis is needed to confirm this, we suspect that the model selection procedure (Section 2.1, step 3) permits highly collinear representations to be chosen. This is because multicollinearity primarily affects the standard errors ofβ BID2, and is not necessarily detrimental to validation error. Therefore it could be incorrect to expect the model selection procedure to effectively choose more disentangled representations. Besides improving the model selection procedure, it may be fruitful to pressure disentanglement at other stages of the search process. For example, the variation process could prune highly correlated features, or the disentanglement metric could be combined with error into a single loss function with a tunable parameter. We hope to pursue these ideas in future studies. This work was supported by NIH grants AI116794 and LM012601.A APPENDIX A.1 ADDITIONAL EXPERIMENT INFORMATION TAB1 details the hyperparameters for each method used in the experimental described in Sections 4 and 5.Runs are conducted in a heterogenous computing environment, with one core assigned to each CV training per dataset. As such, wall-clock times are a flawed measure of computational complexity. With this caveat in mind, we report the wall-clock run times for each method in FIG5. The Feat variants are terminated at 200 generations or 60 minutes, which explains their uniformity. Note that methods are unable to take advantage of parallelization in this experiment. Our initial analysis sought to determine how different SO approaches performed within this framework. We tested five methods: 1) NSGA2, 2) Lex, 3) LexNSGA2, 4) Simulated annealing, and 5) random search. The simulated annealing and random search approaches are described below. Simulated annealing Simulated annealing (SimAnn) is a non-evolutionary technique that instead models the optimization process on the metallurgical process of annealing. In our implementation, offspring compete with their parents; in the case of multiple parents, offspring compete with the program with which they share more nodes. The probability of an offspring replacing its parent in the population is given by the equation DISPLAYFORM0 The probability of offspring replacing its parent is a function of its fitness, F, in our case the mean squared loss of the candidate model. In Eqn. 7, t is a scheduling parameter that controls the rate of "cooling", i.e. the rate at which steps in the search space that are worse are tolerated by the update rule. In accordance with BID31, we use an exponential schedule for t, defined as t g = (0.9) g t 0, where g is the current generation and t0 is the starting temperature. t0 is set to 10 in our experiments. Random search We compare the selection and survival methods to random search, in which no assumptions are made about the structure of the search space. To conduct random search, we randomly sample S using the initialization procedure. Since FEAT begins with a linear model of the process, random search will produce a representation at least as good as this initial model on the internal validation set. A note on archiving When FEAT is used without a complexity-aware survival method (i.e., with Lex, SimAnn, Random), a separate population is maintained that acts as an archive. The archive maintains a Pareto front according to minimum loss and complexity (Eqn 4). At the end of optimization, the archive is tested on a small hold-out validation set. The individual with the lowest validation loss is the final selected model. Maintaining this archive helps protect against overfitting ing from overly complex / high capacity representations, and also can be interpreted directly to help understand the process being modelled. We benchmarked these approaches in a separate experiment on 88 datasets from PMLB BID48. The are shown in FIG0. FIG6, we see that LexNSGA2 achieves the best average R 2 value while producing small solutions in comparison to Lex. NSGA2, SimAnneal, and Random search all produce less accurate models. The runtime comparisons of the methods in FIG0 show that they are mostly within an order of magnitude, with NSGA2 being the fastest (due to its maintenance of small representations) and Random search being the slowest, suggesting that it maintains large representations during search. The computational behavior of Random search suggests the variation operators tend to increase the average size of solutions over many iterations. We perform pairwise comparisons of methods according to the procedure recommended by BID12 for comparing multiple estimators. In TAB3, the CV R 2 rankings are compared. In 1.00e+00 KernelRidge 1.16e-09* 1.18e-04* 4.37e-03* 1.14e-03* MLP 5.24e-09* 3.80e-04* 2.08e-02* 1.28e-03* 1.00e+00 RF 1.08e-09* 2.09e-07* 2.19e-05* 1.30e-06* 1.00e+00 1.00e+00 XGB 1.47e-13* 1.00e+001.00e+00 1.00e+00 3.41e-04* 1.60e-03* 8.49e-13* 1.58e-07* MLP 6.24e-18* 4.26e-17* 3.09e-17* 3.98e-17* RF 9.28e-20* 2.05e-17* 5.61e-18* 2.37e-17* 3.54e-17* XGB 9.14e-18* 4.05e-17* 2.46e-17* 3.71e-17* 1.00e+00 3.94e-18*' == 'Representing the network architecture as a set of syntax trees and optimizing their structure leads to accurate and concise regression models. ' != 'We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.'

Question: In this paper, we present a technique for generating artificial datasets that retain statistical properties of the real data while providing differential privacy guarantees with respect to this data. We include a Gaussian noise layer in the discriminator of a generative adversarial network to make the output and the gradients differentially private with respect to the training data, and then use the generator component to synthesise privacy-preserving artificial dataset. Our experiments show that under a reasonably small privacy budget we are able to generate data of high quality and successfully train machine learning models on this artificial data. Following recent advancements in deep learning BID28 BID12 BID30, more and more people and companies are interested in putting their data in use as they see that machine learning is able to generate a wide range of benefits, including financial, social, medical, security, and so on. At the same time, however, such models are often able to capture a fine level of detail in training data potentially compromising privacy of individuals who's features sharply differ from others. This problem is partially mitigated by the use of regularisation techniques that "smooth out" outstanding details and avoid overfitting, but it does not give any theoretical privacy guarantees. Recent research by BID8 suggests that even without access to internal model parameters, by using hill climbing on output probabilities of a neural network, it is possible to recover (up to a certain degree) individual faces from a training set. The latter is especially disturbing knowing that deep learning models are becoming an integral part of our lives, making its way to phones, smart watches, cars, and appliances. And since these models are often trained on customers data, such training set recovery techniques will endanger privacy even without access to the manufacturer's servers where these models are being trained. In order to protect privacy while still benefiting from the use of statistics and machine learning, a number of techniques for data anonymisation has been developed over the years, including kanonymity BID29, l-diversity BID18, t-closeness BID17, and differential privacy BID2 BID3 BID7. The latter has been recognised as a strong standard and is widely accepted by the research community. We study the task of publishing datasets in a differentially private manner. In particular, we are interested in solving two problems. First, we want to be able to benefit from the use of machine learning by third parties while protecting sensitive information of individuals in our dataset. Second, we want to be sure that even if adversaries get access to the third-party model trained on our data, they would not be able to recover private information. An additional challenge is to be able to publish an entire dataset, as opposed to being required to use a query interface like in a typical differentially private framework. In this paper, we propose a simple solution to this problem. The main idea of our approach is to use generative adversarial networks (GANs) introduced in BID9, trained with addition of Gaussian noise in the embedding space, to create artificial datasets that follow the same distribution as the real data while providing differential privacy guarantees. This method has a number of advantages over the methods proposed earlier. First of all, this solution is simple to implement, e.g. it does not require training ensembles of models on disjoint data. Second, it can be done on a user side, and not on the side of the machine learning service provider, which eliminates the necessity of trusting this service provider or implementing privacy-preserving models locally. Third, similarly to, privacy cannot be compromised even if the entire trained model is accessible to an adversary. Our contributions in this paper are the following:• we propose a novel mechanism for non-interactive differentially private data release, and to the best of our knowledge this is the first practical solution for complex real-world data; • we introduce a new technique of preserving privacy in neural networks via adding noise in the forward pass during training; • we show that this technique guarantees differential privacy for both the outputs and the learned weights of the network; • we demonstrate that we are able to achieve high accuracy in learning tasks while maintaining a reasonable (single-digit) privacy budget. The remainder of the paper is structured as follows. In Section 2, we give an overview of related work. Section 3 contains necessary on differential privacy and generative adversarial networks. In Section 4, we describe our approach and provide its theoretical analysis and some practical aspects. Experimental and implementation details are presented in Section 5, and Section 6 concludes the paper. The theorem proofs and additional details can be found in the Appendix. Given the level of attention to deep learning and the rising importance of privacy, it is unsurprising that there has been a significant increase in the number of publications on the topic of privacypreserving deep learning (and machine learning in general) in recent years. One take on the problem is to distribute training and use disjoint sets of training data. An example of such approach is the paper of BID27, where they propose to train in a distributed manner by communicating sanitised updates from participants to a central authority. Such a method, however, yields high privacy losses as pointed out by and BID23. An alternative technique, also using disjoint training sets, suggested by BID23, applies an ensemble of independently trained teacher models and semi-supervised knowledge transfer to a student model to achieve almost state-of-the-art (non-private) accuracy on MNIST BID16 and SVHN BID20 with single-digit differential privacy bounds. This work was based on a paper by BID11 and extends their method to generic learning models with any type of loss functions or optimisation algorithms. To the best of our knowledge, this is the most accurate privacy-preserving learning to date, although one has to make sure that all the teaching ensemble and the aggregator are inaccessible to an adversary and the model is queried for teachers' votes only a small number of times. A somewhat different approach is taken in. They suggest using differentially private stochastic gradient descent (for brevity, we will refer to it as DP-SGD in the remainder of the paper) to train deep learning models in a private manner. This approach allows to achieve high accuracy while maintaining low differential privacy bounds, and does not require distributed training. As stated above, our goal is to enable data usage by third party machine learning service providers to benefit from their expertise. All of the aforementioned methods, however, require every provider of such service to comply with the chosen privacy-preserving procedure which is not realistic. An alternative solution to this problem is to focus on sanitising data and making sure that training machine learning models on it would not compromise privacy. This direction is taken, for example, by BID1. The authors use a graphical probabilistic model to learn an underlying data distribution and transform real data points (seeds) into synthetic data points. Synthetic data is then filtered by a privacy test based on a plausible deniability criterion, which can be equivalent to differential privacy under certain conditions. Our approach, on the other hand, is to generate private data without requiring any real seeds. Thus, there is no need for privacy tests at the release stage, and the only requirement is that the generative model is privacy-preserving. By using GANs BID9 we ensure that our method is scalable and applicable to complex real-world data. This section gives a short introduction to GANs and differential privacy. Another important notion is the moments accountant method used to compute actual privacy bounds during training. However, since it is not essential for understanding the paper, we defer its description to the Appendix. In recent years, generative adversarial networks BID9 BID26 and its extensions, such as DCGAN BID24 and EBGAN BID31, have received great attention and pushed the boundaries for deep generative models along with variational autoencoders (VAEs) BID15 BID25 BID10 and recursive neural networks (e.g. PixelRNN by Oord et al. FORMULA1). The most successful application for such generative models so far has been realistic image generation, perhaps due to abundance of training data and inherent geometric structure. In our work, we decided to choose GANs for several reasons. Firstly, GANs have shown very good in practice, generating sharper images compared to other generative models. Secondly, the forward pass for generating data is much faster than that of, for instance, RNNs. Thirdly, the generator part of the model, the one we eventually interested in, does not interact with real training data at any point in the learning process, only getting gradients from the discriminator. In short, GANs can be described as follows. The model consists of two separate components: the generator G(z) and the discriminator D(x). The generator's goal is to produce realistic samples of data based on a random variable z ∼ p z (z), while the discriminator is tasked with distinguishing real data samples x ∼ p data (x) from generated samplesx ∼ p g (x). These two models are trained in an adversarial fashion, essentially playing a two-player game, with the goal to converge to the Nash equilibrium. Since training GANs in practice can be challenging, there is a number of commonly used tricks to improve convergence, such as using the Adam optimisation method BID14, feature matching, batch normalisation, and one-sided label smoothing BID26. We also observe improvements with adding labels to the discriminator BID21 and unrolling discriminator updates BID19. The notion of differential privacy has been introduced and extended in a series of papers by Dwork et al. BID2 BID3 BID7, and is regarded as a strong privacy standard. It is defined for two adjacent datasets that differ by a single element: DISPLAYFORM0 DISPLAYFORM1 Among the mechanisms to achieve differential privacy, two of the most widely used are Laplacian and Gaussian noise mechanisms. We are primarily interested in the latter, because of the improved privacy bounds analysis provided by the moments accountant method described in the Appendix. The Gaussian noise mechanism is defined as follows: DISPLAYFORM2 where s f is the sensitivity of f (i.e. DISPLAYFORM3 is the Gaussian distribution with the mean 0 and the standard deviation s f σ. Sensitive data X is fed into a discriminator D with a privacypreserving layer (dashed line). This discriminator is used to train a differentially private generator G to produce a private artificial dataset X. In this section, we describe our solution and provide a theoretical proof of privacy guarantees, as well as discuss limitations of the method. Let us begin with the formal problem statement. Problem Statement. Given the dataset X ∼ p data (x), generate an artificial dataset X = M(X) using the privacy mechanism M: X → X, such that 1. it follows the same data distribution: DISPLAYFORM0 2. it provides differential privacy guarantees: DISPLAYFORM1 ε Pr [M(X) ∈ S] + δ for any adjacent datasets X, X, and for any S ⊆ X.Here X = {X | X ∼ p data (x)} is the space of all datasets formed by points drawn from the same distribution p data (x).In most real-world problems, the true data distribution p data (x) is unknown and needs to be estimated empirically. Since we are primarily interested in data synthesis, we will turn to generative models, and in particular we are going to use GANs as the mechanism to estimate p data (x) and draw samples from it. If trained properly, GAN will provide a solution to the sub-problem.Despite the fact that the generator does not have access to the real data X in the training process, one cannot guarantee differential privacy because of the information passed through with the gradients from the discriminator. A simple high level example will illustrate such breach of privacy. Let the datasets X, X contain small real numbers. The only difference between these two datasets is the number x ∈ X, which happens to be extremely large. Since the gradients of the model depend on x, one of the updates of the discriminator trained on X may be very different from the rest, and this difference will the be propagated to the generator breaking privacy in general case. In order to maintain differential privacy guarantees, we propose the following solution. Proposition. Introduce a Gaussian noise layer in the discriminator network of GAN, so that its output, and therefore the weights of the trained generator, are differentially private with respect to the input data X. Use this generator to create a publishable differentially private dataset. The components of our solution are depicted in FIG1 To validate the proposed solution, we first analyse it theoretically and show that the addition of a Gaussian noise layer in the discriminator network yields differential privacy in the generator. We will take the following steps to do that:1. analyse privacy of the output of the noise layer w.r.t. the inputs X and X; 2. determine privacy bounds on the output of the whole network; 3. show that the same bounds hold for gradient updates. Let us start by describing the setting and notation used in the remainder of the section. We are given two adjacent datasets (X, y) and (X, y) and a deterministic feed-forward neural network N with a Gaussian noise layer π. We denote the inputs of the layer π as x π and x π, and the outputs of the final layer of the networkŷ = N (X) andŷ = N (X) correspondingly. To ensure (ε, δ)-differential privacy of π, the standard deviation of the noise has to be at least σ = C 2 log(1.25/δ)/ε, where C is the sensitivity of the preceding layer's output x π. Lemma 1. If the output of the noise layer π(x π) is (ε, δ)-differentially private w.r.t. x π and the network layers before π preserve adjacency of X and X, then π(X) is also (ε, δ)-differentially private w.r.t. X.The proof of this lemma and the following Theorems 1 and 2 can be found in the appendix. Using Lemma 1, we are able demonstrate that the outputs of a feed-forward neural network with a Gaussian noise layer are differentially private with respect to the input data, which is expressed in the following theorem. Theorem 1. (Forward pass) The outputŷ of a deterministic feed-forward neural network N with (ε, δ)-differentially private layer π, is also (ε, δ)-differentially private with respect to X. Now, given that the forward pass is differentially private, we can formulate the main theoretical of the paper: differential privacy of the gradients, and thus, the weights of the network N. Theorem 2. (Backward pass) Given a feed-forward neural network N with (ε, δ)-differentially private outputsŷ, weight updates ω DISPLAYFORM0 X are also (ε, δ)-differentially private with respect to X in each iteration i of gradient descent. Since we are interested in generating data using GANs, we will also need the following corollary to finalise the theoretical foundation for our framework. Corollary 1. (GANs) Given a generative adversarial network consisting of the generator G and the discriminator D with a privacy-preserving layer, gradient updates of G will have the same privacy bounds as gradient updates of D.Proof. This trivially follows from Theorem 2 once we observe that generator updates are a function of discriminator updates. The above analysis is applicable for each individual iteration of the gradient descent, and privacy bounds on the final parameters can be obtained using composition theorems or a more efficient moments accountant method.Note that Theorems 1 and 2 define differential privacy of the neural network with respect to the inputs X only, not taking into account the labels y. In certain cases, when labels of interest are already a public knowledge and do not reveal any information about data, it may be sufficient. However, if labels privacy is required, it is possible to incorporate it in the proposed approach in two ways. A first solution is to modify the learning problem so that labels become a part of data. For example, if one wants to train a face recognition model with privacy-breaking labels (e.g. specific namesJohn, Bob, Julia, etc.), it is possible to add these labels to X, and instead use True and False labels in y, indicating whether the input image and the input name correspond to each other. This way, label privacy will be handled by the same framework. Alternatively, one can use a separate privacy-preserving mechanism to retrieve labels during training. In this case, the eventual privacy w.r.t. the pair (X, y) may be derived from a composition of two mechanisms, which is shown in the theorem below. One possible candidate for such mechanism is the noisy voting scheme as used in BID23. Theorem 3. (Private labels) Given a feed-forward neural network N with (ε 1, δ 1)-differentially private outputsŷ, and the training labels y satisfying (ε 2, δ 2)-differential privacy w.r.t. the true labels y, the gradient updates ω (i) X are (ε 1 + ε 2, δ 1 + δ 2)-differentially private with respect to (X, y) on each iteration i of gradient descent. Proof. There are two privacy mechanisms M 1 and M 2 applied to X and y correspondingly. Observe that M 1 does not have access to y, and thus, y cannot influence the output probabilities of M 1. The same is true for M 2 and X. Consequently, we can assume that both mechanisms are applied to a pair (X, y). This allows us to employ a basic sequential composition theorem for differential privacy BID4 ) to obtain the privacy bounds. While it may appeal to use parallel composition instead of sequential composition to obtain a tighter bound, since X and y appear to be disjoint, it would be incorrect. The reason is that X and y are strongly correlated and breaking privacy of one can reveal the other. Alternatively, one could use advanced composition theorems (see e.g. BID6 ; BID13) to prove tighter privacy bounds, but it is not the goal of our paper. Based on the analysis above, we can do a number of important observations regarding applicability of this technique. First of all, the analysis is performed for feed-forward networks. Other architectures, such as RNNs, LSTMs, or memory networks, require additional investigation. Second, we focused on deterministic networks, meaning that the only two sources of stochasticity are data shuffling and privacypreserving noise layer π. Additional randomness in the network would complicate the proofs by introducing uncertainty in mappings. Third, conditions of Lemma 1 dictate that the network layers prior to π must preserve adjacency of the input. One layer breaking this condition is batch normalisation, because it introduces interdependencies between examples inside a batch, and just one different instance can change an entire batch. Summarising these limitations, the neural network under question must• be a feed-forward network;• not have randomised layers, e.g. dropout;• not have adjacency breaking layers before the privacy layer, e.g. batch normalisation. In the following section, we will touch upon some implications of it that affect practical performance. Note that these restrictions only apply to the network, in which we insert a privacypreserving layer, i.e. only the discriminator in our case. In this section, we provide some implementation details and discuss evaluation obtained on MNIST BID16 and SVHN BID20 ) datasets. We evaluate our solution as follows. First, we train a generative model on original datasets (using only training parts of each) with differential privacy by adding a Gaussian noise layer to the discriminator. We will call this model a teacher, analogously to BID23. Then, we generate an artificial dataset of comparable size using the obtained model. Finally, we train a separate (nonprivate) classifier, which we call a student, on generated data and test it using held-out test sets. The last step is important from two perspectives: we can quantify the quality of generated samples as opposed to visual inspection typically done with GANs, and we can compare test errors to previously reported values. Note that there is no dependencies between the teacher and the student models. Moreover, student models are not constrained to neural networks and can be implemented as any type of machine learning algorithm. We choose two commonly used image classification datasets for our experiments: MNIST and SVHN. MNIST is a handwritten digit recognition dataset consisting of 60'000 training examples Table 1: Accuracy of student models for given privacy bounds for our method and semi-supervised knowledge transfer approach of BID23. In both cases, we restricted our method to have tighter privacy bounds. and 10'000 test examples, each example is a 28x28 size greyscale image. SVHN is also a digit recognition task, with 73'257 images for training and 26'032 for testing. The examples are coloured 32x32 pixel images of house numbers from Google Street View. Implementation was done in Python using Pytorch 1. For generative model, we used a modified version of DCGAN by BID24. More specifically, the discriminator consists of five (four for MNIST) convolutional layers followed by leaky ReLU activations and a linear classifier with sigmoid output. We clip the output of the third convolutional layer (to ensure bounded sensitivity) and add Gaussian noise before passing it to the remaining convolutions with batch normalisation. The generator has two linear layers in front of five deconvolutions with batch normalisation and ReLU activations, ensued by fractional max pooling with tanh activation at the end. Both networks were trained using Adam optimiser BID14 with parameters typical for GAN training: learning rate set to 0.0002, β 1 = 0.5, β 2 = 0.999, and a batch size of 32. Privacy bounds were evaluated using the moments accountant and the privacy amplification theorem, and therefore, are data-dependent and are tighter than using normal composition theorems. The student network is constructed of two convolutional layers with ReLU activations, batch normalisation and max pooling, followed by two fully connected layers with ReLU, and a softmax output layer. Again, training is performed by Adam algorithm. It is worth mentioning that this network does not achieve state-of-the-art performance on the used datasets, but we are primarily interested in evaluating the performance drop compared to a non-private model rather than getting the best test score. Using the experimental setup and implementation described above, we were able to get close to BID23 although not quite matching their accuracy for the same privacy bounds on SVHN. A performance gap is expected due to more generic nature of our method and a simpler privacy-preserving procedure. Overall, we managed to achieve 98.19% accuracy on MNIST and 83.49% accuracy on SVHN while maintaining approximately (3.45, 10 −5) and (8, 10 −6)-differential privacy. These numbers, along with the corresponding of BID23, can be found in Table 1. It is also worth noting that we did not perform rigorous hyper-parameter tuning due to limited computational resources; even better accuracy could be achieved have we had done that. Additionally, we trained a simple logistic regression model on MNIST, and obtained 88.96% accuracy on privately generated data compared to 92.58% on the original data, which confirms that any model can be used as a student. Examples of real and generated privacy-preserving images for MNIST and SVHN data are depicted on FIG2. It can be seen that generated images don't have the same contrast and dynamic range as real examples, which is not a problem in non-private GANs. We attribute it to the lack of batch normalisation in the discriminator. In addition to quantitative analysis of test errors and privacy bounds, we perform visual inspection of generated examples and corresponding nearest neighbours in real data. FIG3 depicts a set of generated private examples and their nearest real counterparts. We observe that while some generated images are very close to real examples they don't match exactly, differing either in shape, colour or surrounding digits. Moreover, a lot of pairs come from entirely different classes. We investigate the problem of non-interactive private data release with differential privacy guarantees. We employ generative adversarial networks to produce artificial privacy-preserving datasets. Contrary to existing privacy protection work in deep learning, this method allows to publish sanitised data and train any non-private models on it. The choice of GANs as a generative model ensures scalability and makes the technique suitable for real-world data with complex structure. Moreover, this method does not require running privacy tests on generated data before releasing it. Additionally, we introduce a novel method for preserving privacy of training data specific to deep neural networks based on adding noise in the embedding space during forward pass. It provides differential privacy guarantees and allows to construct privacy-preserving models in a simple and straightforward fashion, without modifying optimisation algorithms. In our experiments, we show that student models trained on artificial data can achieve high utility on MNIST dataset, while maintaining performance costs of added privacy and flexibility at acceptable levels on a more complicated SVHN data. Adding privacy directly to the trained model still provides better accuracy, and therefore, one of the possible directions for future work is to improve the quality of generated data for given privacy bounds. Extending presented technique and analysis to other types of deep neural networks provides another exciting opportunity for further research. In this appendix, we state again and prove lemmas and theorems from Section 4.1. Lemma 2. If the output of the noise layer π(x π) is (ε, δ)-differentially private w.r.t. x π and the network layers before π preserve adjacency of X and X, then π(X) is also (ε, δ)-differentially private w.r.t. X.Proof. By definition of differential privacy: DISPLAYFORM0 for all adjacent x π and x π.We need to show that the same holds for all adjacent inputs X, X, i.e. DISPLAYFORM1 Observe that we defined our network as deterministic (i.e. not having any randomness apart from initial data shuffling). Therefore, P [X π |X] = δ xπ (X π), where δ x (X) is a Dirac delta function. Conceptually, it means that the entire mass of the distribution of X π is concentrated on the point x π.Using the above observation, DISPLAYFORM2 Remark. Allowing randomised layers in the network would complicate the proof due to marginalisation over all possible outcomes X π corresponding to the input X. Theorem 1. (Forward pass) The outputŷ of a deterministic feed-forward neural network N with (ε, δ)-differentially private layer π, is also (ε, δ)-differentially private with respect to X.Proof. Using the lemma above, we can show that outputs of the layer π are (ε, δ)-differentially private w.r.t. the inputs X, i.e. DISPLAYFORM0 Since we require all the layers of N (except π) to be deterministic, there is a deterministic mapping from the outputs of π toŷ. Let us denote this mapping f (π), and the preimage of a set S under this mapping DISPLAYFORM1 Note that we treat X and X as points in the space of all datasets X, and thus, π and f are not set-valued functions. Also, to avoid confusion, let us restate that f −1 [S] is a preimage of a set S under f, and not a function inverse. Hence, we do not require f to be bijective, or even injective. Using the above, P [ŷ ∈ S] = P [f (π(X)) ∈ S] DISPLAYFORM2 ≤ e ε P [π(X) ∈ f −1 [S]] + δ = e ε P [f (π(X)) ∈ S] + δ = e ε P [ŷ ∈ S] + δ,for any pair of adjacent datasets X and X (differing in one training example), thus, proving the theorem. Theorem 2. (Backward pass) Given a feed-forward neural network N with (ε, δ)-differentially private outputsŷ, weight updates ω Since gradient is a function of network outputs and labels, we have DISPLAYFORM0 Combining the above , DISPLAYFORM1 = e ε P [ω (i) DISPLAYFORM2 for any pair of adjacent datasets X and X, demonstrating that weight updates stay (ε, δ)-differentially private w.r.t to the input. The privacy bound produced by the strong composition theorem is often too loose, and therefore, we exploit the moments accountant technique developed by for analysing their DP-SGD algorithm. To give the main idea of the method, let us start with defining the privacy loss. Definition 2. Let M: D → R be a randomized mechanism and d, d a pair of adjacent databases. Let aux denote an auxiliary input. For an outcome o ∈ R, the privacy loss at o is defined as: The moments accountant is then defined as follows: Definition 3. Again, let M: D → R be a randomized mechanism, d, d a pair of adjacent databases, and aux denote an auxiliary input. The moments accountant is α M (λ) max DISPLAYFORM0 DISPLAYFORM1 where α M (λ; aux, d, d) log E[exp(λC(M, aux, d, d))] is a moment-generating function. In short, the moments accountant method tracks the bounds on the moments of the privacy loss random variable and then uses Markov inequality to obtain the tail bound on this random variable corresponding to the values of ε and δ.

Represent the short summary for finding the corresponding scientific article (long).

Train GANs with differential privacy to generate artificial privacy-preserving datasets.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Paper analyzes the problem of designing adversarial attacks against multiple classifiers, introducing algorithms that are optimal for linear classifiers and which provide state-of-the-art results for deep learning.


Question: Represent the article for finding a single-sentence TLDR

Question: This paper presents two methods to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network. Unlike convolutional studies that visualize image appearances corresponding to the network output or a neural activation from a global perspective, our research aims to clarify how a certain input unit (dimension) collaborates with other units (dimensions) to constitute inference patterns of the neural network and thus contribute to the network output. The analysis of local contextual effects w.r.t. certain input units is of special values in real applications. In particular, we used our methods to explain the gaming strategy of the alphaGo Zero model in experiments, and our method successfully disentangled the rationale of each move during the game. Interpreting the decision-making logic hidden inside neural networks is an emerging research direction in recent years. The visualization of neural networks and the extraction of pixel-level inputoutput correlations are two typical methodologies. However, previous studies usually interpret the knowledge inside a pre-trained neural network from a global perspective. For example, BID17 BID14 BID10 mined input units (dimensions or pixels) that the network output is sensitive to; BID2 visualized receptive fields of filters in intermediate layers; BID33 BID15 BID24 BID5 BID6 BID20 illustrated image appearances that maximized the score of the network output, a filter's response, or a certain activation unit in a feature map. However, instead of visualizing the entire appearance that is responsible for a network output or an activation unit, we are more interested in the following questions.• How does a local input unit contribute to the network output? Here, we can vectorize the input of the network into a high-dimensional vector, and we treat each dimension as a specific "unit" without ambiguity. As we know, a single input unit is usually not informative enough to make independent contributions to the network output. Thus, we need to clarify which other input units the target input unit collaborates with to constitute inference patterns of the neural network, so as to pass information to high layers.• Can we quantitatively measure the significance of above contextual collaborations between the target input unit and its neighboring units?Method: Therefore, given a pre-trained convolutional neural network (CNN), we propose to disentangle contextual effects w.r.t. certain input units. As shown in Fig. 1, we design two methods to interpret contextual collaborations at different scales, which are agnostic to the structure of CNNs. The first method estimates a rough region of contextual collaborations, i.e. clarifying whether the target input unit mainly collaborates with a few neighboring units or most units of the input. This method distills knowledge from the pre-trained network into a mixture of local models (see Fig. 2), where each model encodes contextual collaborations within a specific input region to make predictions. We hope that the knowledge-distillation strategy can help people determine quantitative contributions from different regions. Then, given a model for Extracting fine-grained contextual effects from a student net A lattice within the Go board Figure 1: Explaining the alphaGo model. Given the state of the Go board and the next move, we use the alphaGo model to explain the rationale of the move. We first estimate a rough region of contextual collaborations w.r.t. the current move by distilling knowledge from the value net to student nets that receive different regions of the Go board as inputs. Then, given a student net, we analyze fine-grained contextual collaborations within its region of the Go board. In this figure, we use a board state from a real Go game between humans for clarity.local collaborations, the second method further analyzes the significance of detailed collaborations between each pair of input units, when we use the local model to make predictions on an image. The quantitative analysis of contextual collaborations w.r.t. a local input unit is of special values in some tasks. For example, explaining the alphaGo model BID22 BID7 October 2017) is a typical application. The alphaGo model contains a value network to evaluate the current state of the game-a high output score indicates a high probability of winning. As we know, the contribution of a single move (i.e. placing a new stone on the Go board) to the output score during the game depends on contextual shapes on the Go board. Thus, disentangling explicit contextual collaborations that contribute to the output of the value network is important to understand the logic of each new move hidden in the alphaGo model. More crucially, in this study, we explain the alphaGo Zero model BID7, which extends the scope of interests of this study from diagnosing feature representations of a neural network to a more appealing issue letting self-improving AI teach people new knowledge. The alphaGo Zero model is pre-trained via self-play without receiving any prior knowledge from human experience as supervision. In this way, all extracted contextual collaborations represent the automatically learned intelligence, rather than human knowledge. As demonstrated in well-known Go competitions between the alphaGo and human players (alp, Retrieved 17 March 2016; 2017-05-27), the automatically learned model sometimes made decisions that could not be explained by existing gaming principles. The visualization of contextual collaborations may provide new knowledge beyond people's current understanding of the Go game. Contributions of this paper can be summarized as follows.(i) In this paper, we focus on a new problem, i.e. visualizing local contextual effects in the decisionmaking of a pre-trained neural network w.r.t. a certain input unit.(ii) We propose two new methods to extract contextual effects via diagnosing feature representations and knowledge distillation.(iii) We have combined two proposed methods to explain the alphaGo Zero model, and experimental have demonstrated the effectiveness of our methods. Understanding feature representations inside neural networks is an emerging research direction in recent years. Related studies include 1) the visualization and diagnosis of network features, 2) disentangling or distilling network feature representations into interpretable models, and 3) learning neural networks with disentangled and interpretable features in intermediate layers. Network visualization: Instead of analyzing network features from a global view BID30 BID19 BID16, BID2 BID33 BID15 BID24 BID5 BID32 BID34 showed the appearance that maximized the score of a given unit. BID5 used up-convolutional nets to invert CNN feature maps to their corresponding images. Pattern retrieval: Some studies retrieved certain units from intermediate layers of CNNs that were related to certain semantics, although the relationship between a certain semantics and each neural unit was usually convincing enough. People usually parallel the retrieved units similar to conventional mid-level features BID25 of images. BID37 selected units from feature maps to describe "scenes". BID23 discovered objects from feature maps. Model diagnosis and distillation: Model-diagnosis methods, such as the LIME BID17, the SHAP , influence functions BID11, gradientbased visualization methods BID6 BID20, and BID12 extracted image regions that were responsible for network outputs. BID29 BID36 ) distilled knowledge from a pre-trained neural network into explainable models to interpret the logic of the target network. Such distillation-based network explanation is related to the first method proposed in this paper. However, unlike previous studies distilling knowledge into explicit visual concepts, our using distillation to disentangle local contextual effects has not been explored in previous studies. A new trend is to learn networks with meaningful feature representations in intermediate layers BID9 BID26 BID13 in a weakly-supervised or unsupervised manner. For example, capsule nets BID18 and interpretable RCNN learned interpretable middle-layer features. InfoGAN BID3 and β-VAE BID8 learned meaningful input codes of generative networks. BID35 ) developed a loss to push each middle-layer filter towards the representation of a specific object part during the learning process without given part annotations. All above related studies mainly focused on semantic meanings of a filter, an activation unit, a network output. In contrast, our work first analyzes quantitative contextual effects w.r.t. a specific input unit during the inference process. Clarifying explicit mechanisms of how an input unit contributes to the network output has special values in applications. In the following two subsections, we will introduce two methods that extract contextual collaborations w.r.t. a certain input unit from a CNN at different scales. Then, we will introduce the application that uses the proposed methods to explain the alphaGo Zero model. Since the input feature usually has a huge number of dimensions (units), it is difficult to accurately discover a few input units that collaborate with a target input unit. Therefore, it is important to first approximate the rough region of contextual collaborations before the unit-level analysis of contextual collaborations, i.e. clarifying in which regions contextual collaborations are contained. Given a pre-trained neural network, an input sample, and a target unit of the sample, we propose a method that uses knowledge distillation to determine the region of contextual collaborations w.r.t. the target input unit. Let I ∈ I denote the input feature (e.g. an image or the state in a Go board).Note that input features of most CNNs can be represented as a tensor I ∈ R H×W ×D, where H and W indicate the height of the width of the input, respectively; D is the channel number. We clip different lattices (regions) Λ 1, Λ 2,..., Λ N ∈ Λ from the input tensor, and input units within the i-th lattice are given as I Λi ∈ R h×w×D, h ≤ H, w ≤ W. Different lattices overlap with each other. The core idea is that we use a mixture of models to approximate the function of the given pretrained neural network (namely the teacher net), where each model is a student net and uses input information within a specific lattice I Λi to make predictions. DISPLAYFORM0 Generate weights 2x2 lattices for the first type of student nets 3x3 lattices for the second type of student nets. We only illustrate three of the nine lattices for clarity. Figure 2: Division of lattices for two types of student nets. We distill knowledge from the value net into a mixture of four/nine student nets to approximate decision-making logic of the value net. whereŷ = f (I) and y i = f i (I Λi) denote the output of the pre-trained teacher net f and the output of the i-th student net f i, respectively. α i is a scalar weight, which depends on the input I. Because different lattices within the input are not equally informative w.r.t. the target task, input units within different lattices make different contributions to final network output. More crucially, given different inputs, the importance for the same lattice may also change. For example, as shown in BID20, the head appearance is the dominating feature in the classification of animal categories. Thus, if a lattice corresponds to the head, then this lattice will contribute more than other lattices, thereby having a large weight α i. Therefore, our method estimates a specific weight α i for each input I, i.e. α i is formulated as a function of I (which will be introduced later).Significance of contextual collaborations: Based on the above equation, the significance of contextual collaborations within each lattice Λ i w.r.t. an input unit can be measured as DISPLAYFORM0 Impacts from the first lattice Λ1 DISPLAYFORM1 where we revise the value of the target unit in the input and check the change of network outputs, DISPLAYFORM2 If contextual collaborations w.r.t. the target unit mainly localize within the i-th lattice Λ i, then α i · ∆y i can be expected to contribute the most to the change ofŷ. We conduct two knowledge-distillation processes to learn student nets and a model of determining {α i}, respectively. The first process distills knowledge from the teacher net to each student net f i with parameters θ i based on the distillation loss min θi I∈I y I,i −ŷ I 2, where the subscript I indicates the output for the input I. Considering that Λ i only contains partial information of I, we do not expect y I,i to reconstructŷ I without any errors. Distilling knowledge to weights: Then, the second distillation process estimates a set of weights α = [α I,1, α I,2, . . ., α I,n] for each specific input I. We use the following loss to learn another neural network g with parameters θ g to infer the weight. DISPLAYFORM0 3.2 FINE-GRAINED CONTEXTUAL COLLABORATIONS w.r.t. AN INPUT UNITIn the above subsection, we introduce a method to distill knowledge of contextual collaborations into student nets of different regions. Given a student net, in this subsection, we develop an approach to disentangling from the student net explicit contextual collaborations w.r.t. a specific input unit u, i.e. identifying which input unit v collaborates with u to compute the network output. We can consider a student net as a cascade of functions of N layers, i.e. DISPLAYFORM1, where x (l) denotes the output feature of the l-th layer. In particular, x and x (n) indicate the input and output of the network, respectively. We only focus on a single scalar output of the network (we may handle different output dimensions separately if the network has a high-dimensional output). If the sigmoid/softmax layer is the last layer, we use the score before the softmax/sigmoid operation as x (n) to simplify the analysis. As preliminaries of our algorithm, we extend the technique of BID21 to estimate the quantitative contribution of each neural activation in a feature map to the final prediction. We use C x ∈ R H l ×W l ×D l to denote the contribution distribution of neural activations on the l-th layer x ∈ R H l ×W l ×D l. The score of the i-th element C xi denotes the ratio of the unit x i's score contribution w.r.t. the entire network output score. Because x (n) is the scalar network output, it has a unit contribution C x (n) = 1. Then, we introduce how to back-propagate contributions to feature maps in low layers. The method of contribution propagation is similar to network visualization based on gradient backpropagation BID15 BID32. However, contribution propagation reflects more objective distribution of numerical contributions over {x i}, instead of biasedly boosting compacts of the most important activations. Without loss of generality, in this paragraph, we use o = φ(x) to simplify the notation of the function of a certain layer. If the layer is a conv-layer or a fully-connected layer, then we can represent the convolution operation for computing each elementary activation score o i of o in a vectorized form DISPLAYFORM0 We consider x j w j as the numerical contribution of x j to o i. Thus, we can decompose the entire contribution of o i, C oi, into elementary contributions of x j, i.e. C oi→xj = C oi · xj wj oi+max{−b,0}, which satisfies C oi→xj ∝ x j w j (see the appendix for details). Then, the entire contribution of x j is computed as the sum of elementary contributions from all o i in the above layer, i.e. C xj = i C oi→xj.A cascade of a conv-layer and a batch-normalization layer can be rewritten in the form of a single conv-layer, where normalization parameters are absorbed into the conv-layer 1. For skip connections, a neural unit may receive contributions from different layers, C x DISPLAYFORM1. If the layer is a ReLU layer or a Pooling layer, the contribution propagation has the same formulation as gradient back-propagations of those layers 1. As discussed in BID2, each neural activation o i of a middle-layer feature o can be considered as the detection of a mid-level inference pattern. All input units must collaborate with neighboring units to activate some middle-layer feature units, in order to pass their information to the network output. Therefore, in this research, we develop a method to 1. determine which mid-level patterns (or which neural activations o i) the target unit u constitutes; 2. clarify which input units v help the target u to constitute the mid-level patterns; 3. measure the strength of the collaboration between u and v. Let o bfr and o denote the feature map of a certain conv-layer o = f (x) when the network receives input features with the target unit u being activated and the feature map generated without u being activated, respectively. In this way, we can use |o − o bfr | to represent the absolute effect of u on the feature map o. The overall contribution of the i-th neural unit C oi depends on the activation score o i, C oi ∝ max{o i, 0}, where max{o i, 0} measures the activation strength used for inference. The proportion of the contribution is affected by the target unit u can be roughly formulated asC o. DISPLAYFORM0 where C oi = 0 and thusC oi = 0 if o i ≤ 0, because negative activation scores of a conv-layer cannot pass information through the following ReLU layer (o is not the feature map of the last conv-layer before the network output).In this way,C oi highlights a few mid-level patterns (neural activations) related to the target unit u. C o measures the contribution proportion that is affected by the target unit u. We can useC o to replace C o and use techniques in Section 3.2.1 to propagateC o back to input units DISPLAYFORM1 represents a map of fine-grained contextual collaborations w.r.t. u. Each element in the mapC DISPLAYFORM2 j's collaboration with u. We can understand the proposed method as follows. The relative activation change DISPLAYFORM3 can be used as a weight to evaluate the correlation between u and the i-th activation unit (inference pattern). In this way, we can extract input units that make great influences on u's inference patterns, rather than affect all inference patterns. Note that both u and v may either increase or decrease the value of o i. It means that the contextual unit v may either boost u's effects on the inference pattern, or weaken u's effects. We use the ELF OpenGo BID28 BID30 as the implementation of the alphaGo Zero model. We combine the above two methods to jointly explain each move's logic hidden in the value net of the alphaGo Zero model during the game. As we know, the alphaGo Zero model contains a value net, policy nets, and the module of the Monte-Carlo Tree Search (MCTS). Generally speaking, the superior performance of the alphaGo model greatly relies on the enumeration power of the policy net and the MCTS, but the value net provides the most direct information about how the model evaluates the current state of the game. Therefore, we explain the value net, rather than the policy net or the MCTS. In the ELF OpenGo implementation, the value net is a residual network with 20 residual blocks, each containing two conv-layers. We take the scalar output 2 before the final (sigmoid) layer as the target value to evaluate the current state on the Go board. Given the current move of the game, our goal is to estimate unit-level contextual collaborations w.r.t. the current move. I.e. we aim to analyze which neighboring stones and/or what global shapes help the current move make influences to the game. We distill knowledge from the value net to student networks to approximate contextual collaborations within different regions. Then, we estimate unitlevel contextual collaborations based on the student net. Determining local contextual collaborations: We design two types of student networks, which receive lattices at the scales of 13 × 13 and 10 × 10, respectively. In this way, we can conduct two distillation processes to learn neural networks that encode contextual collaborations at different scales. As shown in Fig. 2, we have four student nets {f i |i = 1, . . ., 4} oriented to 13 × 13 lattices. Except for the output, the four student nets have the same network structure as the value net. The four student nets share parameters in all layers. The input of a student net only has two channels corresponding to maps of white stones and black stones, respectively, on the Go board. We crop four overlapping lattices at the four corners of the Go board for both training and testing. Note that we rotate the board state within each lattice I Λi to make the top-left position corresponds to the corner of the board, before we input I Λi to the student net. The neural network g has the same settings as the value net. g receives a concatenation of [I Λ1, . . ., I Λ4] as the input. g outputs four scalar weights {α i} for the four local student networks {y i}. We learn g via knowledge distillation. Student nets for 10×10 lattices have similar settings as those for 13×13 lattices. We divide the entire Go board into 3 × 3 overlapping 10 × 10 lattices. Nine student nets encode local knowledge from nine local lattices. We learn another neural network g, which uses a concatenation of [I Λ1, . . ., I Λ9] to weight for the nine local lattices. Finally, we select the most relevant 10 × 10 lattice and the most relevant 13 × 13 lattice, via max i s i, for explanation. Estimating unit-level contextual collaborations: In order to obtain fine-grained collaborations, we apply the method in Section 3.2.2 to explain two student nets corresponding to the two selected relevant lattices. We also use our method to explain the value net. We compute a map of contextual collaborations for each neural network and normalize values in the map. We sum up maps of the three networks together to obtain the final map of contextual collaborationsĈ.More specifically, given a neural network, we use the feature of each conv-layer to compute the initialC o in Equation and propagatedC o to obtain a map of collaborationsC x. We sum up maps based on the 1st, 3rd, 5th, and 7th conv-layers to obtain the collaboration map of the network. In experiments, we distilled knowledge of the value network to student nets, and disentangled finegrained contextual collaborations w.r.t. each new move. We compared the extracted contextual collaborations and human explanations for the new move to evaluate the proposed method. In this section, we propose two metrics to evaluate the accuracy of the extracted contextual collaborations w.r.t. the new move. Note that considering the high complexity of the Go game, there is no exact ground-truth explanation for contextual collaborations. Different Go players usually have different analysis of the same board state. More crucially, as shown in competitions between the alphaGo and human players (alp, Retrieved 17 March 2016; 2017-05-27), the knowledge encoded in the alphaGo was sometimes beyond humans' current understanding of the Go game and could not be explained by existing gaming principles. In this study, we compared the similarity between the extracted contextual collaborations and humans' analysis of the new move. The extracted contextual collaborations were just rough explanations from the perspective of the alphaGo. We expected these collaborations to be close to, but not exactly the same as human understanding. More specifically, we invited Go players who had obtained four-dan grading rank to label contextual collaborations. To simplify the metric, Go players were asked to label a relative strength value of the collaboration between each stone and the target move (stone), no matter whether the relationship between the two stones was collaborative or adversarial. Considering the double-blind policy, the paper will introduce the Go players if the paper is accepted. Let Ω be a set of existing stones except for the target stone u on the Go board. p v ≥ 0 denotes the labeled collaboration strength between each stone v ∈ Ω and the target stone u. q v = |Ĉ v | is referred to as the collaboration strength estimated by our method, whereĈ v denotes the final estimated collaboration value on the stone v. We normalized the collaboration strength,p v = p v / v p v, q v = q v / v q v and computed the Jaccard similarity between the distribution of p and the distribution of q as the similarity metric. In addition, considering the great complexity of the Go game, different Go players may annotate different contextual collaborations. Therefore, we also required Go players to provide a subjective rating for the extracted contextual collaborations of each board state, i.e. selecting one of the five ratings: 1-Unacceptable, 2-Problematic, 3-Acceptable, 4-Good, and 5-Perfect. FIG0 shows the significance of the extracted contextual collaborations, as well as possible explanations for contextual collaborations, where the significance of the stone v's contextual collaboration was reported as the absolute collaboration strength q v instead of the original scoreĈ v in experiments. Without loss of generality, let us focus on the winning probability of the black. Considering the complexity of the Go game, there may be two cases of a positive (or negative) value of the collaboration scoreĈ v. The simplest case is that when a white stone had a negative value ofĈ v, it means that the white stone decreased the winning probability of the black. However, sometimes a white stone had a positiveĈ v. It may be because that this white stone did not sufficiently exhibit its power due to its contexts. Since the white and the white usually had a very similar number of stones in the Go board, putting a relatively ineffective white stone in a local region also wasted the opportunity of winning advantages in other regions in the zero-sum game. Similarly, the black stone may also have either a positive or a negative value ofĈ v. The Jaccard similarity between the extracted collaborations and the manually-annotated collaborations was 0.3633. Nevertheless, considering the great diversity of explaining the same game state, The black stone at has a high value, because it collaborates with the new stone to escape from the surrounding of the white. The white stone at has a high value, because it is about to be eaten by the new black stone. The black stone at has a high value, because it collaborates with the new stone to eat the white stone at. Explanations for the estimated collaborations The black stone at has a high value, because it collaborates with the new stone, because it collaborates with the new black stone to get a head out of the white's regime. The two black stone also communicate with black stones on the top. The white stone at has a high value, because future white stones can only be placed to the right to escape from the regime of the new black stone. The black stone at has a high value, because it collaborates with the new black stone to separate white stones into the left and right groups, which increases the probability of attacking white stones on the left in the future. The white stone at has a high value, because the new black stone reduces the white stone's space of "making eyes" in the future. The black stone at has a high value, because the new black stone helps this black stone to get a head out of the white's regime. The white stone at has a high value, because the new black stone limits the potential of the white's future development in its neighboring area. the average rating score that was made by Go players for the extracted collaborations was 3.7 (between 3-Acceptable and 4-Good). Please see the appendix for more . In this paper, we have proposed two typical methods for quantitative analysis of contextual collaborations w.r.t. a certain input unit in the decision-making of a neural network. Extracting fine-grained contextual collaborations to clarify the reason why and how an input unit passes its information to the network output is of significant values in specific applications, but it has not been well explored before, to the best of our knowledge. In particular, we have applied our methods to the alphaGo Zero model, in order to explain the potential logic hidden inside the model that is automatically learned via self-play without human annotations. Experiments have demonstrated the effectiveness of the proposed methods. Note that there is no exact ground-truth for contextual collaborations of the Go game, and how to evaluate the quality of the extracted contextual collaborations is still an open problem. As a pioneering study, we do not require the explanation to be exactly fit human logics, because human logic is usually not the only correct explanations. Instead, we just aim to visualize contextual collaborations without manually pushing visualization towards human-interpretable concepts. This is different from some previous studies of network visualization BID15 BID32 that added losses as the natural image prior, in order to obtain beautiful but biased visualization . In the future, we will continue to cooperate with professional Go players to further refine the algorithm to visualize more accurate knowledge inside the alphaGo Zero model. Let o = ω ⊗ x + β denote the convolutional operation of a conv-layer. We can rewrite the this equation in a vectorized form as DISPLAYFORM0 If the conv-layer is a fully-connected layer, then each element W ij corresponds to an element in ω. Otherwise, W is a sparse matrix, i.e. W ij = 0 if o i and x j are too far way to be covered by the convolutional filter. Thus, we can write o i = j x j w j + b to simplify the notation. Intuitively, we can propagate the contribution of o i to its compositional elements x j based on their numerical scores. Note that we only consider the case of o i > 0, because if o i ≤ 0, o i cannot pass information through the ReLU layer, and we obtain C oi = 0 and thus C oi→xj = 0. In particular, when b ≥ 0, all compositional scores just contribute an activation score o i − b, thereby receiving a total contribution of C oi oi−b oi. When b < 0, we believe the contribution of C oi all comes from elements of {x j}, and each element's contribution is given a C oi · xj wj oi−b. Thus, we get DISPLAYFORM1 When a batch-normalization layer follows a conv-layer, then the function of the two cascaded layers can be written as DISPLAYFORM2 Thus, we can absorb parameters for the batch normalization into the conv-layer, i.e. w j ← For ReLU layers and Pooling layers, the formulation of the contribution propagation is identical to the formulation for the gradient back-propagation, because the gradient back-propagation and the contribution propagation both pass information to neural activations that are used during the forward propagation. Considering the great complexity of the Go game, there do not exist ground-truth annotations for the significance of contextual collaborations. Different Go players may have different understanding of the same Go board state, thereby annotating different heat maps for the significance of contextual collaborations. More crucially, our reflect the logic of the automatically-learned alphaGo Zero model, rather than the logic of humans. Therefore, in addition to manual annotations of collaboration significance, we also require Go players to provide a subjective evaluation for the extracted contextual collaborations. 94.63% Figure 6: We show the significance of contextual collaborations within a local lattice. The score for the i-th lattice is reported as si j sj.

Represent the TLDR for finding the corresponding scientific article

This paper presents methods to disentangle and interpret contextual effects that are encoded in a deep neural network.

Represent the short summary for finding the corresponding scientific article (long)

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.