Question: Represent the article for finding a single-sentence TLDR

Question: We introduce an unsupervised structure learning algorithm for deep, feed-forward, neural networks. We propose a new interpretation for depth and inter-layer connectivity where a hierarchy of independencies in the input distribution is encoded in the network structure. This in structures allowing neurons to connect to neurons in any deeper layer skipping intermediate layers. Moreover, neurons in deeper layers encode low-order (small condition sets) independencies and have a wide scope of the input, whereas neurons in the first layers encode higher-order (larger condition sets) independencies and have a narrower scope. Thus, the depth of the network is automatically determined---equal to the maximal order of independence in the input distribution, which is the recursion-depth of the algorithm. The proposed algorithm constructs two main graphical models: 1) a generative latent graph (a deep belief network) learned from data and 2) a deep discriminative graph constructed from the generative latent graph. We prove that conditional dependencies between the nodes in the learned generative latent graph are preserved in the class-conditional discriminative graph. Finally, a deep neural network structure is constructed based on the discriminative graph. We demonstrate on image classification benchmarks that the algorithm replaces the deepest layers (convolutional and dense layers) of common convolutional networks, achieving high classification accuracy, while constructing significantly smaller structures. The proposed structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU. Over the last decade, deep neural networks have proven their effectiveness in solving many challenging problems in various domains such as speech recognition BID17, computer vision BID28 BID16 BID46 and machine translation BID9. As compute resources became more available, large scale models having millions of parameters could be trained on massive volumes of data, to achieve state-of-the-art solutions for these high dimensionality problems. Building these models requires various design choices such as network topology, cost function, optimization technique, and the configuration of related hyper-parameters. In this paper, we focus on the design of network topology-structure learning. Generally, exploration of this design space is a time consuming iterative process that requires close supervision by a human expert. Many studies provide guidelines for design choices such as network depth BID46, layer width BID55, building blocks, and connectivity BID20 BID23. Based on these guidelines, these studies propose several meta-architectures, trained on huge volumes of data. These were applied to other tasks by leveraging the representational power of their convolutional layers and fine-tuning their deepest layers for the task at hand BID21 BID33. However, these meta-architecture may be unnecessarily large and require large computational power and memory for training and inference. The problem of model structure learning has been widely researched for many years in the probabilistic graphical models domain. Specifically, Bayesian networks for density estimation and causal discovery BID42 BID50. Two main approaches were studied: score-based (search-and-score) and constraint-based. Score-based approaches combine a scoring function, such as BDe BID10 and BIC BID44, with a strategy for searching through the space of structures, such as greedy equivalence search BID6. BID1 introduced an algorithm for sampling deep belief networks (generative model) and demonstrated its applicability to high-dimensional image datasets. Constraint-based approaches BID42 BID50 find the optimal structures in the large sample limit by testing conditional independence (CI) between pairs of variables. They are generally faster than score-based approaches BID54 ) and have a well-defined stopping criterion (e.g., maximal order of conditional independence). However, these methods are sensitive to errors in the independence tests, especially in the case of high-order conditional-independence tests and small training sets. Motivated by these methods, we propose a new interpretation for depth and inter-layer connectivity in deep neural networks. We derive a structure learning algorithm such that a hierarchy of independencies in the input distribution is encoded in the network structure, where the first layers encode higher-order independencies than deeper layers. Thus, the number of layers is automatically determined. Moreover, a neuron in a layer is allowed to connect to neurons in deeper layers skipping intermediate layers. An example of a learned structure, for MNIST, is given in Figure 1.We describe our recursive algorithm in two steps. In Section 2 we describe a base case-a singlelayer structure learning. In Section 3 we describe multi-layer structure learning by applying the key concepts of the base case, recursively (proofs are provided in Appendix A). In Section 4 we discuss related work. We provide experimental in Section 5, and conclude in Section 6. DISPLAYFORM0 a set of latent variables, and Y a class variable. Our algorithm constructs three graphical models and an auxiliary graph. Each variable is represented by a single node and a single edge may connect two distinct nodes. Graph G is a generative DAG defined over the observed and latent variables X ∪ H. Graph G Inv is called a stochastic inverse of G. Graph G D is a discriminative model defined over the observed, latent, and class variables X ∪ H ∪ Y. An auxiliary graph G X is defined over X (a CPDAG; an equivalence class of a Bayesian network) and is generated and maintained as an internal state of the algorithm. The parents set of a node X in G is denoted P a(X; G). The order of an independence relation is defined to be the condition set size. For example, if X 1 and X 2 are independent given X 3 and X 4, denoted X 1 ⊥ ⊥ X 2 |{X 3, X 4}, then the independence order is two. Figure 1: An example of a structure learned by our algorithm (classifying MNIST digits). Neurons in a layer may connect to neurons in any deeper layer. Depth is determined automatically. Each gather layer selects a subset of the input, where each input variable is gathered only once. A neural route, starting with a gather layer, passes through densely connected layers where it may split (copy) and merge (concatenate) with other routes in correspondence with the hierarchy of independencies identified by the algorithm. All routes merge into the final output layer (e.g., a softmax layer). We start by describing the key concepts of our approach using a simple scenario: learning the connectivity of a single-layer neural network. Assume the input joint distribution p(X) complies with the following property. Assumption 1. The joint distribution p(X) is faithful to a DAG G over observed X and latent nodes H, where for all X ∈ X and H ∈ H, P a(X; G) ⊆ H and P a(H; G) ⊆ H\H. DISPLAYFORM0 Note that the generative graphical model G can be described as a layered deep belief network where parents of a node in layer m can be in any deeper layer, indexes greater than m, and not restricted to the next layer m + 1. This differs from the common definition of deep belief networks BID22 BID1 where the parents are restricted to layer m + 1.It is desired to learn an efficient graph G having small sets of parents and a simple factorization of p(H) while maintaining high expressive power. We first construct an auxiliary graph, a CPDAG BID50, G X over X (an equivalence class of a fully visible Bayesian network) encoding only marginal independencies 1 (empty condition sets) and then construct G such that it can mimic BID42. That is, preserving all conditional dependencies of X in G X. DISPLAYFORM1 The simplest connected DAG that encodes statistical independence is the v-structure, a structure with three nodes X 1 → X 3 ← X 2 in which X 1 and X 2 are marginally independent X 1 ⊥ ⊥ X 2 and conditionally dependent X 1 ⊥ ⊥X 2 |X 3. In graphs encoding only marginal independencies, dependent nodes form a clique. We follow the procedure described by BID54 and decompose X into autonomous sets (complying with the Markov property) where one set, denoted X D (descendants), is the common child of all other sets, denoted X A1,..., X AK (ancestor sets). We select X D to be the set of nodes that have the lowest topological order in G X. Then, by removing X D from G X (temporarily for this step), the ing K disjoint sets of nodes (corresponding to K disjoint substructures) form the K ancestor sets DISPLAYFORM2. See an example in FIG1. Next, G is initialized to an empty graph over X. Then, for each ancestor set X Ai a latent variable H i is introduced and assigned to be a common parent of the pair (X Ai, X D). Thus, DISPLAYFORM3 Note that the parents of two ancestor sets are distinct, whereas the parents set of the descendant set is composed of all the latent variables. In the auxiliary graph G X, for each of the ing v-structures (X Ai → X D ← X Aj), a link between a parent and a child can be replaced by a common latent parent without introducing new independencies. For example, in Algorithm 1 summarizes the procedure of constructing G having a single latent layer. Note that we do not claim to identify the presence of confounders and their inter-relations as in BID14; BID45; BID2. Instead, we augment a fully observed Bayesian network with latent variables, while preserving conditional dependence. [a] DISPLAYFORM4 A stochastic inverse generated by the algorithm presented by.[c] A stochastic inverse generated by our method where the graph is a projection of a latent structure. A dependency induced by a latent Q is described using a bi-directional edge DISPLAYFORM5 having a class node Y that provides an explaining away relation for H A ↔ H B. That is, the latent Q is replaced by an observed common child Y. It is important to note that G represents a generative distribution of X and is constructed in an unsupervised manner (class variable Y is ignored). Hence, we construct G Inv, a graphical model that preserves all conditional dependencies in G but has a different node ordering in which the observed variables, X, have the highest topological order (parentless)-a stochastic inverse of G. Note that conditional dependencies among X are not required to be preserved in the stochastic inverse as these are treated (simultaneously) as observed variables (highest topological order).; BID41 presented a heuristic algorithm for constructing such stochastic inverses where the structure is a DAG (an example is given in Figure 3 -[b] ). However, these DAGs, though preserving all conditional dependencies, may omit many independencies and add new edges between layers. We avoid limiting G Inv to a DAG and instead limit it to be a projection of another latent structure BID42. That is, we assume the presence of additional hidden variables Q that are not in G Inv but induce dependency 2 among H. For clarity, we omit these variables from the graph and use bi-directional edges to represent the dependency induced by them. An example is given in Figure 3 -[c] where a bi-directional edge represents the effect of some variable Q ∈ Q on H A and H B. We construct G Inv in two steps:1. Invert all G edges (invert inter-layer connectivity).2. Connect each pair of latent variables, sharing a common child in G, with a bi-directional edge. This simple procedure ensures G G Inv over X ∪ H while maintaining the exact same number of edges between the layers (Proposition 1, Appendix A). XD ←− nodes having the lowest topological order identify autonomous sets DISPLAYFORM0 set each Hi to be a parent of {XA 1 ∪ XD} connect 12 return G Recall that G encodes the generative distribution of X and G Inv is the stochastic inverse. We further construct a discriminative graph G D by replacing bi-directional dependency relations in G Inv, induced by Q, with explaining-away relations by adding the observed class variable Y. Node Y is set in G D to be the common child of the leaves in G Inv (latents introduced after testing marginal independencies) (see an example in Figure 3 - [d] ). This preserves the conditional dependency relations of G Inv. That is, G D can mimic G Inv over X and H given Y (Proposition 2, Appendix A). It is interesting to note that the generative and discriminative graphs share the exact same inter-layer connectivity (inverted edge-directions). Moreover, introducing node Y provides an "explaining away" relation between latents, uniquely for the classification task at hand. We construct a neural network based on the connectivity in G D. Sigmoid belief networks BID38 have been shown to be powerful neural network density estimators BID29 BID15. In these networks, conditional probabilities are defined as logistic regressors. Similarly, for G D we may define for each latent variable H ∈ H, DISPLAYFORM0 where sigm(x) = 1/(1 + exp(−x)), X = P a(H ; G D), and (W, b) are the parameters of the neural network. BID37 proposed replacing each binary stochastic node H by an infinite number of copies having the same weights but with decreasing bias offsets by one. They showed that this infinite set can be approximated by DISPLAYFORM1 where v = W X + b. They further approximate this function by max(0, v +) where is a zerocentered Gaussian noise. Following these approximations, they provide an approximate probabilistic interpretation for the ReLU function, max(0, v). As demonstrated by BID25 and BID37, these units are able to learn better features for object classification in images. In order to further increase the representational power, we represent each H by a set of neurons having ReLU activation functions. That is, each latent variable H in G D is represented in the neural network by a dense (fully-connected) layer. Finally, the class node Y is represented by a softmax layer. We now extend the method of learning the connectivity of a single layer into a method of learning multi-layered structures. The key idea is to recursively introduce a new and deeper latent layer by testing n-th order conditional independence (n is the condition set size) and connect it to latent layers created by previous recursive calls that tested conditional independence of order n + 1. The method is described in Algorithm 2. It is important to note that conditional independence is tested only between input variables X and condition sets do not include latent variables. Conditioning on latent variables or testing independence between them is not required as the algorithm adds these latent variables in a specific manner, preserving conditional dependencies between the input variables. Algorithm 2: Recursive Latent Structure Learning (multi-layer)1 RecurLatStruct (GX, X, Xex, n) Input: an initial DAG GX over observed X & exogenous nodes Xex and a desired resolution n. Output: G, a latent structure over X and H 2 if the maximal indegree of GX (X) is below n + 1 then exit condition DISPLAYFORM0 to be a parent of {HA The algorithm maintains and recursively updates an auxiliary graph G X (a CPDAG) over X and utilizes it to construct G. BID54 introduced an efficient algorithm (RAI) for constructing a CPDAG over X by a recursive application of conditional independence tests with increasing condition set sizes (n). Our algorithm is based on this framework for updating the auxiliary graph G X (Algorithm 2, lines 5 and 6).The algorithm starts with n = 0, G X a complete graph, and a set of exogenous nodes X ex = ∅. The set X ex is exogenous to G X and consists of parents of X.The function IncreaseResolution (Algorithm 2-line 5) disconnects (in G X) conditionally independent variables in two steps. First, it tests dependency between X ex and X, i.e., X ⊥ ⊥ X |S for every connected pair X ∈ X and X ∈ X ex given a condition set S ⊂ {X ex ∪ X} of size n. Next, it tests dependency within X, i.e., X i ⊥ ⊥ X j |S for every connected pair X i, X j ∈ X given a condition set S ⊂ {X ex ∪ X} of size n. After removing the corresponding edges, the remaining edges are directed by applying two rules BID42 BID50. First, v-structures are identified and directed. Then, edges are continually directed, by avoiding the creation of new v-structures and directed cycles, until no more edges can be directed. Following the terminology of BID54, we say that this function increases the graph d-separation resolution from n − 1 to n. The function SplitAutonomous (Algorithm 2-line 6) identifies autonomous sets in a graph in two steps, as described in Algorithm 1 lines 7 and 8. An autonomous set in G X includes all its nodes' parents (complying with the Markov property) and therefore a corresponding latent structure can be constructed independently using a recursive call. Thus, the algorithm is recursively and independently called for the ancestor sets (Algorithm 2 lines 7-8), and then called for the descendant set while treating the ancestor sets as exogenous (Algorithm 2 line 9).[a] Each recursive call returns a latent structure for each autonomous set. Recall that each latent structure encodes a generative distribution over the observed variables where layer H (n+1), the last added layer (parentless nodes), is a representation of the input X ⊂ X. By considering only layer H (n+1) of each latent structure, we have the same simple scenario discussed in Section 2-learning the connectivity between H (n), a new latent layer, and H (n+1), treated as an "input" layer. Thus, latent variables are introduced as parents of the H (n+1) layers, as described in Algorithm 2 lines 11-13. A simplified example is given in Figure 4.Next, a stochastic inverse G Inv is constructed as described in Section 2-all the edge directions are inverted and bi-directional edges are added between every pair of latents sharing a common child in G. An example graph G and a corresponding stochastic inverse G Inv are given in Figure 5. A discriminative structure G D is then constructed by removing all the bi-directional edges and adding the class node Y as a common child of layer H, the last latent layer that is added (Figure 5-[c] ). Finally, a neural network is constructed based on the connectivity of G D. That is, each latent node, H ∈ H (n), is replaced by a set of neurons, and each edge between two latents, H ∈ H (n) and DISPLAYFORM1, is replaced by a bipartite graph connecting the neurons corresponding to H and H. Recent studies have focused on automating the exploration of the design space, posing it as a hyperparameter optimization problem and proposing various approaches to solve it. BID34 learns the topology of an RNN network introducing structural parameters into the model and optimize them along with the model weights by the common gradient descent methods. BID47 takes a similar approach incorporating the structure learning into the parameter learning scheme, gradually growing the network up to a maximum size. A common approach is to define the design space in a way that enables a feasible exploration process and design an effective method for exploring it. BID56 (NAS) first define a set of hyper-parameters characterizing a layer (number of filters, kernel size, stride). Then they use a controller-RNN for finding the optimal sequence of layer configurations for a "trainee network". This is done using policy gradients (REINFORCE) for optimizing the objective function that is based on the accuracy achieved by the "trainee" on a validation set. Although this work demonstrates capabilities to solve large-scale problems (Imagenet), it comes with huge computational cost. In a following work, BID57 address the same problem but apply a hierarchical approach. They use NAS to design network modules on a small-scale dataset (CIFAR-10) and transfer this knowledge to a large-scale problem by learning the optimal topology composed of these modules. BID3 use reinforcement learning as well and apply Q-learning with epsilon-greedy exploration strategy and experience replay. BID39 propose a language that allows a human expert to compactly represent a complex search-space over architectures and hyperparameters as a tree and then use methods such as MCTS or SMBO to traverse this tree. Smithson et al. FORMULA1 present a multi objective design space exploration, taking into account not only the classification accuracy but also the computational cost. In order to reduce the cost involved in evaluating the network's accuracy, they train a Response Surface Model that predicts the accuracy at much lower cost, reducing the number of candidates that go through actual validation accuracy evaluation. Another common approach for architecture search is based on evolutionary strategies to define and search the design space. BID43 BID35 use evolutionary algorithm to evolve an initial model or blueprint based on its validation performance. Common to all these recent studies is the fact that structure learning is done in a supervised manner, eventually learning a discriminative model. Moreoever, these approaches require huge compute resources, rendering the solution unfeasible for most applications given limited compute and time resources. We evaluate the quality of the learned structure in two experiments:• Classification accuracy as a function of network depth and size for a structure learned directly from MNIST pixels.• Classification accuracy as a function of network size on a range of benchmarks and compared to common topologies. All the experiments were repeated five times where average and standard deviation of the classification accuracy were recorded. In all of our experiments, we used a ReLU function for activation, ADAM BID26 for optimization, and applied batch normalization BID24 followed by dropout BID51 to all the dense layers. All optimization hyperparameters that were tuned for the vanilla topologies were also used, without additional tuning, for the learned structures. For the learned structures, all layers were allocated an equal number of neurons. Threshold for independence tests, and the number of neurons-per-layer were selected by using a validation set. Only test-set accuracy is reported. Our structure learning algorithm was implemented using the Bayesian network toolbox BID36 and Matlab. We used Torch7 BID8 and Keras BID7 with the TensorFlow BID0 back-end for optimizing the parameters of both the vanilla and learned structures. We analyze the accuracy of structures learned by our algorithm as a function of the number of layers and parameters. Although network depth is automatically determined by the algorithm, it is implicitly controlled by the threshold used to test conditional independence (partial-correlation test in our experiments). For example, a high threshold may cause detection of many independencies leading to early termination of the algorithm and a shallow network (a low threshold has the opposite effect). Thus, four different networks having 2, 3, 4, and 5 layers, using four different thresholds, are learned for MNIST. We also select three configurations of network sizes: a baseline (normalized to 1.00), and two configurations in which the number of parameters is 0.5, and 0.375 of the baseline network (equal number of neurons are allocated for each layer).Classification accuracies are summarized in TAB0. When the number of neurons-per-layers is large enough (100%) a 3-layer network achieves the highest classification accuracy of 99.07% (standard deviation is 0.01) where a 2-layer dense network has only a slight degradation in accuracy, 99.04%. For comparison, networks with 2 and 3 fully connected layers (structure is not learned) with similar number of parameters achieve 98.4% and 98.75%, respectively. This demonstrates the efficiency of our algorithm when learning a structure having a small number of layers. In addition, for a smaller neuron allocation (50%), deeper structures learned by our algorithm have higher accuracy than shallower ones. However, a decrease in the neurons-per-layer allocation has a greater impact on accuracy for deeper structures. MNIST images as a function of network depth and number of parameters (normalized). For comparison, when a structure is not learned, networks with 2 and 3 dense layers, achieve 98.4% and 98.75% accuracy, respectively (having the same size as learned structures at configuration "100%"). We evaluate the quality of learned structures using five image classification benchmarks. We compare the learned structures to common topologies (and simpler hand-crafted structures), which we call "vanilla topologies", with respect to network size and classification accuracy. The benchmarks and vanilla topologies are described in Table 2. In preliminary experiments we found that, for SVHN and ImageNet, a small subset of the training data is sufficient for learning the structure (larger training set did not improve classification accuracy). As a , for SVHN only the basic training data is used (without the extra data), i.e., 13% of the available training data, and for ImageNet 5% of the training data is used. Parameters were optimized using all of the training data. Table 2: Benchmarks and vanilla topologies. MNIST-Man and SVHN-Man topologies were manually created by us. MNIST-Man has two convolutional layer (32 and 64 filters each) and one dense layer with 128 neurons. SVHN-Man was created as a small network reference having reasonable accuracy compared to Maxout-NiN. In the first row we indicate that in one experiment a structure for MNIST was learned from the pixels and feature extracting convolutional layers were not used. Convolutional layers are powerful feature extractors for images exploiting domain knowledge, such as spatial smoothness, translational invariance, and symmetry. We therefore evaluate our algorithm by using the first convolutional layers of the vanilla topologies as "feature extractors" (mostly below 50% of the vanilla network size) and learning a deep structure from their output. That is, the deepest layers of the vanilla network (mostly over 50% of the network size) is removed and replaced by a structure learned by our algorithm in an unsupervised manner. Finally, a softmax layer is added and the entire network parameters are optimized. First, we demonstrate the effect of replacing a different amount of the deepest layers and the ability of the learned structure to replace feature extraction layers. Table 3 describes classification accuracy achieved by replacing a different amount of the deepest layers in VGG-16. For example, column "conv.10" represents learning a structure using the activations of conv.10 layer. Accuracy and the normalized number of network parameters are reported for the overall network, e.g., up to conv.10 + the learned structure. Column "vanilla" is the accuracy achieved by the VGG-16 network, after training under the exact same setting (a setting we found to maximize a validation-set accuracy for the vanilla topologies). Table 3: Classification accuracy (%) and overall network size (normalized number of parameters). VGG-16 is the "vanilla" topology. For both, CIFAR 10/100 benchmarks, the learned structure achieves the highest accuracy by replacing all the layers that are deeper than layer conv.10. Moreover, accuracy is maintained when replacing the layers deeper than layer conv.7.One interesting phenomenon to note is that the highest accuracy is achieved at conv. 10 rather than at the "classifier" (the last dense layer). This might imply that although convolutional layers are useful at extracting features directly from images, they might be redundant for deeper layers. By using our structure learning algorithm to learn the deeper layers, accuracy of the overall structure increases with the benefit of having a compact network. An accuracy, similar to that of "vanilla" VGG-16, is achieved with a structure having 85% less total parameters (conv. 7) than the vanilla network, where the learned structure is over 50X smaller than the replaced part. Next, we evaluate the accuracy of the learned structure as a function of the number of parameters and compare it to a densely connected network (fully connected layers) having the same depth and size. For SVHN, we used the Batch Normalized Maxout Network in Network topology BID4 and removed the deepest layers starting from the output of the second NiN block (MMLP-2-2). For CIFAR-10, we used the VGG-16 and removed the deepest layers starting from the output of conv.10 layer. For MNIST, a structure was learned directly from pixels. Results are depicted in FIG6. It is evident that accuracy of the learned structures is significantly higher (error bars represent 2 standard deviations) than a set of fully connected layers, especially in cases where the network is limited to a small number of parameters.[a] Finally, in Table 4 we provide a summary of network sizes and classification accuracies, achieved by replacing the deepest layers of common topologies (vanilla) with a learned structure. In the first row, a structure is learned directly from images; therefore, it does not have a "vanilla" topology as reference (a network with 3 fully-connected layers having similar size achieves 98.75% accuracy). In all the cases, the size of the learned structure is significantly smaller than the vanilla topology, and generally has an increase in accuracy. Comparison to other methods. Our structure learning algorithm runs efficiently on a standard desktop CPU, while providing structures with competitive classification accuracies and network sizes. For example, the lowest classification error rate achieved by our unsupervised algorithm for CIFAR 10 is 4.58% with a network of size 6M (WRN-40-4 row in Table 4). For comparison, the NAS algorithm BID56 achieves error rates of 5.5% and 4.47% for networks of sizes 4.2M and 7.1M, respectively, and requires optimizing thousands of networks using hundreds of GPUs. For AlexNet network, recent methods for reducing the size of a pre-trained network (pruning while maintaining classification accuracy) achieve 5× and 9× BID18 BID34 Table 4: A summary of network sizes and classification accuracies (and standard deviations), achieved by replacing the deepest layers of common topologies (vanilla) with a learned structure. The number of parameters are reported for "feature extraction" (first layers of the vanilla topology), removed section (the deepest layers of the vanilla topology), and the learned structure that replaced the removed part. The sum of parameters in the "feature extraction" and removed parts equals to the vanilla topology size. The first row corresponds to learning a structure directly from image pixels. We presented a principled approach for learning the structure of deep neural networks. Our proposed algorithm learns in an unsupervised manner and requires small computational cost. The ing structures encode a hierarchy of independencies in the input distribution, where a node in one layer may connect another node in any deeper layer, and depth is determined automatically. We demonstrated that our algorithm learns small structures, and maintains high classification accuracies for common image classification benchmarks. It is also demonstrated that while convolution layers are very useful at exploiting domain knowledge, such as spatial smoothness, translational invariance, and symmetry, they are mostly outperformed by a learned structure for the deeper layers. Moreover, while the use of common topologies (meta-architectures), for a variety of classification tasks is computationally inefficient, we would expect our approach to learn smaller and more accurate networks for each classification task, uniquely. As only unlabeled data is required for learning the structure, we expect our approach to be practical for many domains, beyond image classification, such as knowledge discovery, and plan to explore the interpretability of the learned structures.

Represent the TLDR for finding the corresponding scientific article

A principled approach for structure learning of deep neural networks with a new interpretation for depth and inter-layer connectivity.

represent the tldr for finding the corresponding scientific article

We cast causal structure discovery as a Bayesian model selection in a way that allows us to discriminate between Markov equivalent graphs to identify the unique causal graph.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: L1 and L2 regularizers are critical tools in machine learning due to their ability to simplify solutions. However, imposing strong L1 or L2 regularization with gradient descent method easily fails, and this limits the generalization ability of the underlying neural networks. To understand this phenomenon, we investigate how and why training fails for strong regularization. Specifically, we examine how gradients change over time for different regularization strengths and provide an analysis why the gradients diminish so fast. We find that there exists a tolerance level of regularization strength, where the learning completely fails if the regularization strength goes beyond it. We propose a simple but novel method, Delayed Strong Regularization, in order to moderate the tolerance level. Experiment show that our proposed approach indeed achieves strong regularization for both L1 and L2 regularizers and improves both accuracy and sparsity on public data sets. Our source code is published. Regularization has been very common for machine learning to prevent over-fitting and to obtain sparse solutions. Deep neural networks (DNNs), which have shown huge success in many tasks such as computer vision BID9 BID15 BID5 and speech recognition, often contain a number of parameters in multiple layers with non-linear activation functions, in order to gain enough expressive power. However, DNNs with many parameters are often prone to over-fitting, so the need for regularization has been emphasized. While new regularization techniques such as dropout BID16 and pruning BID2 have been proposed to solve the problem, the traditional regularization techniques using L1 or L2 norms have cooperated with them to further improve the performance significantly. L1 regularization, often called Lasso BID17, obtains sparse solutions so that the required memory and power consumption are reduced while keeping reasonable accuracy. On the other hand, L2 regularization smooths the parameter distribution and reduces the magnitude of parameters, so the ing solution is simple (i.e., less prone to over-fitting) and effective. Indeed, our empirical show that applying strong L2 regularization to the deep neural networks that already has dropout layers can reduce the error rate by up to 24% on a public data set. Strong regularization is especially desired when the model contains too many parameters for the given amount of training data. This is often the case for deep learning tasks in practice because DNNs often contain millions of parameters while labeled training data set is limited and expensive. However, imposing strong L1 or L2 regularization on DNNs is difficult for gradient descent method due to the vanishing gradient problem. If we impose too strong regularization, the gradient from regularization becomes dominant, and DNNs stop learning. In this paper, we first study the interesting phenomenon that strong regularization fails in learning. We also provide an analysis why the gradients diminish so quickly that learning completely fails. Then, we propose a simple yet effective solution, Delayed Strong Regularization, which carries a time-dependent schedule of regularization strength. We find that we can overcome the failure in learning by waiting for the model to reach an "active learning" phase, where the gradients' magnitudes are significant, and then enforcing strong regularization. Delayed Strong Regularization enables us to obtain the superior performance that is otherwise hidden by learning failure in deep networks. The proposed approach is general and does not require any additional computation. The experiment indicate that the proposed approach indeed achieves strong regularization, consistently yielding even higher accuracy and higher compression rate that could not be achieved. 2.1 Let us denote a generic DNN by y = f (x; w) where x ∈ R d is an input vector, w ∈ R n is a flattened vector of all parameters in the network f, and y ∈ R c is an output vector after feed-forwarding x through multiple layers in f. The network f is trained by finding optimal set of w by minimizing the cost function within the training data DISPLAYFORM0 as follows. DISPLAYFORM1 where L is the loss function, which is usually cross-entropy loss for classification tasks. Here, the regularization term λΩ(w) is added to simplify the solution, and λ is set to zero for non-regularized cost function. A higher value of λ means that stronger regularization is imposed. The most commonly used regularization function is a squared L2 norm: Ω(w) = ||w|| 2 2, which is also called as weight decay in deep learning literature. This L2 regularizer has an effect of reducing the magnitude of the parameters w, and the simpler solution becomes less prone to over-fitting. On the other hand, the L1 regularizer Ω(w) = ||w|| 1 is often employed to induce sparsity in the model (i.e., make a portion of w zero). The sparse solution is often preferred to reduce computation time and memory consumption for deep learning since DNNs often require heavy computation and big memory space. With the gradient descent method, each model parameter at time t, w (t) i, is updated with the following formula: DISPLAYFORM2 where α is a learning rate. As L1 norm is not differentiable at 0, the formula doesn't have value when w (t) i = 0, but in practice, the subgradient 0 is often used. Please see Section 2.3 for more details. From the formula, we can see that L2 regularizer continuously reduces the magnitude of a parameter proportionally to it while L1 regularizer reduces the magnitude by a constant. In both regularizers, strong regularization thus means greatly reducing the magnitude of parameters. Strong regularization is especially useful for deep learning because the DNNs often contain a large number of parameters while the training data is limited in practice. However, we have observed a phenomenon where learning suddenly fails when strong regularization is imposed for gradient descent method, which is the most commonly used solver for deep learning. The example of the phenomenon is depicted in Figure 1. In the example, the architectures VGG-16 BID15 and AlexNet BID9 were employed for the data set CIFAR-100 BID8.2 As shown, the accuracy increases as we enforce more regularization. However, it suddenly drops to 1.0% after enforcing a little more regularization, which means that the model entirely fails to learn. The depicted training loss also indicates that it indeed learns faster with stronger regularization (λ = 1 × 10 −3), but the training loss does not improve at all when even stronger regularization is imposed (λ = 2 × 10 −3).In order to look at this phenomenon in more detail, we show how gradients and their proportion change in Figure 2. As depicted in Figure 2a −3 ) follows a path that has a relatively steep slope during the first 150 epoch, and then it converges with a gentle slope. However, a model with a little stronger L2 regularization (λ = 2 × 10 −3) does not follow a path that has a good slope, so it does not really have a chance to learn from gradients. A close-up view of this in the first 20 epochs is depicted in Figure 2b. The models with moderate L1 and L2 regularization seem to follow a good path in a couple of epochs. Through following the good path, the models keep the proportion of the gradients from L to all gradients dominant, especially for the first 150 epochs (Figure 2c). On the other hand, the models with a little stronger regularization fail to follow such path and the gradients from L decrease exponentially (Figure 2b). Since the magnitude of gradients from L decreases faster than that from Ω, the proportion of the latter to all gradients becomes dominant (Figure 2c), and it in failure in learning. From this observation, we can see that there exists a tolerance level of regularization strength, which decides success or failure of entire learning. Why does the magnitude of the gradient from L decrease so fast? It is not difficult to see why the magnitude of ∂L ∂wi decreases so fast when the regularization is strong. In deep neural networks, the gradients are dictated by back-propagation. It is well known that the gradients at the l th layer are given by DISPLAYFORM0 where a (l−1) is the output of the neurons at the (l − 1) th layer and δ (l) is the l th -layer residual which follows the recursive relation DISPLAYFORM1 where ⊙ and a ′ denote the element-wise multiplications and derivatives of the activation function respectively. Using the recursive relation, we obtain DISPLAYFORM2 If the regularization is too strong, the weights would be significantly suppressed as shown in Figure 5b. From, since the gradients are proportional to the product of the weights at later layers (whose magnitudes are typically much less than 1 for strong regularization), they are even more suppressed. In fact, the suppression is more severe than what we have deduced above. The factor a (l−1) in could actually lead to further suppression to the gradients when the weights are very small, for the following reasons. First of all, we use ReLU as the activation function and it could be written as DISPLAYFORM3 where Θ(x) is the Heaviside step function. Using this, we could write DISPLAYFORM4 Applying FORMULA7 recursively, we can see that a (l−1) is proportional to the product of the weights at previous layers. Again, when the weights are suppressed by strong regularization, a (l−1) would be suppressed correspondingly. Putting everything together, we can conclude that in the presence of strong regularization, the gradients are far more suppressed than the weights. Strictly speaking, the derivations above are valid only for fully-connected layers. For convolutional layers, the derivations are more complicated but similar. Our above would still be valid. Normalization Normalization techniques such as batch normalization BID7 and weight normalization BID13 can be possible approaches to prevent the L gradients from diminishing quickly. However, it has been shown that L2 regularization has no regularizing effect when combined with normalization but only influences on the effective learning rate, ing in good performance BID18. In other words, the normalization techniques do not really simplify the solution as the decrease of parameter magnitude is canceled by normalization. This does not meet our goal, which is to heavily simplify solutions to reduce over-fitting, so we propose an approach that meets our goal. Since we have seen that stronger regularization can in better performance in Figure 1, we propose a method that is able to accommodate strong regularization. Specifically, we introduce a time-dependent regularization strength, λ t, to the equation FORMULA2, and it is defined as DISPLAYFORM0 where epoch(t) gets the epoch number of the time step t, and γ is a hyper-parameter that is set through cross-validation. The formula means that we do not impose any regularization until γ th epoch, and then impose the strong regularization in each training step. The underlying hypothesis is that once the model follows a good learning path, i.e., the gradient from L is big enough, it won't easily change its direction because of the steep slope, and thus, it can learn without failure. We empirically verify our hypothesis in the experiment section. The hyper-parameter γ is relatively easy to set because the models often follow the good path in a couple of epochs, and once they follow such path, learning does not fail. We recommend using 2 ≤ γ ≤ 20. Please note that our approach is different from imposing a slightly weaker regularization throughout the whole training. The reduced amount by not skipping regularization for the first few epochs is negligible compared to the total reduced amount by regularization. In addition, we empirically show that our approach can achieve a much higher sparsity than the baseline in the parameter space. The proposed method is easy to implement, and the hyper-parameter is easy to set. Also, the method is very close to the traditional regularization method so that it inherits the traditional one's good performance for non-strong regularization while it also achieves strong regularization. Although the method is very simple, we found that it shows the best accuracy among the approaches we tried in our preliminary experiments while it is the simplest. The preliminary experiments are further discussed in Appendix B.Proximal gradient algorithm for L1 regularizer Meanwhile, since L1 norm is not differentiable at zero, we employ the proximal gradient algorithm BID11, which enables us to obtain proper sparsity (i.e., guaranteed convergence) for non-smooth regularizers. We use the following update formulae: DISPLAYFORM1 where S is a soft-thresholding operator. Basically, the algorithm assigns zero to a parameter if its next updated value is smaller than αλ. In other cases, it just decreases the magnitude of the parameter as usual. We first evaluate the effectiveness of our proposed method with popular architectures, AlexNet BID9 and VGG-16 BID15 on the public data sets CIFAR-10 and CIFAR-100 BID8 ). Then, we employ variations of VGG on another public data set SVHN BID10, in order to see the effect of the number of hidden layers on the tolerance level. Please note that we do not employ architectures that contain normalization techniques such as batch normalization BID7, for the reason described in Section 2.2. The data set statistics are described in TAB0. VGG-11 and VGG-19 for SVHN contain 9.8 and 20.6 millions of parameters. Regularization is applied to all network parameters except bias terms. We use PyTorch 3 framework for all experiments, and we use its official computer vision library 4 for the implementations of the networks. In order to accommodate the data sets, we made some modifications to the networks. The kernel size of AlexNet's max-pooling layers is changed from 3 to 2, and the first convolution layer's padding size is changed from 2 to 5. All of its fully connected layers are modified to have 256 neurons. For VGG, we modified the fully connected layers to have 512 neurons. The output layers of both networks have 10 neurons for CIFAR-10 and SVHN, and 100 neurons for CIFAR-100. The networks are learned by stochastic gradient descent algorithm with momentum of 0.9. The parameters are initialized according to BID4. The batch size is set to 128, and the initial learning rate is set to 0.05 and decays by a factor of 2 every 30 epochs during the whole 300-epoch training. In all experiments, we set γ = 5. We did not find significantly different for 2 ≤ γ ≤ 20. Please note that we still use drop out layers (with drop probability 0.5) and pre- AlexNet and VGG-16 are experimented for different regularization methods (L1 and L2) and different data sets (CIFAR-10 and CIFAR-100), yielding 8 combinations of experiment sets. Then, VGG-11, VGG-16, and VGG-19 are experimented for L1 and L2 regularization methods on SVHN, yielding 6 experiment sets. For each experiment set, we set the baseline method as the one with well-tuned L1 or L2 regularization but without our time-dependent regularization strength. For each regularization, we try more than 10 different values of λ, and for each value, we report average accuracy of three independent runs and report 95% confidence interval. We perform statistical significance test (t-test) for the improvement over the baseline method and report the p-value. We also report sparsity of each trained model, which is the proportion of the number of zero-valued parameters to the number of all parameters. Please note that we mean the sparsity by the one actually derived by the models, not by pruning parameters with threshold after training. The experiment by VGG-16 are depicted in FIG1. As we investigated in Section 2.2, the baseline method suddenly fails beyond certain values of tolerance level. However, our proposed method does not fail for higher values of λ. As a , our model can achieve higher accuracy as well as higher sparsity. In practice, L2 regularization is used more often than L1 regularization due to its superior performance, and this is true for our VGG-16 experiments too. Using L2 regularization, our model improves the model without L1 or L2 regularization but with dropout, by 14.4% in accuracy, which is about 24% of error rate improvement. Tuning L2 regularization parameter is difficult as the curves have somewhat sharp peak, but our proposed method ease the problem to some extent by preventing the sharp drop. Our L1 regularizer obtains much better sparsity for the similar level of accuracy FIG1 ), which means that strong regularization plays an important role in compressing neural networks. The improvement is more prominent on CIFAR-100 than on CIFAR-10, and we think this is because over-fitting can more likely occur on CIFAR-100 as there are less images per class than on CIFAR-10.The experiment by AlexNet are depicted in FIG2. Again, our proposed method achieves higher accuracy and sparsity in general. Unlike VGG-16, we obtain more improvement over baseline with L1 regularization than with L2 regularization. In addition, the curves make sharper peaks than those by VGG-16 especially for the sparsity regularizer (L1). Interestingly, our proposed method often obtains higher accuracy even when the baseline does not fail on CIFAR-10, and this is only prominent when the regularization strength is relatively strong (better shown in FIG1). This may be because avoiding strong regularization in the early stage of training can help the model to explore more spaces freely, and the better exploration in finding superior local optima. The overall experiment are shown in TAB1. It shows that there is more performance improvement by L1/L2 regularization on VGG-16 than on AlexNet, which is reasonable since VGG-16 contains about 6 times more parameters so that it is more prone to over-fitting. Our proposed model always improves the baselines by up to 3.89%, except AlexNet with L1 regularization on CIFAR-10, and most (6 out of 7) improvements are statistically significant (p¡0.05). Our L1 regularization models always obtain higher sparsity with compression rate up to 4.2× than baselines, meaning that our model is promising for compressing neural networks. We also show in Figure 5 how gradients and weights change when our method and the baseline are applied. We hypothesized that if the model reaches an "active learning" phase with an elevated gradient amount, it does not suffer from vanishing gradients any more even when strong regularization is enforced. The Figure 5a shows that our model indeed reaches there by skipping strong regularization for the first five epochs, and and it keeps learning even after strong regularization is enforced. In Figure 5b, although the same strong regularization is enforced since epoch 5, the magnitude of weights in our model stops decreasing around epoch 20, while that in baseline (green dotted line) keeps decreasing towards zero. This means that our model can cope with strong regularization, and it maintains its equilibrium between gradients from L and those from regularization. The analysis in Section 2.2 implies that the number of hidden layers would affect the tolerance level when strong regularization is imposed. That is, if there are more hidden layers in the neural network architecture, the learning will fail more easily by strong regularization. In order to check the hypothesis empirically, we employ variations of the VGG architecture, i.e., which contain 11, 16, and 19 hidden layers, respectively. We experiment them on the SVHN data set. The by L2 regularization are depicted in Figure 6. As shown, the peak of our method's performance is formed around λ = 1 × 10 −3. As more hidden layers are added to the network, the tolerance level where the performance suddenly drops by the baseline is shifted to left, as hypothesized by our analysis. The by L1 regularization are in Appendix A, and it is shown that VGG-19 more easily fails as the parameters become more sparse. The overall experiment are shown in TAB2. As the method without L1/L2 regularization already performs well on this data set and there are relatively many training images per class, the improvement by L1/L2 regularization is not big. Our method still outperforms the baseline in all experiments (6 out of 6), but the improvement is less statistically significant compared to CIFAR-10 and CIFAR-100 data sets. The compression rate is especially good for VGG-19 mainly because its tolerance level is low so that the baseline can only achieve low sparsity. The related work is partially covered in Section 1, and we extend other related work here. It has been shown that L2 regularization is important for training DNNs BID9 BID1. Although there has been a new regularization method such as dropout, L2 regularization has been shown to reduce the test error effectively when combined with dropout BID16. Meanwhile, L1 regularization has also been used often in order to obtain sparse solutions. To reduce computation and power consumption, L1 regularization and its variations such as group sparsity regularization has been promising for deep neural networks BID19 BID14 ). However, for both L1 and L2 regularization, the phenomenon that learning fails with strong regularization has not been emphasized previously. BID0 showed that tuning hyper-parameters such as L2 regularization strength can be effectively done through random search instead of grid search, but they did not study how and why learning fails or how strong regularization can be successfully achieved. visualized activations to understand deep neural networks and showed that strong L2 regularization fails to learn. However, it was still not shown how and why learning fails and how strong regularization can be achieved. To the best of our knowledge, there is no existing work that is dedicated to studying the phenomenon that learning fails with strong regularization and to proposing a method that can avoid the failure. In this work, we studied the problem of achieving strong regularization for deep neural networks. Strong regularization with gradient descent algorithm easily fails for deep neural networks, but few work addressed this phenomenon in detail. We provided investigation and analysis of the phenomenon, and we found that there is a strict tolerance level of regularization strength. To avoid this problem, we proposed a novel but simple method: Delayed Strong Regularization. We performed experiments with fine tuning of regularization strength. Evaluation show that our model successfully achieves strong regularization on deep neural networks, verifying our hypothesis that the model will keep learning once it reaches an "active learning" phase, with strong regularization, our model obtains higher accuracy and sparsity, the number of hidden layers in neural networks affects the tolerance level, and L1/L2 regularization is difficult to tune, but it can yield great performance boost when tuned well. There are limitations in this work. Our proposed method can be especially useful when strong regularization is desired. For example, deep learning projects that cannot afford a huge labeled data set can benefit from our method. However, strong regularization may not be necessary in some other cases where the large labeled data set is available or the networks do not contain many parameters. In addition, our experiments were not performed on a bigger data set such as ImageNet data set. We need to fine-tune the models with different regularization parameters, and we also need multiple training sessions of each model to obtain confidence interval. For example, the experiment in FIG1 and 4 include 750 training sessions in total. This is something we cannot afford with ImageNet data set, which requires several weeks of training for EACH session (unless we have GPU clusters). Our approach cannot be applied to architectures containing normalization techniques for the reason in Section 2.2. We actually tried to intentionally exclude normalization part from Residual Networks BID5 ) and train the model to see if we can apply our method to non-normalized Residual Networks. However, we could not control the exploding gradients caused by the exclusion of normalization. Our work can be further extended in several ways. Since our model can achieve strong regularization, it will be interesting to see how the strongly regularized model performs if combined with pruning-related methods BID2. We applied our approach to only L1 and L2 regularizers, but applying it to other regularizers such as group sparsity regularizers will be promising as they are often employed for DNNs to compress networks. Lastly, our proposed Delayed Strong Regularization is very simple, so one can easily extend it to more complicated methods. All these directions are left as our future work. To empirically check the effect of the number of hidden layers on the tolerance level, we experimented variations of VGG on SVHN, and we showed the by L2 regularizer in Section 3.2. Here, we show the by L1 regularizer in FIG4. As more hidden layers are included to the network, the tolerance level where the baseline method suddenly fails is shifted to left. Such pattern in baseline method is more clearly shown in the accuracy vs. sparsity plots. VGG-19 fails to learn even when it loses only 27% of its parameters, whereas VGG-11 can still learn after losing 84% of its parameters. The reason why we proposed a very simple method is that it is effective while it is simple to implement. The only additional hyper-parameter, which is the number of initial epochs to skip regularization, is also not difficult to set. We think that the proposed method is very similar to the traditional regularization method so that it inherits the traditional one's good performance for non-strong regularization while it also achieves strong regularization. We actually tried a couple more approaches other than the proposed one in our preliminary experiments. We found that the proposed one shows the best accuracy among the approaches we tried while it is the simplest. For example, we tried an approach that can be regarded as a warm-start strategy. It starts with the regularization parameter λ t = 0, and then it gradually increases λ t to λ for γ epochs, where γ >= 0 and it is empirically set. We found that it can achieve strong regularization, but its best accuracy is similar to or slightly lower than that of our proposed approach. We also tried a method that is similar to Ivanov regularization BID12. In this method, the regularization term is applied only when the L1 norm of the weights is greater than a certain threshold. To enforce strong regularization, we set λ just above the tolerance level that is found by the baseline method. However, this method did not accomplish any learning. The reason is that, to reach the level of L1 norm that is low enough, the model needs to go through the strong regularization for the first few epochs, and the neurons already lose its learning ability during this period like the baseline method. If we set λ below the tolerance level, it cannot reach the desired L1 norm without strong regularization, and thus the performance is inferior to our proposed method. Meanwhile, an approach that applies strong regularization first and then continuously reduces the regularization strength is used in sparse learning for convex optimization. This approach is opposite to our approach in that ours avoids strong regularization for the first few epochs and then apply strong regularization afterwards. We performed a simple experiment with VGG-16 on CIFAR-100 to see if the approach can perform well for deep neural networks. We set the initial regularization parameter λ = 2 × 10 −3 and λ = 6 × 10 −5 for L2 and L1 regularization, respectively, which are just above the "tolerance level". Then, we continuously reduced λ t to zero throughout the training session. The trained models didn't show any improvement over "random guess", which means that they were not able to learn. Once the strong regularization is enforced in the beginning, the magnitudes of weights decrease quickly. This in turn drives the magnitudes of gradients to diminish exponentially in deep neural networks as explained in Section 2.2, and thus, the model loses its ability to learn after a short period of strong regularization.

Represent the TLDR for finding the corresponding scientific article:

We investigate how and why strong L1/L2 regularization fails and propose a method than can achieve strong regularization.

Represent the TLDR for finding the corresponding scientific article.

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Despite neural network’s high performance, the lack of interpretability has been the main bottleneck for its safe usage in practice. In domains with high stakes (e.g., medical diagnosis), gaining insights into the network is critical for gaining trust and being adopted. One of the ways to improve interpretability of a NN is to explain the importance of a particular concept (e.g., gender) in prediction. This is useful for explaining reasoning behind the networks’ predictions, and for revealing any biases the network may have. This work aims to provide quantitative answers to \textit{the relative importance of concepts of interest} via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interests and test hypotheses using examples (e.g., a set of pictures that illustrate the concept). We show that CAV can be learned given a relatively small set of examples. Testing with CAV, for example, can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other set of concepts. Interpreting with CAV does not require any retraining or modification of the network. We show that many levels of meaningful concepts are learned (e.g., color, texture, objects, a person’s occupation), and we present CAV’s \textit{empirical deepdream} — where we maximize an activation using a set of example pictures. We show how various insights can be gained from the relative importance testing with CAV. Neural networks (NNs) are capable of impressively good performance, yet understanding and interpreting their behavior remains a significant challenge. Solving this challenge is an important problem for several reasons. For example, explaining a system's behavior may be necessary to establish acceptability and see adoption for critical applications, such as those in the medical domain. For scientists and engineers, any greater understanding of how neural networks function is appreciated, since it may lead to better models and help with debugging (30; 19).Recent work suggests that linear combinations of neurons may encode meaningful, insightful information (2; 19; 27). However, we lack methods to 1) identify which linear combinations (if any) relate to a given concept, and 2) how these can aid in our quantitative understanding of concepts and classification decisions. For example, we may hypothesize that an image model that successfully classifies zebras may naturally encode concepts for'stripe' and'animal', somewhere in its internal representations, using a linear combination of neurons. How can we formalize this notion, and test such a hypothesis?Neural networks build internal representations that are far richer than the input features or output classes explicit in their training data. Unfortunately, many machine learning interpretation methods provide only in terms of input features. For example, the learned coefficients in linear classifiers or logistic regression can be interpreted as each feature's classification importance. Similar first-order importance measures for neural networks often use first derivatives as a proxy for input feature importance, as is done for pixel importance in saliency maps (8; 22).It is critical that model understanding and interpretation not be limited to only the concepts explicit in training data. This can be seen by considering classification fairness-an increasingly relevant, difficult problem where interpretability can be useful-and noting that no input features may identify discriminated-against groups. For example, the Inception model BID24 has an output class for'doctor' but no input features identifying the concepts of'man' or'woman' in a way that would allow existing interpretability approaches to quantify gender bias in classification. This work introduces the method of concept activation vectors (CAV) for the following purposes. First, CAV can be used to identify linear combinations of neurons in a layer of a model that correspond to given semantic concepts, even for new, user-provided concepts not explicit in the model's training data. Second, CAV provides quantitative measures of the relative importance of userprovided concepts, which allows for hypothesis testing of the relationship between given concepts and the model's predictions. Testing with CAV (TCAV) is designed with the following desiderata in mind.1. accessibility: Requires little to no user expertise in machine learning. 2. customization: Adapt to any concept of interest (e.g., gender) on the fly without pre-listing a set of concepts before training. 3. plug-in readiness: Work without retraining or modifying the model. BID2. quantification: Provide quantitative explanation that are tied to human-relatable concept, and not input features. One of key ideas for TCAV is that we can test the relative importance between small set of concepts, rather than ranking the importance of all possible features/concepts. For example, we can gain insights by testing whether the concept of gender was used more than the'wearing scrubs' concept for the classification of doctor. We can also test whether or not a given concept was relevant to the classification of a certain class. Similar forms of sparsity (i.e., only considering a few concepts at a time) are used in many existing interpretable models (12; 7; 28; 31; 29; 4). Note that interpretability does not mean understanding the entire network's behavior on every feature/concept of the input BID4. Such a goal may not be achievable, particularly for ML models with super-human performance BID21.TCAV satisfies these desiderata-accessibility, customization, plug-in readiness and quantification -it enables quantitative relative importance testing for non-ML experts, for user-provided concepts without retraining or modifying the network. Users express their concepts of interest using examples-a set of data points exemplifying the concept. For example, if gender is the concept of interest, users can collect pictures of women. The use of examples has been shown to be powerful medium for communication between machine learning (ML) models and non-expert users (16; 12; 13). Cognitive studies on experts also support this approach (e.g., experts think in terms of examples BID13).The structure of this paper is as follows: Section 2 relates this work to existing interpretability methods. Section 3 explains the details of the TCAV method. In Section 4, we show 1) how this framework can be used to identify semantically meaningful directions in a layer and 2) the relative importance testing that measure the relevance of concepts of interest to the classification output by the network. In this section, we provide a brief overview of existing related interpretability methods and their relation to our desiderata. We also discuss the need and the challenges of desiderata 3): plug-in readiness. One of the most popular approaches in interpreting NN is saliency methods (24; 22; 25; 8; 5). These techniques seek to identify regions of the input most relevant to the classification output by the network. Qualitatively, these methods often successfully label regions of the input which seem semantically relevant to the classification. Unfortunately, these methods do not satisfy our desiderata of 2) customization and 4) quantification. Recent work has also demonstrated that the saliency map these methods produced may be very sensitive to completely uninteresting properties of the data distribution. In particular, BID12 showed that simply applying a mean shift to the dataset may cause some saliency methods to in significant changes in the given explanation. also showed that saliency methods can be easily tricked. There are techniques, such as DeepDream, which can be used to visualize patterns that maximally activates each neuron of a neural network. The technique starts from an image of random noise and iteratively modifies the image in order to maximally activate a neuron or a linear combination of neurons of interest (17; 18). This technique has offered some insights into the information encoded in a neuron's activation. This technique also has opened up opportunities for AI-aided art BID15.However, the DeepDream method does not satisfy our desiderata 1) accessibility, 2) customization, and 4) quantification. It does not satisfy 1) because in order to apply it a user must first understand what a neuron is, and second be familiar enough with the the internal architecture in order to choose which neurons to visualize. It does not satisfy 2) because no current method exists to find which neurons correspond to semantically meaningful concepts such as gender, and it is unclear whether or not such a neuron even exists. It does not satisfy 4) because we do not understand these pictures and there is currently no method to quantify how these pictures relate to the output of the network. This method again does not provide actionable insight. As we show later, DeepDream may be combined with TCAV in order to identify and visualize interesting directions in a layer. Prior work on DeepDream has typically chosen neurons or linear combinations of neurons at random to visualize. Note that a user of TCAV does need to pick a layer in the network for which to apply TCAV to. However, if only the final prediction is concerned, the last layer can be used by default. To achieve interpretability, we have two options: restrict ourselves to inherently interpretable models or post-process our models in way that yields insights. Users may choose option as there are a number of methods to build inherently interpretable models (12; 7; 28; 31; 29; 4). If the users are willing and are able to adapt these models, then this is the gold standard. Although building inherently interpretable machine learning models may be possible for some domains, doing so may in decreased performance. Furthermore, changing the model may be costly for users who already have a model implemented. A method that can be applied without retraining or modifying the network could be instantly used to interpret existing models. Increasingly, attention is turning to the importance of providing explanations for ML decisions (for one example, consider). As a , there is a growing need for interpretation techniques that can be applied "out of the box," that is, without rebuilding or retraining existing systems. One of many challenges of building a post-processing interpretion method is to ensure that the explanation is truthful to the model's behavior. By "truthful" we mean that explanations are roughly consistent with model's internal state. For example, if the explanations are created completely independently of the model FORMULA0, it has high probability of having such inconsistencies. The plug-in readiness desiderata poses interesting challenge for explanations to remain consistent with the model's behavior. Recently, there has been work showing that saliency methods contains such inconsistencies. For instance, BID12 show that saliency methods are vulnerable to constant shift in input that does not affect classification. It has also shown that the methods can be easily tricked BID7.One way to improve consistency between explanations and the model's reasoning is to use the generated explanation as an input, and check the network's output for validation. This is typically used in perturbation-based interpretability methods (16; 20). These methods perturb the input, and use the network's response to generate explanations. They maintain the consistency either locally or globally 1 by construction. TCAV is a type of perturbation method. Even a truthful explanation may be misleading if it is only locally truthful BID18. For example, since the importance of features only needs to be truthful in the vicinity of the data point of interest, there is no guarantee that the method will not generate two completely conflicting explanations. These inconsistencies may in decreased user trust at a minimum. On the other hand, making a globally truthful explanation may be difficult as the networks decision boundaries may be complex. TCAV produces globally explanations, and uses model's output to generate explanations to maintain consistency between explanations and the model's reasoning. We introduce a method that allows for global linear interpretability of a highly flexible class of models, namely deep feedforward neural networks trained on classification tasks. As a form of explanation, TCAV uses concepts that are provided by users, instead of using predetermined set of input features or input classes. These concepts are tied to real-world data that represents concepts of interest. Users express their concepts of interest using examples -a set of data points exemplifying the concept. These concepts enable testing the relative importance of concepts used in classification. Informally, the key idea is the following: although feedforward networks learn highly nonlinear functions there is evidence that they work by gradually disentangling concepts of interest, layer by layer (2; 3). It has also been shown that representations are not necessarily contained in individual neurons but more generally in linear combinations of neurons (19; 27). Thus the space of neuron activations in layers of a neural network may have a meaningful global linear structure. Furthermore, if such a structure exists, we can uncover it by training a linear classifier mapping the representation in a single layer to a human selected set of concepts. We now formalize this intuition. First, we formally define a concept activation vector. Let us imagine that an analyst is interested in a given concept C (e.g., striped textures) and has gathered two sets of data points, P C and N, that represent positive and negative examples of this concept (say, photos of striped objects, versus a set of random photos). There is a lot of flexibility in how to choose N, we often choose a random set of images of the same size as P C. Here we represent the input to the network as vector in R n, and P C, N ⊂ R n. Consider a layer l of the feedforward network consisting of m neurons. Then running inference on an input example and looking at the activations at layer l yields a function f l: R n → R m. For a set of inputs X ⊆ R n we denote by f l (X) to be the set of layer activations {f l (x): x ∈ X}. Note that for convolutional networks we view a layer in it's flattened form, thus a layer of width w, height h, and c channels becomes a flat vector of m = w × h × c activations. The two sets P C and N then naturally give rise to two sets of activation vectors in layer l, namely f l (P C) and f l (N). We can then train a linear classifier on the binary classification task of distinguishing between these two sets. The weights of this classifier are an element v l C ∈ R m. We call this the concept activation vector for the concept C.A variation on this idea is a relative concept activation vector. Here the analyst selects two sets of inputs that represent two different concepts, C and D. Training a classifier on f l (P C) and f l (P D) yields a vector v A key benefit of this technique is the flexibility allowed in choosing the set P C. Rather than being tied to the class labels in the training data, an analyst can-after training-select sets that correspond to any concept of interest. We now describe two ways that this technique be used to interpret a feedforward neural network trained on an image classification task. First, we show relative importance testing between M number of concepts, P C i, where i ∈ M. In this case, each v l C is learned by treating all P C i, i = j as N. Second, we can also test one concept, P C, against a random concept, where N are set of random pictures. In the next section we discuss the of experiments with these methods. The real value of CAVs comes from the fact that they may be used to test the relative importance of concepts. With CAVs, we can formulate generating explanation as a task of performing two-tailed statistical significance test (in particular, z-test). Given samples of class images (e.g., zebra pictures) and two concept vectors A and B, we perform two-tailed z-testing to invalidate the null hypothesis that there is no difference in importance of concepts A and B for the class. We perform this testing for each pair of concepts. For example, an analyst might ask about a photo of a zebra, "was the presence of stripes in the image relevant to the model's classification of the image as a zebra?" In some cases, with tedious effort in a photo retouching program, it might be possible answer questions like this directly. However, CAVs provide a faster and more general technique. Consider an input x and concept C of interest. At inference time, this will give rise to activations f l (x) at layer l. We can then modify this vector of activations in various ways using the concept vector v l C. For example, we might create a new vector w − = f l (x)−v l C, which might be interpreted as a counterfactual of "x with less of concept C". Performing inference starting at layer l + 1 based on w − will give a new classification y w. We can also create a new vector w + = f l (x) + v l C, which might be interpreted as a counterfactual of "x with more of concept C".Thus to test the relevance of stripes to the classification of a zebra, we would either add or subtract a'stripe' concept activation vector from the zebra image embedding, run forward propagation the ing embedding and examine the output of the network. Large changes in the probability of zebra would indicate that stripes played a key role in the classification. Simply adding the vector is a bit ad-hoc, especially when the norm of the vector may depend on how exactly the CAV was trained. However, we found this naive method to empirically yield that were consistently semantically meaningful. We also note that this addition is loosely related to directional derivative -saliency maps take the derivative of the logits with respect to each pixel, while our work takes derivatives with respect to a concept direction. Future work should explore more principled methods, to measure relevance of concepts to classification. Quantitatively, the metric of the influence of the concept C to class k can be measured by DISPLAYFORM0 where p k (y) represents the probability of class k of prediction y and i = {1, 2, . . ., N} where N is the number of images of the class for inspection (e.g., zebra). Intuitively speaking, Iup/down w measures the ratio of data points that become'more/less like class k' after the modification with concept vector v l C. We can perform statistical significance testing in order to quantify the concept importance; we can test a hypothesis that one concept is more/less important than another concept. In other words, the null hypothesis that no color is significant. To test this hypothesis we can do z-testing on the measured importance values, and ask the question what is the probability that random concept vectors would observe the measured difference. As a simple demonstration of the value of CAVs, we describe a technique for localizing where a concept is disentangled in a network. A common view of feedforward networks is that different concepts are represented at different layers. For example, many image classification networks are said to detect textures at low levels, with increasingly complex concepts recognized at later layers. Typically evidence for this view comes from various methods of reverse-engineering (3; 30; 17) the behavior of individual neurons in the network. Using concept activation vectors, however, we can approach this question from the opposite direction. We can pick out a particular concept C in advance, and consider the concept vectors v 2; 3; 19 ). In the next section we describe the of an experiment based on this idea. In this section, we first show evidence that the learned CAV indeed detect the concepts of interest. Then we show the for hypothesis testing using these CAVs, and the insights one can gain from it. All our experiments are based on the model from using the publicly available model parameters from. The pictures used to learn the CAVs are collected from the latest ImageNet Fall 2011 release BID19. Each concept vector is learned using between 30-500 pictures as input. We intentionally chose not to use all the available pictures in order to mirror realistic scenarios where users may not be able to collect a large amount of data. The class'arms' only had 30 pictures (all manually collected), since ImageNet does not include this as a part of the dataset. The pictures used to learn texture concepts are taken from the data set. In order to learn concept vectors for colors, we generated 500 pictures synthetically by generating the color channels randomly. We provide experiments both where CAV's are trained to distinguish between a set of concepts (e.g. red, yellow, green, blue) and when one CAV is trained per concept (with N chosen to be a random collection of images). In this section we describe experiments that indicate our linearly learned CAVsalign with their intended concepts. First, the linear maps used to construct the CAVsare accurate in predicting the concepts. The point in the network where these concepts in the networks are learned (i.e., when accuracy becomes high) is consistent with general knowledge about the representations learned by neural networks. Low level features such as edges and textures are detected in the early layers and higher level concepts are only reliably detected in the upper layers. Next, we use activation maximization techniques BID16 to visualize each of the CAVs and observe that the patterns are consistent with the provided concept. Finally, we show the top k images that are most similar in terms of cosine similarity to each CAV for further qualitative confirmation. Figure 1 shows the accuracy of the linear classifiers at each layer in the network for each type of CAV. Each classifier is trained to distinguish one concept from other concepts (e.g., textures1 set contains 'stripes', 'zigzagged' and 'dotted' texture concepts). Overall, we observe high accuracy as measured by a held out test set of size 1/3 that of the training size. This is evidence that the given concepts are linearly separable in many layers of the network. Note that the accuracy of more abstract CAV (e.g., objects) increases in higher layers of the network. The accuracy of a simpler CAV, such as color, is high throughout the entire network. This agrees with prior work on visualizing the learned representations at different layers of neural networks BID28.This experiment does not yet show that these CAVs align with the concepts that makes sense semantically to humans. We demonstrate this with the next set of experiments. In this section, we use the activation maximization technique BID15 to visualize the learned representations in the direction of the CAV. We use the same techniques from BID16. As typically done, we use a random image as a starting point BID15 for the optimization to avoid choosing an arbitrary image as a starting point. Figure 2 shows highly recognizable features, such as knitted textures, and corgi faces. Figure 3: Deepdreamed CAV texture concepts for each layer (each row) in inception. We can identify textures better in the mid-layer (mixed4d), then later layers. We can also observe how each concept is represented as the depth increases. Images in Fig. 3 show the for sets of textures. Interestingly, there is a layer (around mixed 4d) where textures are clearly identifiable, after which they become less and less recognizable. The left images in FIG3 show the for set of colors -green, blue, yellow and red (each column) and for the set of layers (lower to higher layers from top to bottom). The training set for the color CAVs are generated by simply replacing each RGB channel with randomly sampled values around 0.5, while leaving other channels empty. Note how the color CAVs also contain many random textures, this is perhaps not surprising -there is no reason to expect any direction in a layer to be associated purely with one concept while being orthogonal to all other concepts. The right image in FIG3 shows higher level concepts. The columns represent zebra, siberian huskies, and corgis. It is interesting to note how the zebra CAVs include textures suggestive of water, trees, and grass. This suggests that the model associates all of these concepts with the classi- fication of zebra and is likely a learned bias that from the of zebra pictures in the training data. These visualizations provide some qualitative confirmation that the learned directions align with the given concepts. This also shows that DeepDream may be combined with TCAV in order to identify and visualize interesting directions in a layer. Prior work on DeepDream has typically chosen neurons or linear combinations of neurons at random to visualize. The next section provides further evidence that the CAVs are indeed aligned with the concept of interest using real data. In order to qualitatively confirm that the learned CAV aligns with meaningful concepts, we compute cosine similarity between a set of pictures (all from one class) to the CAV. FIG4 shows that the top corgi images that aligns with striped CAV selects pictures with striped objects (e.g., a striped tube or vertical blinds in the ). Thus being similar to the striped CAV meant being highly similar to one of the other concepts in the CAV training set. We also show that CAVs for more abstract concepts can be constructed (e.g., CEO). Recall that many of these learned concepts are not classes that the NN learned to classify. We also observe that if there is not much relation between pictures and the concept, the cosine similarity is low. This experiment qualitatively supports our argument that the alignment between CAVs and the meaningful concepts. Note that the images least similar to striped appear with a checkered or knitted pattern, likely due to the fact that the striped CAV was trained relative to checkered and knitted images. When using TCAV, in order to ensure that the provided examples were sufficient to learn CAV, we recommend users to perform this qualitative test by checking cosine similarity with each concepts to the class. In this section, we describe how concept activation vectors may be used for quantitative testing of the relative importance of concepts. In FIG5 we show I up w + for eight classes. We do not list I up w −, as they always appear to be the inverse of I up w +. This means that when CAVs are added, the probability of the class rarely stays the same. This confirms that adding or subtracting v l C clearly impacts the classification task. In particular the v l C direction is aligned with the direction that measures the probability of the target class. Note that adding random directions should be expected to change the probability of the target class in some way. We find, however, that for several CAV's which are semantically relevant to the class (e.g., red to fire engine) the CAVdirection will consistently increase the probability of class for most images of this class. On the other hand, we show that random directions (e.g., a concept learned from random set of images) tend to be much less consistent. This difference allows us to perform a more rigorous hypothesis test in a following section. This testing identifies relationships between concepts and the target class that agree with human intuition (e.g., fire engines are red, cucumbers are bumpy, zebras are striped, CEOs wear suits). In the next section, we describe insights we gained from these tests. In this section, we show that confirm common-sense intuitions about training images, as a kind of sanity check that TCAV provides valid . We then describe that surprised us, but lead to insights about the dataset that we did not initially have. In FIG5, the yellow color was more influential to cab class than any other colors. For people who have lived in cities where this is the standard color for taxis, this is no shock -and indeed, most ImageNet pictures of cabs are yellow. Similarly, the'women' concept is important to'bikini' class, likely due to the fact that training images have humans wearing the bikini rather than pictures of the clothing. More subtly, we also discovered that the'model women' concept is important to'bikini' This lead us to realize that the most of'bikini' class pictures feature professional models, typically very slim, posing in their bikinis -probably not well representative samples of bikinis in the real world. Note that the network was trained with bikini and cab classes, and the concept'yellow','women' and'model women' are not part of classes used to train this network 2.The graph for'dumbbell' class in FIG5 shows that'arms' concept was more important to predict dumbbell class than other concepts. This finding is consistent with previous qualitative findings from BID15, where they discovered that the DeepDream picture of a dumbbell also showed an arm holding it. TCAV allows for quantitative confirmation of this previously qualitative finding. Moreover, unlike all other concepts in this figure, we only collected 30 pictures of each concept (ImageNet did not have arms as a label). Despite the small number of examples, the TCAV method is able to identify this relationship. The flexibility of the TCAV method makes it easy to explore a network for other surprising associations. For example, we saw that'baby' concept was important to'school bus' class. It turns out that some school bus pictures include young children riding or standing in front of the bus. Another discovery was to find that'red' concept is important for cucumber class. We believe this due to the fact that a number of cucumber pictures include other red vegetables, such as tomato and carrot. We believe that TCAV can be useful in discovering these insights for many applications, including for example, to improve fairness in the model. When constructing the CAV, the choice of negative samples may cause spurious suggestions that a concept is relevant to a class when it actually is not. In other words, the learned CAV may accidentally be aligned with something related to the class, and cause high I up w +. For instance, when we tested unrelated concepts and classes, such as zebra to a set of textures, honeycombed, lace-like and bumpy textures, we found that lace-like concept shows high I up w +. One might argue that lace is vaguely related to a zebra's stripes, but in fact even if we choose both P C and N to be independent random sets of images we still often observe I up w + to be relatively high. This is not surprising: there are directions in a layer l which are highly aligned with the zebra concept, and randomly chosen directions will typically have small but non-zero projection along these directions. One way to filter out these spurious is to do statistical testing against random concepts. Using different sets of random images for the negative class and striped images for P C, we make 50-100 CAVs, all of which represent concept'striped'. For each'striped' CAV we can measure I up w +. We can also generate a set of'random' CAVs by choosing random images for both P C and N. Then we performed a z-test to see if the mean I up w + s from striped CAVs are statistically different from the mean I up w + of random CAVs. We can successfully filter out some spurious correlations, including lace-like concepts with zebras by using this method, see the histogram 4.4. We have introduced the notion of a "concept activation vector," or CAV, which is a flexible way to probe the internal representation of a concept in a classification network. Since CAVs may be defined via a set of example inputs, rather than custom coding, they are well suited to use by non-experts. We then described a technique (Testing with CAVs, or TCAV) for quantifying the relation between a CAV and a particular class. The TCAV technique allows us to provide quantitative answers to questions such as, "How important are the stripes to the classification of a zebra?"To provide evidence for the value of the TCAV technique, we described a series of experiments which supported common-sense intuition, for example, that stripes are indeed important to the identification of zebras. In addition, we used the DeepDream technique to create images whose internal representations approximate certain CAVs. The ing pictures were strongly evocative of the original concepts. Finally, we described how the TCAV technique may be used to find associations between concepts, both obvious ("yellow" and "taxi") and non-obvious ("red" and "cucumber").In addition to analyzing a single network, TCAV can be also used to compare and contrast a pair of networks. For example, one can compare the relative importance of concepts to determine how the different choices of training process or architecture influences learning of each concept. Based on the , users can perform model selection based on the concepts that are more or less important for the task. An interesting direction for future work may be to explore applications of using CAVs to adjust the of a network during inference time. Adding a scalar multiple of a CAV to the activations of an intermediate layer can, as shown in our experiments, allow us to deemphasize or enhance conceptual aspects of an input. One potential application, for example, might be to reduce bias the network has learned from training data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.

Represent the short summary for finding the corresponding scientific article (long)

We propose new objective measurement for evaluating explanations based on the notion of adversarial robustness. The evaluation criteria further allows us to derive new explanations which capture pertinent features qualitatively and quantitatively.