Question: Represent the article for finding a single-sentence TLDR

Question: Deep artificial neural networks can achieve an extremely small difference between training and test accuracies on identically distributed training and test sets, which is a standard measure of generalization. However, the training and test sets may not be sufficiently representative of the empirical sample set, which consists of real-world input samples. When samples are drawn from an underrepresented or unrepresented subset during inference, the gap between the training and inference accuracies can be significant. To address this problem, we first reformulate a classification algorithm as a procedure for searching for a source code that maps input features to classes. We then derive a necessary and sufficient condition for generalization using a universal cognitive similarity metric, namely information distance, based on Kolmogorov complexity. Using this condition, we formulate an optimization problem to learn a more general classification function. To achieve this end, we extend the input features by concatenating encodings of them, and then train the classifier on the extended features. As an illustration of this idea, we focus on image classification, where we use channel codes on the input features as a systematic way to improve the degree to which the training and test sets are representative of the empirical sample set. To showcase our theoretical findings, considering that corrupted or perturbed input features belong to the empirical sample set, but typically not to the training and test sets, we demonstrate through extensive systematic experiments that, as a of learning a more general classification function, a model trained on encoded input features is significantly more robust to common corruptions, e.g., Gaussian and shot noise, as well as adversarial perturbations, e.g., those found via projected gradient descent, than the model trained on uncoded input features. Generalization error in deep learning is typically defined as the difference between training and test errors measured on identically distributed training and test sets. This traditional approach fails to take into account how representative these sets are of the empirical sample set from which input samples are drawn at inference time. When the training and test sets are not sufficiently representative of the empirical sample set, the difference between training and inference errors can be significant, thus rendering the learned classification function ineffective. The lack of the latter kind of generalization in unreliable decisions, raising questions about how robust, fair, and safe a learned classification function is . A natural question then arises: is there a necessary and sufficient condition ensuring that deep learning classifiers generalize in this broader sense? If so, how can this condition be satisfied in a real-world setting? To answer these questions, we draw on algorithmic information theory, which proposes a complexity measure, Kolmogorov complexity, as the absolute information content of any object, e.g., a computer program, function, or set. After deriving a necessary and sufficient condition for generalization using the information distance , which is a universal cognitive similarity metric based on Kolmogorov complexity, and formulating an optimization problem for generalization, we turn our attention to coding theory in order to learn a more general classification function by extending the input features to a classifier with systematically generated encodings of the original features. For a classification task, we assume that there exists a true classification function. Given training and test sets, neither of which are sufficiently representative of the the empirical sample set from which input samples are drawn during inference, a learning algorithm is asked to find the true classification function. In this work, we study how well the learned classification function generalizes with respect to the true classification function. In other words, we study the problem of how to minimize the generalization error, which we define as the difference between the training error and inference error measured on the empirical sample set, as opposed to the difference between the training error and test error. We use robustness to both common corruptions and adversarial robustness to measure how well a learned classification function generalizes on the empirical sample set, which contains corrupted or perturbed samples. Universal cognitive similarity metric. In order to find a necessary and sufficient condition for generalization in deep learning, we use the normalized information distance. A key finding in algorithmic information theory is that the normalized information distance is a universal cognitive similarity metric: the normalized information distance between two objects minorizes any other admissible distance up to an additive logarithmic term . In other words, although different learning algorithms will pick up on different dominating input features, depending on the classification task that they perform, every such dominating feature will be detected by the normalized information distance. Classification function as a source code. We formulate a learning algorithm as a procedure for searching for a source code based on training examples. We show that the learned classification function is a lossy compressor: the classifier discards some information. The input features thus cannot be recovered from the class label. We use the normalized information distance between the true source code (true classification function) and the learned source code (learned classification function) to find a necessary and sufficient condition ensuring generalization, and then formulate the problem of learning a more general classification function as an optimization problem. Compression-based similarity metric. The normalized information distance provides the theoretical tools needed to learn more general source codes, but in practice the normalized information distance is not effectively computable. We therefore use a compression-based similarity metric (Cilibrasi & Vitányi, 2005) based on a real-world compressor to approximate this theoretical construct. Specifically, we use the normalized compression distance between the true source code and learned source code to derive an effectively computable condition on the compressed size of the learned source code to identify encodings of the input features that help to learn a more general source code. Encoding input features. In a typical communication system, a source code is followed by a channel code which is then followed by a physical channel. In this paper, the learned source code (learned classification function) is preceded by one or more input codes that help ensure the learned classifier is more general by generating relations between input features that are not captured by the set of available input features. In order to showcase our findings for a specific classification task, we use channel codes on the input features for CIFAR-10 image classification. Precisely, we use a four-dimensional (4-D) five-level pulse-amplitude modulation (5-PAM) trellis-coded modulation (TCM) scheme (; ;) to systematically generate multiple encodings of the set of available input features. In doing so, we enable the deep neural network (DNN) to learn information from the empirical sample set which it could not learn from the uncoded input features alone. The generalization error is thereby reduced. The impact of generalization. Through image classification experiments, we show that a model trained on arbitrarily encoded input features is significantly more robust to common corruptions, such as Gaussian noise and shot noise, and to adversarial perturbations, like those generated via projected gradient descent (PGD) , than a model trained on uncoded input features. The role of code design. The code used on the input features can be designed in various ways for a classification task, and designing input codes is an important step to learning a more general classification function from the set of available input features. We show that merely increasing the number of input channels of a DNN does not confer any robustness to Gaussian noise or to PGD. How to design efficient input codes to build encoded DNNs is an intriguing research direction for achieving generalization in deep learning. The literature on generalization, e.g. , is largely concerned with minimizing the generalization error, defined as the difference between training and test errors measured on identically distributed training and test sets. Minimizing this form of generalization error does not address the problem of generalizing to input samples drawn from an empirical sample set of which the training and test sets are not sufficiently representative, as we do herein. In this subsection, we compare our work with domain-generalization, domain-adaptation, and dataaugmentation techniques to highlight their differences. There is a substantial body of literature on domain generalization (; ; ;), which aims to better generalize to unknown domains by training on samples drawn from different domains, not a single source, which is a limitation that our work does not have. In this work, there is no need to draw training samples from a different domain. We show that encoding the given training set enables a DNN to learn different relations between features that it could not learn from the uncoded training set alone. There has also been much work on domain adaptation (Daumé ; ; ; ; ; ; a) that addresses the problem of generalization to a priori fixed target domains, which is a different approach from ours because these algorithms need to access samples from the target distributions during an adaptation phase. Importantly, our approach does not require accessing new samples during an adaptation phase in order to achieve generalization to the empirical sample set. Similar to the domain adaptation work, there has been some work on adversarial training (; ;), which aims to achieve robustness to adversarial perturbations by using training samples perturbed by a specific adversarial-perturbation method. Adversarial training can be computationally costly because it requires generating adversarially perturbed training samples in each epoch of training, unlike in our work where input encodings need to be generated only once before training. Furthermore, as there are numerous adversarial-perturbation methods (; b;), an adversarially trained DNN does not necessarily generalize well to samples subjected to an adversarial perturbation method that was not used for training . There is also a substantial body of work on data-augmentation techniques (; b), which perform simple label-preserving transformations of the training samples to provide a DNN with additional data points to learn from. In this work, we do not generate new samples to increase the diversity of the training set; instead, we take a theoretically-grounded approach to extend the input features with their encodings in order to enable a DNN to learn a sufficiently complex classification function from the set of available input samples. Our goal is to minimize the generalization error (defined in Appendix B) for a classification task, defined as the difference between training error and inference error, given a training set and a test set, both of which are not sufficiently representative of the empirical sample set from which input samples are drawn at inference time. To accomplish this goal, we derive a necessary and sufficient condition under which a classifier will generalize well, and, based on that condition, cast the search for a classifier with good generalization (defined in Appendix B) as an optimization problem. Our approach requires that we describe and compute the absolute information content of any object, e.g., a computer program, function, or set, in order to determine which of a pair of learned classification functions contains more information of the true classification function. The appropriate tool here is a concept in algorithmic information theory: Kolmogorov complexity (defined in Appendix B). Defining the amount of information in individual objects in terms of their Kolmogorov complexity has the advantage that it refers to these objects in isolation, not as outcomes of a known random source. In contrast, quantifying the amount of information in individual objects based on their Shannon entropy requires that these objects be treated as members of a set of objects with an associated probability distribution. This understanding is fundamental to our study because applying Shannon entropy to "an estimate of the quantity of information contained in a novel or in the translation of a novel into another language relative to the original" would not be clear . As a DNN may be employed to learn a classification function from a set of features contained in such an object as, for example, a document, image, video, or sound, we study the Kolmogorov complexity of the set of input features, model, and outputs of the DNN. In our quest to find a condition ensuring our running definition of generalization, we require a distance function that measures how similar two objects are in any aspect so we can decide which of two learned classification functions is closer to the true classification function. The closer a learned classification function is to the true classification function, the better its generalization error. This distance function should satisfy the metric (in)equalities in order for it to have a meaning in the context of generalization. For example, this distance function would have to be symmetric; i.e., the distance from object a to object b must be equal to that from object b to object a. The normalized information distance between objects a and b, defined as where K(a) denotes the Kolmogorov complexity of object a and K(a|b) denotes the Kolmogorov complexity of object a given b, satisfies the metric (in)equalities and is also a universal cognitive similarity metric because D I (a, b) minorizes all other normalized admissible distances up to a negligible additive error term. This means that all effective similarities between a pair of objects are discovered by the normalized information distance; i.e., two objects that are close according to some effective similarity are also close according to the normalized information distance. The main intuition behind normalizing the information distance max(K(a|b), K(b|a)) is that two larger objects that differ by a small amount are closer than two smaller objects that are different by the same amount: the absolute difference between two objects does not measure similarity as such, but the relative difference does (Cilibrasi & Vitányi, 2005). A successful DNN distills information useful for its classification task T from its input features x. In doing so, the DNN has to learn a classification function f from the set X n of its input features to an m-ary alphabet A of classes u in such a way that some information in its input features is given less weight in determining its relevance to the class decisionû, and then entirely discarded by the arg max operation . A deep learning classifier is thus acting as a source code C (defined in Appendix B). Proofs of the following mathematical statements are given in Appendix A. Lemma 1. For a classification task T wherein each n-dimensional input sample x is mapped to a class u drawn from an m-ary signal alphabet A, the true output function f (·) of a learning algorithm is a source code C for a multivariate random variable X. Lemma 1 reformulates a learning algorithm as a procedure for searching for a source code C for a multivariate random variable X, which compresses the values that this random variable takes, namely the input samples x. When a DNN generalizes well with respect to the true classification function f (·), it is able to decide which information in its input features is more relevant to making a particular class decision. A DNN is a lossy compressor when the absolute information content of any of its input samples x is larger than that of the class u to which it is mapped. Corollary 1. The true source code C = f (·) of a learning algorithm used for the classification task T is a lossy compressor when the Kolmogorov complexity K(x) of one of its input samples is larger than the number of bits required to represent the corresponding class u. Corollary 1 formalizes a deep learning classifier as a lossy compressor, so the source code C that corresponds to the true output function f (·) is not uniquely decodable; i.e., its input samples x cannot be recovered from the class u to which they are mapped. A DNN can be trained to learn a source code that generalizes well with respect to the true source code, but first we will analyze the similarity between these two source codes by using the normalized information distance. Source codes are designed for the most efficient representation of data . Whether it is designed for a data-transmission or a data-storage system, a source code, whether lossless or lossy, should retain information about the data necessary to accomplish a given task. The same consideration applies to a learning system. The information in the input features of a learning system is represented by the classification function that it learns; thus, a neural network can be viewed as a source code that encodes inputs features for its classification task. The reformulation of a learning algorithm as a procedure for searching for a source code allow us to exploit theoretical from algorithmic information theory and coding theory for deep learning, thereby avoiding the necessity to reinvent theory that is already established in these fields. Given that source codes are designed for the most efficient representation of data , we will exploit the duality of a source code and a channel code to learn a classification function that represents the input features more efficiently for the classification task T; i.e., a more general classification function. Showing that a deep learning classifier is a non-uniquely decodable source code is also fundamental to understanding that the normalized information distance between the input features and the output cannot be used to derive a condition for generalization in deep learning. This from the fact that deriving such a condition would require finding the conditional Kolmogorov complexity K(x|y) of the input features with respect to the output, which is impossible because the source code is not uniquely decodable; i.e., the program to go from the output to the input features cannot be found. A necessary and sufficient condition for generalization based on the normalized information distance can hence be found only between a learned source code and the true source code. The normalized information distance between the true source code C and learned source codeC reveals how generalC is with respect to C. A necessary and sufficient condition ensuring that learned source codeC 0 is more general than learned source codeC 1 with respect to the true source code C is Equation 3 is a direct of using the normalized information distance as a universal cognitive similarity metric to determine whether learned source codeC 0 orC 1 is more general with respect to the true source code C. Because the normalized information distance is a metric that uncovers all effective similarities between the true source code and a learned source code, learning a source code that is closer to the true source code C under this metric ensures achieving generalization. The normalized information distance D I (C,C) between the true source code C and the learned source codeC must thus be minimized in order to minimize the generalization error. Theorem 1. When a learning algorithm used for the classification task T finds a suboptimal source codeC instead of the true source code C, the optimization problem for the generalization ofC is minC(D I (C,C)) = minC max(K(C|C), K(C|C)). Theorem 1 has formulated the optimization objective for generalization as the minimization of D I (C,C) and states that to achieve generalization we should make the learned function sufficiently complex for the classification task T. Theorem 1 states that the Kolmogorov complexity K(C|C) of the program that computes how to go from the learned source codeC to the true source code C or the Kolmogorov complexity K(C|C) of the program that computes how to go from the true source code C to the learned source codeC, whichever is larger, must be minimized in order to minimize generalization error. Thus, the goal is to increase the complexity of the learned source codeC, but not beyond the complexity of the true source code C. Therefore, Occam's first razor still holds: simpler classifiers generalize better than complex ones. However, a classifier that input features encoded input features Encoded Model does not perform well on its empirical sample set X n is too simple for its classification task. Ideally, the learning algorithm would learn the true source code C, achieving the best possible performance metrics determined by its classification task T. In practice, because the learning algorithm will see only a small subset X n S of the possible inputs at training time, the learned source codeC will be a partial function of the true source code C at perfect training accuracy (that is, when the classifier has sufficient capacity to memorize the training samples ). Whether a model is over-fit or under-fit is conventionally determined on a cross-validation set and/or test set that are/is identically distributed with the training set, all of which are subsets of the empirical sample set. Being more general on such a cross-validation set and/or test set does not as such guarantee generalization on the empirical sample set X n because the latter may contain corrupted or perturbed samples and/or there may be samples in the empirical sample set that are out of distribution of the cross-validation set and test set. Therefore, whether a model is over-fit or under-fit does not have a consequence for Theorem 1. Next, we target learning a source code that is more general on the empirical sample set X n, not only on a cross-validation set and/or test set. In this work, we increase the complexity of the learned source codeC by generating I encodings E 0, E 1,... E I−1 of the available input features x S that capture relations between the features which are not learned well from the original features, and then append these encodings to the original features. Note that the available input features are denoted by x S, which are drawn from the set X n S of available features; i.e., x S ∈ X n S, which is a subset of the empirical sample set X n. By providing a different view of the relations between the features, the encodings E i help the learning algorithm to learn a more complex source codeC E whose normalized information distance D I (C,C E) to the true source code C is less than D I (C,C). This in learning a more general source code. Theorem 2. For classification task T, a more general suboptimal codeC E is learned from the concatenation {x S, E i ( x S)}, where The effective capacity of several successful DNN architectures is sufficiently large to memorize the set X n S of available input samples . Any encoding is the set of available encoded samples such that Y n S ⊆ X n S, when concatenated with the uncoded input samples x S, thus increases the Kolmogorov complexity of the learned source code, which is now calledC E. The task of the source code is to find the most efficient representation of its input data. In a typical communication system, the source code compresses the input, then a channel code adds redundancy to guard against noise in the channel, then the encoded information is transmitted over the physical channel. The design goal for the source and channel codes is to achieve the channel capacity (maximum mutual information between the channel input and output). In contrast, Theorem 2 considers a learning system in which an input code is followed by a learned source code, the classification function, and the design goal is for the composition of the input and source codes to generalize as well as possible (see Figure 1). In other words, in a learning system the "physical channel" precedes the source code, and it can be seen as a process whereby the empirical sample set X n is reduced to the set X n S of available input samples and/or whereby common corruptions, such as Gaussian noise, and adversarial perturbations, such as those generated by PGD, are applied to the set of available input samples. Because the "physical channel" comes first in a learning system, there is no access to the set of information bits. Only a subset of these information bits can be accessed, which may have been subjected to common corruptions or adversarial perturbations. It is therefore crucial for a learning algorithm to compress its features while retaining information useful for its classification task. One way to accomplish this is to extend the input features with encodings that capture relations between features that are useful for classification and not captured well by the original set of input features. The classification task T does not change when input features are extended by their encodings because it is defined by the mapping between the input features and the output , which remains the same because the only input to the encoded model is the uncoded input features (see Figure 1). The encoder is simply a new layer in the encoded model, which is designed from an encoder and an uncoded model. The normalized information distance is based on the notion of Kolmogorov complexity, which is not a partial recursive function; i.e., it is not effectively computable. While we can use normalized information distance to analyze whether a source codeC E learned from the concatenation {x S, E i ( x S)} of the encoded input samples E i (x S) with the uncoded input samples x S is more general with respect to the true source code C, in practice we may need to approximate the normalized information distance with the normalized compression distance, so we can determine which of any pair of source codes is more general with respect to the true source code C. Based on a real-world compressor, the normalized compression distance (Cilibrasi & Vitányi, 2005) approximates the normalized information distance D I (C,C E), where Z is a real-world compressor. Thus, the generalization condition and minimization of D I (C,C E) can be cast in effectively computable forms. Note that neither Equation 2 nor Equation 4 is a training criterion, but they specify the normalized information distance and the normalized compression distance between the true source code C and a learned source code, respectively. They are used to derive theoretical , particularly the use of input codes to achieve generalization as illustrated by experiments in Section 3. Proposition 1 states for classification task T that the compressed size Z(C E) of the source codeC E learned from the concatenation {x S, E i ( x S)} of the encoded input samples E i (x S) and the uncoded input samples x S is larger than the compressed size Z(C) of the source codeC learned from the uncoded input samples alone x S. Proposition 2. When a learning algorithm used for classification task T finds a suboptimal source codeC E instead of the true source code C, the effectively computable optimization problem for the generalization ofC E is minC Proposition 2 shows that the compressed size Z(C E) of the source codeC E learned from the concatenation {x S, E i ( x S)} of the encoded input samples E i (x S) and the uncoded input samples x S must be maximized until it reaches the compressed size Z(C) of the true source code C to learn the most general source code with respect to the true source code C for the classification task T. This statement is a consequence of the fact thatC E is a partial function of C at perfect training accuracy. In other words, the source codeC E learned from the concatenation {x S, E i ( x S)} of the encoded input samples E i (x S) and the uncoded input samples x S can be made more general if the encoded input samples E i (x S) bear information of relations between input features that are not represented by its input samples. A channel encoder generates encodings from its input features that enable a classifier to learn relations between these features not captured by the set of available input samples. Concatenated together, these features are then input to a model to produce a class decision. For example, we use a 4-D 5-PAM TCM scheme (; ;) as a systematic way to generate multiple encodings of input features. As shown in Figure 2, the channel encoder flattens the input features by them into 2 × 2 patches of features, then, starting from the upper left feature and ending at the lower left feature, ordering them in a sequence going in the clockwise direction. The features are traversed twice in order to avoid the initialization length of the channel code. This particular scheme is used because it focuses on local relations between features. Exploration of other flattening schemes is left for future research. The features in the CIFAR-10 dataset are represented by eight bits. The flattened features are fed to the convolutional encoder, which produces one extra bit out of the two least significant bits of the eight bits representing each feature. The 4-D 5-PAM TCM symbol mapper then maps each nine bits into four equidistant 5-PAM symbols, which are then mapped to 12 bits by the bit mapper. The bit mapper uses different symbol-to-bit mappings to generate different encodings of the input features, and the matrix used for generating these encodings is given in Appendix C.7. Each encoding has the same size as the original input samples. Figure 5 in Appendix C.1 shows three CIFAR-10 images and four of their encodings, which are arbitrarily chosen. As seen in this figure, each encoding conveys a different view of the input features, which helps the source code (learned classification function) model relations between the features that are useful for the image classification task. Note that using channel codes on the input features is not a data-augmentation technique: the encodings are appended to the input features, not treated as new input samples. These encodings enable the classifier to learn from the set of available input samples a source code that is sufficiently complex for its classification task. As in a data-transmission or data-storage system, the source code is designed for the most efficient representation of the data, which is the set of available input features for the classification task at hand, and the channel code is independently designed for the channel. This combination is key to achieving generalization in deep learning, and how best to design a channel code for a given classification task is an intriguing future research direction. Let the set of available input samples subjected to common corruptions and adversarial perturbations belong to the empirical sample space from which input samples are drawn during inference. To show that using channel codes on the input features in learning a more general source code with respect to the true source code, we conduct experiments on the CIFAR-10 and CIFAR-10-C datasets to show increased robustness to common corruptions and adversarial perturbations. For CIFAR-10 and CIFAR-10-C, we train uncoded VGG-11 and VGG-16 models, encoded VGG-11 and VGG-16 models, and an uncoded ResNet-18 model. The VGG networks are modified only by adding the encoder and increasing the number of input channels. The encoded models use the same training criterion as the uncoded models, namely the cross-entropy loss. The training setup and the achieved test accuracies are given in Appendix C.2. In all experiments conducted on the encoded models, we use arbitrary encodings. The input samples are corrupted or perturbed before they are input to the encoded models, as the uncorrupted or unperturbed input samples would not be accessible by a neural network in a real-world application. Increasing the number of encodings may reduce the generalization error, but at the expense of increased run time. However, encoding the training and test samples is a one-time process that can be done prior to training, unlike adversarial training, which requires generating perturbed input samples in each epoch. In Appendix C.6, we show that increasing the number of input channels does not, as such, confer robustness to Gaussian noise or to PGD. Designing efficient input codes for a given classification task considering the generalization error and the required number of encodings is a direction for future research. To the best of the authors' knowledge, there is no other published method that can achieve robustness to both common corruptions and adversarial perturbations. The set of available input samples may be subjected to common corruptions before reaching a realworld image classifier. For example, Gaussian noise can appear in low-lighting conditions, and shot noise is caused by the discrete nature of light. To show robustness to such corruptions, we conduct experiments on the CIFAR-10-C and CIFAR-10 datasets. We use four common corruptions in our experiments, namely Gaussian noise, shot noise, impulse noise, and speckle noise. The CIFAR-10-C dataset consists of the 10,000-sample CIFAR-10 test set subjected to five different noise levels, called severity, so it has 50,000 samples in all. As shown in Figure 3, increasing the number of arbitrary encodings concatenated to the original input features increases robustness to Gaussian noise, shot noise, and impulse noise. The for speckle noise is given in Appendix C.3. For example, when test samples are subjected to impulse noise with a severity level of 4, we see a sharper increase in the number of test errors for the uncoded VGG-11 model than that for the VGG-11 model with 32 encodings. Note that the vertical axis in these plots is cumulative: the number of test errors made at the previous severity level is added to that at the current severity level. Table 1 in Appendix C.4 compares the encoded VGG-11 model with 32 encodings with previously published methods on the CIFAR-10-C dataset, which shows that the encoded VGG-11 model achieves the highest inference accuracy (defined in Appendix B) against shot noise with a severity level of 5 compared with all the other works listed in this table. Additional experimental on Gaussian noise are shown in Figure 6 in Appendix C.3. To show robustness to adversarial perturbations without adversarial training, we conduct experiments on the CIFAR-10 dataset. We use the white-box PGD attack and transfer attacks from an uncoded VGG-16 and an uncoded ResNet-18 model to evaluate the adversarial robustness of the encoded VGG-16 models. The for the black-box boundary attack are given in Appendix C.3. The white-box PGD attacks use the gradient of the loss function with respect to the uncoded input features in the encoded VGG-16 models because the channel encoder is part of the encoded VGG-16 models; i.e., the only input to the encoded model is the uncoded input features. The encoder is simply a new layer of the neural network architecture, whose outputs are computed directly from the uncoded input features. Changing the outputs of the encoder layer is tantamount to changing the outputs of any other layer of the model, which is a threat model that falls out of the scope of our work. For the CIFAR-10 experiments, we use different numbers of encodings, and robustness to all adversarial perturbations in our experiments systemically increased with an increasing number of arbitrary encodings concatenated to the input features. Figure 4 shows the for the white-box PGD and transfer PGD attacks. The plot on the left shows the increase in robustness to white-box PGD starting from a random perturbation around the natural example and using 20 iterations and a step size of 0.003. . The difficult problem of achieving adversarial robustness in a real-world application requires a holistic approach. Our approach, which does not depend on adversarial training, can be readily used in combination with adversarial training and other known methods to achieve greater robustness to adversarial perturbations. We presented a theoretical and experimental framework for defining and understanding generalization in deep learning, defined as the difference between training and inference errors. The theoretical findings and experimental show that a learned classification function must be sufficiently complex for a classification task in order to be closer to the true classification function. Another insight from this study is that concatenating encodings of input features to the original input features helps to achieve generalization in deep learning by enabling the classifier to learn relations between features not captured by the original inputs. Experiments demonstrate that a model trained on arbitrarily encoded input features is more robust to common corruptions and adversarial perturbations and that using more encodings may be beneficial to minimize the generalization error. Designing input codes to help a DNN learn a more general classification function with a minimum number of encodings is an intriguing research direction to achieve reliability in machine learning. Proof of Lemma 1. For classification task T, a learning algorithm is asked to produce the true output function f (·): X n → A. There exists a source code C for a random variable X, which is also a mapping from the sample space X n of X to the m-ary signal alphabet A from which a class u is drawn. The true output function f (·) is equivalent to the source code C for the random variable X because their domain X n and codomain A are equal and the image of both functions is the same for each input sample x in the domain X n. Proof of Corollary 1. If the Kolmogorov complexity K(x) of an input sample x is larger than the number of bits required to describe the class u to which it is mapped, which is at most log 2 m bits, then some information about the input sample x is lost. Satisfying this condition, the true source code C is a lossy compressor. Proof of Theorem 1. The normalized information distance is a universal cognitive similarity metric that minorizes all other admissible distances up to a negligible additive error term. This means that decreasing the normalized information distance D I (C,C) ensures that the true source code C and the learned source codeC are more similar; i.e., the learned source codeC is more general with respect to the true source code C. In a real-world setting, because the empirical sample space X n may be too large, the learning algorithm sees an input sample x S drawn from a subset X n S of X n; i.e., X n S ⊂ X n. Put differently, the set X n S of available input samples on which a neural network is trained and tested is a subset of the empirical sample set X n which the trained neural network sees during inference. This means that true source code C bears information of all possible relations between input features that are useful for the classification task T, whereas the learned source codeC bears information of a subset of all possible relations between the input features. The Kolmogorov complexity of the true source code is thus larger than that of a source code learned from the set of available input samples by a sufficiently high-capacity neural network, which can memorize its input samples ; i.e., K(C) > K(C). Therefore, is an optimization problem for the generalization of the learned source codeC with respect to the true source code C. Proof of Theorem 2. Any encoding E i: X n S → Y n S that bears information useful for the classification task T that is not entirely represented by the subset X n S of uncoded input samples; i.e., Y n S ⊆ X n S, when concatenated with the uncoded input samples x S, increases the Kolmogorov complexity of the learned source code, which is now calledC E, because a sufficiently high-capacity neural network can memorize its input samples . Put differently, the Kolmogorov complexity K(C E) of the source codeC E learned from a concatenation {x S, E i ( x S)} of uncoded and encoded input samples is larger than that of the source codeC learned from uncoded input samples alone if the encodings bear information of relations between input features that are not represented by the uncoded input samples. As the the true source code C bears information of all possible relations between input features, the Kolmogorov complexity K(C E) of the source codeC E learned from a concatenation {x S, E i ( x S)} of uncoded input samples and their encodings bearing information of a subset of all possible relations between input features is upper bounded by the Kolmogorov complexity K(C) of the true source code C; i.e., K(C) > K(C E). In other words, a sufficiently high-capacity neural network can memorize its input samples without being assisted by encodings E i. However, the encodings E i bear information of relations between input features, which help to increase the Kolmogorov complexity of the learned source code if they are useful for the classification task T; i.e., they are contained in the empirical sample set X n, of the neural network and if the information in the mappings contained in the input code, which is used to generate the encodings E i, is not represented in the set X n S of available input samples. The conditional Kolmogorov complexities {K(C|C), K(C|C)} are thus both larger than {K(C|C E), K(C E |C)}, respectively, because the program that computes how to go fromC E to C is shorter in length than the program that computes how to go fromC to C. The same holds in the reverse direction. Therefore, The source codeC E learned from the concatenation {x S, E i ( x S)} is thus more general than the source codeC learned from x S. Proof of Proposition 1. As the normalized information distance D I (C,C E) is not effectively computable, it can be approximated for practical purposes by the normalized compression distance where Z is a real-world compressor. The learning algorithm sees an input sample x S drawn from a subset X n S of X n as the empirical sample space X n may be too large. Because a sufficiently high-capacity neural network can memorize its input samples , the compressed size of the true source code is larger than that of the learned source code; i.e., Z(C) > Z(C E). At perfect training accuracy, the compressed size Z({C,C E}) of the concatenation {C,C E} is equal to Z(C), asC E is a partial function of C. For a sufficiently high training accuracy, we can consider |Z({C,C E}) − Z(C)| to be negligible for the purposes of generalization. As the generalization condition D I (C,C E) < D I (C,C) is not effectively computable, an equivalent effectively computable condition is useful for practical purposes. As for the purposes of generalization, the effectively computable condition is equivalent to Proof of Proposition 2. By the Proof of Proposition 1, the effectively computable optimization problem for the generalization ofC E with respect to C is B DEFINITIONS Inference Accuracy. The classification accuracy measured on a subset of the empirical sample set X n, which may be subjected to common corruptions or adversarial perturbations and which may be out of distribution of the training set, is defined as inference accuracy. The definition of inference accuracy can be contrasted with that of test accuracy by considering that the former is measured on a subset of the empirical sample set X n which consists of corrupted or perturbed samples which may be out of distribution of the training set and that the latter is measured on the test set which consists of uncorrupted and unperturbed samples that are presumed to come from the same distribution as the training set. Generalization Error. The difference between the training error measured on the training set and inference error measured on a subset of the empirical sample set X n, which may be subjected to common corruptions or adversarial perturbations and which may be out of distribution of the training set, is defined as the generalization error. This definition is different from that of prior works , which define generalization error as the difference between the training error measured on the training set and test error measured on the test set. Generalization. A learned classification function is said to be more general with a decreasing generalization error. This definition is different from that of prior works , which define a learned classification function to be more general with a decreasing difference between the training error measured on the training set and test error measured on the test set. Source Code. A source code C for a random variable X is a mapping from the sample space X n of X to an m-ary signal alphabet A. Source codes can be designed for the most efficient representation of the data . Channel codes appropriate for a channel can be designed separately and independently. This combination is as efficient as any other method that can be designed by considering both problems together. We refer the reader to a textbook such as for a detailed understanding of source codes. Kolmogorov Complexity. The Kolmogorov complexity K U (x) of a string x with respect to a universal computer U is defined as where p denotes a program, and l(p) denotes the length of the program p. Thus, K U (x) is the shortest description length of x over all descriptions interpreted by computer U. We fix such a universal computer U as reference and write K U (x) = K(x). We refer the reader to a textbook such as and (; Cilibrasi & Vitányi, 2005) for a detailed understanding of Kolmogorov complexity. C.1 ENCODED CIFAR-10 IMAGES Figure 5 shows three CIFAR-10 images and four of their encodings, which are arbitrarily chosen. This figure shows that each encoding conveys a different view of the input features, which helps the learned source code model relations between the features that are useful for the image classification task. All models are trained in PyTorch with 16 random initializations. We train the networks over 450 epochs with a batch size of 128 and with a dynamic learning rate equal to 0.1 until epoch 150, 0.01 until epoch 250, and 0.001 until epoch 450 (To show the robustness of the encoded VGG-11 models to Gaussian noise beyond the noise levels included in the CIFAR-10-C dataset, we apply Gaussian noise with zero mean and variance σ 2 w to the CIFAR-10 test set. The average input-feature energy equals where x i is a feature of the input sample x, k is the number of input samples in the test set, and n is the number of features in an input sample. We define the signal-to-noise ratio to be In Figure 6, we show on the left plot that increasing the number of arbitrary encodings concatenated to the input features significantly increases robustness to Gaussian noise applied to the CIFAR-10 test set with signal-to-noise ratios from 25 to 0 dB. For example, at a signal-to-noise ratio of 12 dB, the inference accuracy of the VGG-11 model with 32 encodings is 61.15%, whereas that of the uncoded VGG-11 model is 21.49%. On the right plot, the experimental for the VGG-16 model with 32 encodings tested on the samples in the CIFAR-10-C dataset corrupted by Gaussian noise and shot noise are given. The indicate that using a larger encoded model does not necessarily confer more robustness to such common corruptions as Gaussian noise and shot noise than a smaller encoded model. In Figure 7, the of the experiments conducted on the CIFAR-10-C dataset corrupted by speckle noise and black-box boundary attack experiments on the CIFAR-10 dataset are shown. On the right plot, as with the other type of common-corruption experiments, we see that increasing the number of encodings increases robustness to speckle noise. On the left plot, we see that the encoded model is significantly more robust to the boundary attack than the uncoded model. For example, at a normalized 2 distance of 0.01, an inference accuracy of approximately 50% is achieved by the model with 32 encodings, whereas the inference accuracy of the uncoded model already drops to 0% at an 2 distance much closer to 0. Table 1 compares the encoded VGG-11 model with 32 encodings with previously published methods on the CIFAR-10-C dataset. At a severity level of 5, the encoded VGG-11 model achieves the highest inference accuracy against shot noise compared with all the other works listed in this table, which use a ResNet-18 or ResNet-26 model. The highest inference accuracy (77.30%) against Gaussian noise is attained by the adversarial logit pairing (ALT) method published in , but the test accuracy of this method is 83.50%, whereas the encoded VGG-11 model achieves the second highest inference accuracy (75.22%) against Gaussian noise with a test accuracy of 90.19%. Our seem to indicate that using a larger number of encodings improves robustness to common corruptions, so the inference accuracy achieved by the channel-coding method may be improved by merely increasing the number of encodings or designing higher performance codes. C.5 PERFORMANCE COMPARISON: ADVERSARIAL PERTURBATIONS Table 2 compares the encoded VGG-16 model with previously published defenses on the CIFAR-10 dataset. The experimental do not imply that encoded input features are more robust to common corruptions and adversarial perturbations than uncoded features. The encoded input features simply bear information of relations between input features that are not represented by the uncoded input features, so a source code learned from a concatenation of uncoded and encoded input features bears more information of the true source code than a source code learned from uncoded input features alone. A source code learned from a concatenation of uncoded and encoded input features is thus more robust than a source code learned from uncoded input features alone. To study the impact of increasing the number of input channels of the uncoded VGG-11 and VGG-16 models, we conducted experiments on the encoded VGG-11 and and VGG-16 models that use identical encodings; i.e., the input features are replicated across additional input channels (the "encoders" are just identity functions). In Figure 8, we see on the left that increasing the number of input channels of the uncoded VGG-11 model confers no robustness to Gaussian noise whatsoever. The plot on the right shows that increasing the number of input channels of the uncoded VGG-16 model does not confer robustness to white-box PGD either. Robustness to Gaussian noise (left) and the PGD attack (right) is tested by providing identical samples from the CIFAR-10 test set to the increased number of input channels. The bit mapper in Figure 2 uses the matrix to map four 5-PAM symbols into 12 bits. In this symbol-to-bit mapping matrix, the i th row corresponds to the encoding E i, where 0 ≤ i ≤ 31. Each symbol in the 5-PAM symbol alphabet is converted into three bits by using the corresponding three columns in this matrix. For example, the first symbol in the 5-PAM symbol alphabet for the encoding E 3 is converted to by drawing the bits from the third row and third, fourth, and fifth columns of the symbol-to-bit mapping matrix. After all four of the 5-PAM symbols are converted into their respective three bits, these bits are concatenated to each other, determining the value of the corresponding feature in the encoded sample.

Represent the short summary for finding the corresponding scientific article (long):

We present a theoretical and experimental framework for defining, understanding, and achieving generalization, and as a result robustness, in deep learning by drawing on algorithmic information theory and coding theory.

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: Represent the article for finding a single-sentence TLDR.

Question: Many approaches to causal discovery are limited by their inability to discriminate between Markov equivalent graphs given only observational data. We formulate causal discovery as a marginal likelihood based Bayesian model selection problem. We adopt a parameterization based on the notion of the independence of causal mechanisms which renders Markov equivalent graphs distinguishable. We complement this with an empirical Bayesian approach to setting priors so that the actual underlying causal graph is assigned a higher marginal likelihood than its alternatives. Adopting a Bayesian approach also allows for straightforward modeling of unobserved confounding variables, for which we provide a variational algorithm to approximate the marginal likelihood, since this desirable feat renders the computation of the marginal likelihood intractable. We believe that the Bayesian approach to causal discovery both allows the rich methodology of Bayesian inference to be used in various difficult aspects of this problem and provides a unifying framework to causal discovery research. We demonstrate promising in experiments conducted on real data, supporting our modeling approach and our inference methodology. Causal networks (CNs) are special Bayesian networks where all edges reflect causal relations . The aim of causal structure learning is identifying the CN underlying the observed data. In this paper, we focus on the problem of scoring causal graphs using marginal likelihood in a way that identifies the unique causal generative graph. Succeeding to do so is very valuable, since once the correct CN is selected, various causal inference tasks such as estimating causal effects or examining confounder distributions becomes straightforward in a Bayesian framework. A central challenge in such an attempt, however, is adopting a prior selection policy that not only allows discriminating between Markov equivalent graphs but also assigns higher marginal likelihood score to the actual underlying CN. The key notion underlying our solution to first part of this challenge is the widely accepted principle of independence of the cause-effect mechanisms , that is, the natural mechanisms that generate the cause and the effect (based on cause) must be independent of each other. We embody this assumption by assuming the mutual independence of the parameters pertaining to cause and effect distributions in a Bayesian model, a line of reasoning that is natural to this modeling perspective, where parameters are modeled as random variables (; ; ;). By assigning independent priors to the cause and effect variables, we render them statistically independent. Critically, this assignment of independent priors also breaks the likelihood equivalence between Markov equivalent graphs. This is contrast to other ways of selecting independent priors such as the BDeu prior, which leads to assigning equal marginal likelihood to Markov equivalent graphs . As mentioned above, though breaking likelihood equivalence does not necessarily lead to assigning a higher marginal likelihood to the actual underlying CN, it is a prerequisite for doing so 1. The second part of the problem is adapting a prior selection policy that leads to assigning a higher marginal likelihood to the actual CN compared to its alternatives. In this work, we use an empirical Bayesian approach in selecting the hyperparameters of the independent priors described above, as we learn the priors that lead to assigning higher marginal likelihood to the actual CN from labeled data. The current approach is in the intersection of various other approaches in the literature, thereby combining many of their respective advantages . It is based on the notion of mechanism independence similar to; , does not assume causal sufficiency similar to;; Janzing et al. (, 2012 ; ; Schölkopf et al., can theoretically work on arbitrary graph structures that possibly include latent variables similar to , and can discriminate between Markov equivalent structures similar to; Zhang and Hyvärinen;;;. Our approach diverges from other Bayesian methods (; ;) in various dimensions such as by being able to distinguish between Markov equivalent causal graphs, using marginal likelihood (or approximations thereof) instead of surrogate scores such as BIC, or being able to model non-linear relationships. In Section 2, we introduce an example model for continuous observations and latent categorical confounders. To approximate the marginal likelihood in graphs which include latent confounders, we present a variational inference algorithm in Section 3. After testing our approach on various real data sets in Section 4, we present our and further avenues of research in Section 5. A general causal graph G(V G, E G) is a combination of a vertex set V G, which is the set of observed and latent random variables, and a set of directed edges E G ⊆ V G × V G where directed edges imply immediate cause-effect relationships between these variables. Let {x 1, . . ., x n, . . ., x N} ⊆ V G denote the set of continuous random variables, and similarly {r 1 . . ., r k, . . ., r K} ⊆ V G denote the discrete latent variables of the network where each x n and each r k are defined in the domains X n and R k, respectively. The set of parent vertices of a vertex v ∈ V G is denoted by π(v), while we denote its continuous parents by x π(v), and discrete parents by r π (v). For the scope of this text, we specify conditional distributions for the graphs as follows: we assume categorical distributions on the discrete variables r 1:K and linear basis functions models with Gaussian noise on the continuous variables x 1:N. Though these choices are by no means mandatory for our framework, we define latent variables as categorical. Furthermore, we restrict our attention to the graphical structures that do not include a continuous variable as a parent of a categorical variable for inferential convenience , and construct the following generative model for T independent and identically distributed observations from the network G: where 1 ≤ t ≤ T, φ is an arbitrary basis function with the convention φ({}) = 1, and's are the parameters of the conditional distributions. Namely, θ k is the conditional distribution table of r k, w n is the weights of the basis functions, and ρ n is the precision parameter of the conditional distribution of x n. Notice that declaring parameters as random variables simplifies the notion of independent cause-effect mechanisms as follows: Since the conditional distributions are the functions of the parameters, independence of the conditional distributions boils down to the independence of the parameters. Therefore, we complete our generative model by defining independent conjugate prior distributions on the parameters ∀n, r π(xn): where γ k|r π(r k), m n|r π(xn), Λ n|r π(xn), a n|r π(xn), b n|r π(xn) are the prior parameters, i.e. hyperparameters, of our generative model. Variational Bayesian inference (VB) is a technique where an intractable posterior distribution P is approximated by a variational distribution Q via minimizing Kullback-Leibler divergence KL(Q||P). In the context of Bayesian model selection, minimization of the KL(Q||P) corresponds to establishing a tight lower bound for the marginal log-likelihood, which we refer to as evidence lower bound (ELBO). This correspondence is due to the following decomposition of marginal log-likelihood log p(x where P = p(r 1:T 1:K, θ 1:K, ρ 1:N, w 1:N | x 1:T 1:N) is the full posterior distribution, and ELBO is denoted by B P [Q]. In a typical scenario of VB, Q is assumed to be a member of a restricted family of distributions. In its most common form, also known as mean-field approximation, Q is assumed to factorize over some partition of the latent variables, in a way that is reminiscent to a rank-one approximation in the space of distributions Q(r 1:T 1:K, θ 1:K, ρ 1:N, w 1:N) = q(r 1:T 1:K) q(θ 1:K, ρ 1:N, w 1:N) ELBO is then maximized with respect to Q which is restricted to the class of factorized distributions. Due to conjugacy, maximization of Q in further factorized variational distributions which also belong to the same family as the prior To calculate variational parameter updates, we need to calculate the expected sufficient statistics. In its final form, our variational algorithm becomes equivalent to iteratively calculating the expected sufficient statistics and updating the parameters. The explicit forms for the variational parameters and ELBO can be found in Appendix C. In Section 4.1 we test the performance of our approach in bivariate causal discovery. Then in Section 4.2 we identify the cardinality and distribution of a latent confounder in a multivariate data set, exemplifying the versatility of a Bayesian approach to causality. In the first part we measured the accuracy of VB for the causal direction determination problem. The data set in this part is CEP , frequently used in causal discovery research, which includes 100 data sets, vast majority of which is bivariate. For the hyperparameters of the model, we created 36 different settings by varying the critical hyperparameters systematically. We detail this hyperparameter creation process in the Appendix D.1. In making a decision between two causal directions in a given hyperparameter setting, we choose the model which obtains a higher ELBO 2. We tested our algorithm on the data set by using 10 × 3 cross-validation. That is, for each test, we separated the data set into three, detected the hyperparameter setting (of 36) that obtained the best accuracy score on the first two thirds, and tested our model on the last third of the data set, which corresponds to an empirical Bayesian approach to prior selection. We conducted the same process two more times, each fold becoming the test set once. We conducted this split and tested randomly 10 times. We report the accuracy and AUC values according to these 10 runs. the CEP data set, we obtained a mean accuracy of.78±.09 and AUC score of.84±.13 (the values following the mean values correspond to 68% CI) where the accuracy and AUC calculations are performed by using the weights mentioned by. also compared most recent methods on their performance on the data set; our correspond to a state-of-the-art performance in bivariate causality detection. Using a different data set, we next examine the ability of our approach to identify a latent confounder. For this purpose, we use the smallest database in the Thyroid data set from the UCI repository . This data involves five different diagnostic measurements from patients with low, normal, and high thyroid activity. This being a diagnostic data set, the causal structure is known, where the thyroid activity is the cause of the rest of the variables (Figure 1(a) ). In our experiments we ignore the thyroid activity variable, thus it becomes a latent confounder. This way we can test how well our approach identifies the latent confounder. To assess our method's performance, we first examine whether the latent variable cardinality our method favors corresponds to the cardinality of the actual variable that we held out. Figure 1 (b) shows that the ELBO of the model is maximized at the latent cardinality which corresponds to the actual cardinality of thyroid activity variable (which is 3). Then, to ascertain that the inferred latent variable indeed corresponds to thyroid activity variable, we compare the assignments of our model to actual patient thyroid activity levels. The demonstrate an accuracy of.93, thus we conclude that our method accurately identified the latent causal variable. Overall, we show that Bayesian model selection is a promising framework that can facilitate causal research significantly both through conceptual unification and increased performance. Given that Bayesian modeling is agnostic to specific variable types, conditional distributions, and to approximate inference methodology, the value of a successful Bayesian modeling approach for causal research is immense. Though our empirical Bayesian approach to setting priors can be useful in various contexts (e.g. in data sets where only some of the bivariate causal directions are known), finding other principled ways of assigning (or integrating out) priors that do not require labeled data is an important direction for future research. Conducting causal discovery with different variable types, and/or different distributions would also be beneficial for demonstrating current approach's viability in various contexts. When constructing a generative model for causal inference, our aim is making Markov equivalent graph structures identifiable. However, the model that is described only by Equations and is not necessarily identifiable . To be more precise, consider the case where we have two continuous variables and no latent categorical variable, which is equivalent to the following structural equation model: One can also construct the following equivalent structural equation model in which the dependence structure is reversed: These two models are not identifiable with the descriptions above, since they both correspond to linear models with Gaussian noise. However, by assuming priors on the parameters we can break the symmetry and make these Markov equivalent models identifiable. For instance, assuming Gaussian priors on the weights of the first model implies non-Gaussian priors on the second model, which makes these two models distribution inequivalent . Moreover, even when two Markov equivalent models are also distribution equivalent, choosing appropriate prior parameters that violate likelihood equivalence still makes them identifiable . Indeed, for a model with a parameterization as described, only a very specific choice of priors leads to likelihood equivalence between the Markov equivalent models , and we will avoid following such a constraint. Choosing arbitrary priors almost always leads to likelihood inequivalent, hence identifiable models. In this section, we define the appropriate graphical structures for causal structure learning in the bivariate case. As we stated in Section 1, we do not assume causal sufficiency and allow the existence of possibly many exogenous variables. Luckily, we can combine the effects of exogenous variables into a single latent variable with an arbitrary cardinality. As a , the relationship between two observable dependent variables x 1 and x 2 boils down to one of three cases due to causal Markov condition : 2. x 2 causes x 1, 3. they do not cause each other, but a latent variable r 1 causes both of them. Associated causal networks corresponding to each of these hypotheses are depicted in Figure 2, where latent variable r 1 represents the overall effect of the all unobserved variables. For the spurious relationship (Figure 2(a) ), marginally correlated variables x 1 and (a) Spurious correlation. Figure 2: Graphical models for bivariate causality. x 2 become independent once the latent common cause variable r 1 is known. However in direct causal relationships (Figures 2(b) and 2(c)), even when the latent common cause is known, two variables are still dependent and the direction of cause-effect relationship is implicit in the parameterization of the models. The identifiability of these models resides in the fact that modelling parameters explicitly as random variables makes these graphs Markov inequivalent. If we were considering only the marginal models of the observed variables, then we would end up with three Markov equivalent graphs. However, including latent variables and independent parameters renders distinctive conditional independence properties for each graph. For instance, when x 2 and r 1 are known, x 1 and the parameters of x 2 are dependent only in the case of x 1 → x 2, or knowing r 1 makes x 1 and x 2 independent only if they have a spurious relationship. These distinctive conditional independence properties are the underlying reasons making all of these graphs identifiable. In this section, we supply the brief descriptions of the basic distributions that we mentioned in the main part of the manuscript. which is equal to (z − 1)! for nonnegative integer z. Gamma(ρ; a, b) = exp((a − 1) log ρ − bρ − log Γ(a) + a log b) where a is the shape and b is the rate parameter. 3. Expected sufficient statistics: 4. Cross entropy: Here, ψ(x) is the digamma function which is defined as ψ(x) = d log Γ(x) dx. 1. Multivariate Beta function: 2. Dirichlet density: 4. Cross entropy: where µ is the mean parameter and ρ is the precision parameter, i.e. ρ −1 is the variance. B.1.5. Multivariate Normal Distribution 1. Multivariate Normal density: where µ is the mean vector and Λ is the precision matrix, i.e. Λ −1 is the covariance matrix. for any symmetric matrix A. B.1.6. Normal-Gamma Distribution 1. Normal-Gamma density: which can be equivalently decomposed into a marginal Gamma distribution and a conditional Normal distribution: 2. Expected sufficient statistics: 3. Cross entropy: B.1.7. Multivariate Normal-Gamma Distribution 1. Multivariate Normal-Gamma density: which can be equivalently decomposed into a marginal Gamma distribution and a conditional Multivariate Normal distribution: 2. Expected sufficient statistics: for any symmetric matrix A. E N G(m,Λ,â,b) {− log N G(w, ρ; m, Λ, a, b)} = −a log b + log Γ(a) − 1 2 log det(Λ) + 1 2 tr(Λ −1 Λ) + M 2 log 2π − a + M 2 − 1 (ψ(â) − logb) +â b b +â 2b (m − m) T Λ(m − m) In this section we summarize the basic conjugate models that are closely related to our example model. 2. Posterior of θ: where γ * r = γ r + 2. Posterior of µ and ρ: where 1. Generative model: An equivalent description with Normal-Gamma priors is 2. Posterior of w and ρ: where In this section, we will explicitly evaluate these equations to derive closed form expressions for the variational posteriors: 1. We first simplify the In order to keep the notation uncluttered, from now on we will omit the implicit subscripts in expectation operators. So each individual factor q(r t 1:K) above is equal to 2. We now pursue the same strategy for the expression in q(θ 1:K, ρ 1:N, w 1: where each individual factor turns out to be Finally, we match the coefficients of the sufficient statistics in above equations with the natural parameters and find the following variational parameters in terms of the expected sufficient statistics: Update logθ t Update expected sufficient statistics end for A simplified sketch of our variational inference algorithm VB-CN is also presented in Algorithm 1. ELBO can be expressed as a sum of expectation terms most of which are in the form of negative cross entropy or negative entropy: In this section we will evaluate each of those expectations explicitly. We start our derivation with the trickier Gaussian log-likelihood term, then the rest of the expectations will correspond to negative cross entropy values of standard exponential family distributions: Variational distribution Q treats r t 1:K and θ 1:K as independent variables. So, the expectations of the categorical log-likelihood terms admit the following form The rest of the terms are related to cross entropy or entropy of the well-known exponential family distributions, and closed form expressions for them are supplied in Appendix B. So here, we only modify these expressions by changing their parameters with the appropriate variational parameters. 1. By using the negative cross entropy formulation in Appendix B.1.3 for categorical distributions: r 1 ∈ R 1, m n|r 1 's were set to 0, and Λ n|r 1 's were set to 1 10 I each; while for all values of r 1 ∈ R 1, γ 1 (r 1)'s were set to 10. We next describe the remaining hyperparameters with respect to the causal graph in Figure 2 (b) in which x 1 causes x 2. Their adaptation to other two graphs is straightforward due to symmetry. The hyperparameters of the Gamma distributions, (a 1, b 1, a 2, b 2), from which the precision of the observed variables were drawn, were allowed to take different values with the condition that a n|r 1 ≥ b n|r 1 at all times, but again every element of these vectors corresponding to different values of r 1 assumed to be constant within the vector. This is because the mean of a Gamma distribution Gamma(a, b) is a/b and its variance is a/b 2, therefore when b is allowed to take a greater value than a, this in a close to zero precision value for the relevant distribution for the observed variable. Obeying the constraint, the a and b's were allowed to take values among 1, 10, and 100 each. The a parameter was not allowed to be larger than 100 since this leads to an equivalent sample size much larger than the sample size of certain data sets used in experiments, effectively rendering the observations unimportant. The b parameter was not allowed to be smaller than 1 since this again implies extremely imprecise Gaussian distributions for the observed variables to which the Gamma distribution provided the precision variable. The combinations with these constraints lead to a total of 36 sets of hyperparameters. While doing model comparison in a hyperparameter setting, we expect several criteria to be satisfied for maintaining consistency. For instance, in the spurious model (Figure 2(a) ) there is no reason to assign different priors on variables x 1 and x 2. Otherwise, just by permuting the labels of the pairs, we would obtain inconsistent marginal likelihoods. Likewise, when the labels of a pair are permuted, e.g. 1, x 1:T 2 ) given the relation x 1 → x 2 to be equal to the marginal likelihood of the permuted pair (x 1:T 1,x 1:T 2) given the relationx 2 →x 1. The rule we used to solve inconsistency issues in such situations is the following: the prior parameters of two variables must be identical whenever the parental graphs of them are homomorphic. So, if we are calculating the marginal likelihood of the relation x 1 → x 2 with a particular hyperparameter setting, say (a 1 = 100, b 1 = 10, a 2 = 10, b 2 = 1), then the corresponding consistent hyperparameter setting for x 2 → x 1 should be (a 1 = 10, b 1 = 1, a 2 = 100, b 2 = 10), whereas the corresponding consistent hyperparameters for the spurious relationship should be (a 1 = 100, b 1 = 10, a 2 = 100, b 2 = 10). For this experiment, for each of 36 hyperparameter combinations, and for each rank values of |R 1 | = 1 to 5 for the linear model, a total of 3 different data pairs (one for each different graphical model) with 2000 observations were generated. This amounted to a total of 540 data pairs. For each synthetic data pair, the corresponding hyperparameters were used to compare the three hypotheses demonstrated in Figure 2 using the marginal likelihood estimate of the variational Bayes algorithm. The ing ROC curves can be seen in the Figure 3. With an overall accuracy of.961 and AUC of.998, the demonstrate that our method can identify the data generating graph comfortably, given the correct hyperparameter settings. The CEP data set is not labeled as to the spurious relationships, therefore it is not possible to conduct hyperparameter selection with cross-validation. However, we ran the experiments again, this time including the spurious relationship hypothesis in the experiments, for all 36 parameter settings, and recorded the pairs for which the marginal likelihood of the spurious hypothesis was the highest. We observed that, using the hyperparameter setting that achieved the highest accuracy in the previous experiment, these four data sets were found to be spurious: 19, 91, 92, and 98. The scatter plots of these data sets are presented in Figure 4. Visual examination of the first three pairs reveals that, although each of these pairs are correlated, they can be separated into two clusters in which X and Y axes become independent. In other words, once the confounding variables governing the cluster affiliations are decided, then the variables X and Y generated independently, so their correlation is indeed spurious. As we lack the expertise, we do not know what these confounding variables correspond in reality, but the existence of such variables is evident from the scatter plots. The case of the fourth spurious pair is slightly different than other correlated pairs. The fourth pair consists of the measurements of initial and final speeds of a ball on a ball track where initial speed is thought as the cause of final speed. However, our variational algorithm selected the spurious model with a latent variable having cardinality |R 1 | = 1, which actually corresponds to the marginal independence of X and Y. Such an explanation makes sense considering the plot in Figure 4, as the initial speed of the ball does not seem related to its final speed.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We cast causal structure discovery as a Bayesian model selection in a way that allows us to discriminate between Markov equivalent graphs to identify the unique causal graph.

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.