Question: represent the text to find a very short summary (1-2 sentences max)!

Question: One can substitute each neuron in any neural network with a kernel machine and obtain a counterpart powered by kernel machines. The new network inherits the expressive power and architecture of the original but works in a more intuitive way since each node enjoys the simple interpretation as a hyperplane (in a reproducing kernel Hilbert space). Further, using the kernel multilayer perceptron as an example, we prove that in classification, an optimal representation that minimizes the risk of the network can be characterized for each hidden layer. This removes the need of backpropagation in learning the model and can be generalized to any feedforward kernel network. Moreover, unlike backpropagation, which turns models into black boxes, the optimal hidden representation enjoys an intuitive geometric interpretation, making the dynamics of learning in a deep kernel network simple to understand. Empirical are provided to validate our theory. Any neural network (NN) can be turned into a kernel network (KN) by replacing each artificial neuron , i.e., learning machine of the form f (x) = σ(w x + b), with a kernel machine, i.e., learning machine of the form f (x) = w, φ(x) + b with kernel function k(x, y) = φ(x), φ(y). This combination of connectionism and kernel method enables the learning of hierarchical, distributed representations with kernels. In terms of training, similar to NN, KN can be trained with backpropagation (BP) . In the context of supervised learning, the need for BP in learning a deep architecture is caused by the fact that there is no explicit target information to tune the hidden layers . Moreover, BP is usually computationally intensive and can suffer from vanishing gradient. And most importantly, BP in hidden representations that are notoriously difficult to interpret or assess, turning deep architectures into "black boxes".The main theoretical contribution of this paper is the following: Employing the simplest feedforward, fully-connected KN as an example, we prove that in classification and under certain losses, the optimal representation for each hidden layer that minimizes the risk of the network can be explicitly characterized. This removes the need for BP and makes it possible to train the network in a feedforward, layer-wise fashion. And the same idea can be generalized to other feedforward KNs. The layer-wise learning algorithm gives the same optimality guarantee as BP in the sense that it minimizes the risk. But the former is much faster and evidently less susceptible to vanishing gradient. Moreover, the quality of learning in the hidden layers can be directly assessed during or after training, providing more information about the model to the user. For practitioners, this enables completely new model selection paradigms. For example, the bad performance of the network can now be traced to a certain layer, allowing the user to debug the layers individually. Most importantly, the optimal representation for each hidden layer enjoys an intuitive geometric interpretation, making the learning dynamics in a deep KN more transparent than that in a deep NN. A simple acceleration method that utilizes the "sparse" nature of the optimal hidden representations is proposed to further reduce computational complexity. Empirical on several computer vision benchmarks are provided to demonstrate the competence of the model and the effectiveness of the greedy learning algorithm. Figure 1: (a) Any NN (left, presented in the usual weight-nonlinearity abstraction) can be abstracted as a "graph" (right) with each node representing a neuron and each edge the input-output relationship between neurons. If a node receives multiple inputs, we view its input as a vector in some Euclidean space, as indicated by the colored rectangles. Under this abstraction, each neuron (f (x) = σ(w x + b)) can be directly replaced by a kernel machine (f (x) = w, φ(x) + b with kernel k(x, y) = φ(x), φ(y) ) mapping from the same Euclidean space into the real line without altering the architecture and functionality of the model. (b) Illustration for layer-wise optimality drifting away from network-optimality. Consider a two-layer network and let T 1, T 2 be the target function of the first and second layer, respectively. If the first layer creates error, which is illustrated by F (x) being far away from T 1 (x), the composed solution F • F on the right is better than that on the left and hence the F on the right corresponds to the network-wise optimality of the second layer. But the F on the left is clearly a better estimate to the layer-wise optimality T 2 if the quality of estimation is measured by the supremum distance. In this section, we discuss how to build a KN using a given NN. First, the generic approach is described in FIG4. Note that KN inherits the expressive power of the original NN since a kernel machine is a universal function approximator under mild conditions and the two models share the same architecture. However, KN works in a more intuitive way since each node is a simple linear model in a reproducing kernel Hilbert space (RKHS).We now concretely define the KN equivalent of an l-layer Multilayer Perceptron (MLP), which we shall refer to as the kernel MLP (kMLP).1 Given a random sample (x n, y n) N n=1, where (x n, y n) ∈ X 1 × Y ⊂ R d0 × R, denote (x n) N n=1 as S and (y n) N n=1 as Y S for convenience. For i = 1, 2,..., l, consider kernel k (i): X i × X i → R, X i ⊂ R di−1 (for i > 1, d i−1 is determined by the width of the i − 1 th layer). k (i) (x, y) = φ (i) (x), φ (i) (y) Hi, where φ (i) is a mapping into RKHS H i.For i ≥ 1, the i th layer in a kMLP, denoted F (i), is an array of d i kernel machines: DISPLAYFORM0 2,..., f (i) di ), a d i -tuple. Let F be the identity map on R d0, each f DISPLAYFORM1 j, where the α DISPLAYFORM2 ∈ R are the learnable parameters. 2 The set of mappings DISPLAYFORM3 j ∈ R for all admissible n, j, i} defines an l-layer kMLP. In the rest of this paper, we shall restrict our discussions to this kMLP. We now specify the assumptions that we impose on all kernels considered in this paper. First, we consider real, continuous, symmetric kernels only and we call a kernel positive semidefinite (PSD) or positive definite (PD) if for any S, the kernel matrix defined as (G) mn = k(x m, x n) is PSD or PD, respectively. We shall always assume that any kernel considered is at least PSD and that DISPLAYFORM0 It is straightforward to check using Cauchy-Schwarz inequality that the first condition implies max x,y∈Xi k (i) (x, y) = c. For each fixed x ∈ X i, we assume that DISPLAYFORM1 x -Lipschitz with respect to the Euclidean metric on DISPLAYFORM2, which we assume to be finite. The following notations will be used whenever convenient: We use the shorthand DISPLAYFORM3 (S) and the same is true with S substituted by any x. Throughout this paper, notations such as F (i) can either be used to denote a set of functions or a specific function in some set depending on the context. Also, when there is no confusion, we shall suppress the dependency of any loss function on the example for brevity, i.e., for a loss function, instead of writing (f (x), y), we shall write (f). To simplify discussion, we shall restrict ourselves to binary classification (Y = {+1, −1}) and directly give the on classification with more than two classes in the end. A generalization to regression is left as future work. Again, we only focus on kMLP although the idea can be directly generalized to all feedforward KNs. We now discuss the layer-wise learning algorithm, beginning by addressing the difficulties with training a deep architecture layer-by-layer. There are two fundamental difficulties with learning a deep architecture layer-wise. First, the hidden layers do not have supervision (labels) to learn from. And it depends on BP to propagate supervision from the output backward . We shall prove that for kMLP, one can characterize the optimal target representation for each hidden layer, which induces a risk for that layer. The target is optimal in the sense that it minimizes the risk of the subsequent layer and eventually that of the network if all layers succeed in learning their optimal representations. This optimal representation defines what we call "layer-wise optimality".The other difficulty with layer-wise learning is that for any given hidden layer, when the upstream layers create error, layer-wise optimality may not coincide with "network-wise optimality", i.e., the solution of this layer that eventually leads to the composed solution that minimizes the risk of the network in this suboptimal case. Indeed, when a hidden layer creates error, the objective of any layer after it becomes learning a solution that is a compromise between one that is close to the layer-wise optimality and one that prevents the error from the "bad" layer before it from "getting through" easily. And the best compromise is the network-wise optimality. The two solutions may not coincide, as shown in the toy example in FIG4. Clearly, we would like to always learn the network-wise optimality at each layer, but the learner is blind to it if it is only allowed to work on one layer at a time. By decomposing the overall error of the network into error at each layer, we prove that in fact, network-wise optimality is learnable for each hidden layer even in a purely layer-wise fashion and that the proposed layer-wise algorithm learns network-wise optimality at each layer. We now address the first difficulty in layer-wise learning. The basic idea is first described in Section 4.2.1. Then we provide technical in Section 4.2.2 and Section 4.2.3 to fill in the details. DISPLAYFORM0 and a loss function l defined for this network which induces a risk that we wish to minimize: R l = E l (F). BP views this problem in the following way: R l is a function of F. The learner tries to find an F that minimizes R l using the random sample S with labels Y S according to some learning paradigm such as Empirical Risk Minimization (ERM) or Structural Risk Minimization (SRM) . S is considered as fixed in the sense that it cannot be adjusted by the learner. Alternatively, one can view R l as a function of F (l) and the learner tries to find an F (l) minimizing R l using random sample S l−1 with labels Y S according to some learning paradigm, where S l−1:= DISPLAYFORM1 The advantage is that the learner has the freedom to learn both the function F (l) and the random sample S l−1. And since S l−1 determines the decision of the learning paradigm, which then determines R l, R l is now essentially a function of both F (l) and S l−1: DISPLAYFORM2 The key is that independently of the actual learning of F (l), one can characterize the sufficient condition on S l−1 under which R l, as a function of S l−1, is minimized, as we shall prove. In other words, the "global minimum" of R l w.r.t. S l−1 can be explicitly identified prior to any training. This gives the optimal S l−1, which we denote as S l−1.Moreover, the characterization of S l−1 gives rise to a new loss function l−1 and thus also a new risk R l−1 that is a function of DISPLAYFORM3. Consequently, the same reasoning would allow us to deduce S l−2 before the learner learns F (l−1). And this analysis can be applied to each layer, eventually leading to a greedy learning algorithm that sequentially learns DISPLAYFORM4 *, in that order, where the asterisk on the superscript indicates that the corresponding layer has been learned and frozen. The layer-wise learning algorithm provides a framework that enjoys great flexibility. To be specific, one could stop the above analysis at any layer i, then learn layers i + 1,..., l in a greedy fashion but still learn layers 1,..., i together with BP. Thus, it is easy to see that BP can be brought under this framework as a special case. Nevertheless, in later text, we shall stay on the one end of the spectrum where each layer is learned individually for clarity. We now present the formal that give the optimal hidden representations. By the reasoning above, the analysis starts from the last hidden layer (layer l − 1) and proceeds backward. To begin with, we need to approximate the true classification error R l since it is not computable. To this end, we first review a well-known complexity measure. Definition 4.1 (Gaussian complexity ). Let P be a probability distribution on a metric space X and suppose x 1,..., x N are independent random elements distributed as P. Let F be a set of functions mapping from X into R. Definê DISPLAYFORM0 where g 1,..., g N are independent standard normal random variables. The Gaussian complexity of DISPLAYFORM1 Intuitively, Gaussian complexity quantifies how well elements in a given function class can be correlated with a noise sequence of length N, i.e., the g n . Based on this complexity measure, we have the following bound on the expected classification error. DISPLAYFORM2, with probability at least 1 − δ and for any N ∈ N, every function DISPLAYFORM3, the empirical hinge loss. Given the assumptions on k (l), for any F, we have DISPLAYFORM4 Without loss of generality, we shall set hyperparameter γ = 1. We now characterize S l−1. Note that for a given f (l), A = w f (l) H l is the smallest nonnegative real number such that f (l) ∈ F l,A and it is immediate that this gives the tightest bound in Theorem 4.2. DISPLAYFORM5, where τ is any positive constant satisfying τ < 2(c − a) min(κ, 1 − κ). Denote as S l−1 any representation satisfying DISPLAYFORM6 for all pairs of x +, x − from distinct classes in S and all pairs of x, x from the same class. Suppose the learning paradigm returns f (l) under this representation. Let S DISPLAYFORM7 The optimal representation S l−1, characterized by Eq. 1, enjoys a straightforward geometric interpretation: Examples from distinct classes are as distant as possible in the RKHS whereas examples from the same class are as concentrated as possible (see proof (C) of Lemma 4.3 for a rigorous justification). Intuitively, it is easy to see that such a representation is the "easiest" for the classifier. The conditions in Eq. 1 can be concisely summarized in an ideal kernel matrix G defined as DISPLAYFORM8 And to have the l − 1 th layer learn S l−1, it suffices to train it to minimize some dissimilarity measure between G and the kernel matrix computed from k (l) and F (l−1) (S), which we denote G l−1. Empirical alignment , L 1 and L 2 distances between matrices can all serve as the dissimilarity measure. To simplify discussion, we let the dissimilarity measure be the DISPLAYFORM9 This specifiesR l−1 (F (l−1) ) as the sample mean of (l−1 (F (l−1), (x m, y m), (x n, y n))) N m,n=1 and R l−1 as the expectation of l−1 over (X 1, Y)×(X 1, Y). Note that due to the boundedness assumption on k (l), l−1 ≤ 2 max(|c|, |a|). S l−2,..., S 1 Similar to Section 4.2.2, we first need to approximate R l−1. DISPLAYFORM0, where F l−1 is a given hypothesis class. There exists an absolute constant C > 0 such that for any N ∈ N, with probability at least 1 − δ, DISPLAYFORM1 We are now in a position to characterize S l−2. For the following lemma only, we further assume that k (l−1) (x, y), as a function of (x, y), depends only on and strictly decreases in x − y 2 for all x, y ∈ X l−1 with k (l−1) (x, y) > a, and that the infimum inf x,y∈X l−1 k (l−1) (x, y) = a is attained in X l−1 at all x, y with x − y 2 ≥ η. Also assume that inf x,y∈X l−1; x−y 2<η ∂k (l−1) (x, y)/∂ x − y 2 = ι (l−1) is defined and is positive. Consider an F mapping S into X l−1, let DISPLAYFORM2, it is immediate that DISPLAYFORM3 is the smallest nonnegative real number such that f Under review as a conference paper at ICLR 2019Lemma 4.5 (optimal S l−2). Given a learning paradigm minimizingR l−1 (DISPLAYFORM4, where τ is any positive constant satisfying τ < 2d l−1 (c − a)ψι (l−1). Denote as S l−2 any representation satisfying DISPLAYFORM5 for all pairs of x +, x − from distinct classes in S and all pairs of x, x from the same class. Suppose the learning paradigm returns DISPLAYFORM6 under this representation. Let S • l−2 be another representation under which the learning paradigm returns DISPLAYFORM7 achieves zero loss on at least one pair of examples from distinct classes, then for any N ∈ N, DISPLAYFORM8 Applying this analysis to the rest of the hidden layers, it is evident that the i th layer, i = 1, 2,..., l − 1, should be trained to minimize the difference between G and the kernel matrix computed with k DISPLAYFORM9 and DISPLAYFORM10 Generalizing to classification with more than two classes requires no change to the algorithm since the definition of G is agnostic to the number of classes involved in the classification task. Also note that the sufficiency of expanding the kernel machines of each layer on the training sample (see Section 2) for the learning objectives in Lemma 4.3 and Lemma 4.5 is trivially justified since the generalized representer theorem directly applies (Schölkopf et al., 2001). Now since the optimal representation is consistent across layers, the dynamics of layer-wise learning in a kMLP is clear: The network maps the random sample sequentially through layers, with each layer trying to map examples from distinct classes as far as possible in the RKHS while keeping examples from the same class in a cluster as concentrated as possible. In other words, each layer learns a more separable representation of the sample. Eventually, the output layer works as a classifier on the final representation and since the representation would be "simple" after the mappings of the lower layers, the learned decision boundary would generalize better to unseen data, as suggested by the bounds above. We now discuss how to design a layer-wise learning algorithm that learns network-wise optimality at each layer. A rigorous description of the problem of layer-wise optimality drifting away from network-wise optimality and the search for a solution begins with the following bound on the total error of any two consecutive layers in a kMLP.Lemma 4.6. For any i = 2,..., l, let the target function and the approximation function be T i, F (i) *: DISPLAYFORM0 DISPLAYFORM1 By applying the above bound sequentially from the input layer to the output, we can decompose the error of an arbitrary kMLP into the error of the layers. This gives a formal description of the problem: The hypothesis with the minimal norm minimizes the propagated error from upstream, but evidently, this hypothesis is not necessarily close to the layer-wise optimality T i.Moreover, this bound provides the insight needed for learning network-wise optimality individually at each layer: For the i th layer, i ≥ 2, searching for network-wise optimality amounts to minimizing the r.h.s. of Eq. 2. Lemma 4.3 and Lemma 4.5 characterized T i for i < l and learning objectives that bound i were provided earlier in the text accordingly. Based on those , the solution that minimizes the new learning objectiveR i (DISPLAYFORM2, where τ > 0 is a hyperparameter, provides a good approximate to the minimizer of the r.h.s. of Eq. 2 if, of course, τ is chosen well. Thus, taking this as the learning objective of the i th layer produces a layer-wise algorithm that learns network-wise optimality at this layer. Note that for BP, one usually also needs to heuristically tune the regularization coefficient for weights as a hyperparameter. There is a natural method to accelerate the upper layers (all but the input layer): The optimal representation F (S) is sparse in the sense that φ(F (x m)) = φ(F (x n)) if y m = y n and φ(F (x m)) = φ(F (x n)) if y m = y n (see the proof (C) of Lemma 4.3). Since a kernel machine built on this representation of the given sample is a function in the RKHS that is contained in the span of the image of the sample, retaining only one example from each class would in exactly the same hypothesis class because trivially, we have {DISPLAYFORM0 Thus, after training a given layer, depending on how close the actual kernel matrix is to the ideal one, one can (even randomly) discard a large portion of centers for kernel machines of the next layer to speed up the training of it without sacrificing performance. As we will later show in the experiments, randomly keeping a fraction of the training sample as centers for upper layers produces performance comparable to or better than that obtained with using the entire training set. The idea of combining connectionism with kernel method was initiated by. In their work, an "arc cosine" kernel was so defined as to imitate the computations performed by a one-layer MLP. extended the idea to arbitrary kernels with a focus on MKL, using an architecture similar to a two-layer kMLP. As a further generalization, Zhang et al. FORMULA8 independently proposed kMLP and the KN equivalent of CNN. However, they did not extend the idea to any arbitrary NN. proposed to reparameterize each nonlinearity in an NN with a kernel expansion, ing in a network similar to KN but is trained with BP. There are other works aiming at building "deep" kernels using approaches that are different in spirit from those above. proposed to learn the covariance matrix of a Gaussian process using an NN in order to make the kernel "adaptive". This idea also underlies the now standard approach of combining a deep NN with an SVM for classification, which was first explored by and. Such an interpretation can be given to KNs as well, as we point out in Appendix B.5. proposed to learn hierarchical representations by learning mappings of kernels that are invariant to irrelevant variations in images. Much works have been done to improve or substitute BP in learning a deep architecture. Most aim at improving the classical method, working as add-ons for BP. The most notable ones are perhaps the unsupervised greedy pre-training techniques proposed by and. Among works that try to completely substitute BP, none provided a comparable optimality guarantee in theory as that given by BP. pioneered the idea of greedily learn the architecture of an NN. In their work, each new node is added to maximize the correlation between its output and the residual error signal. Several authors explored the idea of approximating error signals propagated by BP locally at each layer or each node (; ; ; ;). proposed to train NN layer-wise using an ideal kernel matrix that is a special case of that in our work. No theoretical were provided to justify its optimality for NN. proposed a BP-free deep architecture based on decision trees, but the idea is very different from ours. attempted to quantify the quality of hidden representations toward learning more interpretable deep architectures, sharing a motivation similar to ours. We compared kMLP learned using the proposed greedy algorithm with other popular deep architectures including MLP, Deep Belief Network (DBN) and Stacked Autoencoder (SAE) , with the last two trained using a combination of unsupervised greedy pre-training and standard BP . Note that we only focused on comparing with the standard, generic architectures because kMLP, as the KN equivalent of MLP, does not have a specialized architecture or features designed for specific application domains. Several optimization and training techniques were applied to the MLPs to boost performance. These include Adam , RMSProp , dropout and batch normalization (BN) . kMLP accelerated using the proposed method (kMLP FAST) was also compared. For these models, we randomly retained a subset of the centers of each upper layer before its training. As for the benchmarks used, rectangles, rectangles-image and convex are binary classification datasets, mnist (10k) and mnist (10k) rotated are variants of MNIST . And fashion-mnist is the Fashion-MNIST dataset . To further test the proposed layer-wise learning algorithm and the acceleration method, we compared greedily-trained kMLP with MLP and kMLP trained using BP using the standard MNIST . Two popular acceleration methods for kernel machines were also compared on the same benchmark, including using a parametric representation (i.e., for each node in a kMLP, f (x) = k(w, x), w learnable) (kMLP PARAM) and using random Fourier features (kMLP RFF) . More details for the experiments can be found in Appendix A 4.From TAB0, we see that the performance of kMLP is on par with some of the most popular and most mature deep architectures. In particular, the greedily-trained kMLPs compared favorably with their direct NN equivalents, i.e., the MLPs, even though neither batch normalization nor dropout was used for the former. These also validate our earlier theoretical on the layer-wise learning algorithm, showing that it indeed has the potential to be a substitute for BP with an equivalent optimality guarantee. Results in TAB0 further demonstrate the effectiveness of the greedy learning scheme. For both the single-hidden-layer and the two-hidden-layer kMLPs, the layer-wise algorithm consistently outperformed BP. It is worth noting that the proposed acceleration trick, despite being extremely simple, is clearly very effective and even produced models outperforming the original ones. This shows that kMLP together with the greedy learning scheme can be of practical interest even when dealing with the massive data sets in today's machine learning. Last but not least, we argue that it is the practical aspects that makes the greedy learning framework promising. Namely, this framework of learning makes deep architectures more transparent and intuitive, which can serve as a tentative step toward more interpretable, easy-to-understand models with strong expressive power. Also, new design paradigms are now possible under the layer-wise framework. For example, each layer can now be "debugged" individually. Moreover, since learning becomes increasingly simple for the upper layers as the representations become more and more well-behaved, these layers are usually very easy to set up and also converge very fast during training. The first data set, known as rectangles, has 1000 training images, 200 validation images and 50000 test images. The learning machine is required to tell if a rectangle contained in an image has a larger width or length. The location of the rectangle is random. The border of the rectangle has pixel value 255 and pixels in the rest of an image all have value 0. The second data set, rectangles-image, is the same with rectangles except that the inside and outside of the rectangle are replaced by an image patch, respectively. rectangles-image has 10000 training images, 2000 validation images and 50000 test images. The third data set, convex, consists of images in which there are white regions (pixel value 255) on black (pixel value 0) . The learning machine needs to distinguish if the region is convex. This data set has 6000 training images, 2000 validation images and 50000 test images. The fourth data set contains 10000 training images, 2000 validation images and 50000 test images taken from MNIST. The fifth is the same as the fourth except that the digits have been randomly rotated. Sample images from the data sets are given in FIG3 The experimental setup for the greedily-trained kMLPs is as follows, kMLP-1 corresponds to a one-hidden-layer kMLP with the first layer consisting of 15 to 150 kernel machines using the same Gaussian kernel (k(x, y) = e − x−y 2 /σ 2 ) and the second layer being a single or ten (depending on the number of classes) kernel machines using another Gaussian kernel. Note that the Gaussian kernel does not satisfy the condition that the infimum a is attained (see the extra assumptions before Lemma 4.5), but for practical purposes, it suffices to set the corresponding entries of the ideal kernel matrix to some small value. For all of our experiments, we set (G) mn = 1 if y m = y n and 0 otherwise. Hyperparameters were selected using the validation set. The validation set was then used in final training only for early-stopping based on validation error. For the standard MNIST and Fashion-MNIST, the last 5000 training examples were held out as validation set. For other datasets, see . kMLP-1 FAST is the same kMLP for which we accelerated by randomly choosing a fraction of the training set as centers for the second layer after the first had been trained. The kMLP-2 and kMLP-2 FAST are the two-hidden-layer kMLPs, the second hidden layers of which contained 15 to 150 kernel machines. We used Adam as the optimization algorithm of the layer-wise scheme. Although some of the theoretical presented earlier in the paper were proved under certain losses, we did not notice a significant performance difference between using L 1, L 2 and empirical alignment as loss function for the hidden layers. And neither was such difference observed between using hinge loss and cross-entropy for the output layer. This suggests that these may be proved in more general settings. To make a fair comparison with the NN models, the overall loss functions of all models were chosen to be the cross-entropy loss. Settings of all the kMLPs trained with BP can be found in . Note that because it is extremely time/memory-consuming to train kMLP with BP without any acceleration method, to make training possible, we could only randomly use 10000 examples from the entire training set of 55000 examples as centers for the kMLP-2 (BP) from TAB0.We compared kMLP with a one/two-hidden-layer MLP (MLP-1/MLP-2), a one/three-hidden-layer DBN (DBN-1/DBN-3) and a three-hidden-layer SAE (SAE-3). For these models, hyperparameters were also selected using the validation set. For the MLPs, the sizes of the hidden layers were chosen from the interval. All hyperparameters involved in Adam, RMSProp and BN were set to the suggested default values in the corresponding papers. If used, dropout and BN was added to each hidden layer, respectively. For DBN-3 and SAE-3, the sizes of the three hidden layers varied in intervals, and, respectively. DBN-1 used a much larger hidden layer than DBN-3 to obtain comparable performance. A simple calculation shows that the total numbers of parameters in the kMLPs were fewer than those in the corresponding DBNs and SAEs by orders of magnitude in all experiments. Like in the training for the kMLPs, the validation set were also reserved for early-stopping in final training. The DBNs and SAEs had been pre-trained unsupervisedly before the supervised training phase, following the algorithms described in . More detailed settings for these models were reported in . In this section, we provide some further analysis on kMLP and the layer-wise learning algorithm. Namely, in Appendix B.1, we give a bound on the Gaussian complexity of an l-layer kMLP, which describes the intrinsic model complexity of kMLP. In particular, the bound describes the relationship between the depth/width of the model and the complexity of its hypothesis class, providing useful information for model selection. In Appendix B.2, we give a constructive stating that the dissimilarity measure being optimized at each hidden layer will not increase as training proceeds from the input layer to the output. This also implies that a deeper kMLP performs at least as well as its shallower counterparts in minimizing any loss function they are trained on. In Appendix B.3, a similar to Lemma 4.3 is provided, stating that the characterization for the optimal representation can be made much simpler if one uses a more restricted learning paradigm. In fact, in contrast to Lemma 4.3, both necessary and sufficient conditions can be determined under the more restricted setting. In Appendix B.4, we provide a standard, generic method to estimate the Lipschitz constant of a continuously differentiable kernel, as this quantity has been repeatedly used in many of our in this paper. In Appendix B.5, we state some advantages of kMLP over classical kernel machines. In particular, empirical are provided in Appendix B.5.1, in which a two-layer kMLP consistently outperforms the classical Support Vector Machine (SVM) as well as several SVMs enhanced by Multiple Kernel Learning (MKL) algorithms (; Gönen & Alpaydın, 2011). We first give a on the Gaussian complexity of a two-layer kMLP. Lemma B.1. Given kernel k: DISPLAYFORM0 where the x ν are arbitrary examples from X 2. DISPLAYFORM1 where Ω is a given hypothesis class that is closed under negation, i.e., if DISPLAYFORM2 If the range of some element in Ω contains 0, we have DISPLAYFORM3 The above can be easily generalized to kMLP with an arbitrary number of layers. Lemma B.2. Given an l-layer kMLP, for each f DISPLAYFORM4 1 ≤ A i and let d l = 1. Denote the class of functions implemented by this kMLP as F l, we have DISPLAYFORM5 Proof. It is trivial to check that the hypothesis class of each layer is closed under negation and that there exists a function in each of these hypothesis classes whose range contains 0. Then the follows from repeatedly applying Lemma B.1. Lemma B.3. For i ≥ 2, assume k (i) is PD and fix layers 1, 2,..., i − 1 at arbitrary states F, F,..., F (i−1). Let the loss functions i, i−1 be the same up to their domains, and denote both i and i−1 as. Suppose layer i is trained with a gradient-based algorithm to minimize the loss (F (i) ). Denote the state of layer i after training by DISPLAYFORM0 Calculation for this initialization is specified in the proof. For i = 1, under the further assumption that DISPLAYFORM1 is the identity map on X 1.Remark B.3.1. For the greedily-trained kMLP, Lemma B.3 applies to the hidden layers and implicitly requires that k (i+1) = k (i) since the loss function for layer i, when viewed as a function of DISPLAYFORM2 ) and can be rewritten as DISPLAYFORM3 ). Since Lemma B.3 assumes to be the same across layers (otherwise it does not make sense to compare between layers), this forces DISPLAYFORM4 Further, if k (i+1) and k (i) have the property that k(x, y) = k(x,ȳ), wherex,ȳ denote the images of x, y under an embedding of R p into R q (p ≤ q) defined by the identity map onto a p-dimensional subspace of R q, then the condition DISPLAYFORM5 This lemma states that for a given kMLP, when it has been trained upto and including the i th hidden layer, the i + 1 th hidden layer can be initialized in such a way that the value of its loss function will be lower than or equal to that of the i th hidden layer after training. In particular, the actual hidden representation "converges" to the optimal represetation as training proceeds across layers. On the other hand, when comparing two kMLPs, this implies that the deeper kMLP will not perform worse in minimizing the loss function than its shallower counterpart. In deep learning literature, analogous to Lemma B.3 generally state that in the hypothesis class of a NN with more layers, there exists a hypothesis that approximates the target function nontrivially better than any hypothesis in that of another shallower network . Such an existence for kMLP can be easily deduced from the earlier bound on its Gaussian complexity (see Lemma B.2). However, these proofs of existence do not guarantee that such a hypothesis can always be found through learning in practice, whereas Lemma B.3 is constructive in this regard. Nevertheless, one should note that Lemma B.3 does not address the risk R = E. Instead, it serves as a handy that guarantees fast convergence of upper layers during training in practice. The following lemma states that if we are willing to settle with a more restricted learning paradigm, the necessary and sufficient condition that guarantees the optimality of a representation can be characterized and is simpler than that described in Lemma 4.3. The setup for this lemma is the same as that of Lemma 4.3 except that the assumption that the numbers of examples from the two classes are equal is not needed. Lemma B.4. Consider a learning paradigm that minimizesR l (f (l) ) + τ G N (F l,A) using represen- DISPLAYFORM0 is minimized over all linearly separable representations if and only if the representation F (S) satisfies DISPLAYFORM1 for all pairs of x +, x − from distinct classes in S. In general, for a continuously differentiable function f: R → R with derivative f and any a, b ∈ R, a < b, we have DISPLAYFORM0 This simple can be used to bound the Lipschitz constant of a continuously differentiable kernel. For example, for Gaussian kernel k: DISPLAYFORM1, we have ∂k(x, y)/∂y = 2(x − y)k(x, y)/σ 2. Hence for each fixed x ∈ X, k(x, y) is Lipschitz in y with Lipschitz constant bounded by sup y∈X 2(x − y)k(x, y)/σ 2. In practice, X is always compact and can be a rather small subspace of some Euclidean space after normalization of data, hence this would provide a reasonable approximation to the Lipschitz constant of Gaussian kernel. There are mainly two issues with classical kernel machines besides their usually high computational complexity. First, despite the fact that under mild conditions, they are capable of universal function approximation and that they enjoy a very solid mathematical foundation BID0, kernel machines are unable to learn multiple levels of distributed representations , yet learning representations of this nature is considered to be crucial for complicated artificial intelligence (AI) tasks such as computer vision, natural language processing, etc. . Second, in practice, performance of a kernel machine is usually highly dependent on the choice of kernel since it governs the quality of the accessible hypothesis class. But few rules or good heuristics exist for this topic due to its extremely task-dependent nature. Existing solutions such as MKL (; Gönen & Alpaydın, 2011) view the task of learning an ideal kernel for the given problem to be separate from the problem itself, necessitating either designing an ad hoc kernel or fitting an extra trainable model on a set of generic base kernels, complicating training.kMLP learns distributed, hierarchical representations because it inherits the architecture of MLP.To be specific, first, we see easily that the hidden activation of each layer, i.e., F (i) (x) ⊂ R di, is a distributed representation . Indeed, just like in an MLP, each layer of a kMLP consists of an array of identical computing units (kernel machines) that can be activated independently. Further, since each layer in a kMLP is built on top of the previous layer in exactly the same way as how the layers are composed in an MLP, the hidden representations are hierarchical .Second, kMLP naturally combines the problem of learning an ideal kernel for a given task and the problem of learning the parameters of its kernel machines to accomplish that task. To be specific, kMLP performs nonparametric kernel learning alongside learning to perform the given task. Indeed, for kMLP, to build the network one only needs generic kernels, but each layer F (i) can be viewed as a part of a kernel of the form DISPLAYFORM0 The fact that each F (i) is learnable makes this kernel "adaptive", mitigating to some extent any limitation of the fixed generic kernel k (i+1). The training of layer i makes this adaptive kernel optimal as a constituent part of layer i + 1 for the task the network was trained for. And it is always a valid kernel if the generic kernel k (i+1) is. Note that this interpretation has been given in a different context by and , we include it here only for completeness. We now compare a single-hidden-layer kMLP using simple, generic kernels with SVMs enhanced by MKL algorithms that used significantly more kernels to demonstrate the ability of kMLP to automatically learn task-specific kernels out of standard ones. The standard SVM and seven other SVMs enhanced by popular MKL methods were compared , including the classical convex MKL with kernels learned using the extended level method proposed in Eleven binary classification data sets that have been widely used in MKL literature were split evenly for training and test and were all normalized to zero mean and unit variance prior to training. 20 runs with identical settings but random weight initializations were repeated for each model. For each repetition, a new training-test split was selected randomly. For kMLP, all were achieved using a greedily-trained, one-hidden-layer model with the number of kernel machines ranging from 3 to 10 on the first layer for different data sets. The second layer was a single kernel machine. All kernel machines within one layer used the same Gaussian kernel, and the two kernels on the two layers differed only in kernel width σ. All hyperparameters were chosen via 5-fold cross-validation. As for the other models compared, for each data set, SVM used a Gaussian kernel. For the MKL algorithms, the base kernels contained Gaussian kernels with 10 different widths on all features and on each single feature and polynomial kernels of degree 1 to 3 on all features and on each single feature. For 2LMKL INF, one Gaussian kernel was added to the base kernels at each iteration. Each base kernel matrix was normalized to unit trace. For L p MKL, p was selected from {2, 3, 4}. For MKM, the degree parameter was chosen from {0, 1, 2}. All hyperparameters were selected via 5-fold cross-validation. From TAB4, kMLP compares favorably with other models, which validates our claim that kMLP learns its own kernels nonparametrically hence can work well even without excessive kernel parameterization. Performance difference among models can be small for some data sets, which is expected since they are all rather small in size and not too challenging. Nevertheless, it is worth noting that only 2 Gaussian kernels were used for kMLP, whereas all other models except for SVM used significantly more kernels. Proof of Lemma 4.3. Throughout this proof we shall drop the layer index l for brevity. Given that the representation satisfies Eq. 1, the idea is to first collect enough information about the returned f = (w, b) such that we can computeR(f) + τ w H and then show that for any other F (S) satisfying the condition in the lemma, suppose the learning paradigm returns f = (w, b) ∈ F A, thenR(f) + τ w H ≥R(f) + τ w H. We now start the formal proof. First, note that in the optimal representation, i.e., an F (S) such that Eq. 1 holds, it is easy to see that φ(F (x −)) − φ(F (x +)) H is maximized over all representations for all x −, x +.Moreover, note that given the representation is optimal, we have φ(F (x)) = φ(F (x)) if y = y and φ(F (x)) = φ(F (x)) if y = y: Indeed, by Cauchy-Schwarz inequality, for all x, x ∈ S, k(F (x), F (x)) = φ(F (x)), φ(F (x)) H ≤ φ(F (x)) H φ(F (x)) H and the equality holds if and only if φ(F (x)) = pφ(F (x)) for some real constant p. Using the assumption on k, namely, that φ(F (x)) H = √ c for all F (x), we further conclude that the equality holds if and only if p = 1. And the second half of the claim follows simply from c > a. Thus, all examples from the + and − class can be viewed as one vector φ(F (x +)) and φ(F (x −)), respectively. The returned hyperplane f cannot pass both F (x +) and F (x −), i.e., f (F (x +)) = 0 and f (F (x −)) = 0 cannot happen simultaneously since if so, first subtract b, rotate while keeping w H unchanged and add some suitable b to get a new f such that f (F (x −)) < 0 and f (F (x +)) > 0, then it is easy to see thatR(f) + τ w H <R(f) + τ w H. But by construction of the learning paradigm, this is not possible. Now suppose the learning paradigm returns an f such that DISPLAYFORM0 First note that for an arbitrary θ F,w, ζ is less than or equal to 2 since one can always adjust b such that y + f (F (x +)) = y − f (F (x −)) without changing ζ and hence having a larger ζ will not further reduceR(f), which is 0 when ζ = 2, but will in a larger w H according to Eq. 4. On the other hand, θ F,w must be 0 since this gives the largest ζ with the smallest w H. Indeed, if the returned f does not satisfy θ F,w = 0, one could always shift, rotate while keeping w H fixed and then shift the hyperplane back to produce another f with θ F,w = 0 and this f in a larger ζ if ζ < 2 or the same ζ if ζ = 2 but a smaller w H by rescaling. Hencê DISPLAYFORM1 Together with what we have shown earlier, we conclude that 2 ≥ ζ > 0. Then for some t ∈ R, we haveR DISPLAYFORM2 First note that we can choose t freely while keeping w fixed by changing b. If κ = 1/2, we havê DISPLAYFORM3 Evidently, the last two cases both inR(f DISPLAYFORM4 If κ > 1/2,R(f) decreases in t hence t must be 1 for f, which impliesR(f) = (1 − κ)(2 − ζ). Similarly, if κ < 1/2, t = ζ − 1 and henceR(f) = κ(2 − ζ). DISPLAYFORM5, which increases in t and hence t = 1 and DISPLAYFORM6, this combination of κ and t contradicts the optimality assumption of f. DISPLAYFORM7, where the second equality is becauseR(f) decreases in t. Again, κ > 1/2 leads to a contradiction. Combining all cases, we havê DISPLAYFORM8 which, by the assumption on τ, strictly decreases in ζ over. Hence the returned f must satisfy ζ = 2, which impliesR(f) = 0 and we havê DISPLAYFORM9 Now, for any other F (S), suppose the learning paradigm returns f. Let x w +, x w − be the pair of examples with the largest f (F (x +)) − f (F (x −)). We have DISPLAYFORM10 where we have used the assumption that there exists DISPLAYFORM11 This proves the desired . Lemma C.1. Suppose f 1 ∈ F 1,..., f d ∈ F d are elements from sets of real-valued functions defined on all of X 1, X 2,..., X m, where FIG4,..., f d (x 1), f 1 (x 2),..., f d (x m), y), where ω: R md × Y → R + ∪ {0} is bounded and L-Lipschitz for each y ∈ Y with respect to the Euclidean metric on R md. Let ω • F = {ω • f : f ∈ F}. Denote the Gaussian complexity of F i on X j as G j N (F i), if the F i are closed under negation, i.e., for all i, if f ∈ F i, then −f ∈ F i, we have DISPLAYFORM12 DISPLAYFORM13 In particular, for all j, if the x j n upon which the Gaussian complexities of the F i are evaluated are sets of i.i.d. random elements with the same distribution, we have G DISPLAYFORM14 This lemma is a generalization of a on the Gaussian complexity of Lipschitz functions on R k from . And the technique used in the following proof is also adapted from there. Proof. For the sake of brevity, we prove the case where m = 2. The general case uses exactly the same technique except that the notations would be more cumbersome. Let F be indexed by A. Without loss of generality, assume |A| < ∞. Define DISPLAYFORM15 ω(f α,1 (x n),..., f α,d (x n), y n )g n; DISPLAYFORM16 (f α,i (x n)g n,i + f α,i (x n)g N +n,i ), where α ∈ A, the (x n, x n) are a sample of size N from X 1 × X 2 and g 1,..., g N, g 1,1,..., g 2N,d are i.i.d. standard normal random variables. Let arbitrary α, β ∈ A be given, define X α − X β 2 2 = E(X α − X β) 2, where the expectation is taken over the g n. Define Y α − Y β 2 2 similarly and we have DISPLAYFORM17 ω(f α,1 (x n),..., f α,d (x n), y n ) − ω(f β,1 (x n),..., f β,d (x n), y n ) DISPLAYFORM18. By Slepian's lemma and since the F i are closed under negation, DISPLAYFORM19 Taking the expectation of the x n and x n on both sides, we have DISPLAYFORM20 Proof of Lemma 4.4. Normalize l−1 to by dividing 2 max(|c|, |a|). Then the loss function becomes l−1 F (l−1), (x m, y m), (x n, y n) = 1 2 max(|c|, |a|) k (l) F (l−1) (x m), F (l−1) (x n) − (G) mn.For each fixed (G) mn, l−1 F (l−1), (x m, y m), (x n, y n) − l−1 F (l−1), (x m, y m), (x n, y n) DISPLAYFORM21 2 max(|c|, |a|) DISPLAYFORM22 max(|c|, |a|) DISPLAYFORM23 Hence l−1 is L (l) / max(|c|, |a|)-Lipschitz in (F (l−1) (x m), F (l−1) (x n)) with respect to the Euclidean metric on R Proof of Lemma 4.6. First, it is trivial that the so-defined s metric is indeed a metric. In particular, it satisfies the triangle inequality. For i = 2,..., l, Proof of Lemma B.1. Since Ω and F 2 are both closed under negation, we havê DISPLAYFORM24 which proves that, as a function of F, A achieves its minimum if and only if F maximizes φ(F (x +)) − φ(F (x −)) H. Since arg max where we have used the assumption on k, namely, that k(x, x) = φ(x), φ(x) H = φ(x) 2 H = c, for all x. It immediately follows that any minimizer F of A must minimize k(F (x +), F (x −)) for all pairs of examples from opposite classes. This proves the desired .

We combine kernel method with connectionist models and show that the resulting deep architectures can be trained layer-wise and have more transparent learning dynamics.

Represent the TLDR for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Many notions of fairness may be expressed as linear constraints, and the ing constrained objective is often optimized by transforming the problem into its Lagrangian dual with additive linear penalties. In non-convex settings, the ing problem may be difficult to solve as the Lagrangian is not guaranteed to have a deterministic saddle-point equilibrium. In this paper, we propose to modify the linear penalties to second-order ones, and we argue that this in a more practical training procedure in non-convex, large-data settings. For one, the use of second-order penalties allows training the penalized objective with a fixed value of the penalty coefficient, thus avoiding the instability and potential lack of convergence associated with two-player min-max games. Secondly, we derive a method for efficiently computing the gradients associated with the second-order penalties in stochastic mini-batch settings. Our ing algorithm performs well empirically, learning an appropriately fair classifier on a number of standard benchmarks. Machine learning systems are becoming increasingly prevalent in real-world applications, consequently affecting the decisions that determine a person's life and future, such as playing a role in parole conditions BID1, loan applications BID12, and airport screening BID13. Recent work has shown that such machine learning models often have biases which can unfairly disadvantage certain groups. For example, learned word embeddings exhibit gender-specific biases in what should be gender neutral words BID4. In another case, a machine learning model's predictions regarding convict recidivism were found to be unfairly biased against African-Americans (BID1 . While it may seem at first that simply ignoring the features corresponding to these protected traits when training can alleviate this, previous work BID18 has shown that enforcing such blindness is largely ineffective due to redundant encodings in the data. In other words, while the learning algorithm used may not be biased, the data can be inherently biased in complex ways, and this leads to models which perpetuate these undesirable biases. Research into the challenging problem of machine learning fairness is therefore of great interest. To better specify this problem, previous work has elaborated on precise notions of fairness, such as demographic parity BID8, equal opportunity BID14, etc. These notions can often be expressed mathematically as a linear constraint on the output of a machine learning model, taken in expectation over the entire data distribution. Accordingly, a number of recent works have proposed to incorporate fairness during training by expressing the objective as a constrained optimization problem BID24 BID11 . If the original objective is convex, the addition of linear constraints in a problem which may be readily solved by Lagrangian methods. However, modern machine learning models are often not in a convex form. Indeed, the success of deep neural networks over the past decade makes it clear that the most well-performing models are often highly non-convex and optimized via stochastic gradient methods over large amounts of data BID19 BID20 . It is unfortunate that much of the existing work on fairness in machine learning has provided methods which are either focused on the convex, small data-set setting BID24 BID11, or otherwise require sophisticated and complex training methods BID6 .In this paper, we present a general method for imposing fairness conditions during training, such that it is practical in non-convex, large data settings. We take inspiration from the standard Lagrangian method of augmenting the original loss with linear penalties. In non-convex settings, this dual objective must be optimized with respect to both model parameters and penalty coefficients concurrently, and in general is not guaranteed to converge to a deterministic equilibrium. We propose to re-express the linear penalties associated with common fairness criteria as secondorder penalties. Second-order penalties are especially beneficial in non-convex settings, as they may be optimized using a fixed non-negative value for the penalty coefficient λ. When λ → 0 the optimization corresponds to an unconstrained objective, while as λ → ∞, the problem approaches that of a hard equality constraint. This allows us to avoid sophisticated optimization methods for potentially non-convergent two-player games. Instead, we only need to choose a fixed value for the penalty coefficient, which may be easily determined via standard hyperparameter optimization methods, such as cross-validation. As an additional benefit, by choosing the penalty coefficient on a separate validation set, we can improve generalization performance. Second-order penalties, however, potentially introduce a new problem: By squaring an expectation over the entire data distribution, the ing penalized loss is no longer an expectation of loss functions on individual data points sampled from the distribution, and therefore not readily approachable by stochastic gradient methods. We solve this by presenting an equivalent form of the second-order penalty as an expectation of individual loss functions on pairs of independently sampled data points. Our ing algorithm is thus not only more practical to optimize in non-convex settings, using a fixed value for the penalty coefficient, but is also easily optimized in large-data settings via standard stochastic gradient descent. We evaluate the performance of our algorithm in a number of different settings. In each setting, our algorithm is able to adequately optimize the desired constraints, such as encouraging feature orthonormality in deep image autoencoders and imposing predictive fairness across protected data groups. Consider a data domain X and a probability measure µ constituting a data distribution D = (X, µ). Let H be a set of real-valued functions f: X → R endowed with the inner-product, DISPLAYFORM0 A machine learning model is a function d ∈ H. Given a data point x ∈ X it provides a score d(x).When the machine learning model is a predictive binary classifier, the range of d is and a score d(x) corresponds to a machine learning model which on input x returns 1 with probability d(x) and returns 0 with probability 1 − d(x). Many notions of fairness may be written as linear constraints on d. That is, they may be expressed as d, c ∈ [C −, C +], where c ∈ H is some other fixed function and C ∈ R, ∈ R +. We elaborate on a few popular notions below. For simplicity, we restrict the text to refer to a domain X with a single protected group G ⊂ X whose predictions d(x) we desire to be fair with respect to the distribution of predictions on the entire domain X. Nevertheless, all of our apply to the fully general multi-group setting. We assume access to an indicator function g(x) = 1[x ∈ G] and thus the proportion of data in G is Z G = 1, g. We let y: X → {0, 1} be the true label function. Thus, the proportion of examples which are positive is P X = 1, y; the proportion which are positive and in G is P G = g, y.• Demographic parity BID8: A fair classifier d should make positive predictions at the same rate on each group. This constraint may be expressed as d, c = 0, where c(x) = g(x)/Z G − 1.• Equal opportunity BID14: A fair classifier d should have equal true positive rates on each group. This constraint may be expressed as d, c = 0, where DISPLAYFORM1 • Equalized odds BID14: A fair classifier d should have equal true positive and false positive rates on each group. In addition to the linear constraint associated with equal opportunity, this notion applies an additional constraint d, b = 0, where DISPLAYFORM2 • Disparate impact BID10: A fair classifier d should have a rate of positive prediction on a group at least p% as high as the rate of positive prediction on another group. Traditionally, p = 80. Unlike the other notions of fairness, disparate impact may not be expressed as a linear constraint. Nevertheless, previous work BID24 has suggested approximating it as such; i. DISPLAYFORM3 Any linear constraint d, c ∈ [C −, C +] may be equivalently written as a quadratic constraint: DISPLAYFORM0 We first discuss the known approaches based on the penalty-form Lagrangian with linear constraints and list their disadvantages. We then introduce the optimization based on our second-order penalties and how it avoids many of the issues associated with linear constraints. Suppose that we wish to minimize a loss DISPLAYFORM0 The Lagrangian is formulated as follows DISPLAYFORM1 Then, the original constrained optimization problem (the primal problem) would be DISPLAYFORM2 If our loss function is convex, then solving the dual problem would lead to the solution for the original problem; i.e., there is no duality gap. Unfortunately, this is not so in the non-convex case. Not only can the solution to the dual problem not yield an optimal feasible solution, there may not even exist a saddle point in the Lagrangian that we can converge to. Instead, in order to solve such a constrained optimization problem, one must consider it as a two-player game where one player chooses a classifier and the other player chooses a Lagrange multiplier. The final solution will then be a mixed equilibrium: in other words yields a randomized classifier. This line of work has received recent attention; e.g. BID3; BID0; BID6. To summarize:• The Lagrangian when optimizing jointly over model parameters and multipliers may not converge to any saddle point, and even if converged, may not lead to the right solution. This requires us to resort to sophisticated procedures with many parameters and hyperparameters, which may be difficult to train.• The Lagrangian approach with non-convex objectives leads to randomized classifiers, which may not be desirable in practice. Although work has been done to reduce the randomization BID6, it is in general not possible to reduce the solution to a deterministic one while still being an equilibrium.• Another weakness of the Lagrangian approach is that the constraints must be relaxed or approximated (e.g. hinge relaxation BID11) in order to make them convex and differentiable. The introduced slack necessitates hyperparameter tuning to encourage the optimization on approximated constraints to yield a classifier satisfying the original constraints. We propose to optimize the following objective DISPLAYFORM0 where λ ≥ 0 is a hyperparameter which decides the fairness-accuracy tradeoff. Note that λ → 0 corresponds to the unconstrained objective, while λ → ∞ corresponds to an optimization with hard constraints (i.e. = 0). Any fixed value of λ > 0 will yield a solution between these two extremes. Since the penalty coefficient λ is fixed during training, it may be treated as an additional hyperparameter. Standard hyperparameter optimization methods may be used to choose λ based on validation so that the final solution gives the desired fairness-accuracy trade-off. While we are not optimizing for the fairness metrics such as demographic parity or equalized odds directly, we will show that empirically our second-order penalties provide a reasonable proxy for many of the popular metrics. Additionally, while popular methods such as the aforementioned Lagrangian with linear penalties appear to directly optimize for the fairness metrics, in practice they require some sort of relaxation of these constraints to make the optimization feasible; thus in essence, such methods also optimize a proxy to the fairness metrics rather than the actual metrics desired and accordingly also require some amount of hyperparameter tuning. It can now also be seen that this alternative way of solving for fairness-constrained classifiers overcomes many of the drawbacks listed earlier associated with methods based on the Lagrangian: For any fixed choice of λ, there will exist a solution to the optimization. Moreover, this solution will be deterministic. Next, by tuning λ to directly satisfy the desired fairness metrics, we decouple much of the inherent difficulty of hyperparameter tuning in the Lagrangian approaches, which rely on the procedure itself to allow a certain amount of slack. Moreover, this decoupling encourages better generalization performance as there is less chance of overfitting to the training set compared to approaches which solve for the model parameters and Lagrange multipliers simultaneously on the same dataset. At face value, it appears that the introduction of second-order penalties may complicate stochastic optimization of the objective. The quadratic penalty is a square of an expectation over the dataset. It is not possible to express such a penalty as an expectation of individual loss functions over the dataset. In this section, we show that despite this obstacle, it is in fact possible to express the secondorder penalty as an expectation of individual loss functions over pairs of data points sampled from the dataset. This derivation is crucial for most modern machine learning applications, in which the data set is exceedingly large, at times presenting itself in an online form, and must be optimized using stochastic mini-batches. The second-order penalty is of the form DISPLAYFORM0 Since µ is a probability measure, we may re-write the integrals as DISPLAYFORM1 We may express each squared integral as a double integral: DISPLAYFORM2 Finally, we may express the sum of these double integrals as a double integral of a sum: DISPLAYFORM3 The gradients of this double integral with respect to parameters of d may be approximated via Monte Carlo estimation, only requiring access to two independent samples w, x from D. Algorithm 1 provides a pseudocode of our fairness-aware training algorithm. The machine learning model is parameterized as d θ, for parameter vector θ. We also assume the training loss is an expectation of individual loss functions: DISPLAYFORM0 Algorithm 1 Fairness training with second-order penalties. DISPLAYFORM1, model parameterization d θ, learning rate η, number of training steps N, batch size B, hyperparameter λ. DISPLAYFORM2. For a fixed penalty coefficient λ, our algorithm performs stochastic gradient descent by sampling batches. Each batch is used to compute an unbiased estimate of the gradient of the loss (d θ) as well as an unbiased estimate of the gradient of the second-order penalty. The optimal choice of λ is determined by standard hyperparameter tuning. Our work builds on the constrained optimization view of fairness in machine learning. This view was first introduced in BID24 and later extended in BID11. These works have focused on the convex setting, where optimality and convergence can be guaranteed. Although less is known in the non-convex case, there is work which frames the constrained optimization problem as a two-player game BID6. The ing classifier in this case is a randomized distribution over classifiers. In contrast, our work proposes the use of second-order penalties in a general setting. This allows one to avoid the two-player game formulation in the non-convex case and accordingly the ing classifier is deterministic. We are not the first to propose training for fairness in this manner: BID7 studies this for kernel methods and BID15 gives for training a linear model with such penalities. In contrast, our methods are applicable to highly non-convex models and previous works do not address how to optimize for these penalities stochastically. Another approach for non-convex settings is the use of adversarial training BID9. In this setting, a predictive model is trained concurrently with an adversary, whose objective is to recover sensitive or protected attributes from the model outputs. The model's loss is then augmented with a penalty based on the adversary's success. This is thus another form of a twoplayer game, and hence also suffers from convergence issues. Our approach avoids these issues by allowing the use of a fixed penalty coefficient in training. Moreover, the form of our constraints may be seen as equivalent to an adversarial formulation in which the adversary is parameterized by a linear model. The quadratic penalties we propose are similar to previous notions of orthogonality in machine learning, which is generally useful when one desires diversity of features BID21 BID16. The specific penalties we impose may be interpreted as penalizing the Frobenius norm of the Gram matrix of model outputs, as mentioned in BID23. Many of these methods propose optimization schemes which are not amenable to stochastic mini-batching. In contrast, one of the key contributions of our work is showing that a second-order penalty may be optimized stochastically. Our hinges on a standard calculus identity relating a product of integrals to a double integral. Similar techniques have been used previously in the context of reinforcement learning. Specifically, double-sampling is a known technique for unbiased minimization of the Bellman error, also a square of an expectation BID2. We use the Iris dataset BID17 and train a simple model which is a network with a single intermediate 2-node layer. We show the effects of adding a penalty to encourage orthogonality on this layer. That is, we wish for the two learned features to be decorrelated over the dataset. We show the in Figure 1 as weight on the penalty increases. We further increase the difficulty task by using small stochastic batches of size 4. While this is only a toy example, it clearly shows the possibility of stochastically training with these second-order penalties. Figure 1: The learned 2d representation of the datapoints (colored by label) as the penalty weight changes. We see that indeed, the two features become decorrelated as we increase the weight. We now move to applying our technique to a highly non-convex, neural network model. We use a convolutional autoencoder applied to MNIST images, where the encoder consists of 3 convolutional and max-pooling layers and the decoder consists of 4 convolutional and 3 upsampling layers. The encoded representation is 128-dimensional. We train this autoencoder with added penalties that encourage orthonormality. That is, in additional to orthogonality (as in the previous simulation), we also encourage the feature columns to be of unit norm. For training, we trained with the Adam optimizer and batch-size 128. We show in FIG0 that indeed, we can stochastically optimize with the quadratic penalty and that feature correlation on the test set decreases as we increase the penalty coefficient. We see that we can dramatically decrease the correlation of features while suffering small sacrifices in reconstruction error. 6 EXPERIMENTS: FAIRNESS 6.1 DATASETS Adult (48842 examples). Each datapoint corresponds to an individual and the task is to predict whether the person's income is more than 50k per year. We use 2 protected groups based on gender and preprocess the dataset and use a linear model, consistent with previous works, e.g. BID24; BID11. The 2 fairness constraints here are the equal opportunity constraints for the 2 protected classes with slack 0.05, that is, the positive prediction rate on the positively-labeled examples for each protected class must be at least 95% of the overall positive prediction rate over all positively-labeled examples. Bank Marketing BID17 (45211 examples). The data is based on a direct marketing campaign of a banking institution. The task is to predict whether someone will subscribe to a bank product. We use age as a protected feature and we have 5 protected groups, based on age quantiles and the 5 fairness constraints are demographic parity. Communities and Crime BID17 (Lichman et al.,) (1994 . Each datapoint represents a community and the task is to predict whether a community has high (above the 70-th percentile) or low crime rate. We preprocess the data and use a linear model, consistent with previous works, e.g. BID5 and form the protected group based on race in the same way as done in BID5. We use four race features as real-valued protected attributes corresponding to White, Black, Asian and Hispanic. We threshold each at the median to form 8 protected groups. There is one fairness constraint for each of the 8 protected groups, which constrains the groups false positive rate to be at most the overall false positive rate. ProPublicas COMPAS recidivism data (7, 918 examples). The task is to predict recidivism based on criminal history, jail and prison time, demographics, and risk scores. We preprocess this dataset in a similar way as the Adult dataset and the protected groups are two race-based (Black or White) and two gender-based (Male or Female). We use a 2-layer Neural Network with ReLU activations and 10 hidden units. The 4 fairness constraints here are the equal opportunity constraints for the 4 protected classes each being bounded by at most 0.05. That is, we wish to not predict recidivism more than 5% on top of the overall predicted recidivism rate restricted to examples whose which indeed had recidivism in two years for any protected class. Deterministic Lagrangian: This method BID11 is jointly training the Lagrangian in both model parameters and Lagrange multipliers and uses a Hinge approximation of the constraints to make the Lagrangian differentiable in its input. We then return the "best" iterate selected using a heuristic introduced by BID6, which finds a reasonable accuracy/fairness trade-off. Stochastic Lagrangian: This method BID6 ) returns a stochastic solution to the twoplayer game with the Lagrangian of the previous method as pay-off function. This solution is based on approximating a Nash equilibrium to this two-player game. We optimize over hyperparameters for our method in the following way: we perform a grid search over the weight of the orthogonality penalty as well as the fixed learning rate for the model trained using Adam optimizer. Then, to choose the best model, we find the highest accuracy model on the validation set which satisfies the constraints on the validation set. A final evaluation is performed on a fully un-seen test set. The Lagrangian baselines take in the desired slack and the hyperparameter search is over the fixed learning rate and is chosen using the heuristic on a validation set which chooses the best accuracy and constraint trade-off as done in BID6. We see that our method attains far lower constraint violation compared to training without constraints with almost no trade-off in accuracy. The Lagrangian baselines give solutions that satisfy the constraints but with considerable trade-off in accuracy. We see that our method attains the best testing error compared to the baselines while satisfying the fairness constraints. We show both the violations and overall false positive rates (FPR). The violation is the maximum difference between the FPR for any given protected group and the overall FPR. For example, in the first row, we see that there exists a protected group with a FPR of 0.3482 + 0.1108 = 0.459 by the model. While our method does not attain the lowest violation, it provides a reasonable trade-off. Interestingly, our method attains the lowest overall (i.e. average) FPR across the entire dataset out of all the methods. A low violation at the cost of high overall FPR may be undesirable because ensuring fairness in FPR might make everyone worse off in terms of FPR. We see that our method is able to learn a classifier that is significantly closer to satisfying the fairness constraints than the baselines while trading off a reasonable amount of accuracy. A PARETO CURVES Figure 3: For two of our datasets, we show the pareto frontiers for the testing error vs constraint violation trade-off over the runs of our method obtained from the grid search on the single constant learning rate and the orthogonality penalty term.

We propose a method to stochastically optimize second-order penalties and show how this may apply to training fairness-aware classifiers.

Represent the short summary for finding the corresponding scientific article (long).

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.