Question: Represent the scientific article to find a short TLDR

Question: Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data. However, GCN computes nodes' representation recursively from their neighbors, making the receptive field size grow exponentially with the number of layers. Previous attempts on reducing the receptive field size by subsampling neighbors do not have any convergence guarantee, and their receptive field size per node is still in the order of hundreds. In this paper, we develop a preprocessing strategy and two control variate based algorithms to further reduce the receptive field size. Our algorithms are guaranteed to converge to GCN's local optimum regardless of the neighbor sampling size. Empirical show that our algorithms have a similar convergence speed per epoch with the exact algorithm even using only two neighbors per node. The time consumption of our algorithm on the Reddit dataset is only one fifth of previous neighbor sampling algorithms. Graph convolution networks (GCNs) BID1 generalize convolutional neural networks (CNNs) to graph structured data. The "graph convolution" operation applies same linear transformation to all the neighbors of a node, followed by mean pooling. By stacking multiple graph convolution layers, GCNs can learn nodes' representation by utilizing information from distant neighbors. GCNs have been applied to semi-supervised node classification BID1, inductive node embedding (a), link prediction (; BID1 and knowledge graphs , outperforming multi-layer perceptron (MLP) models that do not use the graph structure and graph embedding approaches (; ;) that do not use node features. However, the graph convolution operation makes it difficult to train GCN efficiently. A node's representation at layer L is computed recursively by all its neighbors' representations at layer L − 1. Therefore, the receptive field of a single node grows exponentially with respect to the number of layers, as illustrated in FIG0. Due to the large receptive field size, BID1 proposed training GCN by a batch algorithm, which computes the representation for all the nodes altogether. However, batch algorithms cannot handle large scale datasets because of their slow convergence and the requirement to fit the entire dataset in GPU memory. Hamilton et al. (2017a) made an initial attempt on developing stochastic algorithms to train GCNs, which is referred as neighbor sampling (NS) in this paper. Instead of considering all the neighbors, they randomly subsample D (l) neighbors at the l-th layer. Therefore, they reduce the receptive field size to l D (l), as shown in FIG0 (b). They found that for two layer GCNs, keeping D = 10 and D = 25 neighbors can achieve comparable performance with the original model. However, there is no theoretical guarantee on the predictive performance of the model learnt by NS comparing with the original algorithm. Moreover, the time complexity of NS is still D D = 250 times larger than training an MLP, which is unsatisfactory. In this paper, we develop novel stochastic training algorithms for GCNs such that D (l) can be as low as two, so that the time complexity of training GCN is comparable with training MLPs. Our methods are built on two techniques. First, we propose a strategy which preprocesses the first graph convolution layer, so that we only need to consider all neighbors within L−1 hops instead of L hops. This is significant because most GCNs only have L = 2 layers BID1; a). Second, we develop two control variate (CV) based stochastic training algorithms. We show that our CV-based algorithms have lower variance than NS, and for GCNs without dropout, our algorithm provably converges to a local optimum of the model regardless of D (l).We empirically test on six graph datasets, and show that our techniques significantly reduce the bias and variance of the gradient from NS with the same receptive field size. Our algorithm with D (l) = 2 achieves the same predictive performance with the exact algorithm in comparable number of epochs on all the datasets, while the training time is 5 times shorter on our largest dataset. We now briefly review graph convolutional networks (GCNs) BID1 and the neighbor sampling (NS) algorithm (a). The original GCN was presented in a semi-supervised node classification task BID1. We follow this setting throughout this paper. Generalization of GCN to other tasks can be found in; BID1 and Hamilton et al. (2017b). In the node classification task, we have an undirected graph G = (V, E) with V = |V| vertices and E = |E| edges, where each vertex v consists of a feature vector x v and a label y v. The label is only observed for some vertices V L and we want to predict the label for the rest vertices V U:= V\V L. The edges are represented as a symmetric V × V adjacency matrix A, where A v,v is the weight of the edge between v and v, and the propagation matrix P is a normalized version of A:Ã = A + I,D vv = v Ã vv, and P =D DISPLAYFORM0. A graph convolution layer is defined as DISPLAYFORM1 where H (l) is the activation matrix in the l-th layer, whose each row is the activation of a graph node. H = X is the input feature matrix, W (l) is a trainable weight matrix, σ(·) is an activation function, and Dropout p (·) is the dropout operation with keep probability p., where f (·, ·) can be the square loss, cross entropy loss, etc., depending on the type of the label. When P = I, GCN reduces to a multi-layer perceptron (MLP) model which does not use the graph structure. Comparing with MLP, GCN is able to utilize neighbor information for node classification. We define n(v, L) as the set of all the L-neighbors of node v, i.e., the nodes that are reachable from v within L hops. It is easy to see from FIG0 that in an L-layer GCN, a node uses the information from all its L-neighbors. This makes GCN more powerful than MLP, but also complicates the stochastic training, which utilizes an approximated gradient ∇L ≈ DISPLAYFORM0 v ), where V B ⊂ V L is a minibatch of training data. The large receptive field size | ∪ v∈V B n(v, L)| per minibatch leads to high time complexity, space complexity and amount of IO. See Table 1 for the average number of 1-and 2-neighbors of our datasets. We introduce alternative notations to help compare different algorithms. Let DISPLAYFORM0 v, we focus on studying how u v is computed based on node v's neighbors. To keep notations simple, we omit all the subscripts and tildes, and exchange the ID of nodes such TAB4: Number of vertexes, edges, average number of 1-and 2-neighbors per node for each dataset. Undirected edges are counted twice and self-loops are counted once. Reddit is already subsampled to have a max degree of 128 following Hamilton et al. (2017a). DISPLAYFORM1 DISPLAYFORM2 | is the number of neighbors. We get the propagation rule u = D v=1 p v h v, which is used interchangeably with the matrix form U (l) = PH (l). To reduce the receptive field size, Hamilton et al. (2017a) propose a neighbor sampling (NS) algorithm. On the l-th layer, they randomly choose D (l) neighbors for each node, and develop an estimator u N S of u based on Monte-Carlo approximation DISPLAYFORM0 In this way, they reduce the receptive field size from DISPLAYFORM1 Neighbor sampling can also be written in a matrix form as DISPLAYFORM2 whereP (l) is a sparser unbiased estimator of P, i.e., EP DISPLAYFORM3 used for testing and for computing stochastic gradient DISPLAYFORM4 CV,v ) during training. The NS estimator u N S is unbiased. However it has a large variance, which leads to biased prediction and gradients after the non-linearity in subsequent layers. Due to the biased gradients, training with NS does not converge to the local optimum of GCN. When D (l) is moderate, NS may has some regularization effect like dropout , where it drops neighbors instead of features. However, for the extreme ease D (l) = 2, the neighbor dropout rate is too high to reach high predictive performance, as we will see in Sec. 5.4. Intuitively, making prediction solely depends on one neighbor is inferior to using all the neighbors. To keep comparable prediction performance with the original GCN, Hamilton et al. (2017a) use relatively large D = 10 and D = 25. Their receptive field size D × D = 250 is still much larger than MLP, which is 1. We first present a technique to preprocess the first graph convolution layer, by approximating ADropout p (X) with Dropout p (AX). The model becomes DISPLAYFORM0 This approximation does not change the expectation because E ADropout p (X) = E Dropout p (AX), and it does not affect the predictive performance, as we shall see in Sec. 5.1.The advantage of this modification is that we can preprocess U = P H = P X and takes Uas the new input. In this way, the actual number of graph convolution layers is reduced by one -the first layer is merely a fully connected layer instead of a graph convolution one. Since most GCNs only have two graph convolution layers BID1 a), this gives a significant reduction of the receptive field size from the number of DISPLAYFORM1 The numbers are reported in Table 1. We now present two novel control variate based estimators that have smaller variance as well as stronger theoretical guarantees than NS. We assume that the model does not have dropout for now and will address dropout in Sec. DISPLAYFORM0 where v is a random neighbor, and ∆h v = h v −h v. For the ease of presentation, we assume that we only use the latest activation of one neighbor, while the implementation also include the node itself besides the random neighbor, so D (l) = 2. Using historical activations is cheap because they need not to be computed recursively using their neighbors' activations, as shown in FIG0. Unlike NS, we apply Monte-Carlo approximation on v p v ∆h v instead of v p v h v. Since we expect h v andh v to be close, ∆h v will be small and u CV should have a smaller variance than u N S. Particularly, if the model weight is kept fixed,h v should be eventually equal with h v, so that , Chapter 5), which has zero mean and large correlation with u N S, to reduce its variance. We refer this stochastic approximation algorithm as CV, and we will formally analyze the variance and prove the convergence of the training algorithm using CV for stochastic gradient in subsequent sections. DISPLAYFORM1 In matrix form, CV computes the approximate predictions as follows, where we explicitly write down the iteration number i and add the subscript CV to the approximate activations DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 CV,i,v stores the latest activation of node v on layer l computed before time i. Formally, let m (l) i ∈ R V ×V be a diagonal matrix, and (m DISPLAYFORM5 After finishing one iteration we update historyH with the activations computed in that iteration as Eq.. With dropout, the activations H are no longer deterministic. They become random variables whose randomness come from different dropout configurations. Therefore, ∆h v = h v −h v is not necessarily small even if h v andh v have the same distribution. We develop another stochastic approximation algorithm, control variate for dropout (CVD), that works well with dropout. Our method is based on the weight scaling procedure to approximately compute the mean DISPLAYFORM0 That is, along with the dropout model, we can run a copy of the model with no dropout to obtain the mean µ v, as illustrated in FIG0. With the mean, we can obtain a better stochastic approximation by separating the mean and variance DISPLAYFORM1 whereμ v is the historical mean activation, obtained by storing µ v instead of h v, and ∆µ = µ v −μ v. u CV D an unbiased estimator of u because the term √ Dp v (h v − µ v) has zero mean, and the Monte-Carlo approximation DISPLAYFORM2 is made by assuming h v's to be independent Gaussians, which we will soon clarify. The pseudocodes for CV and CVD are in Appendix E. We analyze their variance in a simple independent Gaussian case, where we assume that activations are Gaussian random variables Table 2: Variance of different algorithms in the independent Gaussian case.. Without loss of generality, we assume that all the activations h v are one dimensional. We also assume that all the activations h 1,..., h D and historical activationsh 1,...,h D are independent, where the historical DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 We introduce a few more notations. ∆µ v and ∆s Table 2, where the derivations can be found in Appendix C. We decompose the variance as two terms: variance from Monte-Carlo approximation (VMCA) and variance from dropout (VD).If the model has no dropout, the activations have zero variance, i.e., s v =s v = 0, and the only source of variance is VMCA. We want VMCA to be small. As in Table 2, the VMCA for the exact estimator is 0. For the NS estimator, VMCA is DISPLAYFORM3 2, whose magnitude depends DISPLAYFORM4 Similarly, VMCA for both CV and CVD estimators is DISPLAYFORM0, which is likely because ∆µ v should be smaller than µ v. Since CV and CVD estimators have the same VMCA, we adopt the CV estimator for models without dropout, due to its simplicity. The VD of the exact estimator is s 2, which is overestimated by both NS and CV. NS overestimates VD by D times, and CV has even larger VD. Meanwhile, the VD of the CVD estimator is the same as the exact estimator, indicating CVD to be the best estimator for models with dropout. Besides smaller variance, CV also has stronger theoretical guarantees than NS. We can show that during testing, CV's prediction becomes exact after a few testing epochs. For models without dropout, we can further show that training using the stochastic gradients obtained by CV converges to GCN's local optimum. We present these in this section and Sec. 4.5. Note that the analysis does not need the independent Gaussian assumption. Given a model W, we compare the exact predictions (Eq. 1) and CV's approximate predictions (Eq. 5,6) during testing, which uses the deterministic weight scaling procedure. To make predictions, we run forward propagation by epochs. In each epoch, we randomly partition the vertex set V as I minibatches V 1,..., V I and in the i-th iteration, we run a forward pass to compute the prediction for nodes in V i. Note that in each epoch we scan all the nodes instead of just testing nodes, to ensure that the activation of each node is computed at least once per epoch. The following theorem reveals the connection of the exact predictions and gradients, and their approximate versions by CV.Theorem 1. For a fixed W and any i > LI we have: (Exact Prediction) The activations computed by CV are exact, i.e., Z (l) DISPLAYFORM0 Theorem 1 shows that at testing time, we can run forward propagation with CV for L epoches and get the exact prediction. This outperforms NS, which cannot recover the exact prediction. Comparing with directly making exact predictions by a batch algorithm, CV is more scalable because it does not need to load the entire graph into memory. The proof can be found in Appendix A. The following theorem shows that for a model without dropout, training using CV's approximated gradients converges to a local optimum of GCN, regardless of the neighbor sampling size D (l). Therefore, we can choose arbitrarily small D (l) without worrying about the convergence. Theorem 2. Assume that all the activations are ρ-Lipschitz, the gradient of the cost func- DISPLAYFORM0 is the inner product of matrix A and matrix B. We randomly run SGD for R ≤ N iterations, where DISPLAYFORM1. Then, for the updates W i+1 = W i − γ i g CV (W i) and step sizes DISPLAYFORM2 }, there exists constants K 1 and K 2 which are irrelevant with N, s.t. ∀N > LI, DISPLAYFORM3 The proof can be found in Appendix B. Particularly, DISPLAYFORM4 Therefore, our algorithm converges to a local optimum. Finally we discuss the time complexity of different algorithms. We decompose the time complexity as sparse time complexity for sparse-dense matrix multiplication such as PH (l), and dense time complexity for dense-dense matrix multiplication such as U times higher than NS.Our implementation is similar as BID1. We store the node features in the main memory, without assuming that they fit in GPU memory as Hamilton et al. (2017a), which makes our implementation about 2 times slower than theirs. We keep the histories in GPU memory for efficiency since they are only LH < K dimensional. We examine the variance and convergence of our algorithms empirically on six datasets, including Citeseer, Cora, PubMed and NELL from BID1 and Reddit, PPI from Hamilton et al.(2017a), as summarized in Table 1. To measure the predictive performance, we report Micro-F1 for the multi-label PPI dataset, and accuracy for all the other multi-class datasets. We use the same model architectures with previous papers but slightly different hyperparameters (see Appendix D for the details). We repeat the convergence experiments 10 times on Citeseer, Cora, PubMed and NELL, and 5 times on Reddit and PPI. The experiments are done on a Titan X (Maxwell) GPU. We first examine the approximation in Sec. 3 that switches the order of dropout and aggregating the neighbors. Let M0 be the original model (Eq. 1) and M1 be our approximated model (Eq. 3), we compare three settings: M0, D (l) = ∞ is the exact algorithm without any neighbor sampling. M1+PP, D (l) = ∞ changes the model from M0 to M1. Preprocessing does not affect the training for DISPLAYFORM0 uses NS with a relatively large number of neighbors. In Table 3 we can see that all the three settings performs similarly, i.e., our approximation does not affect the predictive performance. Therefore, we use M1+PP, D (l) = 20 as the exact baseline in following convergence experiments because it is the fastest among these three settings. We now study how fast our algorithms converge with a very small neighbor sampling size D (l) = 2. We compare the following algorithms: Exact, which is M1+PP, D We first validate Theorem 2, which states that CV+PP converges to a local optimum of Exact, for models without dropout, regardless of D (l). We disable dropout and plot the training loss with respect to number of epochs as FIG1. We can see that CV+PP can always reach the same training loss with Exact, which matches the of Theorem 2. Meanwhile, NS and NS+PP have a higher training loss because their gradients are biased. Next, we compare the predictive accuracy obtained by the model trained by different algorithms, with dropout turned on. We use different algorithms for training and the same Exact algorithm for testing, and report the validation accuracy at each training epoch. The is shown in FIG5. We find that CVD+PP is the only algorithm that is able to reach comparable validation accuracy with Exact on all datasets. Furthermore, its convergence speed with respect to the number of epochs is comparable with Exact despite its D (l) is 10 times smaller. Note that CVD+PP performs much better than Exact on the PubMed dataset; we suspect it finds a better local optimum. Meanwhile, simper algorithms CV+PP and NS+PP work acceptably on most of the datasets. CV+PP reaches a comparable accuracy with Exact for all datasets except PPI. NS+PP works slightly worse but the final validation accuracy is still within 2%. These algorithms can be adopted if there is no strong need for predictive performance. We however emphasize that exact algorithms must be used for making predictions, as we will show in Sec. 5.4. Finally, the algorithm NS without preprocessing works much worse than others, indicating the significance of our preprocessing strategy. TAB4 reports the average number of epochs, time, and total number of floating point operations to reach a given 96% validation accuracy on the largest Reddit dataset. Sparse and dense computations are defined in Sec. 4.6. We found that CVD+PP is about 5 times faster than Exact due to the significantly reduced receptive field size. Meanwhile, simply setting D (l) = 2 for NS does not converge to the given accuracy. We compare the quality of the predictions made by different algorithms, using the same model trained by Exact in Fig. 4. As Thm. 1 states, CV reaches the same testing accuracy as Exact, while NS and NS+PP perform much worse. Testing using exact algorithms (CV or Exact) corresponds to the weight scaling algorithm for dropout .Finally, we compare the average bias and variance of the gradients per dimension for first layer weights relative to the weights' magnitude in Fig. 5. For models without dropout, the gradient of CV+PP is almost unbiased. For models with dropout, the bias and variance of CV+PP and CVD+PP are ususally smaller than NS and NS+PP, as we analyzed in Sec. 4.3. The large receptive field size of GCN hinders its fast stochastic training. In this paper, we present a preprocessing strategy and two control variate based algorithms to reduce the receptive field size. Our algorithms can achieve comparable convergence speed with the exact algorithm even the neighbor sampling size D (l) = 2, so that the per-epoch cost of training GCN is comparable with training MLPs. We also present strong theoretical guarantees, including exact prediction and convergence to GCN's local optimum, for our control variate based algorithm. DISPLAYFORM0 H (l+1) DISPLAYFORM1 After one more epoch, all the activations h (l+1)CV,i,v are computed at least once for each v, soH DISPLAYFORM2 for all i > (l + 2)I. By induction, we know that after LI steps, we havē DISPLAYFORM3 2. We omit the time subscript i and denote DISPLAYFORM4 CV,v ). By back propagation, the approximated gradients by CV can be computed as follows DISPLAYFORM5 where • is the element wise product and σ (Z DISPLAYFORM6 CV) is the element-wise derivative. Similarly, denote DISPLAYFORM7 v ), the exact gradients can be computed as follows DISPLAYFORM8 Applying EP = EP,...,P (L) to both sides of Eq. 8, and utilizing DISPLAYFORM9 we have DISPLAYFORM10 Comparing Eq. 10 and Eq. 9 we get DISPLAYFORM11 We proof Theorem 2 in 3 steps:1. Lemma 1: For a sequence of weights W,..., W (N) which are close to each other, CV's approximate activations are close to the exact activations.,..., W (N) which are close to each other, CV's gradients are close to be unbiased.3. Theorem 2: An SGD algorithm generates the weights that changes slow enough for the gradient bias goes to zero, so the algorithm converges. The following proposition is needed in our proof DISPLAYFORM0 is the number of columns of the matrix A.• DISPLAYFORM1 Proof. DISPLAYFORM2 Proposition 2. There are a series of T inputs X 1,..., X T, X CV,1,..., X CV,T and weights W 1,..., W T feed to an one-layer GCN with CV DISPLAYFORM3 and an one-layer exact GCN DISPLAYFORM4 2. X CV,i − X CV,j ∞ < and X CV,i − X i ∞ < for all i, j ≤ T and > 0.Then there exists some K > 0, s.t., H CV,i − H CV,j ∞ < K and H CV,i − H i ∞ < K for all I < i, j ≤ T, where I is the number of iterations per epoch. Proof. Because for all i > I, the elements ofX CV,i are all taken from previous epochs, i.e., X CV,1,..., X CV,i−1, we know that DISPLAYFORM5 By triangular inequality, we also know DISPLAYFORM6 Since X CV,1 ∞,..., X CV,T ∞ are bounded, X CV,i ∞ is also bounded for i > I. Then, DISPLAYFORM7 and DISPLAYFORM8 The following lemma bounds CV's approximation error of activations Lemma 1. Given a sequence of model weights W 1,..., W T. If W i − W j ∞ <, ∀i, j, and all the activations are ρ-Lipschitz, there exists K > 0, s.t., DISPLAYFORM9 Proof. We prove by induction. Because DISPLAYFORM10 Repeatedly apply Proposition B.1 for L − 1 times, we get the intended . The following lemma bounds the bias of CV's approximate gradient Lemma 2. Given a sequence of model weights W 1,..., W T, if DISPLAYFORM0 2. all the activations are ρ-Lipschitz, 3. the gradient of the cost function ∇ z f (y, z) is ρ-Lipschitz and bounded, then there exists K > 0, s.t., DISPLAYFORM1 Proof. By Lipschitz continuity of ∇ z f (y, z) and Lemma 1, there exists K > 0, s.t., DISPLAYFORM2 By Eq. 9, Eq. 10 and Lemma 1, we have DISPLAYFORM3 By induction we know that for l = 1,..., L there exists K, s.t., DISPLAYFORM4 Again by Eq. 9, Eq. 10, and Lemma 1, DISPLAYFORM5 Finally, DISPLAYFORM6 Proof. This proof is a modification of , but using biased stochastic gradients instead. We assume the algorithm is already warmed-up for LI steps with the initial weights W 0, so that Lemma 2 holds for step i > 0. DISPLAYFORM0 By smoothness we have DISPLAYFORM1 Consider the sequence of LI + 1 weights W i−LI,..., W i. DISPLAYFORM2 By Lemma 2, there exists K > 0, s.t. DISPLAYFORM3 where DISPLAYFORM4 Taking EP,V B to both sides of Eq. 14 we have DISPLAYFORM5 Summing up the above inequalities and re-arranging the terms, we obtain, DISPLAYFORM6 Dividing both sides by DISPLAYFORM7. DISPLAYFORM8 Particularly, when N → ∞, we have E R∼P R EP,V B ∇L(W R) 2 = 0, which implies that the gradient is asymptotically unbiased. We test 3-layer GCNs on the Reddit dataset. The settings are the same with 2-layer GCNs in Sec. 5.3. To ensure the exact algorithm can run in a reasonable amount of time, we subsample the graph so that the maximum degree is 10. The convergence is shown as FIG7, which is similar with the two-layer models. The time consumption to reach 0.94 testing accuracy is shown in TAB6. We justify the independent Gaussian assumption in Sec. 4.3 by showing that for a 2-layer GCN with the first layer pre-processed, the neighbor's activations are independent. Without loss of generality, suppose that we want to compute z Assumption 1 is not GCN-specific and is discussed in , we now prove assumption 2 by the following lemma. Lemma 3. If a and b are independent random variables, then their transformations f 1 (a) and f 2 (b) are independent. Because for any event A and B, P (f 1 (a) ∈ f 1 (A), f 2 (b) ∈ f 2 (B)) = P (a ∈ A, b ∈ B) = P (a ∈ A)P (b ∈ B) = P (f 1 (a) ∈ f 1 (A))P (f 2 (B) ∈ f 2 (B)), where f 1 (A) = {f 1 (a)|a ∈ A} and f 2 (B) = {f 2 (b)|b ∈ B}. v and hv are independent. The can be further generalized to deeper models. If the receptive fields of two nodes does not overlap, they should be independent.where the indices i, j ∈n (l) (v). Then, we compute the average correlation of all pairs of neighbors i = j. AvgCorr (l,v,d):= 1 n (l) (v) (n (l) (v) − 1) i =j Corr (l,v,d) ij, and define the average neighbor correlation on layer l as AvgCorr (l,v,d) averaged over all the nodes v and dimensions d. We report the average feature correlation and the average neighbor correlation per layer, on the Citeseer, Cora, PubMed and PPI datasets. These quantities are too expensive to compute for NELL and Reddit. On each dataset, we train a GCN with 10 graph convoluation layers until early stopping criteria is met, and compute the average feature correlation and the average neighbor correlation for layer 1 to 9. We are not interested in the correlation on layer 10 because there are no more graph convolutional layers after it. The is shown as FIG10. As analyzed in Sec. G.1, the average neighbor correlation is close to zero on the first layer, but it is not exactly zero due to the finite sample size for computing the empirical covariance. There is no strong tendency of increased correlation as the number of layers increases, after the third layer. The average neighbor correlation and the average feature correlation remain on the same order of magnitude, so bringing correlated neighbors does not make the activations much more correlated than the MLP case . Finally, both correlations are much smaller than one.

Represent the TLDR for finding the corresponding scientific article

A control variate based stochastic training algorithm for graph convolutional networks that the receptive field can be only two neighbors per node.

Represent the short summary for finding the corresponding scientific article (long)

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose to apply Monte Carlo methods and importance sampling to sparsify and quantize pre-trained neural networks without any retraining. We obtain sparse, low bit-width integer representations that approximate the full precision weights and activations. The precision, sparsity, and complexity are easily configurable by the amount of sampling performed. Our approach, called Monte Carlo Quantization (MCQ), is linear in both time and space, while the ing quantized sparse networks show minimal accuracy loss compared to the original full-precision networks. Our method either outperforms or achieves competitive with methods that do require additional training on a variety of challenging tasks. Developing novel ways of increasing the efficiency of neural networks is of great importance due to their widespread usage in today's variety of applications. Reducing the network's footprint enables local processing on personal devices without the need for cloud services. In addition, such methods allow for reducing power consumption -also in data centers. Very compact models can be fully stored and executed on-chip in specialized hardware like for example ASICs or FPGAs. This reduces latency, increases inference speed, improves privacy concerns, and limits bandwidth cost. Quantization methods usually require re-training of the quantized model to achieve competitive . This leads to an additional cost and complexity. The proposed method, Monte Carlo Quantization (MCQ), aims to avoid retraining by approximating the full-precision weight and activation distributions using importance sampling. The ing quantized networks achieve close to the full-precision accuracy without any kind of additional training. Importantly, the complexity of the ing networks is proportional to the number of samples taken. First, our algorithm normalizes the weights and activations of a given layer to treat them as probability distributions. Then, we randomly sample from the corresponding cumulative distributions and count the number of hits for every weight and activation. Finally, we quantize the weights and activations by their integer count values, which form a discrete approximation of the original continuous values. Since the quality of this approximation relies entirely on (quasi)random sampling, the accuracy of the quantized model is directly dependent on the amount of sampling performed. Thus, accuracy may be traded for higher sparsity and speed by adjusting the number of samples. On the challenging tasks of image classification, language modeling, speech recognition, and machine translation, our method outperforms or is competitive with existing quantization methods that do require additional training. The computational cost of neural networks can be reduced by pruning redundant weights or neurons, which has been shown to work well; ). Alternatively, the precision of the network weights and activations may be lowered, potentially introducing sparsity. Using low precision computations to reduce the cost and sparsity to skip computations allows for efficient hardware implementations ). This is the approach used in this paper. BinaryConnect proposed training with binary weights, while XNOR-Net and BNN extended this binarization to activations as well. TWN proposed ternary quantization instead, increasing model expressiveness. Similarly, TTQ used ternary weights with a positive and negative scaling learned during training. LR-Net made use of both binary and ternary weights by using stochastic parameterization while INQ constrained weights to powers of two and zero. FGQ categorized weights in different groups and used different scaling factors to minimize the element-wise distance between full and low-precision weights. used the hardware accelerator's feedback to perform hardware-aware quantization using reinforcement learning. jointly trained quantized networks and respective quantizers. used Bloomier filters to compactly encode network weights. Similarly, quantization techniques can also be applied in the backward pass. Therefore, some previous work quantized not only weights and activations but also the gradients to augment training performance (; ;). In particular, RQ propose a differentiable quantization procedure to allow for gradient-based optimization using discrete values and recently proposed to discretize weights, activations, gradients, and errors both at training and inference time. These quantization techniques have great benefits and have shown to successfully reduce the computation requirements compared to full-precision models. However, all the aforementioned methods require re-training of the quantized network to achieve close to full-precision accuracy, which can introduce significant financial and environmental cost . On the other hand, our method instantly quantizes pre-trained neural networks with minimal accuracy loss as compared to their full-precision counterparts without any kind of additional training. Neural networks make extensive use of randomization and random sampling techniques. Examples are random initialization of network weights, stochastic gradient descent , regularization techniques such as Dropout and DropConnect , data augmentation and data shuffling, recurrent neural networks' regularization (a), or the generator's noise input on generative adversarial networks . Many state-of-the-art networks use ReLU , which has interesting properties such as scale-invariance. This enables a scaling factor to be propagated through all network layers without affecting the network's original output. This principle can be used to normalize network values, such as weights and activations, as further described in Section 3.1. After normalization, these values can be treated as probabilities, which enables the simulation of discrete probability densities to approximate the corresponding full-precision, continuous distributions (Section 3.2). Assuming the exclusive use of the ReLU activation function in the hidden layers, the scale-invariance property of the ReLU activation function allows for arbitrary scaling of the weights or activations without affecting the network's output. Given weights w l−1,i,j connecting the i-th neuron in layer l − 1 to the j-th neuron in layer l, where i ∈ [0, N l−1 − 1] and j ∈ [0, N l − 1], with N l−1 and N l the number of neurons of layer l −1 and l, respectively. Let a l,j be the j-th activation in the l-th layer and f ∈ R +: a l,j = max 0, Biases and incoming weights for neuron j in layer l may then be normalized by f = w l−1,j 1 = N l−1 −1 i=0 |w l−1,i,j |, enabling weights to be seen as a probability distribution over all connections to a neuron. A similar procedure could be used to normalize all activations a l,j of layer l. Propagating these scaling factors forward layer by layer in a single scalar (per output), which converts the outputs of the normalized network to the same range as the original network. This technique allows for the usage of integer weights and activations throughout the entire network without requiring rescaling or conversion to floating point at every layer. Taking advantage of the normalized network, we can simulate discrete probability densities by constructing a probability density function (PDF) and then sampling from the corresponding cumulative density function (CDF). The number of references of a weight is then the quantized integer approximation of the continuous value. For simplicity, the following discussion shows the quantization procedure for weights; activations can be quantized in the same way at inference time. Without loss of generality, given n weights, assuming n−1 k=0 |w k | = w 1 = 1 and defining a partition of the unit interval by P m:= m k=1 |w k | we have the following partitions: Then, given N uniformly distributed samples x i ∈, we can approximate the weight distribution as follows: where j i ∈ {0, . . ., n − 1} is uniquely determined by P ji−1 ≤ x i < P ji. One can further improve this sampling process by using jittered equidistant sampling. Thus, given a random variable ξ ∈, we generate N uniformly distributed samples x i ∈ such that The combination of equidistant samples and a random offset improves the weight approximation, as the samples are more uniformly distributed. The variance of different sampling seeds is discussed in the Appendix. Our approach builds on the aforementioned ideas of network normalization and quantization using random sampling to quantize an entire pre-trained full-precision neural network. As before, we focus on weight quantization; online activation quantization is discussed in Section 4.4. Our method, called Monte Carlo Quantization (MCQ), consists of the following steps, which are executed layer by layer: Create a probability density function (PDF) for all N l,w weights of layer l such that Perform importance sampling on the weights based on their magnitude by sampling from the corresponding cumulative density function (CDF) and counting the number of hits per weight (Section 4.2). Replace each weight with its quantized integer value, i.e. its hit count, to obtain a low bit-width, integer weight representation (Section 4.3). The pseudo-code for our method is shown in Algorithm 1 of the Appendix. Figure 1 illustrates both the normalization and importance sampling processes for a layer with 10 weights and 1 sample per weight, i.e. K = 1.0. Performing normalization neuron-wise, as introduced in Section 3.1 may in an inferior approximation, especially when the number of weights to sample from is small, as for example in convolutional layers with a small number of filters or input channels. To mitigate this, we propose to normalize all neurons simultaneously in a layer-wise manner. This has the additional advantage that samples can be redistributed from low-importance neurons to high-importance neurons (according to and uniformly sample from the corresponding CDF (c). The sampling process produces quantized integer network weights based on the number of hits per weight (d). Note that since weights 7, 8, and 9 were not hit, sparsity is introduced which can be exploited by hardware accelerators. some metric), ing in an increased level of sparsity. Additionally, there is more opportunity for global optimization, so the overall weight distribution approximation improves as well. We use the 1-norm of all weights of a given layer l as the scaling factor f used to perform weight normalization. Thus, each normalized weight can be seen as a probability with respect to all connections between layer l − 1 and layer l, instead of a single neuron. This layer-wise normalization technique is similar to Weight Normalization , which decouples the neuron weight vector magnitude from its direction. As introduced in Section 3.2, we generate ternary samples (hit positive weight, hit negative weight, or no hit), and count such hits during the sampling process. Note that even though the individual samples are ternary, the final quantized values may not be, because a single weight can be sampled multiple times. For jittered sampling, we use one random offset per layer, with a number of samples N = K · N values, where K ∈ R + is a user-specified parameter to control the number of samples and N values represents the number of weights of a given layer. By varying K, the computational cost of sampling can be traded off better approximation (more bits per weight) of the original weight distribution, leading to higher accuracy. In our experiments, K is set the same for all network layers. One simple modification to enhance the quality of the discrete approximation is to sort the continuous values prior to creating the PDF. Applying sorting mechanisms to Monte Carlo schemes has been shown to be beneficial in the past (; . Sorting groups smaller values together in the overall distribution. Since we are using a uniform sampling strategy, smaller weights are then sampled less often, which in both higher sparsity and a better quantized approximation of the larger weights in practice. This effect is particularly significant on smaller layers with fewer weights. Since the quantized integer weights span a different range of values than the original weights, and biases remain unchanged, care must be taken to ensure the activations of each neuron are calculated correctly. After the integer multiply-accumulate (MAC) operation, the must then be scaled by f N before adding the bias. This requires the storage of one floating point scaling value per layer. However, weights are stored as low bit-width integers and the computational cost is greatly reduced since the MAC operations use low-precision integers only instead of floating point numbers. The number of bits required for the weights B W l ∈ N, for layer l and its quantized weights Q(w l,i), corresponds to the bit amount needed to represent the highest hit count during sampling, including its sign: B W l = 1 + log 2 (max 0≤i≤Nw−1 |Q(w l,i)|) + 1. Alternatively, positive and negative weights could be separated into two sets. While weights are quantized offline, i.e. after training and before inference, activations are quantized online during inference time using the same procedure as weight quantization previously described. Thus, in the normalization step (Section 4.1), all N l,a activations of a given layer l are treated as a probability distribution over the output features, such that N l,a −1 j=0 |a l,j | = 1. Then, in the importance sampling step (Section 4.2), activations are sub-sampled using possibly different relative sampling amounts, i.e. K, than the ones used for the weights (we use the same K for both weights and activations in all of our experiments). The required number of bits B A l for the quantized activations Q(a l,j) can also be calculated similarly as described in Section 4.3, although no additional bit sign is required when using ReLU since all activations are non-negative. The proposed method is extensively evaluated on a variety of tasks: for image classification we use CIFAR-10 , SVHN , and ImageNet , on multiple models each. We further evaluate MCQ on language modeling, speech recognition, and machine translation, to assess the preformance of MCQ across different task domains. Due to the automatic quantization done by MCQ, some layers may be quantized to lower or higher levels than others. We indicate the quantization level for the whole network by the average number of bits, e.g.'8w-32a' means that on average 8 bits were used for weights and 32 bits for activations on each layer. Many works note that quantizing the first or last network layer reduces accuracy significantly; ). We use footnotes 1, 2, and 3 to denote the special treatment of first or last layers respectively. For MCQ we report the with both quantized and full-precision first layer. We do not quantize Batch Normalization layers as the parameters are fixed after training and can be incorporated into the weights and biases . Tables 1 to 4 show the accuracy difference ∆ between the quantized and full-precision models. For other compared works this difference is calculated using the baseline models reported in each of the respective works. We didn't perform any search over random sampling seeds for MCQ's . The best accuracies on VGG-7, VGG-14, and ResNet-20 produced by our method using K = 1.0 on CIFAR-10 are shown in Table 1. We refer to the Appendix for model and training details. MCQ outperforms or shows competitive showing minimal accuracy loss on all tested models against the compared methods that require network re-training. The full-precision baselines for BNN and XNOR-Net are from BC as these works use the same model. Similarly, BWN 's on VGG-7 are the ones reported in TWN since they did not report the baseline in the original paper. Figure 2 shows the effects of varying the amount of sampling, i.e. using K ∈ [0.1...2.0].The average percentage of used weights/activations per layer and corresponding bit-widths of the final quantized model is also presented on each graph. We observe a rapid increase of the accuracy even when sparsity levels are high on all tested models. For SVHN, the tested models are identical to the compared methods. Models B, C, and D have the same architecture as Model A but with a 50%, 75%, and 87.5% reduction in the number of filters in each convolutional layer, respectively . We refer to the Appendix for further model and training details. Table 2 shows MCQ's for several models on SVHN using K = 1.0. On bigger models, i.e. VGG-7* and Model A, we see minimal accuracy loss when compared to the full-precision baselines. For the smaller models, we observe a slight accuracy degradation as model size decreases due to the reduction in the sample size, ing in a poorer approximation. However, we used only about 4 bits per weight/activation for such models. Thus, increasing the number of samples would improve Figure 2: Results of quantizing both weights and activations on CIFAR-10 using different sampling amounts. The quantized models reach close to full-precision accuracy at around half the sample size while using only around half the weights and one-third of the activations of the full-precision models. accuracy while still maintaining a low bit-width. Figure 3 illustrates the consequences of varying the number of samples. Less samples are required than on CIFAR-10 for bigger models to achieve close to full-precision accuracy. Potentially this is because layers have a larger number of weights and activations, so a larger sample size reduces quantization noise since the important values being more likely to be better approximated. For ImageNet, we evaluate MCQ on AlexNet, ResNet-18, and ResNet-50 using the pre-trained models provided by Pytorch's model zoo ). Table 3 shows the on ImageNet with K = 5.0 for the different models. The shown for DoReFa, BWN, TWN (; ;) are the ones reported in TTQ . Figure 4 shows the accuracy of the quantized model when using different sample sizes, i.e., K ∈ [0.25, ..., 5.0]. We observe that more sampling is required to achieve a close to full-precision model accuracy on ImageNet. On this dataset, sorting the CDF before sampling didn't in any improvements, so reported are without sorting. All the quantized models achieve close to full-precision accuracy, though more samples are required than for the previous datasets ing in a higher required bit-width. To assess the robustness of MCQ, we further evaluate MCQ on several models in natural language and speech processing. We evaluate language modeling on Wikitext-103 using a Transformer-based model and Wikitext-2 using a 2-layer LSTM , speech recognition on VCTK using Deepspeech2 , and machine translation on WMT-14 English-to-French using a Transformer . Additional details are provided in the Appendix. Table 4 shows the comparison to full-precision models for these various tasks. (1W-32A) +0.14 ----∆ BNN (1W-1A) - Results of quantizing both weights and activations on SVHN using different sampling amounts. The quantized VGG-7* model reaches close to full-precision accuracy using around 0.5 samples per weight/activation, requiring around 8 bits and using 22% of the weights of the original model, with 22% nonzero activations. Model A, B, C, and D are less redundant models that require more sampling to achieve close to full-precision accuracy. The experimental show the performance of MCQ on multiple models, datasets, and tasks, demonstrated by the minimal loss of accuracy compared to the full-precision counterparts. MCQ either outperforms or is competitive to other methods that require additional training of the quantized network. Moreover, the trade-off between accuracy, sparsity, and bit-width can be easily controlled by adjusting the number of samples. Note that the complexity of the ing quantized network is proportional to the number of samples in both space and time. One limitation of MCQ, however, is that it often requires a higher number of bits to represent the quantized values. On the other hand, this sampling-based approach directly translates to a good approximation of the real full-precision values without any additional training. proposed to outlier channel splitting, which is orthogonal work to MCQ and could be used to reduce the bit-width required for the highest hit counts. There are several paths that could be worth following for future investigations. In the importance sampling stage, using more sophisticated metrics for importance ranking, e.g. approximation of the Hessian by Taylor expansion could be beneficial . Automatically selecting optimal sampling levels on each layer could lead to a lower cost since later layers seem to tolerate more sparsity and noise. For efficient hardware implementation, it's important that the quantized Figure 4: Results of quantizing both weights and activations on ImageNet using different sampling amounts. All quantized models reach close to full-precision accuracy at K = 3. Table 4: Evaluation of MCQ on language modeling, speech recognition, and machine translation. All quantized models reach close to full precision performance. Note that, as opposed to the image classification task, we did not study different sampling amounts nor the effect of quantization on specific network layers. A more in-depth analysis could then help to achieve close to full-precision accuracy at a lower bit-width on these additional models. network can be executed using integer operations only. Bias quantization and rescaling, activation rescaling to prevent overflow or underflow, and quantization of errors and gradients for efficient training leave room for future work. In this work, we showed that Monte Carlo sampling is an effective technique to quickly and efficiently convert floating-point, full-precision models to integer, low bit-width models. Computational cost and sparsity can be traded for accuracy by adjusting the number of sampling accordingly. Our method is linear in both time and space in the number of weights and activations, and is shown to achieve similar as the full-precision counterparts, for a variety of network architectures, datasets, and tasks. In addition, MCQ is very easy to use for quantizing and sparsifying any pre-trained model. It requires only a few additional lines of code and runs in a matter of seconds depending on the model size, and requires no additional training. The use of sparse, low-bitwidth integer weights and activations in the ing quantized networks lends itself to efficient hardware implementations. A ALGORITHM An overview of the proposed method is given in Algorithm 1. Input: Pre-trained full-precision network Output: Quantized network with integer weights for K=0 to L-1 do // Update layer's precision B W K ← 1 + f loor(log 2 (max(abs(W K)))) + 1; end Algorithm 1: Monte Carlo Quantization (MCQ) on network weights. L represents the number of trainable layers, K indicates the percentage of samples to be sampled per weight. The process is performed equivalently for quantizing activations at inference time. Our algorithm is linear in both time and space in the number of weights and activations. When using integer weights, care has to be taken to avoid overflows in the activations. For that, activations can be scaled using a dynamically computed shifting factor as in. With Monte Carlo sampling, since we know the expected value of the next-layer activations, we can scale accordingly. With the activation equation presented in Section 3.1 and N I connections from the input layer to every neuron in the second layer: With The activations of a neuron need to be scaled by its number of inputs (the receptive field F in), multiplied with the number of samples per weight and the number of samples per activation. This is also valid for neurons in convolutional layers, where the receptive field is 3D, e.g. 3 × 3 × 128. Moreover, care must be taken to scale biases correctly, by taking both the scaling of weights and activations into account: We trained our full-precision baseline models on the CIFAR-10 dataset , consisting of 50000 training samples. We evaluated both our full-precision and quantized models similarly on the rest of the 10000 testing samples. The full-precision VGG-7 (2×128C3−M P 2−2× 256C3−M P 2−2×512C3−M P 2−1024F C −Sof tmax) and VGG-14 (2×64C3−M P 2−2× 128C3−M P 2−3×256C3−M P 2−3×512C3−M P 2−3×512C3−M P 2−1024F C−Sof tmax) models were trained using the code at https://github.com/bearpaw/pytorch-classification. Each was trained for 300 epochs with the Adam optimizer, with a learning rate starting at 0.1 and decreased by factor 10 at epochs 150 and 225, batch size of 128, and weights decay of 0.0005. The ResNet-20 model uses the standard configuration described, with 64, 128 and 256 filters in the respective residual blocks. We used more filters to increase the number of available weights in the first block to sample from. This could be similarly performed by sampling more on this specific model to reduce the accuracy loss. The ResNet-20 model is trained using the same hyperparameter settings as the VGG models. We trained our full-precision baseline models on the Street View House Numbers (SVHN) dataset , consising of 73257 training samples. We evaluated both our full-precision and quantized models similarly using the 26032 testing samples provided in this dataset. The fullprecision VGG-7* model (2 × 64C3 − M P 2 − 2 × 128C3 − M P 2 − 2 × 256C3 − M P 2 − 1024F C − Sof tmax) was trained for 164 epochs, using the Adam optimizer with learning rate starting at 0.001 and divided by 10 at epochs 80 and 120, weight decay 0.001, and batch size 200. Models A (48C3 − M P 2 − 2 × 64C3 − M P 2 − 3 × 128C3 − M P 2 − 512C3 − Sof tmax), B, C, and D were trained using the code at https://github.com/aaron-xichen/pytorch-playground and the same hyperparameter settings as VGG-7* but trained for 200 epochs. We evaluated both our full-precision and quantized models similarly on the validation set of the ILSVRC12 classification dataset , consisting of 50K valida-tion images. The full-precision pre-trained models are taken from Pytorch's model zoo https://pytorch.org/docs/stable/torchvision/models.html . CSTR's VCTK Corpus (Centre for Speech Technology Voice Cloning Toolkit) includes speech data uttered by 109 native speakers of English with various accents, where each speaker reads out about 400 sentences, most of which were selected from a newspaper. The evaluated model uses 2 convolutional layers and 5 GRU layers of 768 hidden units, using code from https://github.com/SeanNaren/deepspeech.pytorch . The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers -all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. The WikiText-2 model was a 2-layer LSTM with 650 hidden neurons, and an embedding size of 400. It was trained using the setup and code at https://github.com/salesforce/awdlstm-lm (b). The WikiText-102 model was a pretrained model available at https://github.com/pytorch/fairseq/tree/master/examples/language model, along with evaluation code . The dataset is WMT14 English-French, cmobining data from several other corpuses, amongst others the Europarl corpus, the News Commentary corpus, and the Common Crawl corpus . The model was a pretrained model available at https://github.com/pytorch/fairseq/tree/master/examples/scaling nmt, along with evaluation code . Figures 5, 6, and 7 show the effects of varying the amounts of sampling when quantizing only the weights. E QUANTIZING ACTIVATIONS ONLY Figures 8, 9, and 10 show the effects of varying the amounts of sampling when quantizing only the activations. We observe less sampling is required to achieve full-precision accuracy when quantizing only the activations when compared to quantizing the weights only. In a small experiment on CIFAR-10, we observe that using different sampling seeds can in up to a ≈ 0.5% absolute variation in accuracy of the different quantized networks (Figure 11). Grid searching over several sampling seeds may then be beneficial to achieve a better quantized model in the end, depending on the use-case.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Monte Carlo methods for quantizing pre-trained models without any additional training.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose the Information Maximization Autoencoder (IMAE), an information theoretic approach to simultaneously learn continuous and discrete representations in an unsupervised setting. Unlike the Variational Autoencoder framework, IMAE starts from a stochastic encoder that seeks to map each input data to a hybrid discrete and continuous representation with the objective of maximizing the mutual information between the data and their representations. A decoder is included to approximate the posterior distribution of the data given their representations, where a high fidelity approximation can be achieved by leveraging the informative representations. We show that the proposed objective is theoretically valid and provides a principled framework for understanding the tradeoffs regarding informativeness of each representation factor, disentanglement of representations, and decoding quality. A central tenet for designing and learning a model for data is that the ing representation should be compact yet informative. Therefore, the goal of learning can be formulated as finding informative representations about the data under proper constraints. Generative latent variable models are a popular approach to this problem, where a model parameterized by θ of the form p θ (x) = p θ (x|z)p(z)dz is used to represent the relationship between the data x and the low dimensional latent variable z. The model is optimized by fitting the generative data distribution p θ (x) to the training data distribution p(x), which involves maximizing the likelihood for θ. Typically, this model is intractable even for moderately complicated functions p θ (x|z) with continuous z. To remedy this issue, variational autoencoder (VAE) BID13 BID19 proposes to maximize the evidence lower bound (ELBO) of the marginal likelihood objective. However, as was initially pointed out in BID10, maximizing ELBO also penalizes the mutual information between data and their representations. This in turn makes the representation learning even harder. Many recent efforts have focused on resolving this problem by revising ELBO. Generally speaking, these works fall into two lines. One of them targets "disentangled representations" by encouraging the statistical independence between representation components BID9 BID12 BID8 BID4 BID7, while the other line of work seeks to control or encourage the mutual information between data and their representations BID16 BID3 BID1 BID6 ). However, these approaches either in an invalid lower bound for the VAE objective or cannot avoid sacrificing the mutual information. Instead of building upon the generative latent variable model, we start with a stochastic encoder p θ (z|x) and aim at maximizing the mutual information between the data x and its representations z. In this setting, a reconstruction or generating phase can be obtained as the variational inference of the true posterior p θ (x|z). By explicitly seeking for informative representations, the proposed model yields better decoding quality. Moreover, we show that the information maximization objective naturally induces a balance between the informativeness of each latent factor and the statistical independence between them, which gives a more principled way to learn semantically meaningful representations without invalidating ELBO or removing individual terms from it. Another contribution of this work is proposing a framework for simultaneously learning continuous and discrete representations for categorical data. Categorical data are ubiquitous in real-world tasks, where using a hybrid discrete and continuous representation to capture both categorical information and continuous variation in data is more consistent with the natural generation process. In this work, we focus on categorical data that are similar in nature, i.e., where different categories still share similar variations (features). We seek to learn semantically meaningful discrete representations while maintaining disentanglement of the continuous representations that capture the variations shared across categories. We show that, compared to the VAE based approaches, our proposed objective gives a more natural yet effective way for learning these hybrid representations. Recently, there has been a surge of interest in learning interpretable representations. Among them, β-VAE BID9 ) is a popular method for learning disentangled representations, which modifies ELBO by increasing the penalty on the KL divergence between the variational posterior and the factorized prior. However, by using large weight for the KL divergence term, β-VAE also penalizes the mutual information between the data and the latent representations more than a standard VAE does, ing in more severe under utilization of the latent representation space. Several follow up works propose different approaches to address the limitations of β-VAE. BID6 BID1 BID3 BID16 propose to constrain the mutual information between the representations and the data by pushing its upper bound, i.e., the KL divergence term in ELBO, towards a progressively increased target value. However, specifying and tuning this target value can itself be very challenging, which makes this method less practical. Moreover, this extra constraint in an invalid lower bound for the VAE objective. Alternatively, drops the mutual information term in ELBO. By pushing only the aggregated posterior towards a factorial prior, they implicitly encourage independence across the dimensions of latent representations without sacrificing the informativeness of the representations. However, simply removing the mutual information term also violates the lower bound of the VAE objective. Another relevant line of work BID8 BID12 BID4 BID7 seek to learn disentangled representations by explicitly encouraging statistical independence between latent factors. They all propose to minimize the total correlation term of the latent representations, either augmented as an extra term to ELBO or obtained by reinterpreting or re-weighting the terms in the VAE objective, as a way to encourage statistical independence between the representation components. In contrast, we show that our information maximization objective inherently contains the total correlation term while simultaneously seeking to maximize the informativeness of each representation factor. In this paper, we introduce a different perspective to the growing body of the VAE based approaches for unsupervised representation learning. Starting by seeking informative representations for the data, we follow a more intuitive way to maximize the mutual information between the data and the representations. Moreover, we augment the continuous representation with a discrete one, which allows more flexibilities to model real world data that are generated from different categories. We invoke the information maximization principle BID15 BID2 with proper constraints implied by the objective itself to avoid degenerate solutions. The proposed objective gives a theoretically elegant yet effective way to learn semantically meaningful representations. Given data x ∈ R d, we consider learning a hybrid continuous-discrete representation, denoted respectively with variables z ∈ R K1 and y ∈ {1, . . ., K 2}, using a stochastic encoder parameterized by θ, i.e., p θ (y, z|x). We seek to learn compact yet semantically meaningful representations in the sense that they should be low dimensional but informative enough about the data. A natural approach is to maximize the mutual information BID5 ) I θ (x; y, z) between the data and its representations under the constraint K 1, K 2 d. Here the mutual information between two random variables, e.g., x and z, is defined as DISPLAYFORM0 is the entropy of z and H θ (z|x) = −E p θ (x,z) [log p θ (z|x)] is the conditional entropy of z given x. The mutual information can be interpreted as the decrease in uncertainty of one random variable given another random variable. In other words, it quantifies how much information one random variable has about the other. A probabilistic decoder q φ (x|y, z) is adopted to approximate the true posterior p θ (x|y, z), which can be hard to estimate or even intractable. The dissimilarity between them is optimized by minimizing the KL divergence D KL (p θ (x|y, z)||q φ (x|y, z)). In summary, IMAE considers the following, DISPLAYFORM1 Given that H(x) is independent of the optimization procedure, we can show that optimizing is equivalent to optimize the following 1, DISPLAYFORM2 We set β > 0 to balance between maximizing the informativeness of latent representations and maintaining the decoding quality. The second term is often interpreted as the "reconstruction error" which can be optimized using the reparameterization tricks proposed by BID13 and BID11 for continuous representation z and discrete representation y respectively. Now we introduce proper method to optimize the first term I θ (x; y, z) in. We first show that I θ (x; y, z) inherently involves two keys terms that quantify the informativeness of each representation factor and the statistical dependence between these factors. Assuming the conditional distribution of the representation (y, z) given x is factorial, we also assume the marginal distribution of y and z are independent, i.e., DISPLAYFORM0 The first two terms of the RHS quantify how much information each latent factor, i.e., y or z k, carry about the data. The last term is known as the total correlation of z BID21, which quantifies the statistical independence between the continuous latent factors and achieves the minimum if and only if they are independent of each other. As is implied by, maximizing I θ (x; y, z) can be conducted by maximizing informativeness of each latent factor while simultaneously promoting statistical independence between the continuous factors. Various Monte Carlo based sampling strategies have been proposed to optimize the total correlation term BID4 BID7; in this work we follow this line (see Appendix B). Next we proceed by constructing tractable approximations for I θ (x; z k) and I θ (x; y) respectively. Without any constraints, the mutual information I θ (x; z k) between a continuous latent factor and data can be trivially maximized by severely fragmenting the latent space. To be more precise, consider the following proposition. While similar have likely been established in the information theory literature, we include this proposition to motivate our objective design. DISPLAYFORM0 The equality in is attained if and only if z k is Gaussian distributed, given which we have DISPLAYFORM1 Note here both µ k (x) and σ k (x) are random variables. The above implies that z k is more informative about x if it has less uncertainty given x yet captures more variance in data, i.e., σ k (x) is small while µ k (x) disperses within a large space. However, this can in discontinuity of z k, where in the extreme case each data sample is associated with a delta distribution in the latent space. In light of this, we can make what we described above more precise. A vanishing variance of the conditional distribution p(z k |x) leads to a plain autoencoder that maps each data sample to a deterministic latent point, which can fragment the latent space in a way that each data sample corresponds with a delta distribution in the latent space DISPLAYFORM2 On the other hand, Proposition 1 also implies that controlling the variance σ k (x) to be finite, I θ (x; z k) will be maximized by pushing µ k (x) towards two extremes (±∞). To remedy this issue while achieving the upper bound, a natural resolution is to squeeze z k within the domain of a Gaussian distribution with finite mean and variance. By doing so, we can avoid the degenerate solution while achieving a more reasonable trade-off between enlarging the spread of µ k (x) and maintaining the continuity of z. Therefore, we consider the following as the surrogate for maximizing I θ (x; z k), DISPLAYFORM3 Here r(z k) are i.i.d scaled normal distribution with finite variance. That is, we push each p θ (z k) towards a Gaussian distribution r(z k) by minimizing the KL divergence between them. Unlike the continuous representation, the mutual information I θ (x; y) between a discrete representation and data can be well approximated, given the fact that the cardinality of the space of y is typically low. To be more specific, given N i.i.d samples {x n} N n=1 of the data, the empirical estimation of I θ (x; y) under the conditional distribution p θ (y|x n) follows as DISPLAYFORM0 As shown in Proposition 2, with a suitably large batch of samples, the empirical mutual information I θ (x; y) is a good approximation to I θ (x; y). This enables us to optimize I θ (x; y) in a theoretically justifiable way that is amenable to stochastic gradient descent with minibatches of data. Proposition 2. Let y be a discrete random variable that belongs to some categorical class C. Assume the marginal probabilities of the true and the predicted labels are bounded below, i.e. p θ (y), p θ (y) ∈ [1/(CK 2), 1] for all y ∈ C with some constant C > 1. Then for any δ ∈, DISPLAYFORM1 Here N denotes the number of samples used to establish I θ (x; y) according to Eq.Therefore, to maximize the mutual information I θ (x; y), we consider the following: max L θ (y):= I θ (x; y). Maximizing the the mutual information I θ (x; y) provides a natural way to learn discrete categorical representations. To see this, notice that I θ (x; y) contains two fundamental quantities, the category balance term H θ (y) and the category separation term H θ (y|x). In other words, maximizing I θ (x; y) trades off uniformly assigning data over categories and seeking highly confident categorical identity for each sample x. The maximum is achieved if p θ (y|x) is deterministic while the marginal distribution p θ (y) is uniform, that is H θ (y|x) = 0 and H θ (y) = log K 2.Overall Objective As a summary of and, our overall objective is DISPLAYFORM2 The first three terms associate with our information maximization objective, while the last one aims at better approximation of the posterior p θ (x|y, z). A better balance between these two targets can be achieved by weighting them differently. One the other hand, the informativeness of each latent factor can be optimized through L θ (z) and L θ (y), while statistically independent latent continuous factors can be promoted by minimizing the total correlation term D KL p(z)||Π K1 k=1 p(z k). Therefore, trade-offs can be formalized regarding the informativeness of each latent factor, disentanglement of the representation, and better decoding quality. This motivates us to consider the following objective, let β, γ > 0, DISPLAYFORM3 We compare IMAE against various VAE based approaches that are summarized in Figure 1. We would like to demonstrate that IMAE can (i) successfully learn a hybrid of continuous and discrete representations, with y matching the intrinsic categorical information y true well and z capturing the disentangled feature information shared across categories; (ii) outperform the VAE based models by achieving a better trade-off between representation interpretability and decoding quality. We choose the priors r(z) and r(y) to be the isotropic Gaussian distribution and uniform distribution respectively. Detailed experimental settings are provided in Appendix G. Figure 1: Summarization of relevant work. β-VAE modifies ELBO by increasing the penalty on the KL divergence terms. InfoVAE drops the mutual information terms from ELBO. JointVAE seeks to control the mutual information by pushing the their upper bounds (the associated KL divergence terms) towards progressively increased values, C y &C z. We drop the subscripts θ and φ hereafter. DISPLAYFORM0 We first qualitatively demonstrate that informative representations can yield better interpretability. For the continuous representation, FIG0 validates Proposition 1 by showing that, with roughly same amount of variance for each latent variable z k, those achieving high mutual information with the data have mean values µ k (x) of the conditional probability p(z k |x) disperse across data samples and variances σ k (x) decrease to small values for all data samples. As a qualitative evaluation, we traverse latent dimensions corresponding with different levels of I(x, z k). As seen in FIG0 (b)-(d), informative variables in the continuous representation have uncovered intuitive continuous factors of the variation in the data, while the factor z 8 has no mutual information with the data and shows no variation. We observe the same phenomenon for the discrete representation y in FIG0 (e)&(f), which were obtained with two different values of β and γ, where the more informative one discovers matches the natural labels better. This provides further evidence for that interpretable latent factors can be attained by maximizing the mutual information between the representations and the data. We set γ = 2β for IMAE. For each β, we run each method over 10 random initializations. In this section, we perform quantitative evaluations on MNIST , Fashion MNIST and dSprites BID17. We show that IMAE achieves better interpretability vs. decoding quality trade-off. Unsupervised learning of discrete latent factor Before we present our main , we first describe an assumption that we make on the discrete representations. For the discrete representation, a reasonable assumption is that the conditional distribution p(y|x) should be locally smooth so that the data samples that are close on their manifold should have high probability of being assigned to the same category BID0. This assumption is crucial for using neural networks to learn discrete representations, since it's easy for a high capacity model to learn a non-smooth function p(y|x) that can abruptly change its predictions without guaranteeing similar data samples will be mapped to similar y. To remedy this issue, we adopt the virtual adversarial training (VAT) trick proposed by BID18 and augment L θ (y) as follows: DISPLAYFORM0 The second term of RHS regularizes p θ (y|x) to be consistent within the norm ball of each data sample so as to maintain the local smoothness of the prediction model. For fair comparison, we augment all four methods with VAT. As demonstrated in Appendix D, using VAT is essential for all of them except β-VAE to learn interpretable discrete representations. We start by evaluating different methods on MNIST and Fashion MNIST, for which we train over a range of β values (we set γ = 2β for IMAE).Discrete representations For the discrete representations, by simply pushing the conditional distribution p(y|x) towards the uniform distribution r(y), β-VAE sacrifices the mutual information I(x; y) and hence struggles in learning interpretable discrete representation even with VAT. As a comparison, InfoVAE performs much better by dropping I(x; y) from ELBO. For data that are distinctive enough between categories (MNIST), with large β values InfoVAE performs well by uniformly distributing the whole data over categories through minimizing D KL (p(y)||r(y)) while simultaneously encouraging local smoothness with VAT. However, InfoVAE struggles with less distinctive data (Fashion-MNIST), where it cannot give fairly confident category separation by only DISPLAYFORM0 Figure 4: For each image, the first row is the digit type learnt by the model, where each entry is obtained by feeding the decoder with the averaged z values corresponding with the learnt y. The second row is obtained by traversing the "angle" latent factor within [−2, 2] on digit 6. IMAE is capable of uncovering the underlying discrete factor over a wide range of β values. More interpretable continuous representations can be obtained when the method is capable of learning discrete representations, since less overlap between the mainfolds of each category is induced.requiring local smoothness. In contrast, IMAE achieves much better performance by explicitly encouraging confident category separation via minimizing the conditional entropy H(y|x), while using VAT to maintain local smoothness so as to prevent overfitting of neural network. Although JointVAE performs much better than β-VAE by pushing the upper bound of I(x; y) towards a progressively increasing target value C y, we found it can easily get stuck at some bad local optima where I(x; y) is comparatively large while the accuracy is poor. A heuristic is that once JointVAE enters the local region of a local optima, progressively increasing C y only induces oscillation within that region. Informativeness, interpretability and decoding quality As illustrated in Figure 1, by using large β values, β-VAE sacrifices more mutual information between the data and its representations, which in turn (see FIG1) in less informative representations followed by poor decoding quality. In contrast, the other three methods can remedy this issue to different degrees, and hence attains better trade-off regarding informativeness of latent representations and decoding quality. Compared to JointVAE and InfoVAE, IMAE is more capable of learning discrete presentations over a wide range of β, γ values, which implies less overlap between the manifolds of different categories is induced. As a , IMAE is expected to yield better decoding quality for each category. Although InfoVAE and JointVAE can also learn comparatively good discrete representations when using large and small β values respectively, the corresponding of these two regions associate with either poor decoding quality or much lower disentanglement score (see section 4.2.2). In contrast, IMAE consistently performs well with different hyperparameters, especially in the region of interest where the decoding quality as well as the informativeness of latent representations are good enough. In this section, we quantitatively evaluate the disentanglement capability of IMAE on dSprites where the ground truth factors of both continuous and discrete representaions are available. We use the disentanglement metric proposed by BID4, which is defined in terms of the gap between the top two empirical mutual information of each latent representation factor and a ground truth factor. The disentanglement score is defined as the weighted average of the gaps. A high disentanglement score implies that each ground truth factor associates with one single representation factor that is more informative than the others, i.e., the learnt representation factors are more disentangled.4 FIG3 shows that, with large β values, β-VAE penalizes the mutual information too much and this degrades the usefulness of representations. while all other three methods achieve higher disentanglement score with better decoding quality. For JointVAE, higher β values push the upper bound of mutual information converges to the prefixed target value, it therefore can maintain more mutual (a) IMAE performs well regarding the disentanglement score vs. decoding quality trade-off, especially in the region of interest where both decoding quality and informativeness of representations are fairly good.(b) Negative correlation between total correlation and disentanglement score. It also implies that the disentanglement score tends to decrease along with the total correlation if using even larger β, due to the diminishing informativeness of representation factors. In the extreme case, both total correlation and disentanglement score can degrade to zero. information between the data and the whole latent representations and give better decoding quality. However, the disentanglement quality is poor in this region, which implies that simply restricting the overall capacity of the latent representations is not enough for learning disentangled representations. While InfoVAE yields comparatively better disentanglement score by pushing the marginal joint distribution of the representations towards a factorial distribution harder with large values of β, the associated decoding quality and informativeness of latent representations are both poor. In contrast, IMAE is capable of achieving better trade-off between the disentanglement score and the decoding quality in the region of interest where the decoding quality as well as the informativeness are fairly good. We attribute this to the effect of explicitly seeking for statistically independent latent factors by minimizing the total correlation term in our objective. We have proposed IMAE, a novel approach for simultaneously learning the categorical information of data while uncovering latent continuous features shared across categories. Different from VAE, IMAE starts with a stochastic encoder that seeks to maximize the mutual information between data and their representations, where a decoder is used to approximate the true posterior distribution of the data given the representations. This model targets at informative representations directly, which in turn naturally yields an objective that is capable of simultaneously inducing semantically meaningful representations and maintaining good decoding quality, which is further demonstrated by the numerical . Unsupervised joint learning of disentangled continuous and discrete representations is a challenging problem due to the lack of prior for semantic awareness and other inherent difficulties that arise in learning discrete representations. This work takes a step towards achieving this goal. A limitation of our model is that it pursues disentanglement by assuming or trying to encourage independent scalar latent factors, which may not always be sufficient for representing the real data. For example, data may exhibit category specific variation, or a subset of latent factors might be correlated. This motivates us to explore more structured disentangled representations; one possible direction is to encourage group independence. We leave this for future work. H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.S. Zhao, J. Song, and S. Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262, 2017. Balance between posterior inference fidelity and information maximization Notice that we can rewrite the mutual information between the data x and its representations as the following, DISPLAYFORM0 It then follows that, DISPLAYFORM1 Since H(x) is independent of the optimization procedure, we have the following, DISPLAYFORM2 where β trade-off the informativeness of the latent representation and generation fidelity. Decomposition of I θ (x; y, z) Let b = (z, y) denote the joint random variable consisting of the continuous random variable b and discrete random variable y. Note that I θ (x; y, z) = I θ (x; b) can be written as: DISPLAYFORM3 The second term in Eq has the form: DISPLAYFORM4 where ϑ 1 follows by the assumption that p θ (b|x) is factorial. For the first term in Eq, we have: DISPLAYFORM5 Substituting Eqs FORMULA1 & FORMULA1 into Eq yields the : DISPLAYFORM6 Since y and z are assumed to be marginally independent, i.e., p θ (y; z) = p θ (y)p θ (z), then DISPLAYFORM7 1 N N n=1 p θ (y|x n) denote the Monte Carlo estimator of the true probability DISPLAYFORM8 for all x ∈ X, then applying the Hoeffding's inequality for bounded random variables [Theorem 2.2.6, BID20] yields, DISPLAYFORM9 Given Eq, we first establish the concentration of the entropy H p θ (y) with respect to the empirical distribution p θ (y). Assume For all y ∈ C, we have p θ (y), p θ (y) bounded below by 1/(CK 2) for some fixed constant C > 1. This assumption is practical since the distributions of true data and predicted data are approximately uniform and therefore p θ (y), p θ (y) ≈ 1/K 2 for all y ∈ C. Consider the function t log t, with derivative 1 + log t DISPLAYFORM10 (1 + log t)dt DISPLAYFORM11 Summing over C gives DISPLAYFORM12 Let δ = K 2 δ, then Eq together with Eq yield the following, DISPLAYFORM13 Next we are going to bound the divergence between H θ (y|x) and H θ (y|x) which are defined as, DISPLAYFORM14 Note that h log h ∈ [−1/e, 0] for all h ∈, then again applying [Theorem 2.2.6, BID20] yields, DISPLAYFORM15 Following the similar arguments as before, let δ = 2 exp −2t 2 e 2 N, then DISPLAYFORM16 Now let δ = K 2 δ, then applying the union bound we have DISPLAYFORM17 hold with probability 1 − δ. Conclude from Eqs &, we have DISPLAYFORM18 hold with probability at least 1 − 2δ. Computing the marginal distributions of the continuous representations z and z k requires the entire dataset, e.g., DISPLAYFORM0 To scale up our method to large datasets, we propose to estimate based on the minibatch data, e.g., DISPLAYFORM1 Now consider the entropy H(z) of z, which we approximate in the following way, DISPLAYFORM2 We estimate the integral of z by sampling z ∼ p θ (z|x i) and perform the Monte Carlo approximation. Although we minimize the unbiased estimator of the lower bound of the KL divergence, the term inside the logarithm is a summation of probability densities of Gaussians. In particular, we record the distribution of the variances output by our encoder and observe that the mean of the variances of the Gaussians is bounded between 0.2 and 2, which implies that the values of probability densities do not range in a large scale. Since logarithm is locally affine, we argue that our bound in is tight. Other quantities involved in our objective function are estimated in a similar fashion. In VAE, they assume a generative model specified by a stochastic decoder p θ (x|z), taking the continuous representation as an example, and seek an encoder q φ (z|x) as a variational approximation of the true posterior p θ (z|x). The model is fitted by maximizing the evidence lower bound (ELBO) of the marginal likelihood, DISPLAYFORM0 Here the KL divergence term can be further decomposed as BID10, DISPLAYFORM1 That is, minimizing the KL divergence also penalizes the mutual information I θ (x; z), thus reduces the amount of information z has about x. This can make the inference task q φ (z|x) hard and lead to poor reconstructions of x as well. Many recent efforts have been focused on resolving this problem by revising ELBO. Although approaches differ, it can be summarized as either dropping the mutual information term in Eq, or encouraging statistical independence across the dimensions of z by increasing the penalty on the total correlation term extracted from the KL divergence D KL (q φ (z)||r(z)) with respect to q φ (z). However, these approaches either in an invalid lower bound for the VAE objective, or cannot avoid minimizing the mutual information I θ (x; z) between the representation and the data. In contrast, IMAE starts with a stochastic encoder p θ (z|x) and aims at maximizing the mutual information between the data x and the representations z from the very beginning. By following the constraints which are naturally implied by the objective in order to avoid degenerated solutions, IMAE targets at both informative and statistical independent representations. On the other hand, in IMAE the decoder q φ (x|z) serves as a variational approximation to the true posterior p θ (x|z). As we will show in Section 4, being able to learn more interpretable representations allows IMAE to reconstruct and generate data with better quality. (a) IMAE performs well regarding the disentanglement score vs. decoding quality trade-off, especially in the region of interest where both decoding quality and informativeness of representations are fairly good.(b) Negative correlation between total correlation and disentanglement score. It also implies that the disentanglement score tends to decrease along with the total correlation if using even larger β, due to the diminishing informativeness of representation factors. In the extreme case, both total correlation and disentanglement score can degrade to zero. Training procedure: • MNIST & Fashion MNIST: We use momentum to train all models. The initial learning rate is set as 1e-3, and we decay the learning rate by 0.98 every epoch.• dSprites: We use Adam to train all models. The learning rate is set as 1e-3.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Information theoretical approach for unsupervised learning of unsupervised learning of a hybrid of discrete and continuous representations,

represent the tldr for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.