Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Wide adoption of complex RNN based models is hindered by their inference performance, cost and memory requirements. To address this issue, we develop AntMan, combining structured sparsity with low-rank decomposition synergistically, to reduce model computation, size and execution time of RNNs while attaining desired accuracy. AntMan extends knowledge distillation based training to learn the compressed models efficiently. Our evaluation shows that AntMan offers up to 100x computation reduction with less than 1pt accuracy drop for language and machine reading comprehension models. Our evaluation also shows that for a given accuracy target, AntMan produces 5x smaller models than the state-of-art. Lastly, we show that AntMan offers super-linear speed gains compared to theoretical speedup, demonstrating its practical value on commodity hardware. Remarkable advances in deep learning (DL) have produced great models across a wide variety of tasks such as computer vision, machine reading, speech generation and image recognition BID7. However, wide adoption of these models is still limited by their inference performance, cost and memory requirements. On the client side, all pervasive devices like smart-phones, tablets and laptops have limited memory and computational resources to handle large complex DL models. On the server side, intensive computation can render the models too slow to meet responsiveness requirements and too expensive to scale, preventing their deployment in production. Model Compression is a flourishing area that aims to reduce the computational and memory complexity of DL models to address the aforementioned problems without significantly affecting accuracy. Compressing Convolution Neural Networks (CNNs) have already been widely explored in the past few years BID3, while our work focuses on Recurrent Neural Networks (RNNs), which are broadly used among various natural language processing tasks BID17 BID24 BID29. It is well known that large RNN models are computation and memory intensive (Zhang et al.). In particular, their computation increases linearly with sequence length, and their recurrent unit has to be computed sequentially, one step at a time with limited parallelism, both of which makes long execution time a crucial issue for RNN inference computation. Compressing RNNs, however, is challenging, because a recurrent unit is shared across all the time steps in sequence, compressing the unit will aggressively affect all the steps. Inducing sparsity is one of the prominent approaches used for RNN compression. BID18 proposed a pruning approach that deletes up to 90% connections in RNNs. The obtained sparse matrices, however, have an irregular/non-structured pattern of non-zero weights, which is unfriendly for efficient computation in modern hardware systems BID12 BID25. To address this issue, BID19 proposed inducing block-sparsity in RNNs via pruning or group lasso regularization. Similarly, BID26 introduces ISS, intrinsic structured sparsity for LSTMs BID9, a type of RNN, such that a sparse LSTM can be transformed into a dense one but with smaller size. ISS conveniently turns sparsity into efficient execution, but as its sparse structure is quite coarse-grained, it is hard to push out high sparsity without degrading accuracy, especially in RNNs where the hidden dimension is smaller than input dimension (elaborated in Section 5.1).Our work explores a new line of structured sparsity on RNNs, using predefined compact structures as opposed to pruning and regularization based approaches. We take inspiration from predefined compact CNN structures such as group convolutions BID11 and depth-wise separable convolutions BID4. Specifically, we replace matrix-vector multiplications (MVs), the dominant part of RNN computations, with localized group projections (LGP).LGP divides the input and output vectors into groups where the elements of the output group is computed as a linear combination of those from the corresponding input group. In addition, to empower the information flow across multiple groups along the steps of RNN computation, we use a permutation matrix or a dense-square matrix to combine outputs across groups, helping the compact structure to retain accuracy. Furthermore, we combine LGP with low-rank matrix decomposition in order to further reduce the computations. This is possible as low rank and sparsity are complimentary to each other. Low-rank decomposition such as SVD approximates a low-rank multiplication Ax as P Qx, where P and Q are dense. By imposing LGP-based sparsity on P and Q, we reduce the computation further. For a given rank reduction factor of r, we reduce the computation cost and model size by O(r 2), compared to O(r) by using low-rank decomposition methods like SVD BID6 alone. We call our compression approach AntMan -'shrink in scale' by synergistically combining structured sparsity and low-rank decomposition, but'increase in strength' by enabling the flow across structured groups along RNN sequence to retain accuracy. To train RNN models with AntMan, we use teacher-student training paradigm BID1 by combining the label loss with teacher-MSE-loss and teacher-KL-divergence-loss. To improve the training efficiency, we develop a new technique to decide proper coefficients to obtain high accuracy efficiently with minimal trials. We evaluate AntMan on multiple RNN based models for machine reading comprehension and language modeling. For a well-known MRC model BID24, we reduce the computational complexity and model size of LSTMs (a particular type of RNN) by up to 25x with less than 1pt drop in F1 score. For PTB BID29 language model, we achieve a computational reduction of 50x with no drop in perplexity, and 100x with just a single point drop in perplexity. We also construct language models for PTB with perplexities ranging from 64 to 70, but with 3x to 5x fewer overall model weights (5x to 25x reduction in RNN weights) than the state-of-art. Last but not least, we develop efficient implementations of inference kernels on CPUs to serve models compressed by AntMan. We show that unlike computation with unstructured sparsity, AntMan offers significant performance improvement for large RNN models even with modest levels of sparsity. Our evaluations show that a 2x to 10x theoretical reduction in computation can in up to 2x to 30x actual speedup, respectively, for moderate to large RNNs, demonstrating attractive practical value of AntMan on commodity hardware. Compressing RNNs via Sparsity: Described in Section 1 and empirically compared in Section 5.1.Compressing RNNs via Low-Rank Approximations: BID20 BID13 use SVD to compress LSTM models by 3-4x for acoustic modeling and speech recognition tasks with negligible loss in accuracy. AntMan achieves significantly higher compression rate than SVD based methods for the same rank reduction. BID27 uses Block Tensor Decomposition to compress LSTMs for vision tasks. Their work is specifically designed to exploit redundancies present in the image vector (input to the LSTMs) obtained from upstream CNN layers, while AntMan is designed to compress general RNNs, where the inputs do not exhibit such redundancies in many cases. Teacher-Student training paradigm: Knowledge Distillation (KD) technique developed by BID8 is a popular approach to compress deep and wide networks into sparser ones, where the compressed model mimics the function learned by the complex model. KD usually optimizes a weighted average of two different objective functions. The first objective function can be one of the following three: cross entropy, or mean square error, or Kullerback Leiber divergence, all computed with respect to the soft targets, and the second objective function is the cross entropy with the correct labels. Several similar approaches BID23 BID14 BID2 BID0 BID28 extend the idea of KD.In contrast, AntMan optimally combines three objective functions, MSE loss, KL divergence loss and the cross entropy of the true labels, powered by an efficient method of deciding their coefficients. AntMan compresses RNN computation by combining benefits of structured sparsity and low rank decomposition. It consists of three components: i) localized group projections that sparsify matrix multiplications using block diagonal matrices, ii) group mixing that exchanges information across different local groups along the sequence of RNN computation, and iii) low rank approximation that uses SVD like decomposition to reduce the rank of the projection. By composing them, we construct a few variations of AntMan compression modules that exhibit varying degree of compression rate and accuracy impact. We also analyze the cost complexity of AntMan modules and discuss efficient implementation on commodity hardware such as CPUs using off-the-shelf BLAS libraries. AntMan reduces the computation and size of RNNs by replacing dense matrix-vector product (MV) with sparse but structured MV. It divides the input and output vectors into g local groups such that the elements of an output group is a weighted linear sum of the elements in the corresponding input group. Since output elements of one group only depend on the input elements of the corresponding group, we call it localized group projections (LGP). Mathematically, we replace the matrix vector product Ax with D g x, where D g is a block-diagonal matrix with g blocks. In an RNN cell computation, the hidden-state vector at time-step t − 1 is an input to the MV used to compute the hidden-state vector at time-step t. Therefore, using LGP to replace MV in RNN restricts the information flow within a single local group across multiple time steps of the RNN. This restriction reduces the expressibility of the RNN, potentially degrading accuracy. AntMan uses'group mixing' to address this issue. To facilitate the information flow across multiple localized groups along RNN sequence computation, AntMan multiplies the output (or input) vector of LGP with a square matrix, which we call mixing matrix. We develop two types of mixing with varying memory and computational complexity -shuffle mix and dense mix -inspired by the shuffling layer used with group convolutions, or 1x1 convolutions used with depth-separable convolutions BID4.Shuffle mix: The shuffle-mix matrix is a permutation matrix, which evenly distributes the elements of the same group across the entire output vector across different groups. FIG0 shows the effect of shuffle mix following LGP. Mathematically, shuffle mix is equivalent to a transpose operation. If the output vector v ing from the block diagonal MV has m elements, we can represent the vector as a matrix O of size [g, m/g], where each row represents an output group computed from the corresponding input group. Shuffle mix simply transforms DISPLAYFORM0 Dense mix: This technique uses a dense square matrix for group mixing when the matrix in the MV is non-square. Mathematically, given Ax, where size of A is m x n, we can decompose it into M D g x, when m < n, or D g M x, when n < m, and M is a dense-mixing matrix of size m x m, or n x n, respectively. FIG0 shows an example of dense mix preceding LGP.Dense mix has added cost of the dense matrix vector multiply compared to shuffle mix (quadratic vs linear). However, unlike shuffle mix that simply permutes the elements of the output vector, dense mix takes a weighted linear combination, making it more general. It helps retain accuracy at the expense of additional computation. When combined with low-rank decomposition discussed next, dense mix provides high compression while maintaining accuracy, which we elaborate further in evaluation (Table 4). Low-rank decomposition such as SVD approximates a low-rank matrix-vector product Ax as P Qx, where A, P and Q are dense with shapes m x n, m x n r and n r x n, respectively, and n r is the reduced rank. We combine it with LGP by adding LGP-based sparsity on P and Q, further reducing computation. This combination is likely to obtain more compression than using either of the techniques alone because structured sparsity and low-rank decomposition operate in a complimentary fashion. In particular, low rank reduces the computation by factorizing A into smaller matrices P and Q, while LGP reduces computation by sparsifying these matrices without changing their dimensions. Composed from the three components, LGP, group mixing and low-rank decomposition, we construct variations of AntMan compression modules to address varying efficiency and accuracy demand across DL models. FIG0 shows three of such compression modules: (a) LGP-shuffle -LGP with shuffle mix; (b) LGP-dense -LGP with dense mix; (c) LowRank-LGP -low rank with LGP-dense. We elaborate the compression modules by taking FIG0 (c), LowRank-LGP, as an example. LowRank-LGP combines structured sparsity with low rank decomposition. First, it decomposes an MV into an SVD-like form, i.e., Ax ← P Qx, where A is a matrix of size m x n, P and Q are decomposed matrices of size m x n r and n r x n, and n r represents the reduced rank. Next, we replace P and Q using LGP-Dense, i.e., Ax DISPLAYFORM0 DISPLAYFORM1 This module, combining all three components of AntMan, exhibits the potential of achieving significantly higher cost reduction than using SVD alone, which we quantify shortly. LGP-Shuffle TAB0 discusses the reduction in computation and model size over the original Ax, where A is a matrix of size m x n. The third column reports the total number of multiply-add operations, which is also the size of weight matrix in the case of MV. The final column represents the reduction in computation (that is equal to the reduction in model size) compared to the original MV. DISPLAYFORM2 We highlight two key messages: LGP-Dense reduces the total cost by ≈ max(m,n) min(m,n) when g max(m,n) min(m,n), i.e., the larger difference between m and n, the more reduction it gets. When g out and g in are large enough, LowRank-LGP can enable significantly higher cost reduction over SVD, while maintaining the same reduced rank. To see this, let's assume m = n, and g out = g in = g. In this case, the cost reduction from SVD is r/2, while the cost reduction from LowRank-LGP is r 2 g 2r+g. Now, if g ≥ r, then the cost reduction is at least r 2 /3, and it goes up to r 2 when g r. Therefore, the reduction in computational cost scales as O(r) for SVD, while it scales as O(r 2) for LowRank-LGP assuming g ≥ r. As a concrete example, consider a MV of size 1000x400, where the number of parameters and the number of multiply-add (MADD) operations in 400K. Using LGP-Shuffle with g = 10, we can reduce both the number of parameters and MADD operations to 40K. Using LGP-Dense with g = 10, we can reduce them to 200K (40K from LGP + 160K from dense mix). Using LowRank-LGP with g = 10 and r = 4, we can reduce the parameters and MADD operations to 1000 * 400 4 * 10 + 400 * 400 4 * 4 + 400 * 400 4 * 10, which is 24K. We develop efficient implementation of AntMan modules (LGP-Shuffle, LGP-Dense, LowRank-LGP) on CPUs to empower their usage in practice. The implementation consists of three building blocks: i) Regular matrix-vector multiply for dense mix, ii) shuffle-mix multiply and iii) blockdiagonal MVs. BLAS libraries such as Intel MKL already provides efficient implementation of matrix-vector multiplication. Shuffle mix is implemented efficiently as a matrix transpose operation as described in Section 3.1. The block-diagonal MV is viewed as multiple smaller MVs, each corresponding to one of the blocks in the block-diagonal matrix. With multicores, each of these blocks can be computed in parallel using OpenMP for parallelization and Intel MKL for MV computation. In summary, AntMan modules can be implemented efficiently on commodity hardware, such as CPUs, conveniently applicable to various devices on cloud and on edge. We observe that while training AntMan models directly on target labels alone does not generalize well on test data, using knowledge distillation or teacher-student training helps greatly on retaining accuracy. We use the original uncompressed model as the teacher, and train the compressed model (student) to imitate the output distribution of the teacher, in addition to training on the target labels. We describe how we apply and extend teacher-student training. We define the loss function of the compressed model as a weighted combination of three losses -the raw loss from the target labels, and the MSE and the KL divergence losses of the student's output distribution with respect to the teacher's corresponding output distribution: DISPLAYFORM0 where C target, C mse, C kl are the coefficient values corresponding to the target loss, MSE loss and KL divergence loss, respectively. S o, T o are the output distributions of student and teacher model, respectively, whereas T target is the target distribution. Deciding loss coefficients: The final performance of the compressed model significantly depends on the values of the loss coefficients, C target, C mse and C kl. Searching for appropriate values for these coefficients via grid or random search is time and resource consuming. We develop an efficient method to decide them with the following intuition. The direction of the gradient updates to the model is dictated by the relative magnitudes of the individual losses. If one is significantly smaller than the other, then it would have minimal impact on the direction of the updates during training. Therefore, we want to scale each of the three losses such that the overall magnitude of each of the three terms in Eqn. 1 is roughly the same. To this end, we initially train the compressed model separately using each of the three losses and record the loss values once the model converges. Then we use these values as reference to identify loss coefficients such that each of the three terms is roughly the same. We use these coefficients to train the compressed model to optimize accuracy. We demonstrate the effectiveness of our approach by training a 10x compressed language model constructed by replacing the LSTMs in BID29 with LGP-Shuffle with g = 10. For this compressed model, the validation loss values at convergence when training separately with the three individual losses were: target = 4.110, MSE = 0.133 and KL = 0.004. Table 3 shows the test perplexity values (lower the better) obtained by training the compressed model with varying C mse and C KL, while fixing C target = 1. Note that the lowest perplexity is achieved by setting coefficient values of C target = 1, C mse = 30 and C KL = 1000. At these values, each term in Eqn. 1 is roughly equal to 4, demonstrating the effectiveness of our method. Table 3 also shows the benefits of combining three losses. Note that when C mse = 0, the best achievable perplexity is 121.97. Similarly, when C KL = 0, the best achievable perplexity is 75.61. However, combining all three gives the lowest perplexity of 74.69. Table 3: Different choices of coefficients vs test perplexities for student model with 10x computation reduction on the PTB dataset. We evaluate AntMan on three aspects. We use AntMan to obtain order(s) of magnitude computation reduction for language modeling and machine reading comprehension tasks while getting similar accuracy. We use AntMan to construct models with several times fewer parameters than the state-of-art models with the same accuracy targets. Not limited by theoretical speedup, we measure real speedup of AntMan on CPUs and observe super-linear computational efficiency on large RNNs, demonstrating attractive practical value of AntMan on commodity hardware. We evaluate the effectiveness of AntMan on reducing model computation: on BID29 model for word level completion task, we obtain 50x reduction without sacrificing any accuracy; on BID24 model for machine reading compression task, we obtain up to 25x reduction with less than 1pt drop on F1 score. Word level completion task predicts the next word given a partial input sequence. We use Penn Tree Bank(PTB) dataset BID15 ) that consists of 929k training words, 73k validation words and 82k test words. As the teacher model, we chose the model in BID29 consisting of 2 layered LSTMs each with hidden dimension of 1500. For the student model, we replace all the MVs in the LSTMs with LGP-Shuffle, and use g = 10 to g = 100 groups. We do not use any low-rank decomposition for this model. TAB2 shows the perplexity values of the compressed models for different levels of computation reductions. Matching the perplexity of the original model, AntMan (g = 50) achieves 50x computation reduction. With g = 10, AntMan achieves 10x computation reduction while 3pt better perplexity. With g = 100, AntMan achieves 100x computation reduction with only 1pt loss in perplexity. In addition, comparing with the state-of-art compressed model in BID26 using ISS, AntMan reduces computations further by 5-10x under comparable test perplexities. MRC tasks have gained significant popularity in last few years within NLP and computer vision communities. The models answer a query about a given context paragraph, evaluated based on exact match (EM) and F1 score (higher the better).Dataset: We use Stanford Question Answering Dataset (SQuAD) BID22, which consists of a large set of Wikipedia articles and more than 100,000 questions. The answer to every question is always a small excerpt of the article. Teacher Model: We chose our teacher model as the BiDirectional Attention Flow Model (BiDAF) BID24, which is a hierarchical multi-stage model with 6 layers. We focus on compressing the layers having RNNs, which are also the most computationally expensive ones. Specifically, the modeling layer uses 2 layers of bi-directional LSTMs, denoted by ModFwd1, ModBwd1, ModFwd2, ModBwd2, while the output layer has a single bi-directional LSTM, denoted by OutFwd, OutBwd. Compressed Models: We created three compressed models using AntMan with different levels of compression to replace the LSTMs in the BiDAF model: i) LGP-Shuffle (g im = 10, g hm = 4), Table 4: Comparision of computation reduction between AntMan and ISS for BiDAF ii) LowRank-LGP 1 (g im = 10, g hm = 5, r im = 4, r hm = 2), and iii) LowRank-LGP 2 (g im = 5, g hm = 5, r im = 8, r hm = 4). Here, g im and g hm refers to the number of groups, and r im and r hm refers to the low-rank reduction factors for input and hidden MVs of the LSTMs, respectively. The computation reduction for each LSTM is shown in Table 4.Results: Table 4 shows that both LGP-Shuffle and LowRank-LGP achieve significant computation reduction over the original; their reduction is much higher than the existing work ISS BID26 with better EM and F1 scores. ISS compresses an RNN by reducing hidden dimension. The amount of computation per LSTM step for ISS is proportional to (i + h/r) * (h/r), where i is the input dimension, h is the hidden dimension, and 1/r is fraction of hidden dimension removed by ISS. When i h, the compression is proportional to r. In BiDAF, i h in the first modeling layers (800 vs 100). Therefore, compression in these layers is proportional to the reduction in the hidden dimension. However, h = 100 is already very small. By reducing it further, ISS experiences near 2pt drop in F1 score with less than 2.5x compression on the first modeling layers. LGP-Shuffle uses structured sparsity to compress both the input and hidden MVs without reducing hidden dimension. For a comparable EM and F1 scores to ISS, LGP-shuffle achieves significantly higher reduction on the first modeling layers, while doing modestly better on all other layers. LowRank-LGP improves further upon LGP-Shuffle, increasing accuracy by leveraging dense mix to enrich the connection among multiple localized groups, and reducing computation by combining low-rank decomposition. It achieves significantly higher computation reduction across all layers than both ISS and LGP-Shuffle, while achieving nearly 1pt higher F1 scores. Different applications have various accuracy requirements while the devices they are running on also impose different constraints on the model size. For given accuracy targets, smaller models are desired; and for given model sizes, higher accuracy is desired. We show that AntMan improves the Pareto curve of model size against accuracy, providing more compressed models with the same accuracy targets of several recent models at word level completion task. Teacher Model: We use the state-of-art language model as of , AWD-LSTM BID16, consisting of 3-layer LSTMs with 1150 hidden units and an embedding size of 400.Compressed Models: Our compressed models replace all the MVs in the LSTMs of AWD-LSTM with AntMan (LGP-Shuffle with g = 5 to g = 50 groups). FIG3 compares AntMan with other models. LGP-Shuffle (g = 5) achieves perplexity of 63 with 5x fewer LSTM parameters and 3x fewer total parameters than NAS-Cell BID33, the state-of-art model obtaining this range of accuracy. LGP-Shuffle (g = 10) achieves perplexity of 66 with 10x fewer LSTM parameters and 4x fewer total parameters than Var-RHN BID32, and LGP-Shuffle (g = 50) achieves perplexity of 74 with 50x fewer LSTM parameters and 5x fewer total parameters than Var-LSTM-avg1 BID10. These notably improve the Pareto curve of the task by reducing model sizes against different accuracy targets. 1 1 We did not aim to reproduce the state-of-art perplexity (57.3px at 24M parameters) of AWD-LSTM model. AWD-LSTM uses various regularization techniques, each with its own set of hyper-parameters, requiring extensive hyper-parameter tuning to reach its state-of-art perplexity. The AntMan presented in FIG3 was achieved without any regularization. Trying to match AWD-LSTM perplexity using AntMan with regularization could be an exercise in large scale hyper-parameter tuning, which is beyond the scope of this paper. Table 5: Measured speedup on CPU using LGP-Shuffle and LowRank-LGP compared to the theoretical speedup for various input and hidden dimension. For LGP-Shuffle, we use g = 2 and g = 10 to get a theoretical speedup of 2x and 10x. For LowRank-LGP, we use g = 2 and r = 2, and g = 10, and r = 2 to get a speedup of 2.66x and 8x, respectively. By using efficient implementation of AntMan described in Section 3.6, we turn the theoretical speedup (computation reduction) to actual speedup (execution time reduction) in practice. Furthermore, we show that the actual speedup can be significantly higher than the theoretical speedup for large problem sizes. The of our evaluation is shown in Table 5.Problem Configuration: We measure the execution time of LSTMs with and without AntMan varying input and hidden dimensions from 100 to 1600. We use a batch size of 1, which is common in serving scenarios, and a sequence length of 100. We use an efficient implementation as discussed in Elsen: Fuse 4 input MVs across all time steps into a single large matrix multiplication, and fuse 4 hidden MVs within each time step. The experiments are run on a single core of Intel CPU E5-2650 v4 @ 2.20GHz. We use just a single core for two reasons: i) to emulate the limited resource availability in many use cases such as laptops and smart phones, ii) performance of multi-core RNN is highly implementation dependent (Zhang et al.) even for regular RNNs and therefore is difficult to make apple-to-apple comparison. We use Intel MKL library for GEMM implementation. Discussion: Table 5 shows that, for very small problem size, AntMan offers no speedup regardless of the reduction in the computation. This is expected as GEMM performance gets worse as the problem size decreases. However, as the problem size is already very small, memory reduction or performance improvement is less crucial for such problems. For medium sized problems, AntMan offers good actual speedup compared to the theoretical speedup. Notice that unlike unstructured sparsity, where significant levels of sparsity is necessary to see actual performance improvement, with AntMan, even a modest 50% sparsity or 2x computation reduction in significant performance gain at problem size 400 and 800. Furthermore, for large problem sizes the actual speedup is significantly larger than the theoretical speedup. At problem size of 1200 and 1600, the weight matrices in the LSTM are too large to fit in L3 cache (30 MB in this case), thus spilling into memory. These LSTMs have much lower efficiency as the memory bandwidth of a CPU is much lower than the L3 cache bandwidth. By reducing the memory footprint, AntMan-based LSTM fits in L3 cache, leading to an actual speed up that is considerably higher than the theoretical speedup. These demonstrate attractive practical value of AntMan on commodity hardware. We develop AntMan, combining structured sparsity and low-rank decomposition, to reduce the computation, size and execution time of RNN models by order(s) of magnitude while achieving similar accuracy. We hope its compression efficiency and effectiveness would help unblock and enable many great RNN-based models deployed in practice. We discuss and compare AntMan with several compression techniques as below. Quantization: 16 and 8-bit quantization (original 32-bit) can be supported fairly easily on commodity hardware, ing in a maximum compression of 4x. Even more aggressive quantization (e.g., 2-7 bit) hardly provides additional computational benefit because commodity hardware does not support those in their instruction set, while 1-bit quantization does not offer comparable accuracy. In comparison, we demonstrate that AntMan achieves up to 100x reduction in computation without loss in accuracy. Moreover, quantization can be applied to AntMan to further reduce the computation, and vice versa, as quantization and AntMan are complementary techniques. Pruning: Pruning can be used to generate both unstructured and structured sparsity. The former is not computationally efficient while the latter requires specialized implementation for efficient execution. While we did not present pruning in the paper, we did try out techniques on both PTB and BiDAF models to generate random sparsity as well as blocked sparsity. In both cases, we were able to get more that 10x reduction in computation even in the absence of Knowledge distillation. Therefore pruning provides excellent computation reduction. However, as discussed in the paper, those theoretical computational reductions cannot be efficiently converted into practical performance gains: Unstructured sparsity ing from pruning suffers from poor computation efficiency; a 10x theoretical reduction leads to less than 4x improvement in performance while AntMan achieves 30x performance gain with 10x reduction for PTB like models. TAB7 It is possible to achieve structured sparsity such as block sparsity through pruning. However, structured sparsity requires implementing specialized kernels to take advantage of the computation reduction. Its efficiency greatly depends on the implementation, and in general is far from the theoretical computation reduction. On the contrary both ISS and AntMan achieve good computation reduction, and can be efficiently executed using readily available BLAS libraries such as Intel MKL ing in super linear speedups as shown in the paper. Direct Design: We compared AntMan with smaller RNN models (with smaller hidden dimension) trained using the larger teacher model. Our show that for the same level of compression AntMan achieves much higher accuracy. TAB6 SVD RNN:We constructed compressed models by replacing matrix-multiplication with SVD of various rank, and trained the SVD based models using knowledge distillation. Once again, we find that for the same level of compression, AntMan achieves much higher accuracy than SVD. TAB6 Block Tensor Decomposition (BTD): BTD is designed to compress RNNs whose inputs are produced by convolution based models, and contain certain redundancies. AntMan, on the other hand, is generic to all RNN based models. Also, BTD is designed to compress only the input vector and not the hidden vectors. This hinders the performance of BTD over a range of RNNs, where the hidden vectors are also large. Here, we compare the performance of AntMan with ISS, without using any knowledge distillation. Please note that knowledge distillation is part of the training process for AntMan, but it is not for ISS. Nevertheless, it is interesting to see how AntMan performs in the absence of a teacher. When trained without knowledge distillation, our experiments show that AntMan and ISS have complimentary strengths. On the PTB dataset, with a 10x compute reduction, AntMan does not generalize well without a teacher, while ISS incurs less than 1pt loss in perplexity compared to the original model. This is demonstrated by the first row and column in Table 3, and the third row in TAB2. On the contrary, for the BiDAF, AntMan incurs less than 1pt reduction in F1 score for nearly 10x compute reduction 2, while ISS incurs nearly 2pt reduction in F1 score with less than 5x compute reduction on average. This is shown in TAB9.AntMan can successfully compress BiDAF, while ISS fails because ISS compresses an LSTM by effectively reducing its hidden dimension, while AntMan preserves the hidden dimension size. The LSTMs in the BiDAF model have large input dimensions making them computationally expensive, but they have very small hidden dimensions. Therefore, reducing the already small hidden dimension in significant loss of accuracy. On the contrary, the PTB model has large input as well as hidden dimensions, allowing ISS to work effectively. Optimizers All the models in the evaluation section were trained using ADAM optimizer. Hyperparameters For both PTB and SQUAD, all the hyper-parameters for training the respective models were set as default from their standard and recommended implementations on github (PTB 3, BiDAF 4).

Reducing computation and memory complexity of RNN models by up to 100x using sparse low-rank compression modules, trained via knowledge distillation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Graph-structured data such as social networks, functional brain networks, gene regulatory networks, communications networks have brought the interest in generalizing deep learning techniques to graph domains. In this paper, we are interested to design neural networks for graphs with variable length in order to solve learning problems such as vertex classification, graph classification, graph regression, and graph generative tasks. Most existing works have focused on recurrent neural networks (RNNs) to learn meaningful representations of graphs, and more recently new convolutional neural networks (ConvNets) have been introduced. In this work, we want to compare rigorously these two fundamental families of architectures to solve graph learning tasks. We review existing graph RNN and ConvNet architectures, and propose natural extension of LSTM and ConvNet to graphs with arbitrary size. Then, we design a set of analytically controlled experiments on two basic graph problems, i.e. subgraph matching and graph clustering, to test the different architectures. Numerical show that the proposed graph ConvNets are 3-17% more accurate and 1.5-4x faster than graph RNNs. Graph ConvNets are also 36% more accurate than variational (non-learning) techniques. Finally, the most effective graph ConvNet architecture uses gated edges and residuality. Residuality plays an essential role to learn multi-layer architectures as they provide a 10% gain of performance. Convolutional neural networks of BID20 and recurrent neural networks of BID17 are deep learning architectures that have been applied with great success to computer vision (CV) and natural language processing (NLP) tasks. Such models require the data domain to be regular, such as 2D or 3D Euclidean grids for CV and 1D line for NLP. Beyond CV and NLP, data does not usually lie on regular domains but on heterogeneous graph domains. Users on social networks, functional time series on brain structures, gene DNA on regulatory networks, IP packets on telecommunication networks are a a few examples to motivate the development of new neural network techniques that can be applied to graphs. One possible classification of these techniques is to consider neural network architectures with fixed length graphs and variable length graphs. In the case of graphs with fixed length, a family of convolutional neural networks has been developed on spectral graph theory by BID6. The early work of BID5 proposed to formulate graph convolutional operations in the spectral domain with the graph Laplacian, as an analogy of the Euclidean Fourier transform as proposed by BID14. This work was extended by BID16 to smooth spectral filters for spatial localization. BID9 used Chebyshev polynomials to achieve linear complexity for sparse graphs, BID21 applied Cayley polynomials to focus on narrow-band frequencies, and BID27 dealt with multiple (fixed) graphs. Finally, BID19 simplified the spectral convnets architecture using 1-hop filters to solve the semi-supervised clustering task. For related works, see also the works of, and references therein. For graphs with variable length, a generic formulation was proposed by BID12; BID29 based on recurrent neural networks. The authors defined a multilayer perceptron of a vanilla RNN. This work was extended by BID22 using a GRU architecture and a hidden state that captures the average information in local neighborhoods of the graph. The work of BID30 introduced a vanilla graph ConvNet and used this new architecture to solve learning communication tasks. BID25 introduced an edge gating mechanism in graph ConvNets for semantic role labeling. Finally, BID4 designed a network to learn nonlinear approximations of the power of graph Laplacian operators, and applied it to the unsupervised graph clustering problem. Other works for drugs design, computer graphics and vision are presented by BID10; BID1; BID26.In this work, we study the two fundamental classes of neural networks, RNNs and ConvNets, in the context of graphs with arbitrary length. Section 2 reviews the existing techniques. Section 3 presents the new graph NN models. Section 4 reports the numerical experiments. Generic formulation. Consider a standard RNN for word prediction in natural language processing. Let h i be the feature vector associated with word i in the sequence. In a regular vanilla RNN, h i is computed with the feature vector h j from the previous step and the current word x i, so we have: DISPLAYFORM0 The notion of neighborhood for regular RNNs is the previous step in the sequence. For graphs, the notion of neighborhood is given by the graph structure. If h i stands for the feature vector of vertex i, then the most generic version of a feature vector for a graph RNN is DISPLAYFORM1 where x i refers to a data vector and {h j : j → i} denotes the set of feature vectors of the neighboring vertices. Observe that the set {h j} is unordered, meaning that h i is intrinsic, i.e. invariant by vertex re-indexing (no vertex matching between graphs is required). Other properties of f G-RNN are locality as only neighbors of vertex i are considered, weight sharing, and such vector is independent of the graph length. In summary, to define a feature vector in a graph RNN, one needs a mapping f that takes as input an unordered set of vectors {h j}, i.e. the feature vectors of all neighboring vertices, and a data vector x i, FIG0 (a).We refer to the mapping f G-RNN as the neighborhood transfer function in graph RNNs. In a regular RNN, each neighbor as a distinct position relatively to the current word (1 position left from the center). In a graph, if the edges are not weighted or annotated, neighbors are not distinguishable. The only vertex which is special is the center vertex around which the neighborhood is built. This explains the generic formulation of Eq.. This type of formalism for deep learning for graphs with variable length is described by BID29 BID11 with slightly different terminology and notations. Graph Neural Networks of BID29. The earliest work of graph RNNs for arbitrary graphs was introduced by BID12; BID29. The authors proposed to use a vanilla RNN with a multilayer perceptron to define the feature vector h i: DISPLAYFORM2 with DISPLAYFORM3 and σ is the sigmoid function, A, B, U, V are the weight parameters to learn. Minimization of Eq. does not hold a closed-form solution as the dependence computational graph of the model is not a directed acyclic graph (DAG). BID29 proposed a fixed-point iterative scheme: for t = 0, 1, 2,... DISPLAYFORM4 The iterative scheme is guaranteed to converge as long as the mapping is contractive, which can be a strong assumption. Besides, a large number of iterations can be computational expensive. Gated Graph Neural Networks of BID22. In this work, the authors use the gated recurrent units (GRU) of BID7: DISPLAYFORM5 As the minimization of Eq. does not have an analytical solution, BID22 designed the following iterative scheme: DISPLAYFORM6 DISPLAYFORM7 and DISPLAYFORM8 where is the Hadamard point-wise multiplication operator. This model was used for NLP tasks by BID22 and also in quantum chemistry by BID11 for fast organic molecule properties estimation, for which standard techniques (DFT) require expensive computational time. Tree-Structured LSTM of BID31. The authors extended the original LSTM model of BID17 to a tree-graph structure: DISPLAYFORM9 where C(i) refers the set of children of node i. DISPLAYFORM10 Unlike the works of BID29 BID22, Tree-LSTM does not require an iterative process to update its feature vector h i as the tree structure is also a DAG as original LSTM. Consequently, the feature representation can be updated with a recurrent formula. Nevertheless, a tree is a special case of graphs, and such recurrence formula cannot be directly applied to arbitrary graph structure. A key property of this model is the function f ij which acts as a gate on the edge from neighbor j to vertex i. Given the task, the gate will close to let the information flow from neighbor j to vertex i, or it will open to stop it. It seems to be an essential property for learning systems on graphs as some neighbors can be irrelevant. For example, for the community detection task, the graph neural network should learn which neighbors to communicate (same community) and which neighbors to ignore (different community). In different contexts, BID8 added a gated mechanism inside the regular ConvNets in order to improve language modeling for translation tasks, and van den Oord et al. FORMULA1 considered a gated unit with the convolutional layers after activation, and used it for image generation. Generic formulation. Consider now a classical ConvNet for computer vision. Let h ij denote the feature vector at layer associated with pixel (i, j). In a regular ConvNet, h +1 ij is obtained by applying a non linear transformation to the feature vectors h i j for all pixels (i, j) in a neighborhood of pixel (i, j). For example, with 3 × 3 filters, we would have: DISPLAYFORM0 In the above, the notation {h i j : |i − i | ≤ 1 and |j − j | ≤ 1} denote the concatenation of all feature vectors h i j belonging to the 3 × 3 neighborhood of vertex (i, j). In ConvNets, the notion of neighborhood is given by the euclidian distance. As previously noticed, for graphs, the notion of neighborhood is given by the graph structure. Thus, the most generic version of a feature vector h i at vertex i for a graph ConvNet is DISPLAYFORM1 where {h j : j → i} denotes the set of feature vectors of the neighboring vertices. In other words, to define a graph ConvNet, one needs a mapping f G-CNN taking as input a vector h i (the feature vector of the center vertex) as well as an unordered set of vectors {h j} (the feature vectors of all neighboring vertices), see FIG0 (b). We also refer to the mapping f G-CNN as the neighborhood transfer function. In a regular ConvNet, each neighbor as a distinct position relatively to the center pixel (for example 1 pixel up and 1 pixel left from the center). As for graph RNNs, the only vertex which is special for graph ConvNets is the center vertex around which the neighborhood is built. CommNets of BID30. The authors introduced one of the simplest instantiations of a graph ConvNet with the following neighborhood transfer function: DISPLAYFORM2 where denotes the layer level, and ReLU is the rectified linear unit. We will refer to this architecture as the vanilla graph ConvNet. BID30 used this graph neural network to learn the communication between multiple agents to solve multiple tasks like traffic control. Syntactic Graph Convolutional Networks of BID25. The authors proposed the following transfer function: DISPLAYFORM3 where η ij act as edge gates, and are computed by: DISPLAYFORM4 These gated edges are very similar in spirit to the Tree-LSTM proposed in BID31. We believe this mechanism to be important for graphs, as they will be able to learn what edges are important for the graph learning task to be solved. Proposed Graph LSTM. First, we propose to extend the Tree-LSTM of BID31 to arbitrary graphs and multiple layers: DISPLAYFORM5 As there is no recurrent formula is the general case of graphs, we proceed as BID29 and use an iterative process to solve Eq.: At layer, for t = 0, 1,..., T DISPLAYFORM6 DISPLAYFORM7 In other words, the vector h We leverage the vanilla graph ConvNet architecture of BID30, Eq. FORMULA13, and the edge gating mechanism of , Eq., by considering the following model: DISPLAYFORM0 where h =0 i = x i, ∀i, and the edge gates η ij are defined in Eq.. This model is the most generic formulation of a graph ConvNet (because it uses both the feature vector h i of the center vertex and the feature vectors h j of neighboring vertices) with the edge gating property. Residual Gated Graph ConvNets. In addition, we formulate a multi-layer gated graph ConvNet using residual networks (ResNets) introduced by BID15. This boils down to add the identity operator between successive convolutional layers: DISPLAYFORM1 As we will see, such multi-layer strategy work very well for graph neural networks. We consider the subgraph matching problem presented by BID29, see FIG3 (a). The goal is to find the vertices of a given subgraph P in larger graphs G k with variable sizes. Identifying similar localized patterns in different graphs is one of the most basic tasks for graph neural networks. The subgraph P and larger graph G k are generated with the stochastic block model (SBM), see for example BID0. A SBM is a random graph which assigns communities to each node as follows: any two vertices are connected with the probability p if they belong to the same community, or they are connected with the probability q if they belong to different communities. For all experiments, we generate a subgraph P of 20 nodes with a SBM q = 0.5, and the signal on P is generated with a uniform random distribution with a vocabulary of size 3, i.e. {0, 1, 2}. Larger graphs G k are composed of 10 communities with sizes randomly generated between 15 and 25. The SBM of each community is p = 0.5. The value of q, which acts as the noise level, is 0.1, unless otherwise specified. Besides, the signal on G k is also randomly generated between {0, 1, 2}. Inputs of all neural networks are the graphs with variable size, and outputs are vertex classification vectors of input graphs. Finally, the output of neural networks are simple fully connected layers from the hidden states. BID25. The learning schedule is as follows: the maximum number of iterations, or equivalently the number of randomly generated graphs with the attached subgraph is 5,000 and the learning rate is decreased by a factor 1.25 if the loss averaged over 100 iterations does not decrease. The loss is the cross-entropy with 2 classes (the subgraph P class and the class of the larger graph G k) respectively weighted by their sizes. The accuracy is the average of the diagonal of the normalized confusion matrix w.r.t. the cluster sizes (the confusion matrix measures the number of nodes correctly and badly classified for each class). We also report the time for a batch of 100 generated graphs. The choice of the architectures will be given for each experiment. All algorithms are optimized as follow. We fix a budget of parameters of B = 100K and a number of layers L = 6. The number of hidden neurons H for each layer is automatically computed. Then we manually select the optimizer and learning rate for each architecture that best minimize the loss. For this task, BID22; BID30; BID25 and our gated ConvNets work well with Adam and learning rate 0.00075. Graph LSTM uses SGD with learning rate 0.075. Besides, the value of inner iterative steps T for graph is 3.The first experiment focuses on shallow graph neural networks, i.e. with a single layer L = 1. We also vary the level of noise, that is the probability q in the SBM that connects two vertices in two different communities (the higher q the more mixed are the communities). The hyper-parameters are selected as follows. Besides L = 1, the budget is B = 100K and the number of hidden neurons H is automatically computed for each architecture to satisfy the budget. First row of FIG6 FORMULA1 ) performs much better, but they also take more time than the graph ConvNets architectures we propose, as well as BID30; BID25. As expected, all algorithms performances decrease when the noise increases. The second experiment demonstrates the importance of having multiple layers compared to shallow networks. We vary the number of layers L = {1, 2, 4, 6, 10} and we fix the number of hidden neurons to H = 50. Notice that the budget is not the same for all architectures. Second row of FIG6 reports the accuracy and time w.r.t. L (middle figure is a zoom in the left figure). All models clearly benefit with more layers, but RNN-based architectures see their performances decrease for a large number of layers. The ConvNet architectures benefit from large L values, with the proposed graph ConvNet performing slightly better than BID30; BID25. Besides, all ConvNet models are faster than RNN models. In the third experiment, we evaluate the algorithms for different budgets of parameters B = {25K, 50K, 75K, 100K, 150K}. For this experiment, we fix the number of layers L = 6 and the number of neurons H is automatically computed given the budget B. The are reported in the third row of FIG6. For this task, the proposed graph ConvNet best performs for a large budget, while being faster than RNNs. We also show the influence of hyper-parameter T for BID22 and the proposed graph LSTM. We fix H = 50, L = 3 and B = 100K. FIG7 reports the for T = {1, 2, 3, 4, 6}. The T value has an undesirable impact on the performance of graph LSTM. Multi-layer BID22 is not really influenced by T. Finally, the computational time naturally increases with larger T values. In this section, we consider the semi-supervised clustering problem, see FIG3 (b). This is also a standard task in network science. For this work, it consists in finding 10 communities on a graph given 1 single label for each community. This problem is more discriminative w.r.t. to the architectures than the previous single pattern matching problem where there were only 2 clusters to find (i.e. 50% random chance). For clustering, we have 10 clusters (around 10% random chance). As in the previous section, we use SBM to generate graphs of communities with variable length. The size for each community is randomly generated between 5 and 25, and the label is randomly selected in each community. Probability p is 0.5, and q depends on the experiment. For this task, BID22; BID30; BID25 and the proposed gated ConvNets work well with Adam and learning rate 0.00075. Graph LSTM uses SGD with learning rate 0.0075. The value of T for graph is 3.The same set of experiments as in the previous task are reported in FIG8. ConvNet architectures get clearly better than RNNs when the number of layers increase (middle row), with the proposed Gated ConvNet outperforming the other architectures. For a fixed number of layers L = 6, our graph ConvNets and BID25 best perform for all budgets, while paying a reasonable computational cost. Next, we report the learning speed of the models. We fix L = 6, B = 100K with H being automatically computed to satisfy the budget. FIG10 reports the accuracy w.r.t. time. The ConvNet architectures converge faster than RNNs, in particular for the semi-supervised task. To close this study, we are interested in comparing learning based approaches to non-learning variational ones. To this aim, we solve the variational Dirichlet problem with labeled and unlabelled data as proposed by BID13. We run 100 experiments and report an average accuracy of 45.37% using the same setting as the learning techniques (one label per class). The performance of the best learning model is 82%. Learning techniques produce better performances with a different paradigm as they use training data with ground truth, while variational techniques do not use such information. The downside is the need to see 2000 training graphs to get to 82%. However, when the training is done, the test complexity of these learning techniques is O(E), where E is the number of edges in the graph. This is an advantage over the variational Dirichlet model that solves a sparse linear system of equations with complexity O(E 3/2), see BID24. This work explores the choice of graph neural network architectures for solving learning tasks with graphs of variable length. We developed analytically controlled experiments for two fundamental graph learning problems, that are subgraph matching and graph clustering. Numerical experiments showed that graph ConvNets had a monotonous increase of accuracy when the network gets deeper, unlike graph RNNs for which performance decreases for a large number of layers. This led us to consider the most generic formulation of gated graph ConvNets, Eq.. We also explored the benefit of residuality for graphs, Eq.. Without residuality, existing graph neural networks are not able to stack more than a few layers. This makes this property essential for graph neural networks, which receive a 10% boost of accuracy when more than 6 layers were stacked. Future work will focus on solving domain-specific problems in chemistry, physics, and neuroscience.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We compare graph RNNs and graph ConvNets, and we consider the most generic class of graph ConvNets with residuality.

represent the tldr for finding the corresponding scientific article

We propose Janossy pooling, a method for learning deep permutation invariant functions designed to exploit relationships within the input sequence and tractable inference strategies such as a stochastic optimization procedure we call piSGD


Question: represent the scientific article to find a short tldr

Question: Complex-value neural networks are not a new concept, however, the use of real-values has often been favoured over complex-values due to difficulties in training and accuracy of . Existing literature ignores the number of parameters used. We compared complex- and real-valued neural networks using five activation functions. We found that when real and complex neural networks are compared using simple classification tasks, complex neural networks perform equal to or slightly worse than real-value neural networks. However, when specialised architecture is used, complex-valued neural networks outperform real-valued neural networks. Therefore, complex–valued neural networks should be used when the input data is also complex or it can be meaningfully to the complex plane, or when the network architecture uses the structure defined by using complex numbers. In recent years complex numbers in neural networks are increasingly frequently used. ComplexValued neural networks have been sucessfully applied to a variety of tasks specifically in signal processing where the input data has a natural interpretation in the complex domain. In most publications complex-valued neural networks are compared to real-valued architectures. We need to ensure that these architectures are comparable in their ability to approximate functions. A common metric for their capacity are the number of real-valued parameters. The number of parameters of complex-valued neural networks are rarely studied aspects. While complex numbers increase the computational complexity, their introduction also assumes a certain structure between weights and input. Hence, it is not sufficient to increase the number of parameters. Even more important than in real-valued networks is the choice of activation function for each layer. We test 5 functions: identity or no activation function, rectifier linear unit, hyperbolic tangent, magnitude, squared magnitude. This paper explores the performance of complex-valued multi-layer perceptrons (MLP) with varying depth and width in consideration of the number of parameters and choice of activation function on benchmark classification tasks. In section 2 we will give an overview of the past and current developments in the applications of complex-valued neural networks. We shortly present the multi-layer perceptron architecture in section 3 using complex numbers and review the building blocks of complex-valued network. In section 4 we consider the multi-layer perceptron with respect to the number of real-valued parameters in both the complex and real case. We construct complex MLPs with the same number of units in each layer. We propose two methods to define comparable networks: A fixed number of real-valued neurons per layer or a fixed budget of real-valued parameters. In the same section we also consider the structure that is assumed by introducing complex numbers into a neural network. We present the activation function to be used in our experiments in section 5. In section 6 we present our experiments and their settings. Section 7 discuss the of different multi-layer perceptrons on MNIST digit classification, CIFAR-10 image classification, CIFAR-100 image classification, Reuters topic classification and bAbI question answering. We identify a general direction of why and how to use complex-valued neural networks. The idea of artificial neural networks with complex-valued input, complex-valued weights and complex-valued output was proposed in the 1970s BID0. A complex-valued backpropogation algorithm to train complex multi-layer networks was proposed in the 1990s by several authors BID2 BID10 BID4. In the 2000s complex neural networks, like real-valued neural networks, have been successfully applied to a variety of tasks. These tasks included the processing and analysis of complex-valued data or data with an intuitive mapping to complex numbers. Particularly, signals in their wave form were used as input data to complex-valued neural networks BID7 ).Another natural application of complex numbers are complex convolutions BID3, since they have an application in both image and signal processing. While real convolutions are widely used in deep neural networks for image processing, complex convolution can replace realvalued convolutions BID14 BID5 BID12 BID6.The properties of complex numbers and matrices introduce constraints into deep learning models. Introduced by BID1 and developed further by recurrent networks, which constrain their weights to be unitary, reduce the impact of the vanishing or exploding gradient problem. More recently complex numbers have been (re)discovered by a wider audience and used in approaches to other tasks like embedding learning BID15 BID13, knowledge base completion BID16 or memory networks BID9.Despite their success in signal processing tasks, complex-valued neural networks have been less popular than their real-valued counter-parts. This may be due to training and reports of varying in related tasks. The training process and architecture design are less intuitive, which stems from difficulties in differentiability of activation functions in the complex plane (; BID8 BID11).An aspect that has received little attention is an appropriate comparison of real-and complex-valued neural networks. Many publications ignore the number of parameters all together , consider only the number of parameters of the entire model or do not distinguish in complex-or real-valued parameters . While the latter is most confusing for the reader, all three problems lead to an inappropriate comparison of the overall performance. There exists a significant body of work on exploring deep learning architectures for real-valued neural networks. Deep complex-valued neural networks are still to be explored. Previous work has also shown the significance of the activation, not only for the training and gradient computation, but also for the accuracy. Therefore, the width, depth and the choice of activation function need to be considered together. We aim to fill this gap by systematically exploring the performance of multi-layered architectures on simple classification taks. Many fundamental building blocks of neural networks can be used in the complex domain by replacing real-valued parameters with complex parameters. However, there are some differences in training complex-valued neural networks. We introduce the building blocks and consider differences in structure and training. While the convolution on the complex plane using complex-valued filters is natural, it has been investigated in related literature (see section 2). In this work we focus on layers consisting of complex-valued neurons as building blocks and their use in multi-layer architecture. We define a complex-valued neuron analogous to its real-valued counter-part. In consequence we can use projection onto a complex weight matrix to realise complex-numbered embeddings. The complex valued neuron can be defined as: DISPLAYFORM0 with the (real or complex) input x ∈ ¼ n, complex weight w ∈ ¼ n and complex bias b ∈ ¼. Arranging m neurons into a layer: DISPLAYFORM1 Similarly, we can define the projection onto a complex matrix if the input x is a projector (e.g. one-hot vector).The activation function φ in all of the above definitions can be a real function φ: ¼ → or complex function φ: ¼ → ¼, but the function always acts on a complex variable. We will consider the choice of the non-linear activation function φ in more detail in section 5.The loss function J should be a real function J: ¼ → or J:→. Since there is no total ordering on the field of complex numbers, because the i 2 = −1, a complex-valued function may lead to added difficulties in training. To be able to interpret the output of the last layer as probability one, can use an additional activation function. Thus the activation of the output layer is sigmoid(φ(z)) resp. so f tmax(φ(z)) with φ: ¼ → and is used as a real number in a classical loss function (e.g. cross entropy).Both activation and loss functions are not always complex-differentiable. Hence, the training process in the complex domain differs. Similar to a real function, a complex function f: DISPLAYFORM2 A complex-valued function of one or more complex variables that is entire and complex differentiable is called holomorphic. While in the real-valued case the existence of a limit is sufficient for differentiability, the complex definition in equation 3 implies a stronger property. We map ¼ to 2 to illustrate this point. A complex function f (x + iy) = u(x, y) + iv(x, y) with real-differentiable functions u(x, y) and v(x, y) is complex-differentiable if they satisfy the Cauchy-Riemann equations: DISPLAYFORM3 We simply separate a complex number z ∈ ¼ into two real numbers z = x + iy. For f to be holomorphic, the limit not only needs to exist for the two functions u(x, y) and v(x, y), but they must also satisfy the Cauchy-Riemann equations. That also means that a function can be nonholomorphic (not complex-differentiable) in z, but still be analytic in its parts x, y. That is exactly if the two functions are real-differentiable, but do not satisfy the Cauchy-Riemann equations. To be able to apply the chain rule, the basic principle of backpropagation, to non-holomorphic functions, we exploit the fact that many non-holymorphic functions, are still differentiable in their real and imaginary parts. We consider the complex function f to be a function of z and its complex conjugatez. Effectively, we choose a different basis for our partial derivatives. DISPLAYFORM4 These derivatives are a consequence of the Wirtinger calculus (or CR-calculus). With the new basis we are able allow the application of the chain rule to non-holomorphic functions for multiple complex variables z i: DISPLAYFORM5 4 Real and Complex ParametersIn this section we discuss complex-and real-valued parameters in consideration of the number of parameters per layer and the structure assumed by complex numbers. Any complex number z = x + iy = r * e iϕ can be represented by two real numbers: the real part Re(z) = x and the imaginary part Im(z) = y or as length or magnitude |z| = x 2 + y 2 = r and a phase ϕ. Effectively the number of real parameters of each layer is doubled: DISPLAYFORM6 The number of (real-valued) parameters is a metric of capacity or the ability to approximate functions. Too many and a neural network tends to overfit the data, too few and the neural network tends to underfit. For a comparison of architectures the real-valued parameters per layer should be equal (or at least as close as possible) in the real architecture and its complex counter part. This ensures that models have the same capacity and a comparison shows the performance difference due to the added structure, not due to varying capacity. Consider the number of parameters in a fully-connected layer in the real case and in the complex case. Let n be the input dimension and m the number of neurons. DISPLAYFORM7 For a multi-layer perceptron with k the number of hidden layers, and output dimension c the number of real-valued parameters without bias is given by: DISPLAYFORM8 At first glance designing comparable multi-layer neural network architectures, i.e. they have the same number of real-valued parameters in each layer, is trivial. Halving the number of neurons in every layer will not achieve a comparison, because the number of neurons define the output dimensions of the layer and the following layer's input dimension. We adressed this problem by choosing MLP architectures with an even number of hidden layers k and choose the number of neurons per layer to be alternating between m and m 2. Thus we receive the same number of real parameters in each layer compared to a real-valued network. As an example, let us consider the dimensions of outputs and weights in k = 4. For the real-valued case: DISPLAYFORM9 where m i is the number of (complex or real) neurons of the i-th layer. The equivalent with m complex-valued neurons would be: DISPLAYFORM10 Another approach to the design of comparable architectures is to work with a parameter budget. Given a fixed budget of real parameters p we can define real or complex multi-layer perceptron with an even number k of hidden layers such that the network's parameters are within that budget. All k + 2 layers have the same number of real-valued neurons m or complex-valued neurons m ¼. DISPLAYFORM11 DISPLAYFORM12 Despite the straight forward use and representation in neural networks, complex numbers define an additional structure compared to real-valued networks. This interaction of the two parts becomes particularly apparant if we consider operations on complex numbers to be composed of the real and imaginary part or magnitude and phase: DISPLAYFORM13 with complex numbers z 1 = a + ib, z 2 = c + id. In an equivalent representation with Euler's constant e iϕ = cos(ϕ) + isin(ϕ) the real parts do not interact.z 1 z 2 = (r 1 e iϕ 1)(r 2 e iϕ 2) = (r 1 r 2 e iϕ 1 +ϕ 2), DISPLAYFORM14 Complex parameters increase the computational complexity of a neural network, since more operations are required. Instead of a single real-valued multiplication operation, up to four real multplication and two real additions are required. Depending on the implementation and the representation, this can be significantly reduced. Nevertheless, it is not sufficient to double the numbers of real parameters per layer to achieve the same effect as in complex-valued neural networks. This is also illustrated expressing a complex number z = a + ib ∈ ¼ as 2 × 2 matrices M in the ring of M 2 : DISPLAYFORM15 An augmented representation of an input x allows the represention of complex matrix multiplication with an weight matrix W as larger real matrix multiplication: DISPLAYFORM16 This added structure, however, also means that architecture design needs to be reconsidered. A deep learning architecture which performs well with real-valued parameters, may not work for complexvalued parameters and vice versa. In later sections we experimentally investigate what consequences this particular structure has for the overall performance of a model. In any neural network, real or complex, an important decision is the choice of non-linearity. With the same number of parameters in each layer, we are able to study the effects activation functions have on the overall performance. The Liouville Theorem states that any bounded holomorphic function f: ¼to¼ (that is differentiable on the entire complex plane) must be constant. Hence, we need to choose unbounded and/or non-holomorphic activation functions. We chose the identity function to investigate the performance of complex models assuming a function which is linearly separable in the complex parameters by not introducing a non-linearity into the model. The hyperbolic tangents is a well-studied function and defined for both complex and real numbers. The rectifier linear unit is a well-studied function and illustrates the separate application of an activation function. The magnitude and squared magnitude functions are chosen to map complex numbers to real numbers.• Identity (or no activation function): DISPLAYFORM0 • Hyperbolic tangent: DISPLAYFORM1 • Rectifier linear unit (relU): DISPLAYFORM2 • Magnitude squared: DISPLAYFORM3 • Magnitude (or complex absolute): DISPLAYFORM4 In the last layer we apply an activation function φ: ¼ → before using the softmax or sigmoid to use in a receive a real loss. Note that the squared magnitude allows a more efficient implementation than the magnitude. Thus we change the activation function of the last layer to: DISPLAYFORM5 Applying the two functions in the opposite order as in |sigmoid(z)| 2 and |so f tmax(z)| 2 does not return probabilities from the last layer of a network and would take away the direct interpretability of the models output. To compare real and complex-valued neural networks and their architecture we chose various classification tasks and defined experiments. The settings are as follows:• Experiment 1: We tested multi-layer perceptrons (MLP) with with k = 0, 2, 4, 8 hidden layers, fixed width of units in each layer in real-valued architectures and alternating 64, 32 units in complex-valued architectures (see section 4), no fixed budget, applied to Reuters topic classification, MNIST digit classification, CIFAR-10 Image classification, CIFAR-100 image classification. Reuters topic classification and MNIST digit classification use 64 units per layer, CIFAR-10 and CIFAR-100 use 128 units per layer. All tested activation functions are introduced in 5.• Experiment 2: We tested multi-layer perceptrons (MLP) with fixed budget of 500,000 realvalued parameters, no fixed width, applied to MNIST digit classification, CIFAR-10 Image classification, CIFAR-100 image classification and Reuters topic classification. All tested activation functions introduced are in section 5. Used sigmoid(|z| 2) function for the gates.• Experiment 3: We tested the Memory Network architecture introduced by BID17 as complex-valued network in two versions -one below and one above parameter budget of the real-valued network. We used the bAbI question answering tasks with one, two and three supporting facts. Activation functions in each layer were defined by the original publication. The network used a recurrent layer, which defined by replacing the real-valued weight matrices with complex weight matrices. For all of our experiments we used the weight initialisation discussed by BID14. However, to reduce the impact of the initialisation we ran each model at least 10 times. The larger memory networks were initialised 30 times. All models were trained over 100 epochs with an Adam optimisation. We used categorical or binary cross entropy as a loss function for all of our experiments and models. We used sigmoid(|z| 2) or so f tmax(|z| 2) as activation function for the last layer of the complex models.7 Results and Discussion TAB1, 3, 4 show the for experiment 1. Generally, the performance of complex and real neural network in this setting is similar, altough the complex valued neural network tends to perform slightly worse. We found that the best choice for the activation function for the complex neural network is relu applied separatly to the imaginary and real parts. Suprisingly the hyperbolic tangents tanh and squared magnitude |z| 2 perform significantly worse Tables 5, 6, 7, 8 show the for experiment 2. Similar to experiment 1 the show that the best choice for the activation function for the complex neural network is relu applied separatly to the imaginary and real parts. In both experiments with the depth of the architecture the performance of the complex neural network decreases significantly. These experiments illustrate that an increased width per layers outperforms an increased depth in classification tasks. This is true for the real and the complex case. Table 9 shows the for experiment 3. For a single supporting fact in the bABi data set realvalued neural network. In the first bABi task the real-valued version outperforms the two complex version of the memory network. In the more diffcult tasks with two or three supporting facts both, the small and large version, of the complex-valued neural network outperform the real-valued versiondespite the reduce number of parameters. We made the observation that the assumed structure with introducing complex numbers into neural networks has a regularising effect on the training procedure if used in combination with real-valued input. We also found that complex-valued neural networks are more sensitive towards their initialisation than real-valued neural networks. Overall the complex-valued neural networks do not perform as well as expected. This may be due to the nature of the chosen tasks or the simple architecture of a multi-layer perceptron. Complex neural networks should be used if the data is naturally in the complex domain or can be mapped to complex numbers. The architecture should be selected to respect the expected structure complex numbers introduce to the network. In the architecture needs to reflect the interaction of the real and imaginary part. If the structure is ignored, the model will not perform as well as the corresponding real-valued network. Table 5: Test accuaracy of multi-layer perceptron conisting of k + 2 dense layers with an overall budget of 500, 000 real-valued parameters on MNIST digit classification. Selected from best of 10 runs with each run 100 epochs to converge. Table 6: Test accuaracy of multi-layer perceptron conisting of k + 2 dense layers with an overall budget of 500, 000 real-valued parameters on Reuters topic classification. Selected from best of 10 runs with each run 100 epochs to converge. Table 7: Test accuaracy of multi-layer perceptron conisting of k + 2 dense layers with an overall budget of 500, 000 real-valued parameters on CIFAR-10 image classification. Selected from best of 10 runs with each run 100 epochs to converge. Table 8: Test accuaracy of multi-layer perceptron conisting of k + 2 dense layers with an overall budget of 500, 000 real-valued parameters on CIFAR-100 image classification. Selected from best of 10 runs with each run 100 epochs to converge. identity 0.0000 0.0000 tanh 0.0000 0.0000 relU 0.0000 0.0000 |z| 2 0.0000 0.0000 |z| 0.0000 0.0000 Table 9: Test accuaracy of Memory Networks BID17 in complex and real version on the first three bAbI tasks. Selected from best of 30 runs with each run 100 epochs to converge.

Comparison of complex- and real-valued multi-layer perceptron with respect to the number of real-valued parameters.

Represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.