Question: Represent the article for finding a single-sentence TLDR

Question: Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy performance degradation. Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from O(n^2) to O(nlog n). These non-parametric and low computational properties construct extremely efficient networks in the number parameters and operations, enjoying accuracy gain. Our proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 48.4% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset. Large Convolutional Neural Networks (CNNs) (; ; ; b; a) and automatic Neural Architecture Search (NAS) based networks; ) have evolved to show remarkable accuracy on various tasks such as image classification , object detection , benefited from huge amount of learnable parameters and computations. However, these large number of weights and high computational cost enabled only limited applications for mobile devices that require the constraint on memory space being low as well as for devices that require real-time computations . With regard to solving these problems,;; Zhang et al. (2017b); proposed parameter and computation efficient blocks while maintaining almost same accuracy compared to other heavy CNN models. All of these blocks utilized depthwise separable convolution, which deconstructed the standard convolution with the (3 × 3 × C) size for each kernel into spatial information specific depthwise convolution (3 × 3 × 1) and channel information specific pointwise (1 × 1 × C) convolution. The depthwise separable convolution achieved comparable accuracy compared to standard spatial convolution with hugely reduced parameters and FLOPs. These reduced resource requirements made the depthwise separable convolution as well as pointwise convolution (PC) more widely used in modern CNN architectures. Nevertheless, we point out that the existing PC layer is still computationally expensive and occupies a lot of proportion in the number of weight parameters . Although the demand toward PC layer has been and will be growing exponentially in modern neural network architectures, there has been a little research on improving the naive structure of itself. Therefore, this paper proposes a new PC layer formulated by non-parametric and extremely fast conventional transforms. Conventional transforms that we applied on CNN models are Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT), which have widely been used in image processing but rarely been applied in CNNs . We empirically found that although both of these transforms do not require any learnable parameters at all, they show the sufficient ability to capture the cross-channel correlations. This non-parametric property enables our proposed CNN models to be significantly compressed in terms of the number of parameters, leading to get the advantages (i.e. efficient distributed training, less communication between server and clients) referred by. We note that especially DWHT is considered to be a good replacement of the conventional PC layer, as it requires no floating point multiplications but only additions and subtractions by which the computation overheads of PC layers can significantly be reduced. Furthermore, DWHT can take a strong advantage of its fast version where the computation complexity of the floating point operations is reduced from O(n 2) to O(n log n). These non-parametric and low computational properties construct extremely efficient neural network from the perspective of parameter and computation as well as enjoying accuracy gain. Our contributions are summarized as follows: • We propose a new PC layer formulated with conventional transforms which do not require any learnable parameters as well as significantly reducing the number of floating point operations compared to the existing PC layer. • The great benefits of using the bases of existing transforms come from their fast versions, which drastically decrease computation complexity in neural networks without degrading accuracy. • We found that applying ReLU after conventional transforms discards important information extracted, leading to significant drop in accuracy. Based on this finding, we propose the optimal computation block for conventional transforms. • We also found that the conventional transforms can effectively be used especially for extracting high-level features in neural networks. Based on this, we propose a new transformbased neural network architecture. Specifically, using DWHT, our proposed method yields 1.49% accuracy gain as well as 79.4% and 49.4% reduced parameters and FLOPs, respectively, compared with its baseline model (MobileNet-V1) on CIFAR 100 dataset. For reducing computation complexity of the existing convolution methods, several approaches of rethinking and deconstructing the naive convolution structures have been proposed. factorized a large sized kernel (e.g. 5 × 5) in a convolution layer into several small size kernels (e.g. 3 × 3) with several convolution layers. pointed out the limitation of existing convolution that it has the fixed receptive field. Consequently, they introduced learnable spatial displacement parameters, showing flexibility of convolution. Based on , proved that the standard convolution can be deconstructed as a single PC layer with the spatially shifted channels. Based on that, they proposed a very efficient convolution layer, namely active shift layer, by replacing spatial convolutions with shift operations. It is worth noting that the existing PC layer takes the huge proportion of computation and the number of weight parameters in modern lightweight CNN models (; ;). Specifically, MobileNet-V1 requires 94%, 74% of the overall computational cost and the overall number of weight parameters for the existing PC layer, respectively. Therefore, there were attempts to reduce computation complexity of PC layer. Zhang et al. (2017b) proposed ShuffleNet-V1 where the features are decomposed into several groups over channels and PC operation was conducted for each group, thus reducing the number of weight parameters and FLOPs by the number of groups G. However, it was proved in that the memory access cost increases as G increases, leading to slower inference speed. Similarly to the aforementioned methods, our work is to reduce computation complexity and the number of weight parameters in a convolution layer. However, our objective is more oriented on finding out mathe-matically efficient algorithms that make the weights in convolution kernels more effective in feature representation as well as efficient in terms of computation. Quantization in neural networks reduced the number of bits utilized to represent the weights and/or activations. applied 8-bit quantization on weight parameters, which enabled considerable speed-up with small drop of accuracy. applied 16-bit fixed point representation with stochastic rounding. Based on Han et al. (2015b) which pruned the unimportant weight connections through thresholding the values of weight, Han et al. (2015a) successfully combined the pruning with 8 bits or less quantization and huffman encoding. The extreme case of quantized networks was evolved from , which approximated weights with the binary (+1, −1) values. From the milestone of ,; constructed Binarized Neural Networks which either stochastically or deterministically binarize the real value weights and activations. These binarized weights and activations lead to significantly reduced run-time by replacing floating point multiplications with 1-bit XNOR operations. Based on ), Local Binary CNN proposed a convolution module that utilizes binarized nonlearnable weights in spatial convolution based on Local Binary Patterns, thus replacing multiplications with addition/subtraction operations in spatial convolution. However, they did not consider reducing computation complexity in PC layer and remained the weights of PC layer learnable floating point variables. Our work shares the similarity to Local Binary CNN in using binary fixed weight values. However, Local Binary Patterns have some limitations for being applied in CNN since they can only be used in spatial convolution as well as there are no approaches that enable fast computation of them. In general, several transform techniques have been applied for image processing. Discrete Cosine Transform (DCT) has been used as a powerful feature extractor . For N -point input sequence, the basis kernel of DCT is defined as a list of cosine values as below: where m is the index of a basis and captures higher frequency information in the input signal as m increases. This property led DCT to be widely applied in image/video compression techniques that emphasize the powers of image signals in low frequency regions . Discrete Walsh Hadamard Transform (DWHT) is a very fast and efficient transform by using only +1 and −1 elements in kernels. These binary elements in kernels allow DWHT to perform without any multiplication operations but addition/subtraction operations. Therefore, DWHT has been widely used for fast feature extraction in many practical applications, such as texture image segmentation , face recognition , and video shot boundary detection (G. & S., 2014). Further, DWHT can take advantage of a structured-wiring-based fast algorithm (Algorithm 1) as well as allowing very high efficiency in encoding the spatial information . The basis kernel matrix of DWHT is defined using the previous kernel matrix as below: where H 0 = 1 and D ≥ 1. In this paper we denote H D m as the m-th row vector of H D in Eq. 2. Additionally, we adopt fast DWHT algorithm to reduce computation complexity of PC layer in neural networks, ing in an extremely fast and efficient neural network. We propose a new PC layer which is computed with conventional transforms. The conventional PC layer can be formulated as follows: where (i, j) is a spatial index, and m is output channel index. In Eq. 3, N and M are the number of input and output channels, respectively. X ij ∈ R N is a vector of input X at the spatial index (i, j), N is a vector of m-th weight W in Eq. 3. For simplicity, the stride is set as 1 and the bias is omitted in Eq. 3. Our proposed method is to replace the learnable parameters W m with the bases in the conventional transforms. For example, replacing W m with H D m in Eq. 3, we now can formulate the new multiplication-free PC layer using DWHT. Similarly, the DCT basis kernels C m in Eq. 1 can substitute for W m in Eq. 3, formulating another new PC layer using DCT. Note that the normalization factors in the conventional transforms are not applied in the proposed PC layer, because Batch Normalization performs a normalization and a linear transform which can be viewed as a normalization in the existing transforms. The most important benefit of the proposed method comes from the fact that the fast algorithms of the existing transforms can be applied in the proposed PC layers for further reduction of computation. Directly applying above new PC layer gives computational complexity of O(N 2). Adopting the fast algorithms, we can significantly reduce the computation complexity of PC layer from O(N 2) to O(N logN) without any change of the computation . We demonstrate the pseudo-code of our proposed fast PC layer using DWHT in Algorithm 1 based on the Fast DWHT structure shown in Figure 1a. In Algorithm 1, for logN iterations, the evenindexed channels and odd-indexed channels are added and subtracted in element-wise manner, respectively. The ing elements which were added and subtracted are placed in the first N/2 elements and the last N/2 elements of the input of next iteration, respectively. In this computation process, each iteration requires only N operations of additions or subtractions. Consequently, Algorithm 1 gives us complexity of O(N logN) in addition or subtraction. Compared to the existing PC layer that requires complexity of O(N 2) in multiplication, our method is extremely cheaper than the conventional PC layer in terms of computation costs as shown in Figure 1b and in power consumption of computing devices . Note that, similarly to fast DWHT, DCT can also be computed in a fast manner that recursively decomposes the N -point input sequence into two subproblems of N/2-point DCT . Compared to DWHT, DCT takes advantage of using more natural shapes of cosine basis kernels, which tend to provide better feature extraction performance through capturing the frequency information. However, DCT inevitably needs multiplications for inner product between C and X vectors, and a look up table (LUT) for computing cosine kernel bases which can increase the processing time and memory access. On the other hand, as mentioned, the kernels of DWHT consist only of +1, −1 which allows for building a multiplication-free module. Furthermore, any memory access towards kernel bases is not needed if our structured-wiring-based fast DWHT algorithm (Algorithm 1) is applied. Our comprehensive experiments in Section 3.1 and 3.2 show that DWHT is more efficient than DCT in being applied in PC layer in terms of trade-off between the complexity of computation cost and accuracy. Note that, for securing more general formulation of our newly defined PC layer, we padded zeros along the channel axis if the number of input channels is less than that of output channels while truncating the output channels when the number of output channels shrink compared to that of input channels as shown in Algorithm 1. Figure 1a shows the architecture of fast DWHT algorithm described in Algorithm 1. This structuredwiring-based architecture ensures that the receptive field of each output channels is N, which means each output channel is fully reflected against all input channels through log 2 N iterations. This fullyreflected property helps to capture the input channel correlations in spite of the computation process of what channel elements will be added and subtracted being structured in a deterministic manner. For successfully fusing our new PC layer into neural networks, we explored two themes: i) an optimal block search for the proposed PC; ii) an optimal insertion strategy of the proposed block found by i), in a hierarchical manner on the blocks of networks. We assumed that there are an optimal block unit structure and an optimal hierarchy level (high-, middle-, low-level) blocks in the neural networks favored by these non-learnable transforms. Therefore, we conducted the experiments for the two aforementioned themes accordingly. We evaluated the effectiveness for each of our networks by accuracy fluctuation as the number of learnable weight parameters or FLOPs changes. For comparison, we counted total FLOPs with summation of the number of multiplications, additions and subtractions performed during the inference. Unless mentioned, we followed the default experimental setting as 128 batch size, 200 training epochs, 0.1 initial learning rate where 0.94 is multiplied per 2 epochs, and 0.9 momentum with 5e-4 weight decay value. In all the experiments, the model accuracy was obtained by taking an average of Top-1 accuracy values from three independent training . (a) A black circle indicates a channel element, and black and red lines are additions and subtractions, respectively. The number of input channels is set as 8 for simplicity. Best viewed in color. (b) x axis denotes logarithm of the number of input channels which range from 2 0 to 2 n. For simplicity, the number of output channels is set to be same as that of the input channel for all PC layers. Best viewed in color. Algorithm 1 A new pointwise convolution using fast DWHT algorithm ZeroPad1D(X, axis=1) pad zeros along the channel axis 4: end if 5: for i ← 1 to n do From a microscopic perspective, the block unit is the basic foundation of neural networks, and it determines the efficiency of the weight parameter space and computation costs in terms of accuracy. Accordingly, to find the optimal block structure for our proposed PC layer, our experiments are conducted to replace the existing PC layer blocks with our new PC layer blocks in ShuffleNet-V2 . The proposed block and its variant blocks are listed in Figure 2. Comparing the of (c) and (d) in Table 1 informs us the important fact that the ReLU activation function significantly harms the accuracy of our neural networks equipped with the conventional transforms. We empirically analyzed this phenomenon in Section 4.1. Additionally, comparing the accuracy of (b) and (d) in Table 1 denotes that the proposed PC layers are superior to the PC layer which randomly initialized and fixed its weights to be non-learnable. These imply that DWHT and DCT kernels can better extract meaningful information of cross-channel correlations compared to the kernels which are randomly initialized and non-learnable. Compared to the baseline model in Table 1, (d)-DCT w/o ReLU and (d)-DWHT w/o ReLU blocks show accuracy drop by approximately 2.3% under the condition that 42% and 49.5% of learnable weight parameters and FLOPs are reduced, respectively. These imply that the proposed blocks (c) and (d) are still inefficient in trade-off between accuracy and computation costs of neural networks, leading us to more explore to find out an optimal neural network architecture. In the next subsection, we address this problem through applying conventional transforms on the optimal hierarchy level features (See Section 3.2). Based on our comprehensive experiments, we set the block structure (d) as our default proposed block which will be exploited in all the following experiments. In this section, we search on an optimal hierarchy level where our optimal block which is based on the proposed PC layer is effectively applied in a whole network architecture. The optimal hierarchy level will allow the proposed network to have the minimal number of learnable weight parameters and FLOPs without accuracy drop, which is made possible by non-parametric and extremely fast conventional transforms. It is noted that applying our proposed block on the high-level blocks in the network provides much more reduced number of parameters and FLOPs rather than applying on low-level blocks, because channel depth increases exponentially as the layer goes deeper in the network. Figure 2 on CIFAR100 dataset. All the experimented models are based on ShuffleNet-V2 with width hyper-parameter 1.1x which we customized to make the number of output channels in Stage2, 3, 4 as 128, 256, 512, respectively for fair comparison with DWHT which requires 2 n input channels. We replaced all of 13 stride 1 basic blocks (i.e. Figure 3: Performance curve of hierarchically applying our optimal block on CIFAR100, Top: in the viewpoint of the number of learnable weight parameters, Bottom: in the viewpoint of the number of FLOPs. The performance of baseline models was evaluated by ShuffleNet-V2 architecture with width hyper-parameter 0.5x, 1x, 1.1x, 1.5x. Our models were all experimented with 1.1x setting, and each dot in the figures represents mean accuracy of 3 network instances. Note that the blue line denotes the indicator of the efficiency of weight parameters or FLOPs in terms of accuracy. The upper left part from the blue line is the superior region while lower right part from blue line is the inferior region compared to the baseline models. In Figure 3, we applied our optimal block (i.e. (d) block in Figure 2 ) on high-, middle-and low-level blocks, respectively. In our experiments, we evaluate the performance of the networks depending on the number of blocks where the proposed optimal block is applied. The model that we have tested is denoted as (transform type)-(# of the proposed blocks)-(hierarchy level in Low (L), Middle (M), and High (H) where the proposed optimal block is applied). For example, DWHT-3-L indicates the neural network model where the first three blocks in ShuffleNet-V2 consist of the proposed blocks, while the other blocks are the original blocks of ShuffleNet-V2. It is noted that in this experiment, we fix all the blocks with stride = 2 in the baseline model to be original ShuffleNet-V2 stride = 2 blocks. Figure 3 shows the performance of the proposed methods depending on the transform types {DCT, DWHT}, hierarchy levels {L, M, H} and the number of the proposed blocks that replace the original ones in the baseline {3, 6, 10} in terms of Top-1 accuracy and the number of learnable weight parameters (or FLOPs). It is noted that, since the baseline model has only 7 blocks in the middlelevel Stage (i.e. Stage3), we performed the middle-level experiments only for DCT/DWHT-3-M and -7-M models where the proposed blocks are applied from the end of Stage3 in the baseline model. In Figure 3, the performance of our 10-H (or 10-L), 6-H (or 6-L), 3-H (or 3-L) models (7-M and 3-M only for middle-level experiments) is listed in ascending order of the number of learnable weight parameters and FLOPs. As shown in the first column of Figure 3, applying our optimal block on the high-level blocks achieved much better trade-off between the number of learnable weight parameters (FLOPs) and accuracy. Meanwhile, applying on middle-and low-level features suffered, respectively, slightly and severely from the inefficiency of the number of weight parameters (FLOPs) with regard to accuracy. This tendency is shown similarly for both DWHT-based models and DCT-based models, which implies that there can be an optimal hierarchical level of blocks favored by conventional transforms. Also note that our DWHT-based models showed slightly higher or same accuracy with less FLOPs in all the hierarchy level cases compared to our DCT-based models. This is because the fast version of DWHT does not require any multiplication but needs less amount of addition or subtraction operations compared to the fast version of DCT while it also has the sufficient ability to extract cross-channel information with the exquisite wiring-based structure. For verifying the generality of the proposed method, we also applied our methods into MobileNet-V1 . Inspired by the above showing that optimal hierarchy blocks for conventional transforms can be found in the high-level blocks, we replaced high-level blocks of baseline model (MobileNet-V1) and changed the number of proposed blocks that are replaced to verify the effectiveness of the proposed method. The experimental are described in Table 2. Remarkably, as shown in Table 2, our DWHT-6-H model yielded the 1.49% increase in Top-1 accuracy even under the condition that the 79.4% of parameters and 49.4% of FLOPs are reduced compared with the baseline 1x model. This outstanding performance improvement comes from the depthwise separable convolutions used in MobileNet-V1, where PC layers play dominant roles in computation costs and memory space, i.e. they consume 94.86% in FLOPs and 74% in the total number of parameters in the whole network . The full performance for all the hierarchy levels {L, M, H} and the number of blocks {3, 6, 10} (exceptionally, {3, 7} blocks for the middle level experiments) are described in Appendix A. In Appendix A, based on the comprehensive experiments it can be concluded that i) the proposed PC block always shows its better efficiency of number of parameters and FLOPs when applied on high-levels compared to when applied on low-level in the network hierarchy; ii) the performance gain start to decrease when the number of transform based PC blocks exceeded a certain capacity of the networks. Top-1 Acc (%) # of Weights (ratio) # of FLOPs (ratio) Baseline 67.15 ± 0.3 Table 2: Performance of hierarchically applying our optimal block on CIFAR100 dataset. All the models are based on MobileNet-V1 with width hyper-parameter 1x. We replaced both stride 1, 2 blocks in the baseline model with the optimal block that consist of [3 × 3 depthwise convolutionBatch Normalization -ReLU -CTPC -Batch Normalization] in series. In this section, we analyze the significant accuracy degradation of applying ReLU after our proposed PC layer. Additionally, we analyze the active utilization of 3x3 depthwise convolution weight kernel values which takes an auxiliary role for conventional transform being non-learnable. As shown in Table 1, applying ReLU after conventional transforms significantly harmed the accuracy. This is due to the properties of conventional transform basis kernels that both H D m in Eq. 2 and C m in Eq. 1 have the same number of positive and negative parameters in the kernels except for m = 0 and that the distributions of absolute values of positive and negative elements in kernels are almost identical. These properties let us know that the output channel elements that have under zero value should also be considered during the forward pass; when forwarding X ij in Eq. 3 through the conventional transforms if some important channel elements in X ij that have larger values than others are combined with negative values of C m or H D m, the important cross-channel information in the output Z ijm in Eq. 3 can reside in the value range under zero. Figure 4 shows that all the hierarchy level activations from both DCT and DWHT based PC layer have not only positive values but also negative values in almost same proportion. These negative values possibly include important cross-channel correlation information. Thus, applying ReLU on activations of PC layers which are based on conventional transforms discards crucial cross-channel information contained in negative values that must be forwarded through, leading to significant accuracy drop as shown in the of Table 1. Figure 6 empirically demonstrates above theoretical analysis by showing that as the negative value regions are fully ignored (i.e. F = ReLU), the accuracy is significantly degraded while fully reflecting the negative value regions (i.e. g = 1) shows the best accuracy. From above kernel value based analysis and its experiment, we do not use non-linear activation function after the proposed PC layer. In Figure 5 and Appendix B, it is observed that 3 × 3 depthwise convolution weights of last 3 blocks in DWHT-3-H and DCT-3-H have much less near zero values than that of baseline model. That Figure 6: Ablation study of negative slope term g in activation function F, which is defined as F (x) = max(0, x) + g * min(0, x). The performance of models were evaluated based on DCT or DWHT-13-H ShuffleNet-V2 1.1x where we applied F as an activation function after every DCT or DWHT based PC layer and Batch Normalization layer. is, the number of values which are apart from near-zero is much larger on DCT-3-H and DWHT-3-H models than on baseline model. We conjecture that these learnable weights whose values are apart from near-zero were actively fitted to the optimal domain that is favored by conventional transforms. Consequently, these weights are actively and sufficiently utilized to take the auxiliary role for conventional transforms which are non-learnable, deriving accuracy increase compared to the conventional PC layer as shown in Figure 3. To verify the impact of activeness of these 3 × 3 depthwise convolution weights in the last 3 blocks, we experimented with regularizing these weights varying the weight decay values. Higher weight decay values strongly regularize the scale of 3 × 3 depthwise convolution weight values in the last 3 blocks. Thus, strong constraint on the scale of these weight values hinders active utilization of these weights, which in accuracy drop as shown in Figure 7. We propose the new PC layers through conventional transforms. Our new PC layers allow the neural networks to be efficient in complexity of computation and learnable weight parameters. Especially for DWHT-based PC layer, its floating point multiplication-free property enabled extremely efficient in computation overhead. With the purpose of successfully fusing our PC layers into neural networks, we empirically found the optimal block unit structure and hierarchy level blocks in neural networks for conventional transforms, showing accuracy increase and great representability in cross-channel correlations. We further intrinsically revealed the hindrance of ReLU toward capturing the cross-channel representability and the activeness of depthwise convolution weights on the last blocks in our proposed neural network. Figure 8: Performance curve of hierarchically applying our optimal block (See Table 2 for detail settings) on CIFAR100, Top: in the viewpoint of the number of learnable weight parameters, Bottom: in the viewpoint of the number of FLOPs. The performance of baseline models was evaluated by MobileNet-V1 architecture with width hyper-parameter 0.2x, 0.35x, 0.5x, 0.75x, 1x, 1.25x. Our proposed models were all experimented with 1x setting, and each dot in the figures represents mean accuracy of 3 network instances. Our models are experimented with 10-H, 6-H, 3-H models (first column), 7-M, 3-M-Rear, 3-M-Front models (second column) and 10-L, 6-L, 3-L models (final column), listed in ascending order of the number of learnable weight parameters and FLOPs. In Figure 8, for the purpose of finding more definite hierarchy level of blocks favored by our proposed PC layers, we subdivided our middle level experiment scheme; DCT/DWHT-3-M-Front model denotes the model which applied the proposed blocks from the beginning of Stage3 in the baseline while DCT/DWHT-3-M-Rear model denotes the model which applied from the end of Stage3. The performance curves of all our proposed models in Figure 8 show that if we apply the proposed optimal block within the first 6 blocks in the network, the Top-1 accuracy is mildly or significantly deteriorated compared to the required computational cost and number of learnable parameters, informing us the important fact that there are the definite hierarchy level blocks which are favored or not favored by our proposed PC layers in the network. For the purpose of demonstrating the superiority of our proposed DCT/DWHT based PC layers over RCPC layer in all the hierarchical (i.e. low/mid/high) level layers, we compared the performance trade-off in Figure 10. It is noted that DCT/DWHT based PC layers almost always get higher accuracy than RCPC layer in all the hierarchical level layers. Comparing the distance between the orange or green line with the red line in Figure 10, our DCT/DWHT based PC layers showed high efficiency in trade-off between accuracy and the computational costs or number of learnable parameters, compared to RCPC layer in almost all the hierarchical levels. Table 4: Quantitative comparison between the baseline model and our proposed DCT/DWHT-based models on FDDB dataset. AP means the true positive rate at 1,000 false positives and all the models were evaluated with discontinuous criterion in FDDB dataset. In order to demonstrate the domain-generality of the proposed method, we conducted comprehensive experiments on applying our proposed PC layers to object detection, specifically to the face detection task. For face detection schemes such as anchor design, data augmentation and featuremap resolution design, we followed Zhang et al. (2017a) which is one of the baseline methods in face detection field. It is noted that there is a huge demand on real-time face detection algorithms having high detection accuracy, which leads us to applying our PC layers to a lightweight face detection network. Therefore, instead of using VGG16 as backbone network as in Zhang et al. (2017a), we set MobileNet-V1 0.25x as our baseline backbone model where extra depthwise separable blocks are added for detecting more diverse scales of face in the images. In this baseline model, we replaced the conventional PC layers within last 3, 6 blocks with our DCT/DWHT based PC layers. We trained all the models on the WIDER FACE train dataset and evaluated on WIDER FACE validation dataset and Face Detection Data Set and Benchmark (FDDB) dataset . WIDER FACE validation set has Easy, Medium and Hard subsets, which correspond to large, medium and small scale faces, respectively. Validation of the baseline model and our proposed DCT/DWHT models on WIDER FACE are described in Table 3. In Table 3, we note that, overall, our DWHT-3-H and DWHT-6-H models showed comparable or even higher mAP values than the baseline model on all the subsets (Easy, Medium, and Hard) with significantly reduced number of learnable parameters and FLOPs. Especially, DWHT-3-H model achieved 0.27% higher mAP than the baseline model under the condition that 79% of parameters and 16% of FLOPs are reduced on Hard subset. Regarding DCT-3-H and DCT-6-H models, they showed a solid improvement of mAP on Easy and Medium subsets with significantly reduced number of parameters and FLOPs compared to the baseline model. Additionally, we verified the effectiveness of the proposed method on the FDDB dataset in Table 4. We note that our DWHT-6-H and DWHT-3-H models showed comparable or even 0.09% higher AP than the baseline model with significantly reduced number of learnable parameters and FLOPs. On the other hand, our DCT-6-H and DCT-3-H models showed a small degree of degradation on AP compared to the baseline model, which is a mild degradation considering the reduced amount of parameters and FLOPs. Consequently, our comprehensive experiments on both WIDER FACE and FDDB datasets reveal the generality of our proposed method, enabling neural networks to be extremely lightweight and reduce the computational overhead.

Represent the TLDR for finding the corresponding scientific article

We introduce new pointwise convolution layers equipped with extremely fast conventional transforms in deep neural network.

Represent the short summary for finding the corresponding scientific article (long)

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: represent the article for finding a single-sentence tldr:

Question: We introduce the notion of \emph{lattice representation learning}, in which the representation for some object of interest (e.g. a sentence or an image) is a lattice point in an Euclidean space. Our main contribution is a for replacing an objective function which employs lattice quantization with an objective function in which quantization is absent, thus allowing optimization techniques based on gradient descent to apply; we call the ing algorithms \emph{dithered stochastic gradient descent} algorithms as they are designed explicitly to allow for an optimization procedure where only local information is employed. We also argue that a technique commonly used in Variational Auto-Encoders (Gaussian priors and Gaussian approximate posteriors) is tightly connected with the idea of lattice representations, as the quantization error in good high dimensional lattices can be modeled as a Gaussian distribution. We use a traditional encoder/decoder architecture to explore the idea of latticed valued representations, and provide experimental evidence of the potential of using lattice representations by modifying the \texttt{OpenNMT-py} generic \texttt{seq2seq} architecture so that it can implement not only Gaussian dithering of representations, but also the well known straight-through estimator and its application to vector quantization. With a few notable exceptions, the majority of the practical research in representation learning assumes that the representation of the objects of interest (sentences, images, audio signals, etc.) are vectors of real numbers, as this allows us to use powerful optimization algorithms such as variants of gradient descent in the training of computational networks which encode objects into such representations and then use those representations in downstream tasks. Yet, the idea of representing objects using discrete structures (for example, through categorical variables or through the use of quantization of otherwise real valued representations) is rather enticing: sometimes we might inherently believe discrete representations to be the right way to model objects, or may want to use such representations in settings such as reinforcement learning and planning, where discrete actions are important. A classical by for maximum likelihood learning of mixture models tells us that the optimal mixing distribution can be chosen to be discrete (and in fact, the discrete set need not be larger than the amount of training data); this implies also that the optimal associated "approximate posterior" (when seen as a variational inference problem) in fact can be chosen so that it produces discrete representations. The main difficulty associated with discrete representations is that it is not straightforward to train networks that produce and use them because either there is no meaningful sense in which differentiation can be used directly (e.g. in the case of categorical variables), or in the case of quantization, the associated gradient is zero almost everywhere. In spite of these difficulties, notable progress has been made. For example, for categorical variables, , and proposed essentially the same idea, under the names of Gumbel-Softmax and the Concrete Distribution, respectively. This idea, further improved by , uses a continuous approximation to a "one-hot" encoded categorical distribution which can be learned by appropriately setting a parameter controlling the fidelity of the approximation. For the setting where a continuous representation is quantized to obtain the discrete one, an important development is the idea of straight-through estimation , in which quantization is applied in the "forward" direction, but replaces quantization with the identity operator when performing the "backwards" differentiation step -see also recent work attempting to provide theoretical justification of straight-through estimation . Of particular relevance to our work is VQ-VAE (van den) in which general vector quantization is used, together with straight-through estimation, to train the network and the vector quantizers. In fact, the entirety of this paper can be seen as a type of VQ-VAE where the vector. Also shown for each lattice is the cell P 0, containing the origin. The parameter of the cubic lattice ∆ has been set so that the areas of both lattice cells are identical. quantizers are highly structured instead of being free parameters in a network; as we will see, the advantages of introducing structure are that we can bring to bear analytical tools that give insight not only on how to train these networks with an equivalent continuous version, but also shed light on how we can approximate the performance of continuous Gaussian VAEs in a systematic way. In this article, we study the possibility of using lattices for object representations by borrowing from theoretical developments in information and coding theory , in particular as they apply to lossy source coding problems -see Figure 1 for an example of two lattices. In our article, a lattice is defined as the set of all integral linear mixtures of a given set of basis vectors for an Euclidean space. Lattices have been long studied in information theory as they provide powerful means for building structured codes which are analytically tractable and have good space packing and covering properties, which are useful in channel coding and lossy source coding applications, respectively . We note that it has long been known that machine learning and lossy compression are related to each other; for example, we refer the reader to the classical work on the information bottleneck (; ; ;), the connection between maximum likelihood estimation and rate distortion explored by , Lastras-Montaño, and implicitly, Rose, Neal & Hinton (1998 and a line of research on autoencoders (Giraldo & Príncipe, 2013; ;). Our work adds to this general line of research by adopting more specific structures -in this case, lattices -from the information theory field and applying them to machine learning problems. Our primary contribution will be to provide the necessary theory to train computational networks that employ lattices and dithered quantization, leading to a class of algorithms which we call dithered stochastic gradient descent algorithms. Dithering refers to the act of adding random noise to a quantized signal (e.g. an image or a time series) for the purposes of diminishing the effect that quantization has on it. The main reason dithering is important to our problem can be traced to a fundamental in the study of lattices in information theory called the "Crypto-Lemma" (see , G. D.) which to our knowledge, had not been used before to optimize, with gradient techniques, computational networks that employ quantization in a provably correct way. We will additionally connect the task of lattice representation learning with a well known technique in generative modeling using variational autoencoders; specifically, Gaussian approximate posteriors . In our experimental section, we will demonstrate how we can use our to train a continuous VAE that can be re-interpreted, through our main , as a discrete VAE that uses a finite dimensional lattice. To make our work concrete, we will illustrate it in the context of a standard encoder/decoder architecture; for a given object d ∈ D (where D is the space of all possible objects), it's representation is produced using an encoder: which is then decoded using a decoder: g(e(d)). We pause to remark that there is a subtle difference between the way in which the information theory and machine learning communities use the words "encoder" and "decoder"; the reader needs to be aware of this difference so as to not get lost in this article. In information theory, an encoder/decoder is almost always a means to represent and retrieve information digitally, in particular, there is a significant emphasis on the efficiency of such representations. In machine learning, an encoder's output is generally thought of as a continuous vector (of course, as we stated earlier, there are notable exceptions to this); the fact that such a continuous vector is represented digitally (in the form of a computer representation of a floating point number) is in a sense an afterthought. Significantly more important in machine learning is the capability of learning parameters for an encoder and decoder through training examples, in which case the assumption of continuity, and more generally, differentiability is much more important. Our work lies in the middle of the two fields. We are interested in a machine learning application, but want to emphasize the representation cost for the objects being encoded as a first class metric to be optimized for. In our work, the representations are discrete, this is for any d ∈ D, e(d) belongs to a discrete space, which we denote as X. To each element x ∈ X we assign a probability p(x), so that we can talk about a code for the representation space X. In this setting, a code is a mechanism for mapping x ∈ X to l(x) bits (where l(x) stands for the code bit length) so that x can be recovered, without loss, from the code for x of length l(x). It is well known that for any given code one can construct a probability distribution using the equation p(x) = 2 −l(x). It is also known that for a given distribution {p(x)}, the ideal choice for l(x) (in units of bits) is − log 2 p(x), in the sense of minimizing the number of expected bits under the assumption that the x are being generated with a distribution {p(x)}. It is also known that such ideal code length in general can only be achieved with block codes which compress multiple elements of X simultaneously. In our work, we will be indifferent to this -we will simply use l(x) = − log p(x) for the code length; in fact for our theoretical development we will be using the unit of nats (natural logarithms) as it is more convenient. Assume that D is a random quantity uniformly distributed over the training data {d 1, · · ·, d n}. Then the average representation cost for the training data is This is one of the metrics in our work. The other metric is free to be specified by the designer of the architecture. Typically, this will be of the form where is some loss function. This covers both unsupervised and supervised settings; in the latter, we assume that the loss function also includes the label for D. It also includes variational autoencoders when they are trained using the Evidence Lower Bound (ELBO). One way to create discrete representations e(D) is to quantize an otherwise continuous representation e c: where K denotes a quantization operation; for example, K could use the uniform quantizer with spacing ∆ applied to every one of the m dimensions. The encoder e, the decoder g and the representation code − log p all participate in the end-to-end objective function Under review as a conference paper at ICLR 2020 where λ > 0 is a parameter that controls the importance of the representation cost for the overall optimization process. In a nutshell, the problem to be solved is to find a theoretically sound basis for the design of good quantizers K that can be trained in an end-to-end computational network. As stated earlier, this can be seen as an attempt to bring additional mathematical rigor to the VQ-VAE concept. 3 DITHERED STOCHASTIC GRADIENT DESCENT The principles behind the use of lattices in information theory can be found in the excellent book of. We borrow from this exposition in here in order to make this work self contained. Let b 1, · · ·, b m be a basis for R m, this is, m linearly independent column vectors in R m, and define B to be the matrix obtained using these basis vectors as its columns: is defined as the set of all integral linear combinations of the {b 1, · · ·, b m}: where When clear from the context, we will use Λ to refer to a lattice. We define P 0 (Λ) to be the set of all points in R m whose closest lattice point is the origin. Given x ∈ R m, we define K Λ (x) to be the operator (the "quantizer") that takes x and maps it to the closest element of the lattice Λ: One of the main reasons we are interested in lattices is because of a mathematical tool that can be used in their analysis, which in information theory is colloquially referred to as the "Crypto-Lemma" , (G. D.): Lemma 1 (Crypto-Lemma). For a given m dimensional lattice Λ, let U be uniformly distributed over P 0 (Λ). For any x ∈ R m, the distribution of the random vector The Crypto-Lemma will give us the main mechanism for training a computational network using gradient descent algorithms and using such network (during inference) with an explicit quantization step, with a firm theoretical guarantee of equivalence. A few additional mathematical preliminaries are needed. When taking expectations we will explicitly state the probability law being used in the expectation by placing a random quantity (most of the time, a random vector) as a subindex of the expectation. If two random vectors A, B are independent, we will write A ⊥ ⊥ B. If these random vectors are independent conditional on a third random vector C, we will write (A ⊥ ⊥ B|C). We will use an upper case P to denote a probability measure, this is, a function that assigns a probability to an event passed as argument. We will use the notation P A (a) as a summary form of P ([A = a]); we sometimes will use a random vector as an argument, thus, for example, −E A log PÂ(A) can be interpreted as the average cost, in nats, of using a code designed forÂ on the random vector A, which is drawn using a (generally) different probability law. We will use the notation f A to denote density of a continuous random vector A. We will use the notation P A|B and f A|B to denote conditional versions of the objects described above. In a slight overload of notation, recall we use f and e c to denote encoders; the correct usage should be clear from context. If P 1 and P 2 are two probability measures, we say that P 1 P 2 (in words, P 1 is absolutely continuous with respect to P 2) if for any event A such that P 2 (A) = 0 then we have P 1 (A) = 0. Our main contribution will be centered on the following new : Theorem 1 (representation cost for dithered SGD). Let Λ be any lattice. Let X be a random R m vector distributed according to P X, andX be a random R m vector distributed according to PX, where we assume that P X PX. Assume U is uniformly distributed over P 0, and assume that The proof of this is based on a dual application of the Crypto-Lemma, and can be found in the Appendix. In the application of , P X will be associated with the empirical statistics of the encoding of training data (prior to quantization), and PX will be associated with a prior belief on the statistics of the encodings of any data, prior to quantization. Notice that the right hand side of Equation has no quantization involved. Notice also that the expression in the expectation does not use directly any global statistics about X; instead it relies on fX −U, which is a density that the designer can specify when the statistics ofX are specified and when the lattice Λ is designed. By avoiding the explicit quantization we eliminate the problem ing from the gradient of a quantized signal being zero almost everywhere, and by ensuring that the expression being optimized does not directly depend on global statistics about X, we ensure that stochastic gradient descent is feasible, since it requires that you update parameters based on gradients computed on small batches of training data. For these reasons, we shall call the machine learning algorithms that one derives from Theorem 1 dithered stochastic gradient descent algorithms. Now examine the left hand side of Equation, and notice that it does involve explicitly quantization: in fact, Z will become the discrete lattice representation of our work. Notice the conditioning in PẐ |U on the dither value U. This means that the discrete representation not only depends on the dither U, but also is being encoded using a code that depends U as well. Notice also that − log PẐ |U may not be the very best code one could be using which is − log P Z|U, in fact where the former is the "true" best possible representation cost (for a given encoder) and the latter term (a conditional KL divergence which is known to be nonnegative) denotes the excess cost. Thus the representation cost may not be, in a certain sense, "optimum", which is what we pay to make stochastic gradient descent feasible by avoiding having to compute global statistics at every step of the optimization. Having said this, our anecdotal experience is that this is not a significant problem if one allows the parameters of the encoder to also be trained, as then the encoder "adapts" so as to produce representations that approximate the prior that we have set. A reader knowledgable in the literature of lattices in lossy compression theory may be wondering what is the connection between Theorem 1 and the information theoretic linking entropy coded dithered quantization and mutual information . The answer is that Theorem 1 generalizes this to a setting where we aren't necessarily using the optimum code, which happens to be particularly useful in stochastic gradient descent algorithms. We now construct the system that we intend to study. We refer the reader to Figure 2, where we illustrate an encoder/decoder architecture coupled to a loss function. In the left hand side, we have a network that employs pre/post dithered quantization of a representation quantized using a lattice Λ; in the right, there is subtractive dithering but no quantization. Following Figure 2, define With this definition, we can see that in the left hand side of Figure 2, the quantized representation Z matches the definition in Equation of Theorem 1: Pre/post dithered quantization in the context of lattice representation learning. The system in the left is used during inference time, the one in the right during training time. Now in reference to the optimization objective in Equation, note that one of the things we need to do is to define a code that will be used to encode Z. For this purpose, we will use a code constructed using a distribution PẐ |U. The way we will construct this conditional distribution is also given by the equations in Theorem 1. In particular, the designer will be free to specify any distribution PX of their choice, and then construct the random vectorsX, U so that they are independent random vectors distributed according to PX and P U, respectively. Next, as in Equation we definê The distribution PẐ |U is the one associated with this definition. Encoding Z, the lattice representation of D, using the code implied by PẐ |U incurs on the cost Continuing our analysis of the optimization objective in Equation, the designer specified loss function value is (D, g(Z − U)) so that the total objective function is transformed to The difficulty in optimizing this objective function is that Z is the of quantization operation; this affects both terms. To deal with the term in the left, we observe that Lemma 1 implies that P X,Z−U = P X,X−U Now observe that (Z − U ⊥ ⊥ D|X) and (X − U ⊥ ⊥ D|X); and as a consequence This demonstrates how we get rid of the quantization by replacing it with subtractive dithering for the first term in. For the second term (the representation cost), we use Theorem 1 to obtain. so that the new optimization objective is now As indicated earlier, stochastic gradient descent is possible because fX −U does not use any global information about the training data D. The way we propose this objective function be optimized is similar to the optimization algorithms used in Variational Autoencoders (the so called "re-parametrization trick"); in particular, we propose to sample for every D, one dither U and then performing gradient descent regarding the sampled U regarded as a constant in the computational network. Assume that the role of the decoder g is to output a distribution over the data space D, this is, where Q is a distribution over D. Assume that that we choose for the loss function In this setting, the loss function is quantifying the likelihood of the data d given the dithered lattice representation Z − U. From an information theoretic coding perspective, this likelihood is the cost in bits needed to represent the data d, given Z − U. In information theory, U is regarded as common randomness shared between the encoder and decoder, and therefore there is no cost in transmitting it (in practice, pseudo random numbers with a shared seed can be used to implement this, for example). Thus to complete the cost needed to represent the data D, we need the cost of representing Z, which is in our work is given by − log PẐ |U (Z|U). As per our earlier development, we can substitute both of these likelihoods instead with ones which do not employ quantization, arriving to Those versed in the literature of variational autoencoders, might recognize the famous Evidence Lower BOund (ELBO) (for λ = 1) in the expression above. The term in the right-hand side is associated with the KL term (prior to taking expectation). One way to see is by noticing that ) which is the "approximate posterior" and fX −U is the "prior" (which explains why we are giving the designer full control over the distribution ofX). The term on the left, on the other hand, is associated with the "decoder" in the VAE. A very common practice in VAEs is to model the prior and the approximate posterior as Gaussians. It turns out that there is a sense in which the distribution of U approximates that of independently identically distributed (i.i.d.) Gaussian noise. To see this, we need to introduce additional concepts from the theory of lattices. The volume, second moment and and normalized second moment of the lattice cell are defined as respectively. The smallest possible second moment for any lattice on dimension m is defined as G m; we shall call any such lattice optimal. The normalized second moment of an m-dimensional hyper-sphere is denoted by G * m. It is known that A classical in the theory of lattices is that the normalized second moment of an m dimensional optimal lattice converges to the same quantity in other words, there exist lattices whose normalized second order moment approaches that of a hyper-sphere as the lattice dimension m grows to infinity. Now denote Z * to be an m dimensional Gaussian vector with independent entries each with a variance equal to σ 2 Λ, and let U m denote a random vector uniformly distributed over P 0 (Λ). Then, it is not difficult to show that where D KL denotes the KL divergence and as a consequence, combining and, we see that One way to interpret this colloquially is that the distribution of the dither of a good lattice approximates that of a Gaussian as the dimension grows to infinity. This establishes more firmly the connection between variational autoencoders that employ Gaussian approximate posteriors with lattice representations. Suppose for now thatX is chosen also to be i.i.d. Gaussian. In light of, let us analyze the performance of a good high dimensional lattice Λ by assuming U to be also i.i.d. Gaussian with variance σ 2 Λ. Then fX −U is the distribution of another i.i.d. Gaussian. Let us assume that the parameters ofX and U are such thatX − U has unit variance then becomes At this point, the expression is identical to one that one might derive from VAE theory. One important point is to note that in VAEs, the approximate posterior can have general parameters that depend on the encoder output in complex forms; for example, in the case of a Gaussian approximate posterior both the mean and the correlation matrix in the approximate posterior could depend on the encoder output. In the case in which we use lattice representations, only the mean of the approximate posterior can have dependencies on the encoder output. The significance of the observation in this subsection is that many of the (with relatively minor modifications) that the VAE community has obtained likely can be re-interpreted as involving (discrete) lattice quantized representations together with pre/post quantized dithering. The reader may be wondering: so what is new? The point of this section is to suggest that very good lattices approximate the behavior of Gaussian VAEs which are known to be good. To implement a practical system, Theorem 1 can be used to implement finite lattices. We will show this in the experimental section. The reader may also be asking: but why even use lattices at all? Why not use, say, categorical variables and some variant of the Concrete/Gumbel-Softmax trick, or use VQ-VAEs directly? One way to answer this question is: consider using lattices if you interested in a theoretically sound basis obtaining Gaussian VAE-like performance using discrete representations. For our experiments with finite dimensional lattices, we will be using a lattice whose basis is a diagonal matrix with positive entries. This is a simple variant of a cubic lattice in which the spacing parameter can vary per dimension so we will call it the rectangular lattice. To implement training of a VAE using this lattice, we rely on Theorem 1. For this, we need to specify a prior distribution onX. Note that one of the key things we will need is fX −U, the density ofX − U where U is uniformly distributed over the cell of the lattice we are considering here. Therefore, it is highly advantageous to choose a distribution forX so that the distribution ofX − U has a simple form. Our choice for the distribution ofX is for each entry of the vectorX to be a zero mean Laplacian distribution, this is, an origin centered symmetric distribution that decays exponentially on each side. To simplify, assume m = 1 and let ∆ > 0 the width of the uniform distribution so that U is uniformly distributed over (−∆/2, ∆/2). Let b be the parameter for the Laplacian distribution ofX, then In general we assume a collection of {∆ i} and {b i} potentially distinct parameters. Comparison of the performance (negative log likelihood in nats) of a rectangular lattice VAE, a hypothetical high dimensional lattice (in green), approximated using a Gaussian VAE where the variance of the approximate posterior is not data dependent, and and a Gaussian VAE (in red) where the variance of the Gaussian of the approximate posterior is allowed to be data dependent. We use static MNIST in our experiments. For both the encoder we use a three layer network in which the dimensions of the intermediate layer outputs are 784 → 300 → 300 → 40. The first two layers are gated linear networks with no activation function (the only nonlinearity is the multiplicative gating) and the last layer is a simple linear network with no gating or activation. For the decoder the dimensions are 40 → 300 → 300 → 784; as before the first two layers are gated linear networks with no activation function but the last layer is an ungated linear network with a sigmoid activation function. We use a batch size of 100 and Adam with learning rate 5e-4. We keep track of the model with the best validation set performance and if more than 50 epochs pass without being able to improve this performance, we stop the experiment. Then we compute the an improved log likelihood estimate by making use of importance sampling with 1000 samples for each test example. To implement this, we adapted the code made available for the in. When training for the finite dimensional lattice, the training objective for an MNIST image D is where Q is the decoder network, e c denotes the encoder network and [] i denotes the ith element of a vector. When train two types of Gaussian VAEs. In one case, the approximate posterior variance is not allowed to depend on the original MNIST image; we call this the Lattice Gaussian VAE because it denotes the performance of an ideal lattice VAE. The other setting is a Gaussian VAE where the approximate posterior variance is allowed to depend on the data; this we simply term Gaussian VAE. The of our experiments can be seen in Figure 3, where we illustrate the negative log likelihood of the validation set as a function of the number of training epochs. As seen, the performance of the rectangular lattice VAE is not as good as that of either of the two Gaussian VAEs, which is expected; as described in this article, simpler lattices won't perform as well as high dimensional good lattices. Notice however that the performance of the rectangular lattice is quite competitive and furthermore, notice that the performance of the two Gaussian VAEs are rather close to each other. This is encouraging for lattice based VAEs because it implies that it is in principle possible to get rather close to the performance of a Gaussian VAE using higher dimensional lattices. We then evaluated a better upper bound on negative log likelihood through the technique of importance sampling ; the are in Table 1. We see here that the performance of the rectangular lattice VAE remains quite competitive; for example the numbers reported for GumbelSoftmax for MNIST in are above 100 nats. To get closer to the performance of a Gaussian Lattice VAE, what we need to do is to increase the effective lattice dimension; for example we could use the hexagonal lattice or even higher dimensional lattices. Theorem 1 remains the main tool for training such more complex systems; this is left for future work. The present work is inspired by a belief that information theory, and in particular lossy compression theory can be very effective in serving as a theoretical foundation for problems in representation learning, including the design and analysis of highly performant practical algorithms. We have introduced lattices as a possible way to create discrete representations, and proved a fundamental which allows us to train computational networks that use lattice quantized dithering using an equivalent (in an expected sense) computational network which replaces quantization with dithering, thus allowing gradient descent to apply. This also allows us to use only local information during the optimization, thus additionally enabling stochastic gradient descent. We also established a fundamental connection between the use of good high dimensional lattices and the idea of Gaussian dithering, which is common in generative modeling settings such as Variational Autoencoders. Finally, we provided initial experimental evidence of the potential of using lattices in an VAE setting, where we contrasted the performance of a rectangular lattice based VAE and two types of Gaussian VAEs. The bottom line is that if one is interested in getting close to the performance of a Gaussian VAE with discrete representations with a good theoretical basis, we suggest the reader to consider lattices and to train them using dithered stochastic gradient descent. A.1 PROOF OF THEOREM 1 -CORE ON DITHERED SGD For reference purposes, we restate the statement of the theorem: Theorem 1 (representation cost for dithered SGD). Let Λ be any lattice. Let X be a random R m vector distributed according to P X,X be a random R m vector distributed according to PX, where we assume that P X PX. Assume U is uniformly distributed over P 0, and assume that U ⊥ ⊥ X and U ⊥ ⊥X. Define Then To prove this , we will rely on this Lemma: Lemma 2. Let X, U, Z be R m valued random vectors such that (X ⊥ ⊥ U |Z − U) and X ⊥ ⊥ U, with U continuous and Z discrete. Then The proof thus proceeds as follows: where (a) follows from the definition ofẐ which implies that for all x and all u, PẐ |U,X (K Λ (x + u)|u, x) = 1, (b) follows from the definition of Z, (c) follows from an application of Lemma 2, (d) is a change of variables which is feasible since X and Z − U are the only random vectors under the expectation, (e) follows from an application of the Crypto Lemma, (f) is another change of variables, (g) follows from another application of the Crypto Lemma, but this time applied to the joint distribution of X,Ẑ − U. In this appendix, we describe some for the autoencoding setting, where the goal is not to estimate densities but rather to create a representation that describes with high fidelity the original object of interest. In the autoencoding setting we are able to make a stronger statement than in the VAE example in the main body of the paper about the benefits of using better lattices. Let I m denote the m × m identity matrix. The following Lemma describes how the second moment of a lattice whose dither is "white" controls the quantization error as observed through an arbitrary function: Lemma 3 (approximation error due to lattice quantization). For a given dimension m ≥ 1, let η: R m → R be a twice differentiable function. Assume that U is a column vector uniformly distributed over the cell P 0 (Λ) of a lattice Λ, and assume that where H η denotes the Hessian (the matrix of second order derivatives) of η. Proof. It is easy to see that if u ∈ P 0 (Λ), then −u ∈ P 0 (Λ). Due to this property, the first moment of a lattice is zero: Using a Taylor series expansion, assuming σ Λ is small, Lattices that achieve G m have white dithers. This is a by Poltyrev and can be found in : Lemma 4. For any m ≥ 1, if a lattice Λ that achieves G m and U is uniformly distributed over If one keeps constant the volume V 2/m Λ of the lattice cell then by definition, We consider two lattices: a lattice spanned by a matrix proportional to I m, and a lattice with a normalized second moment equal to G m. From the previous discussion, it is easy to see that both lattices have white dithers, and therefore Lemma 3 applies. We also note that the normalized second moment of lattice spanned by a matrix proportional to I m is G 1 = 1/12. Applying Lemma 3 twice, the difference between the two approximations (one for each lattice) can be estimated by The above is an absolute estimate. In relative terms, the size of the error by using the best lattice with dimension m relative to the size of the error incurred by the cubic lattice. This is then given by In the above, η was a generic function but in our case, we would like to identify it with. Thus, we can improve the quantization error in the designer's choice of a loss function by up to 30% by using a better lattice, all at a constant representation cost. The purpose of our experiments is to draw a comparison between two techniques for quantized representation learning. Both techniques will be using stochastic gradient descent, which implies that we need to guarantee that the "backwards" computational network (the one used to compute the gradient of the objective functional with respect to all parameters) needs to be associated with a meaningful gradient, this is, one that can be used to iteratively improve the loss function. The first technique is a network trained with scalar or vector quantization in the forward direction. Because quantization has a zero derivative almost everywhere, it is not "meaningful" in the sense of the paragraph above. Therefore, in the backward direction we use straight-through estimation in order to create a fully differentiable backwards computational network (; van den). The second technique is an estimate of how a very good lattice in high dimensions will perform, using non-data dependent Gaussian dithering to approximate a uniform distribution over this lattice. For the experimental setting, we have chosen a seq2seq autoencoder as implemented in OpenNMT-py (; . This open source package implements a generic seq2seq architecture that encodes a variety of input types (text, audio, images) into a Figure 4: Comparison of the performance of a hypothetical high dimensional lattice, approximated using Gaussian dithering, and several quantizers trained using straight-through estimation. The x-axis is the average word negative log likelihood in bits. The y-axis is the representation cost of the quantized representation vector, averaged per word. Lower is better for both. All numbers plotted are for the test data. vector valued representation, and then decodes this representation into text. For our experiments, we have chosen a strict autoencoder experiment where we encode text into a representation and where we expect that exactly the same text be produced by the decoder from this representation. We modified this open source package (the source code will be released) by intercepting the communication between the encoder and decoder in order to apply the various quantization techniques that we are proposing to evaluate. In a seq2seq setup like the one being described in here, the loss function is the average word negative log likelihood at the output of the decoder. We will maintain the same loss function verbatim. In addition, we computed the representation cost of the quantized vector communicated from the encoder to the decoder, normalized also on a per word basis. There is a tradeoff between these two quantities -the higher the allowed representation cost, in principle the better loss function we can obtain. For the specifics of the architecture, we chose a 2 layer bidirectional GRU recurrent neural network for both the encoder and decoder, with each direction having 250 dimensions in its internal state and in the GRU's output vector (so that the total communication pipe between encoder and decoder is 500 dimensions). The OpenNMT-py package implements an optional global attention which creates additional communication paths between encoder and decoder. We disabled this, as it presents additional complications that are not important for the purposes of this article. We use Adam with an initial learning rate of 1e-4, which decays by a factor of 0.75 every 50k steps after an initial 100k steps where it is kept constant. The total number of training steps is 500k. The parameter λ which weights the representation cost in the total objective function is slowly annealed from a value close to 0 to its (asymptotic) value of 1.0; the value of 0.5 is achieved at 100k steps. When performing quantization and straight-through estimation, the 250 dimensions of each direction of the bi-directional GRU can be quantized in several ways, depending on how we partition the 250 dimensions. One family of such possibilities is to use N = 250 scalar quantizers each using M levels, for various values of M. Another example we demonstrate is N = 125 quantizers, each with M = 4 two dimensional code vectors. In any of the experiments, each quantizer is independent of the other quantizer, in the sense that its M levels are parameters of the model which are optimized using gradient descent. Each of the N codes is associated with a code (a distribution over its M levels), which can also be optimized using gradient descent. The average code length obtain as the representations are encoded with this code is the representation cost. For the experiments using Gaussian dithering, the output of the encoder is first linearly transformed using a linear operator with free parameters, and then it is dithered using an uncorrelated zero mean Gaussian with a diagonal correlation matrix that is also a set of free parameters. The dithered signal is then passed through another linear operator. The representation cost for Gaussian dithering is computed using techniques from Variational Autoencoders; in essence we assume an isotropic unit variance Gaussian as a prior over the representation space and then estimate the KL divergence between the dithered representation conditional on the encoder output and the Gaussian prior. The specific data that we use for the text autoencoding experiment is a set of over 4 million english sentences prepared using scripts that are available in Google's NMT tutorial (https://github.com/tensorflow/nmt, in particular the wmt16_en_de.sh script) for a German/English translation system; as this is an autoencoding setup, we only use the English sentences. For the test data, we use 3003 sentences also typically used in translation experiments for testing (newstest2014). This data set was then processed using OpenNMT-py preprocess.py tool. We only report for the test data. The can be seen in Figure 4. In the lower left hand side we see the tradeoff between the representation cost and the average word negative log likelihood. We can see that the projected performance of a good lattice is significantly better than the performance of the specific quantizers, trained with straight-through estimation that we tested. The reader may be wondering whether the high dimensional assumption that we make on the gaussian dithering approximation to good lattices implies that the projected performance may be unrealistic; we do not know with certainty at this point however we believe that a good lattice in 250 dimensions will likely be very well approximated with a Gaussian.

We propose to use lattices to represent objects and prove a fundamental result on how to train networks that use them.

represent the tldr for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.