Question: Represent the article for finding a single-sentence TLDR

Question: Historically, the pursuit of efficient inference has been one of the driving forces be-hind the research into new deep learning architectures and building blocks. Some of the recent examples include: the squeeze-and-excitation module of , depthwise separable convolutions in Xception , and the inverted bottleneck in MobileNet v2 . Notably, in all of these cases, the ing building blocks enabled not only higher efficiency, but also higher accuracy, and found wide adoption in the field. In this work, we further expand the arsenal of efficient building blocks for neural network architectures; but instead of combining standard primitives (such as convolution), we advocate for the replacement of these dense primitives with their sparse counterparts. While the idea of using sparsity to decrease the parameter count is not new , the conventional wisdom is that this reduction in theoretical FLOPs does not translate into real-world efficiency gains. We aim to correct this misconception by introducing a family of efficient sparse kernels for several hardware platforms, which we plan to open-source for the benefit of the community. Equipped with our efficient implementation of sparse primitives, we show that sparse versions of MobileNet v1 and MobileNet v2 architectures substantially outperform strong dense baselines on the efficiency-accuracy curve. On Snapdragon 835 our sparse networks outperform their dense equivalents by 1.1−2.2x – equivalent to approximately one entire generation of improvement. We hope that our findings will facilitate wider adoption of sparsity as a tool for creating efficient and accurate deep learning architectures. Convolutional neural networks (CNNs) have proven to be excellent at solving a diverse range of tasks . Standard network architectures are used in classification, segmentation, object detection and generation tasks (; ;). Given their wide utility, there has been significant effort to design efficient architectures that are capable of being run on mobile and other low power devices while still achieving high classification accuracy on benchmarks such as ImageNet . For example, MobileNets employ the depthwise separable convolutions introduced in to significantly reduce resource requirements over previous architectures. Inference time and computational complexity in these architectures are dominated by the 1×1 convolutions, which directly map to matrix-matrix multiplications. Weight sparsity is generally known to lead to theoretically smaller and more computationally efficient (in terms of number of floating-point operations) models, but it is often disregarded as a practical means of accelerating models because of the misconception that sparse operations cannot be fast enough to achieve actual speedups during inference. In this work we introduce fast kernels for Sparse Matrix-Dense Matrix Multiplication (SpMM) specifically targeted at the accceleration of sparse neural networks. The main distinction of our SpMM kernel from prior art is that we focus on a different point in the design space. While prior work focused on extremely sparse problems (typically >99%, found in scientific and graph problems), we target the sparsity range of 70-95%, more common when inducing weight sparsity in neural networks. As a our kernels outperform both the Intel MKL and the TACO compiler . Using these kernels, we demonstrate the effectiveness of weight sparsity across three generations of MobileNet (; ; ;) architectures. Sparsity leads to an approximately one generation improvement in each architecture, with a sparse EfficientNet significantly more efficient than all previous models. These models represent a new generation of efficient CNNs, which reduces inference times by 1.1 − 2.2×, parameter counts by over 2× and number of floating-point operations (FLOPs) by up to 3× relative to the previous generations. Improvements in convolutional network architectures (; ; ;), as measured by increased classification accuracy on benchmark tasks such as ImageNet , have generally been concomitant with increases in model parameter counts, FLOPs and memory requirements. Recently this evolution has led to networks found through neural architecture search ) which can achieve over 82% top-1 accuracy, but require nearly 25 GFLOPs for one inference. Given these prohibitive inference costs, there have been many lines of work attempting to improve CNN efficiency, which is often defined as one of three metrics: 1. Inference speedup on real hardware 2. Theoretical speedup through FLOPs reduction 3. Model size reduction These axes are neither parallel nor orthogonal. The effect of and on in particular can be quite complicated and highly varied depending on the hardware in question. The MobileNet family of architectures has focused on improving efficiency by taking advantage of the depthwise separable convolutions introduced in , which can be thought of as a hand-crafted sparsification of full convolutions with a predefined sparse topology, and which are responsible for the parameter efficiency of these architectures. MobileNet v1 (MBv1) used layers of 1 × 1 convolutions followed by depthwise convolutions. MobileNet v2 (MBv2) introduced the inverted residual block which consists of a 1 × 1 convolution expanding the channel count, a depthwise convolution on the expanded channel count, and then a 1 × 1 convolution reducing the parameter count. Across MobileNet architectures, the depthwise convolutions account for only a small fraction of the total FLOPs, parameters, and inference time of these models. In MBv1, they account for less than 2% of the total FLOPs and in MBv2 less than 3%. A different line of work attempted to make more efficient CNNs by directly pruning the weights of full convolutional filters accompanied by the necessary inference kernels (; . was not able to accelerate 1 × 1 convolutions, did not attempt it. The latter also required generating a new set of kernels for each instance of a model, which is often impractical for deployment. Due to the difficultly of accelerating sparse computation, channel pruning approaches have been preferred (; ; ; ; ;). These approaches prune away entire filters leaving the final model dense, and function more as an architecture search over channel counts. Full Neural Architecture Search has also been applied directly to architectures resembling MBv2 ing in MobileNet v3 , FBNet, and EfficientNet . Alternatively, factorizations of the 1×1 convolutions have been considered in ShuffleNet and Learnable Butterfly Factorizations . ShuffleNet factorizes the weight matrix into a product of a permutation matrix and block diagonal matrix. Butterfly Factorizations factorize the weight matrix into a sequence of permutation matrices and weight matrices with special structure that can represent many common O(N logN) transforms such as Fast Fourier Transforms. Work in Text-to-Speech (TTS) demonstrated that increasing sparsity and concomitant increase in state size in RNN models lead to increased model quality for a given non-zero parameter count. They additionally demonstrated fast block-sparse matrix-vector (SpMV) multiplication routines necessary for RNN inference. To understand how to design the most efficient convolutional models, we investigate both how to construct and train sparse MBv1, MBv2 and EfficientNet models and also the performance of our SpMM kernels. We train on the ImageNet dataset with standard augmentation and report top-1 accuracies on the provided 50k example validation set. To make the networks sparse we use the gradual magnitude pruning technique of . We do not prune the first dense convolution at the beginning of all three networks. Its overall contribution to the parameter count, FLOP count, and runtime is small and does not warrant introducing a new sparse operator. Instead, we implement a dense convolutional kernel which takes as input the image in the standard HWC layout and outputs the CHW layout consumed by the sparse operators in the rest of the network. In HWC layout, the values for different channels corresponding to one spatial location are adjacent in memory. In CHW layout, the values of all the spatial locations for one channel are adjacent in memory. We also do not prune the squeeze-excitation blocks in EfficientNet as they contribute <1% of the total FLOPs to the dense model. The last fully-connected layer in all models also contributes insignificantly (<1%) to the total FLOP count, but does contribute a significant fraction (20-50%) of total parameters, especially after the rest of the model is pruned. As we are concerned Figure 3: Visualization of the memory reads and writes of our algorithm. In step 1, we load 8 spatial locations simultaneously for each of the non-zero weights in the first row of the weight matrix. We multiply each scalar weight by its corresponding row, accumulate the , and in the end write them out. Step 2 performs the same calculation for the next output channel. After steps 1 and 2, all values for these spatial locations are in the cache, so future loads in steps 3 and 4 will be fast, despite being random access. with maximizing top-1 accuracy for a given runtime, we do not prune the final layer in MobileNet v1 and v2 as doing so leads to a small decrease in top-1 accuracy. Standard EfficientNets do not scale the number of filters in the last convolution by the width of the model, however we find that when introducing sparsity it is beneficial to do this; in all sparse EfficientNet models we double the units from 1280 to 2560. We also find that it is possible to make the fully-connected layer sparse without loss of accuracy in EfficientNet, so we do so. A diagram of the 1 × 1 convolution as a SpMM is seen in figure 2. Our scheme requires activation tensors be stored in CHW format, in contrast to dense mobile inference libraries (; ;) which favor HWC. There are two key insights enabling the high performance of our kernels: 1. While the weight matrix is sparse, the activation matrix is dense. This means that we can perform vector loads from the activation matrix and process multiple spatial locations simultaneously. 2. By processing the matrix in the right order we can keep values that will be randomly accessed in the L1 cache, from which random access is fast and constant time. In addition to the vectorization in the HW dimension, taking advantage of small amounts of structure in the weight matrix can offer significant performance boosts by increasing data reuse after values are loaded into registers. Constraining the sparsity pattern so that multiple output or input channels all share the same zero/non-zero pattern creates'blocks' in the weight matrix (see figure 3 right). Blocks in the output channel dimension allow for more data reuse than blocks in the input channel dimension. Experiments (see figure 6) show that either choice has the same effect on accuracy, so we implement output channel blocking with sizes of 2 and 4. Our nomenclature for kernels is to give their spatial vectorization width followed by the output channel block size -16x2 means 16 pixels and 2 output channels are processed in the inner loop. We implement the ARM kernels in C with NEON intrinsics unlike current production libraries (; ;) which rely on expert-optimized assembly. As reference, the code for the 4x1 inner loop is available in appendix A. We provide a library that can run sparse models trained with the model pruning library in TensorFlow . This includes conversion from a dense representation to a Block Compressed Sparse Row (BCSR)-like representation suitable for inference. In addition to the high performance 1 × 1 convolutions, we also provide all supporting CHW kernels -depthwise convo- lutions, global average pooling and a 3 × 3 stride-2 dense convolution -necessary for running all three generations of models. While we provide high performance versions of these kernels, we do not detail them here. They are included in end-to-end measurements. In the main text we mainly include for MBv1 and MBv2 due to space limitations. EfficientNets generally follow the same trends as MBv2 models, plots for EfficientNet can be found in appendix C. First we reveal performance for our SpMM kernels, then we show how the networks respond to sparsity and then finally we combine this information to find the models with the lowest inference time. We use Ruy , the current TensorFlow Lite ARM64 backend written largely in handcoded assembly, as the dense baseline. For a sparse baseline we use the kernel generated by the TACO compiler . We present by plotting the FLOPs achieved at each layer in the model, with increasing depth to the right in figure 4. For MBv1 we use a width multiplier of 1.4 and 90% sparse and for MBV2 we use a width multiplier of 1.4 and 80% sparse as these configurations approximately match the top-1 accuracy of the width 1 dense models. The kernel variants that process 16 spatial locations at a time (e.g. 16x1, etc.) are the highest performing and all reported numbers are from these kernel variants. TACO only supports unstructured sparsity and should be compared with the 16x1 kernels. The raw performance of the sparse kernels falls in the range of 40-90% of the dense kernels. And as they must do much less work, when taking the sparsity of the layer into account, the effective FLOPs are in the 2-7× range. In MBv1 performance falls significantly in the last two layers of the model when the number of channels causes the size of one "strip" of spatial locations to exceed the size of the L1 cache. In MBv2 the sawtooth pattern is caused by the alternating expand and contract operations. The performance is higher for the expand kernels due to greater data reuse of each "strip" that is brought into the L1 cache. We implement an AVX-512 version of our scheme with intrinsics to compare with the Intel MKL SpMM. Results are in figure 5. In the majority of layers our scheme outperforms the MKL. The geometric mean speedup over all layers is 1.20 in both MBv1 and MBv2. The hyper-parameters used to train MBv1 and MBv2 are listed in table 2, they were found with a grid search on dense models with a width multiplier of 1.0 to reproduce the original , which used RMSProp, with SGD with momentum. The same hyper-parameters are used to train sparse models. This change allows us to match or exceed the reported accuracies with only 45k iterations of training. The hyper-parameters used to train EfficientNet are largely unmodified from their code release, with the exception of extending training from 350 to 550 epochs and increasing the learning rate decay exponent to.985 from.97 so that the learning rate decays more slowly. These changes do not improve the dense baseline. We induce sparsity in MBv1 and MBv2 by starting the sparsification process at iteration 7,000 and stopping at 28,000 with a pruning frequency of 2,000. For EfficientNet we start at iteration 23,000 and end at iteration 105,000, also with a pruning frequency of 2,000. We train on the ImageNet dataset with standard data augmentation. Top-1 accuracies are reported on the validation set with center single-crops. To understand the effect of block size, we plot in figure 6 accuracy against flops for different block sizes. In these plots, every sparse tensor in the network uses the same output channel block size. The tradeoff for block sparsity only appears to involve how many elements are in each block, and not their configuration. For example, in MBv1, the 1 × 4, 4 × 1 and 2 × 2 curves all lie on top of one another. The loss in accuracy due to blocking seems to decrease slightly for larger width models. Figure 8: The x-axis corresponds to turning that layer and all following layers to block size 4, the prior layers are unstructured. The y-axis is the efficiency of making this change over an unstructured model given as a ratio where the numerator is the speedup of changing the block(s) from unstructured to block size 4 and the denominator is the decrease in top-1 accuracy that occurs by making this change. To understand how the sparsity level affects the efficiency of the models, we train models at 70%, 80% and 90% unstructured sparsity which is constant throughout the model. The are plotted in figure 7. MBv1 and MBv2 are more efficient the more sparse they become, confirming that the of hold not just for RNNs, but also for convolutional models as well. In figure 1 we plot Top-1 accuracy vs. FLOPs for all three generations of sparse and dense models. MobileNet v1 is 90% sparse, the other models are 80% sparse. A sparse MBv1 exceeds MBv2 in terms of FLOP and parameter efficiency; a sparse MBv2 matches EfficientNet in terms of FLOP and parameter efficiency; and a sparse EfficientNet exceeds all other models in both categories. To design the models with the best top-1 accuracy vs. inference time frontiers we make the following assumptions to reduce the search space: 1. We leave the models themselves unchanged. 2. We consider only block size 1 and block size 4 variants. 3. We induce the same level of sparsity in all 1 × 1 convolutions. Then we do a search at width multiplier 1.4 over N models when there are N residual blocks in a model. An x-axis location of n corresponds to a model in which the first n residual blocks are unstructured and the last N − n residual blocks have an output channel block size of 4. We train each model, note its top-1 accuracy and then measure its inference time. From this we can calculate the ratio of inference time reduction relative to a fully unstructured model and top-1 lost, which are plotted in figure 9. We choose the model with the highest ratio and train models at all widths with this choice. This amounts to making layers 6 and deeper block size 4 in MBv1 models and layers 11 and deeper block size 4 in MBv2. A full ) will likely lead to even more efficient models, but we leave this to future work. Table 1 contains the timings for running our sparse models on a single big core of two different processors, a Snapdragon 835 and a Snapdragon 670. We compare them with MBv1 and MBv2 models from their official repositories (a; b) run on the dense-inference TF Lite framework with the standard Ruy backend. We demonstrate that for a constant computational budget, sparse convolutional networks are more accurate than dense ones; this corroborates the findings of , which demonstrated that for a set number of floating-point operations, sparse RNNs are more accurate than dense RNNs. We enable the use of weight sparsity to accelerate state-of-the-art convolutional networks by providing fast SpMM kernels along with all necessary supporting kernels for ARM processors. On Snapdragon 835 the sparse networks we present in this paper outperform their dense equivalents by 1.1 − 2.2× -equivalent to approximately one entire generation of improvement. By overturning the misconception that "sparsity is slow", we hope to open new avenues of research that would previously not be considered. We present the code for the 4x1 kernel here for reference. ARM intrinsics have been renamed for clarity and casts have been removed for brevity. Table 2: Hyper-parameters for MBv1 and MBv2 training. Learning rates are specified in a reduced space and then multiplied by a factor of 16 due to the batch size. Here we present the plots for EfficientNet corresponding to those in the main text for scaling with sparsity and block size. The same trend for block size is observed -the configuration of the blocks isn't important, only the total size of the block. EfficientNet exhibits less improvement as sparsity increases.

represent this text to find a much much longer article about it with say hundreds-thousands of words
To give you a sense - "We show the possibility of pruning to find a small sub-network with significantly higher convergence rate than the full model." should be close to "Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate training steps. While this discovery is insightful, finding proper sub-networks requires iterative training and pruning. The high cost incurred limits the applications of the lottery ticket hypothesis. We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process and thus can mitigate the cost issue. We conduct extensive experiments to show such sub-networks consistently exist across various model structures for a restrictive setting of hyperparameters (e.g., carefully selected learning rate, pruning ratio, and model capacity). As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training, a standard approach to improve robustness, by up to 49% on CIFAR-10 to achieve the state-of-the-art robustness. Pruning has served as an important technique for removing redundant structure in neural networks (b; a; ;). Properly pruning can reduce cost in computation and storage without harming performance. However, pruning was until recently only used as a post-processing procedure, while pruning at initialization was believed ineffective (a;). Recently, proposed the lottery ticket hypothesis, showing that for a deep neural network there exist sub-networks, when trained from certain initialization obtained by pruning, performing equally or better than the original model with commensurate convergence rates. Such pairs of sub-networks and initialization are called winning tickets. This phenomenon indicates it is possible to perform pruning at initialization. However, finding winning tickets still requires iterative pruning and excessive training. Its high cost limits the application of winning tickets. Although shows that winning tickets converge faster than the corresponding full models, it is only observed on small networks, such as a convolutional neural network (CNN) with only a few convolution layers. In this paper, we show that for a variety of model architectures, there consistently exist such sub-networks that converge significantly faster when trained from certain initialization after pruning. We call these boosting tickets. We observe the standard technique introduced in for identifying winning tickets does not always find boosting tickets. In fact, the requirements are more restrictive. We extensively investigate underlining factors that affect such boosting effect, considering three stateof-the-art large model architectures: VGG-16 , ResNet-18 , and WideResNet . We conclude that the boosting effect depends principally on three factors: (i) learning rate, (ii) pruning ratio, and (iii) network capacity; we also demonstrate how these factors affect the boosting effect. By controlling these factors, after only one training epoch on CIFAR-10, we are able to obtain 90.88%/90.28% validation/test accuracy (regularly requires >30 training epochs) on WideResNet-34-10 when 80% parameters are pruned. We further show that the boosting tickets have a practical application in accelerating adversarial training, an effective but expensive defensive training method for obtaining robust models against adversarial examples. Adversarial examples are carefully perturbed inputs that are indistinguishable from natural inputs but can easily fool a classifier . We first show our observations on winning and boosting tickets extend to the adversarial training scheme. Furthermore, we observe that the boosting tickets pruned from a weakly robust model can be used to accelerate the adversarial training process for obtaining a strongly robust model. On CIFAR-10 trained with WideResNet-34-10, we manage to save up to 49% of the total training time (including both pruning and training) compared to the regular adversarial training process. Our contributions are summarized as follows: 1. We demonstrate that there exists boosting tickets, a special type of winning tickets that significantly accelerate the training process while still maintaining high accuracy. 2. We conduct extensive experiments to investigate the major factors affecting the performance of boosting tickets. 3. We demonstrate that winning tickets and boosting tickets exist for adversarial training scheme as well. 4. We show that pruning a non-robust model allows us to find winning/boosting tickets for a strongly robust model, which enables accelerated adversarial training process. 2 AND RELATED WORK Network pruning has been extensively studied as a method for compressing neural networks and reducing resource consumption (; ; ; b; ;). However, it was previously believed that pruned networks cannot be trained from the start (b;). Surprisingly, recent research has shown it is possible to prune a neural network at the initialization and still reach similar performance as the full model . Within this category, the lottery ticket hypothesis states a randomly-initialized dense neural network contains a sub-network that is initialized such that, when trained in isolation, learns as fast as the original network and matches its test accuracy. In, an iterative pruning method is proposed to find such sub-networks. Specifically, this approach first randomly initializes the model. The initialization is stored separately and the model is trained in the standard manner until convergence. Then a certain proportion of the weights with the smallest magnitudes are pruned while remaining weights are reset to the previously stored initialization and ready to be trained again. This train-prune-reset procedure is performed several times until the target pruning ratio is reached. Using this pruning method, they show the ing pruned networks can be trained to similar accuracy as the original full networks, which is better than the model with the same pruned structure but randomly initialized. One of the limitations of the lottery ticket hypothesis, as pointed in, is that winning tickets are found by unstructured pruning which does not necessarily yield faster training or executing time. In addition, finding winning tickets requires training the full model beforehand, which is time-consuming as well, especially considering iterative pruning. In this paper, we aim to show that there exists a subset of winning tickets, namely boosting tickets, that not only performs equally well as the original model but also converges much faster. Given a classifier f: X → {1, . . ., k} for an input x ∈ X, an adversarial example x adv is a perturbed version of x such that D(x, x adv) < for some small > 0, yet being mis-classified as f (x) = f (x adv). D(·, ·) is some distance metric which is often an p metric, and in most of the literature ∞ metric is considered, so as in this paper. The procedure of constructing such adversarial examples is often referred to as adversarial attacks. One of the simplest attacks is a single-step method, Fast Gradient Sign Method (FGSM) , manipulating inputs along the direction of the gradient with respect to the outputs:, where Π x+S is the projection operation that ensures adversarial examples stay in the p ball S around x. Although this method is fast, the attack is weak and can be defended easily. On the other hand, its multi-step variant, Projected Gra-dient Descend (PGD), is one of the strongest attacks :, where x is initialized with a random perturbation. Since PGD requires to access the gradients for multiple steps, it will incur high computational cost. On the defense side, currently the most successful defense approach is constructing adversarial examples via PGD during training and add them to the training sets as data augmentation, which is referred to as adversarial training . One caveat of adversarial training is its computational cost due to performing PGD attacks at each training step. Alternatively, using FGSM during training is much faster but the ing model is robust against FGSM attacks but vulnerable against PGD attacks . In this paper, we show it is possible to combine the advantages of both and quickly train a strongly robust model benefited from the boosting tickets. Prior studies have shown success in achieving both compactness and robustness of the trained networks (; ; ; ; ;). However, most of them will either incur much higher training cost or sacrifice robustness from the full model. On the contrary, our approach is able to reduce training time while obtaining similar/higher robust accuracy than the original full network. We first investigate boosting tickets on the standard setting without considering adversarial robustness. In this section, we show that with properly chosen hyperparameters, we are managed to find boosting tickets on VGG-16 and ResNet that can be trained much faster than the original dense network. Detailed model architectures and the setup can be found in Supplementary Section A. To find the boosting tickets, we use a similar algorithm for finding winning tickets, which is briefly described in the previous section and will be detailed here. First, a neural network is randomly initialized and saved in advance. Then the network is trained until convergence, and a given proportion of weights with the smallest magnitudes are pruned, ing in a mask where the pruned weights indicate 0 and remained weights indicate 1. We call this train-and-prune step pruning. This mask is then applied to the saved initialization to obtain a sub-network, which are the boosting tickets. All of the weights that are pruned (where zeros in the mask) will remain to be 0 during the whole training process. Finally, we can retrain the sub-networks. The key differences between our algorithm and the one proposed in to find winning tickets are (i) we use a small learning rate for pruning and retrain the sub-network (tickets) with learning rate warm-up from this small learning rate. In particular, for VGG-16 we choose 0.01 for pruning and warmup from 0.01 to 0.1 for retraining; for ResNet-18 we choose 0.05 for pruning and warmup from 0.05 to 0.1 for retraining; (ii) we find it is sufficient to prune and retrain the model only once instead of iterative pruning for multiple times. In Supplementary Section B, we show the difference of boosting effects brought from the tickets found by iterative pruning and one-shot pruning is negligible. Note warmup is also used in. However, they propose to use warmup from small learning rate to a large one during pruning as well, which hinders the boosting effect as shown in the following experiments. First, we show the existence of boosting tickets for VGG-16 and ResNet-18 on CIFAR-10 in Figure 1 and compare to the winning tickets. In particular, we show the boosting tickets are winning tickets, in the sense that they outperform the randomly initialized models. When compared to the winning tickets, boosting tickets demonstrate equally good performance with a higher convergence rate. Similar on MNIST can be found in Supplementary Section C. To measure the overall convergence rate, early stopping seems to be a good fit in the literature. It is commonly used to prevent overfitting and the final number of steps are used to measure convergence rates. However, early stopping is not compatible with learning rate scheduling we used in our case where the total number of steps is determined before training. This causes two issues in our evaluation in Figure 1: (i) Although the boosting tickets reach a relatively high validation accuracy much earlier than the winning ticket, the training procedure is and d) for winning tickets, boosting tickets, and randomly initialized weights. In both models, the boosting tickets show faster convergence rate and equally good performance as the winning tickets. then hindered by the large learning rate. After the learning rate drops, the performance gap between boosting tickets and winning tickets becomes negligible. As a , the learning rate scheduling obscures the improvement on convergence rates of boosting tickets; (ii) Due to fast convergence, boosting tickets tend to overfit, as observed in ResNet-18 after 50 epochs. To mitigate these two issues without excluding learning rate scheduling, we conduct another experiment where we mimic the early stopping procedure by gradually increasing the total number of epochs from 20 to 100. The learning rate is still dropped at the 50% and 75% stage. In this way, we can better understand the speed of convergence without worrying about overfitting even with learning rate scheduling involved. In figure 2, we compare the boosting tickets and winning tickets in this manner on VGG-16. While the first two plots in Figure 2 show the general trend of convergence, the improvement of convergence rates is much clearer in the last four plots. In particular, the validation accuracy of boosting tickets after 40 epochs is already on pair with the one trained for 100 epochs. Meanwhile, the winning tickets fall much behind the boosting tickets until 100 epochs where two finally match. We further investigate the test accuracy at the end of training for boosting and winning tickets in Table 1. We find the test accuracy of winning tickets gradually increase as we allow for more training steps, while the boosting tickets achieve the highest test accuracy after 60 epochs and start to overfit at 100 epochs. Summarizing the observations above, we confirm the existence of boosting tickets and state the boosting ticket hypothesis: A randomly initialized dense neural network contains a sub-network that is initialized such that, when trained in isolation, converges faster than the original network and other winning tickets while matches their performance. In the following sections, we investigate three major components that affect the boosting effects. As finding boosting tickets requires alternating learning rates, it is natural to assume the performance of boosting tickets relies on the choice of learning rate. Thus, we extensively investigate the influence of various learning rates. We use similar experimental settings in the previous section, where we increase the total number of epochs gradually and use the test accuracy as a measure of convergence rates. We choose four different learning rates 0.005, 0.01, 0.05 and 0.1 for pruning to get the tickets. All of the tickets found by those learning rates obtain the accuracy improvement over randomly reinitialized sub-model and thus satisfy the definition of winning tickets (i.e., they are all winning tickets). As shown in the first two plots of Figure 3, tickets found by smaller learning rates tend to have stronger boosting effects. For both VGG-16 and ResNet-18, the models trained with learning rate 0.1 show the least boosting effects, measured by the test accuracy after 20 epochs of training. On the other hand, training with too small learning rate will compromise the eventual test accuracy at a certain extent. Therefore, we treat the tickets found by learning rate 0.01 as our boosting tickets for VGG-16, and the one found by learning rate 0.05 as for ResNet-18, which converge much faster than all of the rest while achieving the highest final test accuracy. Pruning ratio has been an important component for winning tickets, and thus we investigate its effect on boosting tickets. Since we are only interested in the boosting effect, we use the validation accuracy at early stages as a measure of the strength of boosting to avoid drawing too many lines in the plots. In Figure 4, we show the validation accuracy after the first and fifth epochs of models for different pruning ratios for VGG-16 and ResNet-18. For both VGG-16 and ResNet-18, boosting tickets always reach much higher accuracy than randomly reinitialized sub-models, demonstrating their boosting effects. When the pruning ratio falls into the range from 60% to 90%, boosting tickets can provide the strongest boosting effects which obtain around 80% and 83% validation accuracy after the first and the fifth training epochs for VGG-16 and obtain 76% and 85% validation accuracy for ResNet-18. On the other hand, the increase of validation accuracy between the first training epoch and the fifth training epoch become smaller when boosting effects appear. It indicates their convergence starts to saturate due to the large learning rate at the initial stage and is ready for dropping the learning rate. We finally investigate how model capacity, including the depth and width of models, affects the boosting tickets. We use WideResNet either with its depth or width fixed and vary the other factor. In particular, we keep the depth as 34 and increases the width from 1 to 10, comparing their boosting effect. Then we keep the width as 10 and increase the depth from 10 to 34. The changes of validation accuracy of the models are shown in Figure 5. Overall, Figure 5 shows models with larger capacity have a more significant boosting effect, though the boosting effects keep the same when the depth is larger than 22. Notably, we find the largest model WideResNet-34-10 achieves 90.88% validation accuracy after only one training epoch. Although the lottery ticket hypothesis is extensively studied in and, the same phenomenon in adversarial training setting lacks thorough understanding. In this section, we show two important facts that make boosting tickets suitable for the adversarial scheme: the lottery ticket hypothesis and boosting ticket hypothesis are applicable to the adversarial training scheme; pruning on a weakly robust model allows to find the boosting ticket for a strongly robust model and save training cost. In the following experiment, we use a naturally trained model, that is trained in the standard manner, and two adversarially trained models using FGSM and PGD respectively to obtain the tickets by pruning these models. Then we retrain these pruned models with the same PGD-based adversarial training from the same initialization. In Figure 6, we report the corresponding accuracy on the original validation sets and on the adversarially perturbed validation examples, noted as clean accuracy and robust accuracy. We further train the pruned model from random reinitialization to validate lottery ticket hypothesis. Unless otherwise stated, in all the PGD-based adversarial training, we keep the same setting as. The PGD attacks are performed in 10 steps with step size 2/255 (PGD-10). The PGD attacks are bounded by 8/255 in its ∞ norm. For the FGSM-based adversarial training, the FGSM attacks are bounded by 8/255. Both models trained from the boosting tickets obtained with FGSM-and PGD-based adversarial training demonstrate superior performance and faster convergence than the model trained from random reinitialization. This confirms the lottery ticket hypothesis and boosting ticket hypothesis are applicable to adversarial training scheme on both clean accuracy and robust accuracy. More interestingly, the performance of the models pruned with FGSM-and PGD-based adversarial training are almost the same. This observation suggests it is sufficient to train a weakly robust model with FGSM-based adversarial training for obtaining the boosting tickets and retrain it with stronger attacks such as PGD. This finding is interesting because FGSM-based adversarial trained models will suffer from label leaking problems as learning weak robustness. In fact, the FGSM-based adversarially trained model from which we obtain our boosting tickets has 89% robust accuracy against FGSM but with only 0.4% robust accuracy against PGD performed in 20 steps. However, Figure 6 shows the following PGD-based adversarial retraining on the boosting tickets obtained from that FGSM-based trained model is indeed robust. Further discussions can be found in Section 5. , the authors argued that the lottery ticket hypothesis fails to hold in adversarial training via experiments on MNIST. We show they fail to observe winning tickets because the models they used have limited capacity. In the adversarial setting bounded by L ∞ ≤ 0.3, small models such as a CNN with two convolutional layers used in can not yield even winning tickets when pruning ratio is large. In Figure 7, plot (a) and (b) are the clean and robust accuracy of the pruned models when the pruning ratio is 80%. The pruned model degrades into a trivial classifier where all example are classified into the same class with 11.42%/11.42% valid/test accuracy. However, when we use VGG-16, as shown in plot (c) and (d), the winning tickets are found again. This can be explained as adversarial training requires much larger model capacity than standard training , thus pruning small models could undermine their performance. Since MNIST is a simple dataset, adversarial training converges quickly at the first few epochs for both the tickets and randomly initialized models. Therefore, there is no winning tickets performing obvious boosting effect which we can identify as a boosting ticket on MNIST. We then conduct the same experiment as in Figure 2 but in the adversarial training setting to better show the improved convergence rates. The for validation accuracy and test accuracy are presented in Figure 8 and Table 2 respectively. It suggests it is sufficient to train 60 epochs to achieve similar robust accuracy as the full model trained for 100 epochs. Until now, we have confirmed that boosting tickets exist consistently across different models and training schemes and convey important insights on the behavior of pruned models. However, in the natural training setting, although boosting tickets provide faster convergence, it is not suitable for accelerating the standard training procedure as pruning to find the boosting tickets requires training full models beforehand. On the other hand, the two observations mentioned in Section 4 enable boosting tickets to accelerate adversarial training. In particular, we can find boosting tickets with FGSM-based adversarial training that and they can significantly accelerate the PGD-based adversarial training. Note that the cost of FGSM-based training is only 1/10 times of the standard 10-step PGD-based one and thus is almost negligible compared to the time saved due to the boosting effect. In Table 3, we apply adversarial training to WideResNet-34-10, which has the same structure used in , with the proposed approach for 40, 70 and 100 epochs and report the best accuracy/robust accuracy under various attacks among the whole training process. In particular, we perform 20-step PGD, 100-step PGD as white-box attacks where the attackers have the access to the model parameters. More experimental are included in the Appendix. We report the time consumption for training each model to measure how much time is saved by boosting tickets. We run all the experiments on a workstation with 2 V100 GPUs in parallel. From Table 3 we observe that while our approach requires pruning before training, it is overall faster as it uses FGSM-based adversarial training. In particular, to achieve its best robust accuracy, original Madry et al.'s training method requires 134,764 seconds on WideResNet-34-10. To achieve that, our boosting ticket only requires 69,552 seconds, including 15,462 seconds to find the boosting ticket and 54,090 seconds to retrain the ticket, saving 49% of the total training time. Not knowledge distillation. It may seem that winning tickets and boosting tickets behave like knowledge distillation where the learned knowledge from a large model is transferred to a small model. This conjecture may explain the boosting effects as the pruned model quickly recover the knowledge from the full model. However, the lottery ticket framework seems to be distinctive to knowledge distillation. If boosting tickets simply transfer knowledge from the full model to the pruned model, then an FGSM-based adversarially trained model should not find tickets that improves the robustness of the sub-model against PGD attacks, as the full model itself is vulnerable to PGD attacks. Yet in Section 4.1 we observe an FGSM-based adversarially trained model still leads to boosting tickets that accelerates PGD-based adversarial training. We believe the cause of boosting tickets requires further investigation in the future. Accelerate adversarial training. propose to reduce the training time for PGD-based adversarial training by recycling the gradients computed for parameter updates and constructing adversarial examples. While their approach focuses on reducing the computational time for each epoch, our method focuses more on the convergence rate (i.e., reduce the number of epochs required for convergence). Therefore, our approach is compatible with theirs, making it a promising future direction to combine both to further reduce the training time. In this paper, we investigate boosting tickets, sub-networks coupled with certain initialization that can be trained with significantly faster convergence rate. As a practical application, in the adversarial training scheme, we show pruning a weakly robust model allows to find boosting tickets that can save up to 49% of the total training time to obtain a strongly robust model that matches the state-ofthe-art robustness. Finally, it is an interesting direction to investigate whether there is a way to find boosting tickets without training the full model beforehand, as it is technically not necessary. Setup. We use , weight decay, decreasing learning rate schedules (×0.1 at 50% and 75%), and augmented training data for training models. We try to keep the setting the same as the one used in except we use one-shot pruning instead of iterative pruning. It allows the whole pruning and training process to be more practical in real applications. On CIFAR-10 dataset, we randomly select 5,000 images out of 50,000 training set as validation set and train the models with the rest. The reported test accuracy is measured with the whole testing set. All of our experiments are run on four Tesla V100s, 10 Tesla P100s, and 10 2080 Tis. For all the time-sensitive experiments like adversarial training on WideResNet-34-10 in Section 4.3, we train each model on two Tesla V100s with data parallelism. For the rest ones measuring the final test accuracy, we use one gpu for each model without parallelism. In Table 4, we summarize the number of parameters and parameter sizes of all the model architectures that we evaluate with including VGG-16 , ResNet-18 , and the variance of. Figure 9, we track the training of models obtained from both iterative pruning and one shot pruning. We find the performance of both, in terms of the boosting effects and final accuracy, is indistinguishable. and. We plot the validation accuracy of models from both approaches and the corresponding randomly initialized models. In this section, we report experiment on MNIST for the standard setting, where we use LeNet with two convolutions and two fully connected layers for the classification task. As for MNIST we do not use learning rate scheduling, early stopping is then used to determine the speed of convergence. In 5, we report the epochs when early stopping happens and the test accuracy to illustrate the existence of boosting tickets for MNIST. While winning tickets converge at the 18th epoch, boosting tickets converge at the 11th epoch, indicating faster convergence.. In addition to the experimental reported in the main text, in Table 6 we include for C&W attacks and transfer attacks; where we attack one model with adversarial examples found by 20-step PGD based on other models. We find the adversarial examples generated from one model can transfer to another model with a slight decrease on the robust error. This indicates our models and Madry's model share adversarial examples and further share decision boundaries. Table 6: Best test clean accuracy (the first row), robust accuracy (the second to fourth rows), transfer attack accuracy (the middle four rows), and training time for PGD-based adversarial training (the last four rows) on boosting tickets obtained by FGSM-based adversarial training in various of numbers of epochs on WideResNet-34-10. Overall, our adversarial training strategy based on boosting tickets is able to save up to 49% of the total training time while achieving higher robust accuracy compared to the regular adversarial training on the original full model."

Sparse MobileNets are faster than Dense ones with the appropriate kernels.

Represent the TLDR for finding the corresponding scientific article:

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: represent the article for finding a single-sentence tldr

Question: In this work, we address the semi-supervised classification of graph data, where the categories of those unlabeled nodes are inferred from labeled nodes as well as graph structures. Recent works often solve this problem with the advanced graph convolution in a conventional supervised manner, but the performance could be heavily affected when labeled data is scarce. Here we propose a Graph Inference Learning (GIL) framework to boost the performance of node classification by learning the inference of node labels on graph topology. To bridge the connection of two nodes, we formally define a structure relation by encapsulating node attributes, between-node paths and local topological structures together, which can make inference conveniently deduced from one node to another node. For learning the inference process, we further introduce meta-optimization on structure relations from training nodes to validation nodes, such that the learnt graph inference capability can be better self-adapted into test nodes. Comprehensive evaluations on four benchmark datasets (including Cora, Citeseer, Pubmed and NELL) demonstrate the superiority of our GIL when compared with other state-of-the-art methods in the semi-supervised node classification task. Graph, which comprises a set of vertices/nodes together with connected edges, is a formal structural representation of non-regular data. Due to the strong representation ability, it accommodates many potential applications, e.g., social network , world wide data , knowledge graph , and protein-interaction network . Among these, semi-supervised node classification on graphs is one of the most interesting also popular topics. Given a graph in which some nodes are labeled, the aim of semi-supervised classification is to infer the categories of those remaining unlabeled nodes by using various priors of the graph. While there have been numerous previous works (; ; ; ;) devoted to semi-supervised node classification based on explicit graph Laplacian regularizations, it is hard to efficiently boost the performance of label prediction due to the strict assumption that connected nodes are likely to share the same label information. With the progress of deep learning on grid-shaped images/videos , a few of graph convolutional neural networks (CNN) based methods, including spectral and spatial methods (; ;), have been proposed to learn local convolution filters on graphs in order to extract more discriminative node representations. Although graph CNN based methods have achieved considerable capabilities of graph embedding by optimizing filters, they are limited into a conventionally semi-supervised framework and lack of an efficient inference mechanism on graphs. Especially, in the case of few-shot learning, where a small number of training nodes are labeled, this kind of methods would drastically compromise the performance. For example, the Pubmed graph dataset consists (b) The process of Graph inference learning. We extract the local representation from the local subgraph (the circle with dashed line The red wave line denote the node reachability from to . d t th h bilit f d t th d Figure 1: The illustration of our proposed GIL framework. For the problem of graph node labeling, the category information of these unlabeled nodes depends on the similarity computation between a query node (e.g., vj) and these labeled reference nodes (e.g., vi). We consider the similarity from three points: node attributes, the consistency of local topological structures (i.e., the circle with dashed line), and the between-node path reachability (i.e., the red wave line from vi to vj). Specifically, the local structures as well as node attributes are encoded as high-level features with graph convolution, while the between-node path reachability is abstracted as reachable probabilities of random walks. To better make the inference generalize to test nodes, we introduce a meta-learning strategy to optimize the structure relations learning from training nodes to validation nodes. of 19,717 nodes and 44,338 edges, but only 0.3% nodes are labeled for the semi-supervised node classification task. These aforementioned works usually boil down to a general classification task, where the model is learnt on a training set and selected by checking a validation set. However, they do not put great efforts on how to learn to infer from one node to another node on a topological graph, especially in the few-shot regime. In this paper, we propose a graph inference learning (GIL) framework to teach the model itself to adaptively infer from reference labeled nodes to those query unlabeled nodes, and finally boost the performance of semi-supervised node classification in the case of a few number of labeled samples. Given an input graph, GIL attempts to infer the unlabeled nodes from those observed nodes by building between-node relations. The between-node relations are structured as the integration of node attributes, connection paths, and graph topological structures. It means that the similarity between two nodes is decided from three aspects: the consistency of node attributes, the consistency of local topological structures, and the between-node path reachability, as shown in Fig. 1. The local structures anchored around each node as well as the attributes of nodes therein are jointly encoded with graph convolution for the sake of high-level feature extraction. For the between-node path reachability, we adopt the random walk algorithm to obtain the characteristics from a labeled reference node v i to a query unlabeled node v j in a given graph. Based on the computed node representation and between-node reachability, the structure relations can be obtained by computing the similar scores/relationships from reference nodes to unlabeled nodes in a graph. Inspired by the recent meta-learning strategy , we learn to infer the structure relations from a training set to a validation set, which can benefit the generalization capability of the learned model. In other words, our proposed GIL attempts to learn some transferable knowledge underlying in the structure relations from training samples to validation samples, such that the learned structure relations can be better self-adapted to the new testing stage. We summarize the main contributions of this work as three folds: • We propose a novel graph inference learning framework by building structure relations to infer unknown node labels from those labeled nodes in an end-to-end way. The structure relations are well defined by jointly considering node attributes, between-node paths, and graph topological structures. • To make the inference model better generalize to test nodes, we introduce a meta-learning procedure to optimize structure relations, which could be the first time for graph node classification to the best of our knowledge. • Comprehensive evaluations on three citation network datasets (including Cora, Citeseer, and Pubmed) and one knowledge graph data (i.e., NELL) demonstrate the superiority of our proposed GIL in contrast with other state-of-the-art methods on the semi-supervised classification task. Graph CNNs: With the rapid development of deep learning methods, various graph convolution neural networks (; ; ; ; ;) have been exploited to analyze the irregular graph-structured data. For better extending general convolutional neural networks to graph domains, two broad strategies have been proposed, including spectral and spatial convolution methods. Specifically, spectral filtering methods develop convolution-like operators in the spectral domain, and then perform a series of spectral filters by decomposing the graph Laplacian. Unfortunately, the spectral-based approaches often lead to a high computational complex due to the operation of eigenvalue decomposition, especially for a large number of graph nodes. To alleviate this computation burden, local spectral filtering methods are then proposed by parameterizing the frequency responses as a Chebyshev polynomial approximation. Another type of graph CNNs, namely spatial methods , can perform the filtering operation by defining the spatial structures of adjacent vertices. Various approaches can be employed to aggregate or sort neighboring vertices, such as diffusion CNNs , GraphSAGE , PSCN , and NgramCNN . From the perspective of data distribution, recently, the Gaussian induced convolution model is proposed to disentangle the aggregation process through encoding adjacent regions with Gaussian mixture model. Semi-supervised node classification: Among various graph-related applications, semi-supervised node classification has gained increasing attention recently, and various approaches have been proposed to deal with this problem, including explicit graph Laplacian regularization and graphembedding approaches. Several classic algorithms with graph Laplacian regularization contain the label propagation method using Gaussian random fields , the regularization framework by relying on the local/global consistency , and the random walkbased sampling algorithm for acquiring the context information . To further address scalable semi-supervised learning issues , the Anchor Graph regularization approach is proposed to scale linearly with the number of graph nodes and then applied to massive-scale graph datasets. Several graph convolution network methods (; ; ; ;) are then developed to obtain discriminative representations of input graphs. For example, Kipf et al. proposed a scalable graph CNN model, which can scale linearly in the number of graph edges and learn graph representations by encoding both local graph structures and node attributes. Graph attention networks (GAT) are proposed to compute hidden representations of each node for attending to its neighbors with a self-attention strategy. By jointly considering the local-and global-consistency information, dual graph convolutional networks are presented to deal with semi-supervised node classification. The critical difference between our proposed GIL and those previous semi-supervised node classification methods is to adopt a graph inference strategy by defining structure relations on graphs and then leverage a meta optimization mechanism to learn an inference model, which could be the first time to the best of our knowledge, while the existing graph CNNs take semi-supervised node classification as a general classification task. Formally, we denote an undirected/directed graph as G = {V, E, X, Y}, where is the finite set of n (or |V|) vertices, E ∈ R n×n defines the adjacency relationships (i.e., edges) between vertices representing the topology of G, X ∈ R n×d records the explicit/implicit attributes/signals of vertices, and Y ∈ R n is the vertex labels of C classes. The edge E ij = E(v i, v j) = 0 if and only if vertices v i, v j are not connected, otherwise E ij = 0. The attribute matrix X is attached to the vertex set V, whose i-th row X vi (or X i·) represents the attribute of the i-th vertex v i. It means that v i ∈ V carries a vector of d-dimensional signals. Associated with each node v i ∈ V, there is a discrete label y i ∈ {1, 2, · · ·, C}. We consider the task of semi-supervised node classification over graph data, where only a small number of vertices are labeled for the model learning, i.e., |V Label | |V|. Generally, we have three node sets: a training set V tr, a validation set V val, and a testing set V te. In the standard protocol of prior literatures , the three node sets share the same label space. We follow but do not restrict this protocol for our proposed method. Given the training and validation node sets, the aim is to predict the node labels of testing nodes by using node attributes as well as edge connections. A sophisticated machine learning technique used in most existing methods is to choose the optimal classifier (trained on a training set) after checking the performance on the validation set. However, these methods essentially ignore how to extract transferable knowledge from these known labeled nodes to unlabeled nodes, as the graph structure itself implies node connectivity/reachability. Moreover, due to the scarcity of labeled samples, the performance of such a classifier is usually not satisfying. To address these issues, we introduce a meta-learning mechanism (; ;) to learn to infer node labels on graphs. Specifically, the graph structure, between-node path reachability, and node attributes are jointly modeled into the learning process. Our aim is to learn to infer from labeled nodes to unlabeled nodes, so that the learner can perform better on a validation set and thus classify a testing set more accurately. 3.2 STRUCTURE RELATION For convenient inference, we specifically build a structure relation between two nodes on the topology graph. The labeled vertices (in a training set) are viewed as the reference nodes, and their information can be propagated into those unlabeled vertices for improving the label prediction accuracy. Formally, given a reference node v i ∈ V Label, we define the score of a query node v j similar to v i as where G vi and G vj may be understood as the centralized subgraphs around v i and v j, respectively. f e, f r, f P are three abstract functions that we explain as follows: dv, encodes the local representation of the centralized subgraph G vi around node v i, and may thus be understood as a local filter function on graphs. This function should not only take the signals of nodes therein as input, but also consider the local topological structure of the subgraph for more accurate similarity computation. To this end, we perform the spectral graph convolution on subgraphs to learn discriminative node features, analogous to the pixel-level feature extraction from convolution maps of gridded images. The details of feature extraction f e are described in Section 4. • Path reachability f P (v i, v j, E) −→ R dp, represents the characteristics of path reachability from v i to v j. As there usually exist multiple traversal paths between two nodes, we choose the function as reachable probabilities of different lengths of walks from v i to v j. More details will be introduced in Section 4. • Structure relation f r (R dv, R dv, R dp) −→ R, is a relational function computing the score of v j similar to v i. This function is not exchangeable for different orders of two nodes, due to the asymmetric reachable relationship f P. If necessary, we may easily revise it as a symmetry function, e.g., summarizing two traversal directions. The score function depends on triple inputs: the local representations extracted from the subgraphs w.r.t. f e (G vi) and f e (G vj), respectively, and the path reachability from v i to v j. In semi-supervised node classification, we take the training node set V tr as the reference samples, and the validation set V val as the query samples during the training stage. Given a query node v j ∈ V val, we can derive the class similarity score of v j w.r.t. the c-th (c = 1, · · ·, C) category by weighting the reference samples C c = {v k |y v k = c}. Formally, we can further revise Eqn. and define the class-to-node relationship function as where the function φ w maps a reachable vector f P (v i, v j, E) into a weight value, and the function φ r computes the similar score between v j and the c-th class nodes. The normalization factor F Cc→vj of the c-th category w.r.t. v j is defined as For the relation function φ r and the weight function φ w, we may choose some subnetworks to instantiate them in practice. The detailed implementation of our model can be found in Section 4. According to the class-to-node relationship function in Eqn., given a query node v j, we can obtain a score vector s C→j = [s C1→j, · · ·, s C C →j] ∈ R C after computing the relations to all classes. The indexed category with the maximum score is assumed to be the estimated label. Thus, we can define the loss function based on cross entropy as follows: where y j,c is a binary indicator (i.e., 0 or 1) of class label c for node v j, and the softmax operation is imposed on s Cc→j, i.e.,ŷ Cc→j = exp(s Cc→j)/ C k=1 exp(s C k →j). Other error functions may be chosen as the loss function, e.g., mean square error. In the regime of general classification, the cross entropy loss is a standard one that performs well. Given a training set V tr, we expect that the best performance can be obtained on the validation set V val after optimizing the model on V tr. Given a trained/pretrained model Θ = {f e, φ w, φ r}, we perform iteratively gradient updates on the training set V tr to obtain the new model, formally, where α is the updating rate. Note that, in the computation of class scores, since the reference node and query node can be both from the training set V tr, we set the computation weight w i→j = 0 if i = j in Eqn.. After several iterates of gradient descent on V tr, we expect a better performance on the validation set V val, i.e., min. Thus, we can perform the gradient update as follows where β is the learning rate of meta optimization . During the training process, we may perform batch sampling from training nodes and validation nodes, instead of taking all one time. In the testing stage, we may take all training nodes and perform the model update according to Eqn. like the training process. The updated model is used as the final model and is then fed into Eqn. to infer the class labels for those query nodes. In this section, we instantiate all modules (i.e., functions) of the aforementioned structure relation. The implementation details can be found in the following. Node Representation f e (G vi): The local representation at vertex v i can be extracted by performing the graph convolution operation on subgraph G vi. Similar to gridded images/videos, on which local convolution kernels are defined as multiple lattices with various receptive fields, the spectral graph convolution is used to encode the local representations of an input graph in our work. Given a graph sample G = {V, E, X}, the normalized graph Laplacian matrix is L = I n − D −1/2 ED −1/2 = UΛU T, with a diagonal matrix of its eigenvalues Λ. The spectral graph convolution can be defined as the multiplication of signal X with a filter g θ (Λ) = diag(θ) parameterized by θ in the Fourier domain: conv(X) = g θ (L) * X = Ug θ (Λ)U T X, where parameter θ ∈ R n is a vector of Fourier coefficients. To reduce the computational complexity and obtain the local information, we use an approximate local filter of the Chebyshev polynomial ,, where parameter θ ∈ R K is a vector of Chebyshev coefficients and T k (Λ) ∈ R n×n is the Chebyshev polynomial of order k evaluated atΛ = 2Λ/λ max − I n, a diagonal matrix of scaled eigenvalues. The graph filtering operation can then be expressed as n×n is the Chebyshev polynomial of order k evaluated at the scaled LaplacianL = 2L/λ max − I n. Further, we can construct multi-scale receptive fields for each vertex based on the Laplacian matrix L, where each receptive field records hopping neighborhood relationships around the reference vertex v i, and forms a local centralized subgraph. Path Reachability f P (v i, v j, E): Here we compute the probabilities of paths from vertex i to vertex j by employing random walks on graphs, which refers to traversing the graph from v i to v j according to the probability matrix P. For the input graph G with n vertices, the random-walk transition matrix can be defined as P = D −1 E, where D ∈ R n×n is the diagonal degree matrix with D ii = i E ij. That is to say, each element P ij is the probability of going from vertex i to vertex j in one step. The sequence of nodes from vertex i to vertex j is a random walk on the graph, which can be modeled as a classical Markov chain by considering the set of graph vertices. To represent this formulation, we show that P t ij is the probability of getting from vertex v i to vertex v j in t steps. This fact is easily exhibited by considering a t-step path from vertex v i to vertex v j as first taking a single step to some vertex h, and then taking t − 1 steps to v j. The transition probability P t in t steps can be formulated as where each matrix entry P t ij denotes the probability of starting at vertex i and ending at vertex j in t steps. Finally, the node reachability from v i to v j can be written as a d p -dimensional vector: where d p refers to the step length of the longest path from v i to v j. Class-to-Node Relationship s Cc→j: To define the node relationship s i→j from v i to v j, we simultaneously consider the property of path reachability f P (v i, v j, E), local representations f e (G vi), and, which is to map the reachable vector f P (v i, v j, E) ∈ R dp into a weight value, can be implemented with two 16-dimensional fully connected layers in our experiments. The computed value w i→j can be further used to weight the local features at node v i, f e (G vi) ∈ R dv. For obtaining the similar score between v j and the c-th class nodes C c in Eqn., we perform a concatenation of two input features, where one refers to the weighted features of vertex v i, and another is the local features of vertex v j. One fully connected layer (w.r.t. φ r) with C-dimensions is finally adopted to obtain the relation regression score. We evaluate our proposed GIL method on three citation network datasets: Cora, Citeseer, Pubmed , and one knowledge graph NELL dataset . The statistical properties of graph data are summarized in Table 1. Following the previous protocol in , we split the graph data into a training set, a validation set, and a testing set. Taking into account the graph convolution and pooling modules, we may alternately stack them into a multi-layer Graph convolutional network. The GIL model consists of two graph convolution layers, each of which is followed by a mean-pooling layer, a class-to-node relationship regression module, and a final softmax layer. We have given the detailed configuration of the relationship regression module in the class-to-node relationship of Section 4. The parameter d p in Eqn. is set to the mean length of between-node reachability paths in the input graph. The channels of the 1-st and 2-nd convolutional layers are set to 128 and 256, respectively. The scale of the respective filed is 2 in both convolutional layers. The dropout rate is set to 0.5 in the convolution and fully connected layers to avoid over-fitting, and the ReLU unit is leveraged as a nonlinear activation function. We pre-train our proposed GIL model for 200 iterations with the training set, where its initial learning rate, decay factor, and momentum are set to 0.05, 0.95, and 0.9, respectively. Here we train the GIL model using the stochastic gradient descent method with the batch size of 100. We further improve the inference learning capability of the GIL model for 1200 iterations with the validation set, where the meta-learning rates α and β are both set to 0.001. We compare the GIL approach with several state-of-the-art methods (; ; ;) over four graph datasets, including Cora, Citeseer, Pubmed, and NELL. The classification accuracies for all methods are reported in Table 2. Our proposed GIL can significantly outperform these graph Laplacian regularized methods on four graph datasets, including Deep walk , modularity clustering , Gaussian fields , and graph embedding methods. For example, we can achieve much higher performance than the deepwalk method , e.g., 43.2% vs 74.1% on the Citeseer dataset, 65.3% vs 83.1% on the Pubmed dataset, and 58.1% vs 78.9% on the NELL dataset. We find that the graph embedding method , which has considered both label information and graph structure during sampling, can obtain lower accuracies than our proposed GIL by 9.4% on the Citeseer dataset and 10.5% on the Cora dataset, respectively. This indicates that our proposed GIL can better optimize structure relations and thus improve the network generalization. We further compare our proposed GIL with several existing deep graph embedding methods, including graph attention network , dual graph convolutional networks , topology adaptive graph convolutional networks , Multi-scale graph convolution , etc. For example, our proposed GIL achieves a very large gain, e.g., 86.2% vs 83.3% on the Cora dataset, and 78.9% vs 66.0% on the NELL dataset. We evaluate our proposed GIL method on a large graph dataset with a lower label rate, which can significantly outperform existing baselines on the Pubmed dataset: 3.1% over DGCN , 4.1% over classic GCN and TAGCN , 3.2% over AGNN , and 3.6% over N-GCN . It demonstrates that our proposed GIL performs very well on various graph datasets by building the graph inference learning process, where the limited label information and graph structures can be well employed in the predicted framework. Table 2: Performance comparisons of semi-supervised classification methods. Cora Citeseer Pubmed NELL Clustering 59.5 60.1 70.7 21.8 DeepWalk 67.2 43.2 65.3 58.1 Gaussian 68.0 45.3 63.0 26.5 G-embedding 75.7 64.7 77.2 61.9 DCNN 76.8 -73.0 -GCN 81.5 70.3 79.0 66.0 MoNet 81.7 -78.8 -N-GCN 83.0 72.2 79.5 -GAT 83.0 72.5 79.0 -AGNN 83.1 71.7 79.9 -TAGCN 83.3 72.5 79.0 -DGCN 83 5.3 ANALYSIS Meta-optimization: As can be seen in Table 3, we report the classification accuracies of semi-supervised classification with several variants of our proposed GIL and the classical GCN method when evaluating them on the Cora dataset. For analyzing the performance improvement of our proposed GIL with the graph inference learning process, we report the classification accuracies of GCN and our proposed GIL on the Cora dataset under two different situations, including "only learning with the training set V tr " and "with jointly learning on a training set V tr and a validation set V val ". "GCN /w jointly learning on V tr & V val " achieves a better than "GCN /w learning on V tr " by 3.6%, which demonstrates that the network performance can be improved by employing validation samples. When using structure relations, "GIL /w learning on V tr " obtains an improvement of 1.9% (over "GCN /w learning on V tr "), which can be attributed to the building connection between nodes. The meta-optimization strategy ("GIL /w meta-training from V tr → V val " vs "GIL /w learning on V tr ") has a gain of 2.9%, which indicates that a good inference capability can be learnt through meta-optimization. It is worth noting that, GIL adopts a meta-optimization strategy to learn the inference model, which is a process of migrating from a training set to a validation set. In other words, the validation set is only used to teach the model itself how to transfer to unseen data. In contrast, the conventional methods often employ a validation set to tune parameters of a certain model of interest. Network settings: We explore the effectiveness of our proposed GIL with the same mean pooling mechanism, but with different numbers of convolutional layers, i.e., "GIL + mean pooling" with one, two, and three convolutional layers, respectively. As can be seen in Table 3, the proposed GIL with two convolutional layers can obtain a better performance on the Cora data than the other two network settings (i.e., GIL with one or three convolutional layers). For example, the performance of'GIL /w 1 conv. layer + mean pooling" is slightly decreased by 1.7% over "GIL /w 2 conv. layers + mean pooling" on the Cora dataset. Furthermore, we report the classification of our proposed GIL by using mean and max-pooling mechanisms, respectively. GIL with mean pooling (i.e., "GIL /w 2 conv layers + mean pooling") can get a better than the GIL method with max-pooling (i.e., "GIL /w 2 conv layers + max-pooling"), e.g., 86.2% vs 85.2% on the Cora graph dataset. The reason may be that the graph network with two convolutional layers and the mean pooling mechanism can obtain the optimal graph embeddings, but when increasing the network layers, more parameters of a certain graph model need to be optimized, which may lead to the over-fitting issue. Influence of different between-node steps: We compare the classification performance within different between-node steps for our proposed GIL and GCN , as illustrated in Fig. 2(a). The length of between-node steps can be computed with the shortest path between reference nodes and query nodes. When the step between nodes is smaller, both GIL and GCN methods can predict the category information for a small part of unlabeled nodes in the testing set. The reason may be that the node category information may be disturbed by its nearest neighboring nodes with different labels and fewer nodes are within 1 or 2 steps in the testing set. The GIL and GCN methods can infer the category information for a part of unlabeled nodes by adopting node attributes, when two nodes are not connected in the graph (i.e., step=∞). By increasing the length of reachability path, the inference process of the GIL method would become difficult and more graph structure information may be employed in the predicted process. GIL can outperform the classic GCN by analyzing the accuracies within different between-node steps, which indicates that our proposed GIL has a better reference capability than GCN by using the meta-optimization mechanism from training nodes to validation nodes. Influence of different label rates: We also explore the performance comparisons of the GIL method with different label rates, and the detailed on the Pubmed dataset can be shown in Fig. 2(b). When label rates increase by multiplication, the performances of GIL and GCN are improved, but the relative gain becomes narrow. The reason is that, the reachable path lengths between unlabeled nodes and labeled nodes will be reduced with the increase of labeled nodes, which will weaken the effect of inference learning. In the extreme case, labels of unlabeled nodes could be determined by those neighbors with the 1 ∼ 2 step reachability. In summary, our proposed GIL method prefers small ratio labeled nodes on the semi-supervised node classification task. Inference learning process: Classification errors of different epochs on the validation set of the Cora dataset can be illustrated in Fig. 3. Classification errors are rapidly decreasing as the number of iterations increases from the beginning to 400 iterations, while they are with a slow descent from 400 iterations to 1200 iterations. It demonstrates that the learned knowledge from the training samples can be transferred for inferring node category information from these reference labeled nodes. The performance of semi-supervised classification can be further increased by improving the generalized capability of the Graph CNN model. Module analysis: We evaluate the effectiveness of different modules within our proposed GIL framework, including node representation f e, path reachability f P, and structure relation f r. Note that the last one f r defines on the former two ones, so we consider the cases in Table 4 by adding modules. When not using all modules, only original attributes of nodes are used to predict labels. The case of only using f e belongs to the GCN method, which can achieve 81.5% on the Cora dataset. The large gain of using the relation module f r (i.e., from 81.5% to 85.0%) may be contributed to the ability of inference learning on attributes as well as local topology structures which are implicitly encoded in f e. The path information f P can further boost the performance by 1.2%, e.g., 86.2% vs 85.0%. It demonstrates that three different modules of our method can improve the graph inference learning capability. Computational complexity: For the computational complexity of our proposed GIL, the cost is mainly spent on the computations of node representation, between-node reachability, and class-tonode relationship, which are about O((n tr + n te) * e * d in * d out ), O((n tr + n te) * e * P ), and O(n tr * n te d 2 out), respectively. n tr and n te refer to the numbers of training and testing nodes, d in and d out denote the input and output dimensions of node representation, e is about the average degree of graph node, and P is the step length of node reachability. Compared with those classic Graph CNNs , our proposed GIL has a slightly higher cost due to an extra inference learning process, but can complete the testing stage with several seconds on these benchmark datasets. In this work, we tackled the semi-supervised node classification task with a graph inference learning method, which can better predict the categories of these unlabeled nodes in an end-to-end framework. We can build a structure relation for obtaining the connection between any two graph nodes, where node attributes, between-node paths, and graph structure information can be encapsulated together. For better capturing the transferable knowledge, our method further learns to transfer the mined knowledge from the training samples to the validation set, finally boosting the prediction accuracy of the labels of unlabeled nodes in the testing set. The extensive experimental demonstrate the effectiveness of our proposed GIL for solving the semi-supervised learning problem, even in the few-shot paradigm. In the future, we would extend the graph inference method to handle more graph-related tasks, such as graph generation and social network analysis.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a novel graph inference learning framework by building structure relations to infer unknown node labels from those labeled nodes in an end-to-end way.

represent the tldr for finding the corresponding scientific article

In this paper, we explore an internal dense yet external sparse network structure of deep neural networks and analyze its key properties.