Question: Represent the scientific article to find a short TLDR

Question: We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Compared to traditional CNNs, we demonstrate substantial parameter savings on standard image classification tasks, while maintaining accuracy. Our simple parameter sharing scheme, though defined via soft weights, in practice often yields trained networks with near strict recurrent structure; with negligible side effects, they convert into networks with actual loops. Training these networks thus implicitly involves discovery of suitable recurrent architectures. Though considering only the aspect of recurrent links, our trained networks achieve accuracy competitive with those built using state-of-the-art neural architecture search (NAS) procedures. Our hybridization of recurrent and convolutional networks may also represent a beneficial architectural bias. Specifically, on synthetic tasks which are algorithmic in nature, our hybrid networks both train faster and extrapolate better to test examples outside the span of the training set. The architectural details of convolutional neural networks (CNNs) have undergone rapid exploration and improvement via both human hand-design BID33 BID11 BID13 BID45 and automated search methods BID46 ). Yet, this vast array of work limits itself to a circuit-like view of neural networks. Here, a CNN is regarded as a fixed-depth feed-forward circuit, with a distinct parameter governing each internal connection. These circuits are often trained to perform tasks which, in a prior era, might have been (less accurately) accomplished by running a traditional computer program coded by humans. Programs, and even traditional hardware circuits, have a more reusable internal structure, including subroutines or modules, loops, and associated control flow mechanisms. We bring one aspect of such modularity into CNNs, by making it possible to learn a set of parameters that is reused across multiple layers at different depths. As the pattern of reuse is itself learned, our scheme effectively permits learning the length (iteration count) and content of multiple loops defining the ing CNN. We view this approach as a first step towards learning neural networks with internal organization reminiscent of computer programs. Though we focus solely on loop-like structures, leaving subroutines and dynamic control flow to future work, this simple change suffices to yield substantial quantitative and qualitative benefits over the standard baseline CNN models. While recurrent neural networks (RNNs) possess a loop-like structure by definition, their loop structure is fixed a priori, rather than learned as part of training. This can actually be a disadvantage in the event that the length of the loop is mismatched to the target task. Our parameter sharing scheme for CNNs permits a mix of loops and feed-forward layers to emerge. For example, trained with our scheme, a 50-layer CNN might learn a 2-layer loop that executes 5 times between layers 10 and 20, a 3-layer loop that runs 4 times from layers 30 to 42, while leaving the remaining layers to assume independent parameter sets. Our approach generalizes both CNNs and RNNs, creating a hybrid. where parameter templates T, T are shared among each layer i, which now only contains a 2-dimensional parameter α (i). Weights W (i) (no longer parameters, illustrated with dotted boxes) used by layer i are generated from α (i) and templates T, T. Right: If weights W (i) are outputs of a linear function (as in our method), learning parameter templates can be viewed as learning layer templates, offering a new (although equivalent) perspective for the middle diagram. Non-linearities are omitted for simplicity. FIG0 diagrams the parameter sharing scheme facilitating this hybridization. Inspired by dictionary learning, different network layers share, via weighted combination, global parameter templates. This re-parameterization is fully differentiable, allowing learning of sharing weights and template parameters. Section 3 elaborates, and also introduces tools for analyzing learned loop structures. Section 4 demonstrates advantages of our hybrid CNNs across multiple experimental settings. Taking a modern CNN design as a baseline, and re-parameterizing it according to our scheme improves:• Parameter efficiency. Here, we experiment with the standard task of image classification using modern residual networks BID11 BID41. This task is a good proxy for general usefulness in computer vision, as high-performance classification architectures often serve as a backbone for many other vision tasks, such as semantic segmentation BID1 BID44. Our parameter sharing scheme drastically reduces the number of unique parameters required to achieve a given accuracy on CIFAR BID20 ) or ImageNet (classification tasks. Re-parameterizing a standard residual network with our scheme cuts parameters, without triggering any drop in accuracy. This suggests that standard CNNs may be overparameterized in part because, by design (and unlike RNNs), they lack capacity to learn reusable internal operations.• Extrapolation and generalization. Here, we explore whether our hybrid networks expand the class of tasks that one can expect to train neural networks to accomplish. This line of inquiry, focusing on synthetic tasks, shares motivations with work on Neural Turing Machines BID5. Specifically, we would like neural networks to be capable of learning to perform tasks for which there are concise traditional solution algorithms. BID5 uses sorting as an example task. As we examine an extension of CNNs, our tasks take the form of queries about planar graphs encoded as image input. On these tasks, we observe improvements to both generalization ability and learning speed for our hybrid CNNs, in comparison to standard CNNs or RNNs. Our parameter sharing scheme, by virtue of providing an architectural bias towards networks with loops, appears to assist in learning to emulate traditional algorithms. An additional side effect, seen in practice in many of our experiments, is that two different learned layers often snap to the same parameter values. That is, layers i and j, learn coefficient vectors α DISPLAYFORM0 and α (j) (see FIG0) that converge to be the same (up to scaling). This is a form of architecture discovery, as it permits representation of the CNN as a loopy wiring diagram between repeated layers. Section 4.3 presents example . We also draw comparisons to existing neural architec-ture search (NAS) techniques. By simply learning recurrent structure as byproduct of training with standard stochastic gradient descent, we achieve accuracy competitive with current NAS procedures. Before delving into the details of our method, Section 2 provides additional context in terms of prior work on recurrent models, parameter reduction techniques, and program emulation. Sections 3 and 4 describe our hybrid shared-parameter CNN, experimental setup, and . Section 5 concludes with commentary on our and possible future research pathways. 1 2 RELATED WORK Recurrent variants of CNNs are used extensively for visual tasks. Recently, BID42 propose utilizing a convolutional LSTM BID32 as a generic feedback architecture. RNN and CNN combinations have been used for scene labeling BID26, image captioning with attention BID39, and understanding video BID3, among others. These works combine CNNs and RNNs at a coarse scale, and in a fixed hand-crafted manner. In contrast, we learn the recurrence structure itself, blending it into the inner workings of a CNN.Analysis of residual networks BID11 reveals possible connections to recurrent networks stemming from their design BID21. BID7 provide evidence that residual networks learn to iteratively refine feature representations, making an analogy between a very deep residual network and an unrolled loop. BID17 further explore this connection, and experiment with training residual networks in which some layers are forced to share identical parameters. This hard parameter sharing scheme again builds a predetermined recurrence structure into the network. It yields successfully trained networks, but does not exhibit the type of performance gains that Section 4 demonstrates for our soft parameter sharing scheme. Closely related to our approach is the idea of hypernetworks BID9, in which one part of a neural network is parameterized by another neural network. Our shared template-based reparameterization could be viewed as one simple choice of hypernetwork implementation. Perhaps surprisingly, this class of ideas has not been well explored for the purpose of reducing the size of neural networks. Rather, prior work has achieved parameter reduction through explicit representation bottlenecks BID15, sparsifying connection structure BID27 BID45, and pruning trained networks.Orthogonal to the question of efficiency, there is substantial interest in extending neural networks to tackle new kinds of tasks, including emulation of computer programs. Some approach this problem using additional supervision in the form of execution traces (Reed & de ; BID0, while other focus on development of network architectures that can learn from input-output pairs alone BID5 BID29 BID43 BID36 . Our experiments on synthetic tasks fall into the latter camp. At the level of architectural strategy, BID36 benefit from changing the form of activation function to bias the network towards correctly extrapolating common mathematical formulae. We build in a different implicit bias, towards learning iterative procedures within a CNN, and obtain a boost on correctly emulating programs. In convolutional neural networks (CNNs) and variants such as residual CNNs (ResNets) BID11 and DenseNets BID13, each convolutional layer i contains a set of parameters W (i), with no explicit relation between parameter sets of different layers. Conversely, a strict structure is imposed to layers of recurrent neural networks (RNNs), where, in standard models , a single parameter set W is shared among all time steps. This leads to a program-like computational flow, where RNNs can be seen as loops with fixed length and content. While some RNN variants BID4 BID19 BID40 are less strict on the length or content of loops, these are still typically fixed beforehand. As an alternative to learning hard parameter sharing schemes -which correspond to the strict structure present in RNNs -our method consists of learning soft sharing schemes through a relaxation of DISPLAYFORM0 Figure 2: Connection between the LSM matrix S where DISPLAYFORM1 and the structure of the network. White and black entries correspond to maximum and minimum similarities (Si,j = 1 and Si,j = 0, respectively). Left: Empirically, CNNs present no similarity between parameters of different layers. Middle: Trained with our method, the layer similarity matrix (LSM) captures similarities between different layers, including pairs with close to maximum similarity. Such pairs (depicted by same-colored coefficients and weights, and by white entries in the LSM) perform similar operations on their inputs. Right: We can tie together parameters of similar layers, creating a hard parameter sharing scheme. The network can then be folded, creating self-loops and revealing an explicit recurrent computation structure.this structure. We accomplish this by expressing each layer's parameters W (i) as a linear combination of parameter templates T,..., T (k), each with the same dimensionality as W (i): DISPLAYFORM2 where k is the number of parameter templates (chosen freely as a hyperparameter) and α (i), a kdimensional vector, is the coefficients of layer i. FIG0 (left and middle) illustrates the difference between networks trained with and without our method. This relaxation allows for coefficients and parameter templates to be (jointly) optimized with gradient-based methods, yielding negligible extra computational cost, with a single constraint that only layers with same parameter sizes can share templates. Note that constraining coefficients α (i) to be one-hot vectors leads to hard sharing schemes, at the cost of non-differentiability. Having k as a free parameter decouples the number of parameters in network from its depth. Typically, L convolutional layers with constant channel and kernel sizes C, K have O(LC 2 K 2) total parameters. Our soft sharing scheme changes the total number of parameters to O(kL + kC 2 K 2) = O(kC 2 K 2). Sections 4.1 and 4.2 show that we can decrease the parameter count of standard models without significantly impacting accuracy, or simply attain higher accuracy with k = L.In the next two subsections, we discuss two consequences of the linearity of Equation. First, it enables alternative interpretations of our method. Second, and a major advantage, as is the case in many linear relaxations of integer problems, we are able to extract hard sharing schemes in practice, and consequently detect implicit self-loops in a CNN trained with our method. For layers i that are linear in W (i) (e.g. matrix multiplication, convolution), we can view our method as learning template layers which are shared among a network. More specifically, for a convolutional layer U (i) (X) = W (i) * X, and considering Equation: DISPLAYFORM0 where T (j) * X, the of a convolution with filter sets T (j), can be seen as the output of a template layer with individual parameters T (j). Such layers can be seen as global feature extractors, and coefficients α (i) determine which features are relevant for the i'th computation of a network. This is illustrated in FIG0 (right diagram).This view gives a clear connection between coefficients α and the network's structure. Having DISPLAYFORM1, and hence layers i and i + 2 are functionally equivalent. Such a network can be folded to generate an equivalent model with two layers and a self-loop, an explicitly recurrent network. While this is also possible for networks without parameter sharing, a learned alignment of C 2 K 2 parameters is required (unlikely in practice), instead of aligning only k ≤ L parameters. To identify which layers in a network perform approximately the same operation, we can simply check whether their coefficients are similar. We can condense this information for all pairs of layers i, j in a similarity matrix S, where S i,j = s(α (i), α (j) ) for some similarity measure s. For networks with normalization layers, the network's output is invariant to weight rescaling. In this setting, a natural measure is s(DISPLAYFORM0 (absolute value of cosine similarity), since it possess this same property. 2 We call S the layer similarity matrix (LSM). Figure 2 illustrates and Section 4.3 shows experimentally how it can be used to extract recurrent loops from trained CNNs. While structure might emerge naturally, having a bias towards more structured (recurrent) models might be desirable. In this case, we can add a recurrence regularizer to the training objective, pushing parameters to values which in more structure. For example, we can add the negative of sum of elements of the LSM: DISPLAYFORM1 where L is the original objective. The larger λ R is, the closer the elements of S will be to 1. At an extreme case, this regularizer will push all elements in S to 1, ing in a network with a single layer and a self-loop. We begin by training variants of standard models with soft parameter sharing, observing that it can offer parameter savings with little impact on performance, or increase performance at the same parameter count. Section 4.3 demonstrates conversion of a trained model into explicitly recurrent form. We then examine synthetic tasks (Section 4.4), where parameter sharing improves generalization. Appendix B contains details on the initialization for the coefficients α. The CIFAR-10 and CIFAR-100 datasets BID20 ) are composed of 60, 000 colored 32×32 images, labeled among 10 and 100 classes respectively, and split into 50, 000 and 10, 000 examples for training and testing. We pre-process the training set with channel-wise normalization, and use horizontal flips and random crops for data augmentation, following BID11.Using Wide ResNets (WRN) BID41 ) as a base model, we train networks with the proposed soft parameter sharing method. Since convolution layers have different number of channels and kernel sizes throughout the network, we create 3 layer groups and only share templates among layers in the same group. More specifically, WRNs for CIFAR consist of 3 stages whose inputs and outputs mostly have a constant number of channels (C, 2C and 4C, for some C). Each stage contains L−4 3 layers for a network with depth L, hence we group layers in the same stage together, except for the first two, a residual block whose input has a different number of channels. Thus, all layers except for the first 2 in each stage perform parameter sharing (illustrated in left diagram of Figure 4). Having k templates per group means that L−4 3 − 2 convolution layers share k parameter templates. We denote by SWRN-L-w-k a WRN with L layers, widen factor w and k parameter templates per group (trained with our method). Setting k = L−4 3 − 2 means we have Table 1: Test error (%) on CIFAR-10 and CIFAR-100. SWRN 28-10, the of training a WRN 28-10 with our method and one template per layer, significantly outperforms the base model, suggesting that our method aids optimization (both models have the same capacity). SWRN 28-10-1, with a single template per sharing group, performs close to WRN 28-10 while having significantly less parameters and capacity. * indicates models trained with dropout p = 0.3 BID34. Results are average of 5 runs. one parameter template per layer, and hence no parameter reduction. We denote SWRN-L-w (thus omitting k) as a model in this setting. Following BID41, we train each model for 200 epochs with SGD and Nesterov momentum of 0.9 and a batch size of 128. The learning rate is initially set to 0.1 and decays by a factor of 5 at epochs 60, 120 and 160. We also apply weight decay of 5 × 10 −4 on all parameters except for the coefficients α. TAB1 present . Networks trained with our method yield superior performance in the setting with no parameter reduction: SWRN 28-10 presents 6.5% and 2.5% lower relative test errors on C-10 and C-100, compared to the base WRN 28-10 model. With fewer templates than layers, SWRN 28-10-1 (all 6 layers of each group perform the same operation), performs virtually the same as the base WRN 28-10 network, while having 1 3 of its parameters. On CIFAR-10, parameter reduction (k = 2) is beneficial to test performance: the best performance is achieved by SWRN 28-18-2 (3.43% test error), outperforming the ResNeXt-29 16x64 model BID37, while having fewer parameters (55M against 68M) and no bottleneck layers. FIG2 shows that our parameter sharing scheme uniformly improves accuracy-parameter efficiency; compare the WRN model family (solid red) to our SWRN models (dotted red). Table 4 presents a comparison between our method and neural architecture search (NAS) techniques BID46 BID38 BID25 BID28 on CIFAR-10 - differ from Table 2 solely due to cutout BID2, which is commonly used in NAS literature; NAS are quoted from their respective papers. Our method outperforms architectures discovered by recent NAS algorithms, such as DARTS, SNAS BID38 and ENAS BID25, while having similarly low training cost. We achieve 2.69% test error after training less than 10 hours on a single NVIDIA GTX 1080 Ti. This accuracy is only bested by NAS techniques which are several orders of magnitude more expensive to train. Being based on Wide ResNets, our models do, admittedly, have more parameters. Comparison to recent NAS algorithms, such as DARTS and SNAS, is particularly interesting as our method, though motivated differently, bears some notable similarities. Specifically, all three methods are gradient-based and use an extra set of parameters (architecture parameters in DARTS and SNAS) to perform some kind of soft selection (over operations/paths in DARTS/SNAS; over templates in our method). As Section 4.3 will show, our learned template coefficients α can often be used to transform our networks into an explicitly recurrent form -a discovered CNN-RNN hybrid. To the extent that our method can be interpreted as a form of architecture search, it might be complementary to standard NAS methods. While NAS methods typically search over operations (e.g. activation functions; 3 × 3 or 5 × 5 convolutions; non-separable, separable, or grouped filters; dilation; pooling), our soft parameter sharing can be seen as a search over recurrent patterns (which layer processes the output at each step). These seem like orthogonal aspects of neural architectures, both of which may be worth examining in an expanded search space. When using SGD to drive architecture search, these aspects take on distinct forms at the implementation level: soft parameter sharing across layers (our method) vs hard parameter sharing across networks (recent NAS methods). We use the ILSVRC 2012 dataset BID30 as a stronger test of our method. It is composed of 1.2M training and 50, 000 validation images, drawn from 1000 classes. We follow BID8, as in BID41 BID13; BID37, and report Top-1 and Top-5 errors on the validation set using single 224 × 224 crops. For this experiment, we use WRN 50-2 as a base model, and train it with soft sharing and no parameter reduction. Having bottleneck blocks, this model presents less uniform number of channels of layer inputs and outputs. To apply our method, we group convolutions in 12 groups: for each of the 4 stages in a WRN 50-2, we create 3 groups, one for each type of layer in a bottleneck unit (C → B, B → B and B → C channel mappings, for bottleneck B). Without any change in hyperparameters, the network trained with our method outperforms the base model and also deeper models such as DenseNets (though using more parameters), and performs close to ResNet-200, a model with four times the number of layers and a similar parameter count. See TAB2. DISPLAYFORM0 Figure 4: Extracting implicit recurrences from a SWRN 28-10-4. Left: Illustration of the stages of a SWRN-28-10-4 (residual connections omitted for clarity). The first two layers contain individual parameter sets, while the other six share four templates. All 3 stages of the network follow this structure. Middle: LSM for each stage after training on CIFAR-10, with many elements close to 1. Hard sharing schemes can be created for pairs with large similarity by tying their coefficients (or, equivalently, their effective weights). Right: Folding stages 2 and 3 leads to self-loops and a CNN with recurrent connections -LSM for stage 2 is a repetition of 2 rows/columns, and folding decreases the number of parameters. Results on CIFAR suggest that training networks with few parameter templates k in our soft sharing scheme in performance comparable to the base models, which have significantly more parameters. The lower k is, the larger we should expect the layer similarities to be: in the extreme case where k = 1, all layers in a sharing scheme have similarity 1, and can be folded into a single layer with a self-loop. For the case k > 1, there is no trivial way to fold the network, as layer similarities depend on the learned coefficients. We can inspect the model's layer similarity matrix (LSM) and see if it presents implicit recurrences: a form of recurrence in the rows/columns of the LSM. Surprisingly, we observe that rich structures emerge naturally in networks trained with soft parameter sharing, even without the recurrence regularizer. Figure 4 shows the per-stage LSM for CIFAR-trained SWRN 28-10-4. Here, the six layers of its stage-2 block can be folded into a loop of two layers, leading to an error increase of only 0.02%. Appendix A contains an additional example of network folding, diversity of LSM patterns across different runs, and an epoch-wise evolution of the LSM, showing that many patterns are observable after as few as 5 epochs of training. While the propensity of our parameter sharing scheme to encourage learning of recurrent networks is a useful parameter reduction tool, we would also like to leverage it for qualitative advantages over standard CNNs. On tasks for which a natural recurrent algorithm exists, does training CNNs with soft parameter sharing lead to better extrapolation?To answer this, we set up a synthetic algorithmic task: computing shortest paths. Examples are 32 × 32 grids containing two query points and randomly (with probability 0.1) placed obstacles. The objective is to indicate which grid points belong to a shortest path between the query points. We use curriculum learning for training, allowing us to observe how well each model adapts to more difficult examples as training phases progress. Moreover, for this task curriculum learning causes faster learning and superior performance for all trained models.(a) Generated example for the synthetic shortest paths task. Blue pixels indicate the query points; red pixels represent obstacles, and white pixels are points in a shortest path (in terms of Manhattan distance) between query pixels. The task consists of predicting the white pixels (shortest paths) from the blue and red ones (queries and obstacles). Training consists of 5 curriculum phases, each one containing 5000 examples. The maximum allowed distance between the two query points increases at each phase, thus increasing difficulty. In the first phase, each query point is within a 5 × 5 grid around the other query point, and the grid size increases by 2 on each side at each phase, yielding a final grid size of 21 × 21 at phase 5.We train a CNN, a CNN with soft parameter sharing and one template per layer (SCNN), and an SCNN with recurrence regularizer λ R = 0.01. Each model trains for 50 epochs per phase with Adam and a fixed learning rate of 0.01. As classes are heavily unbalanced and the balance itself changes during phases, we compare F 1 scores instead of classification error. Each model starts with a 1 × 1 convolution, mapping the 2 input channels to 32 output channels. Next, there are 20 channel-preserving 3 × 3 convolutions, followed by a final 1 × 1 convolution that maps 32 channels to 1. Each of the 20 3 × 3 convolutions is followed by batch normalization BID16, a ReLU non-linearity BID24, and has a 1-skip connection. FIG4 shows one example from our generated dataset and the training curves for the 3 trained models: the SCNN not only outperforms the CNN, but adapts better to harder examples at new curriculum phases. The SCNN is also advantaged over a more RNN-like model: with the recurrence regularizer λ R = 0.01, all entries in the LSM quickly converge 1, as in a RNN. This leads to faster learning during the first phase, but presents issues in adapting to difficulty changes in latter phases. In this work, we take a step toward more modular and compact CNNs by extracting recurrences from feed-forward models where parameters are shared among layers. Experimentally, parameter sharing yields models with lower error on CIFAR and ImageNet, and can be used for parameter reduction by training in a regime with fewer parameter templates than layers. Moreover, we observe that parameter sharing often leads to different layers being functionally equivalent after training, enabling us to collapse them into recurrent blocks. Results on an algorithmic task suggest that our shared parameter structure beneficially biases extrapolation. We gain a more flexible form of behavior typically attributed to RNNs, as our networks adapt better to out-of-domain examples. Our form of architecture discovery is also competitive with neural architecture search (NAS) algorithms, while having a smaller training cost than state-of-the-art gradient-based NAS.As the only requirement for our method is for a network to have groups of layers with matching parameter sizes, it can be applied to a plethora of CNN model families, making it a general technique with negligible computational cost. We hope to raise questions regarding the rigid definitions of CNNs and RNNs, and increase interest in models that fall between these definitions. Adapting our method for models with non-uniform layer parameter sizes BID13 BID45 might be of particular future interest. A ADDITIONAL FOR IMPLICIT RECURRENCES Section 4.3 presents an example of implicit recurrences and folding of a SWRN 28-10-4 trained on CIFAR-10, where, for example, the last 6 layers in the second stage of the network fold into 2 layers with a self-loop. Figure 6 presents an additional example, where non-trivial recurrences (unlike the one in Figure 4) emerge naturally, ing in a model that is rich in structure. − 2 = 10 layers) trained with soft parameter sharing on CIFAR-10. Each stage (originally with 12 layers -the first two do not participate in parameter sharing) can be folded to yield blocks with complex recurrences. For clarity, we use colors to indicate the computational flow: red takes precedence over green, which in turn has precedence over blue. Colored paths are only taken once per stage. Although not trivial to see, recurrences in each stage's folded form are determined by row/column repetitions in the respective Layer Similarity Matrix. For example, for stage 2 we have S5,3 ≈ S6,4 ≈ 1, meaning that layers 3, 4, 5 and 6 can be folded into layers 3 and 4 with a loop (captured by the red edge). The same holds for S7,1, S8,2, S9,3 and S10,4, hence after the loop with layers 3 and 4, the flow returns to layer 1 and goes all the way to layer 4, which generates the stage's output. Even though there is an approximation when folding the network (in this example, we are tying layers with similarity close to 0.8), the impact on the test error is less than 0.3%. Also note that the folded model has a total of 24 layers (20 in the stage diagrams, plus 4 which are not shown, corresponding to the first layer of the network and three 1 × 1 convolutions in skip-connections), instead of the original 40. Figure 7: LSMs of a SWRN 40-8-8 (composed of 3 stages, each with 10 layers sharing 8 templates) trained on CIFAR-10 for 5 runs with different random seeds. Although the LSMs differ across different runs, hard parameter sharing can be observed in all cases (off-diagonal elements close to 1, depicted by white), characterizing implicit recurrences which would enable network folding. Moreover, the underlying structure is similar across runs, with hard sharing typically happening among layers i and i + 2, leading to a "chessboard" pattern. During our initial experiments, we explored different initializations for the coefficients α of each layer, and observed that using an orthogonal initialization BID31 ed in superior performance compared to uniform or normal initialization schemes. Denote A as the L × k matrix (L is the number of layers sharing parameters and k the number of templates) with each i'th row containing the coefficient of the i'th layer α (i). We initialize it such that A T A = I, leading to ∀ i, α (i), α (i) = 1 and ∀ i =j, α (i), α (j) = 0. While our choice for this is mostly empirical, we believe that there is likely a connection with the motivation for using orthogonal initialization for RNNs. Moreover, we discovered that other initialization options for A work similarly to the orthogonal one. More specifically, either initializing A with the identity matrix when L = k (which naturally leads to A T A = I) or enforcing some sparsity (initialize A with a uniform or normal distribution and randomly setting half of its entries to zero) performs similarly to the orthogonal initialization in a consistent manner. We believe the sparse initialization to be the simplest one, as each coefficient α can be initialized independently. Finally, note that having A T A = I in the Layer Similarity Matrix also being the identity at initialization (check that DISPLAYFORM0, so if (A T A) i,j = 1, then S i,j = 1, and the same holds for 0. Surprisingly, even though the orthogonal initialization leads to a LSM that has no structure in the beginning of training, the rich patterns that we observe still emerge naturally after optimization.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method that enables CNN folding to create recurrent connections

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the scientific article to find a short TLDR

Question: Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum. This intuition has been made precise in a line of recent works, which show that suitable clipping can yield significantly faster convergence than vanilla gradient descent. In this paper, we propose a new lens for studying gradient clipping, namely, robustness: informally, one expects clipping to provide robustness to noise, since one does not overly trust any single sample. Surprisingly, we prove that for the common problem of label noise in classification, standard gradient clipping does not in general provide robustness. On the other hand, we show that a simple variant of gradient clipping is provably robust, and corresponds to suitably modifying the underlying loss function. This yields a simple, noise-robust alternative to the standard cross-entropy loss which performs well empirically. In this paper, we propose a new lens with which to study gradient clipping, namely, robustness: intuitively, clipping the gradient prevents over-confident descent steps, which is plausibly beneficial in the presence of noise. Given this intuition, our interest is whether gradient clipping can mitigate the problem of label noise in classification, which has received significant recent interest (; ; ; ; ; ; ; ; ;). We study this question, and provide three main contributions: (a) we show that gradient clipping alone does not endow label noise robustness to even simple models. Specifically, we show that under stochastic gradient descent with linear models, gradient clipping is related to optimising a "Huberised" loss (Lemma 1, 2). While such Huberised losses preserve classification calibration (Lemma 3), they are not robust to label noise (Proposition 4). (b) we propose composite loss-based gradient clipping, a variant that does have label noise robustness. Specifically, for losses comprising a base loss composed with a link function, (e.g., softmax crossentropy), we only clip the contribution of the base loss. The ing partially Huberised loss preserves classification calibration (Lemma 6), while being robust to label noise (Proposition 7). (c) we empirically verify that on both synthetic and real-world datasets, partially Huberised versions of standard losses (e.g., softmax cross-entropy) perform well in the presence of label noise (§5). To illustrate the difference between standard and composite loss-based gradient clipping in (a) and (b), consider the pervasive softmax cross-entropy loss, viz. the logistic loss for binary classification. Recall that the logistic loss comprises the log-loss (or cross-entropy) composed with a sigmoid link. The Huberised loss arises from loss-based gradient clipping, and linearises the entire logistic loss beyond a threshold. The partially Huberised loss arises from composite loss-based gradient clipping, and linearises only the base loss (i.e., log-loss) beyond a threshold (right panel). Once combined with the sigmoid link, the overall partially Huberised loss asymptotically saturates. Our analysis in §3, §4 establishes that Huberised losses are not robust to label noise, but partially Huberised losses are (Proposition 4, 7). In (a), we relate gradient clipping to loss "Huberisation", which linearises the logistic loss when it exceeds a fixed threshold. In (b), we introduce composite loss-based gradient clipping, which is equivalent to a "partial loss Huberisation" that linearises only the cross-entropy loss but leaves the sigmoid link untouched. Figure 1 illustrates the Huberised and partially Huberised logistic loss. We provide on gradient clipping, loss functions for classification, and label noise. Gradient clipping. Consider a supervised learning task over instances X and labels Y, where we have a family of models indexed by θ ∈ Θ, and the quality of a particular model is measured by a loss function θ: X × Y → R. The gradient for a mini-batch {(x n, y n)} N n=1 is g(θ) One may instead compute the clipped gradient, which for user-specified threshold τ > 0 is if ||w|| 2 ≥ τ w else. Employingḡ τ (θ) for optimisation corresponds to clipped gradient descent. This is closely related to normalised gradient descent (NGD) , wherein one employsg(θ) ||g(θ)||2. showed that NGD can lead to convergence for a wider class of functions than standard gradient descent. showed that NGD can escape saddle points for non-convex problems. showed that gradient clipping can lead to accelerated convergence over gradient descent. Gradient clipping has also been explored for privacy , motivated by it preventing any single instance from dominating parameter updates. Loss functions. In binary classification, one observes samples from a distribution D over X × {±1}, and seeks a predictor f: X → R with low risk R(f). = E (y, f (x)) according to a loss which, in an overload of notation, we denote: {±1} × R → R +. For the zero-one loss 01 (y, f) = y · f < 0, R(f) is known as the misclassification risk. Rather than directly using 01, for computational convenience one often employs a margin loss (y, f) = φ(y · f), for convex φ: R → R +. We say that φ is classification calibrated if driving the excess risk over the Bayes-optimal predictor for φ to zero also drives the excess risk for 01 to zero; that is, minimising the φ-risk is statistically consistent for classification. A canonical example is the hinge loss φ(z) = [1 − z] +. We call φ admissible if it is "well-behaved" in the sense of being bounded from below, strictly convex, continuously differentiable, non-increasing, and classification calibrated. We say that φ is proper composite (or for brevity composite) if its minimising scores can be interpreted as probabilities. "Composite" here refers to such losses comprising a base loss ϕ composed with an invertible link function F: R →. While φ accepts as input real-valued scores (e.g., the final layer logits of a neural network), these are internally converted to probabilities via F. A canonical example is the logistic loss φ(z) = − log F (z) with F: z → σ(z) for sigmoid σ(z). = (1 + e −z) −1. The multiclass analogue is the softmax cross-entropy loss, wherein the sigmoid becomes a softmax. See Appendix B for a more technical discussion. Learning under label noise. In classification under label noise, one has samples from distribution D where PD(y | x) is a noisy version of P D (y | x), e.g., all labels are corrupted with a fixed constant probability. The goal remains to ensure low risk with respect to the clean D. This problem has a long history in statistics , and has emerged as a topic of recent interest in machine learning. There has been particular interest in the problem of learning under symmetric label noise, wherein all instances have a constant probability of their labels being flipped uniformly to any of the other classes.; proposed a framework for analysing the more general setting of class-conditional label noise. Takenouchi (; ;) to abstention . We first show that gradient clipping in general does not endow robustness to label noise, even in simple settings. Specifically, we establish that stochastic gradient clipping with linear models is equivalent to modifying the underlying loss (Lemma 1). This modified loss is closely related to a Huberised loss (Lemma 2), which is equivalent to loss-based gradient clipping (L-gradient clipping). Unfortunately, Huberised losses, and thus L-gradient clipping, are not robust to label noise (Proposition 4). Consider a binary classification problem, with Y = {±1}. Suppose we use a scoring model s θ (x) for θ ∈ Θ with margin m θ (x, y). = y · s θ (x), and margin loss θ (x, y) Now suppose we perform stochastic gradient descent (i.e., we use N = 1 in) and use a linear scorer s θ (x) = θ T x. Here, gradient clipping of is equivalent to modifying the loss as follows. Lemma 1. Pick any admissible margin loss φ, and τ > 0. Then, for loss θ (x, y). = φ(m θ (x, y)) with linear scorer, the clipped loss gradient is equivalent to the gradient of a modified loss¯: The loss¯ in is intuitive: if the derivative of the original loss φ exceeds a certain effective clipping threshold, we replace φ with a linear loss function. This effective threshold τ · (∇m θ (x, y) 2 ) −1 takes into account the instance-dependent margin gradient, viz. x 2 for linear s θ (x). We will comment in §4.4 as to the properties of gradient clipping in more general scenarios. For the moment, we note that Lemma 1 is closely related to the "Huber loss" , This replaces the extremes of the square loss with the absolute loss. Both the absolute and Huber losses are widely employed in robust regression (, Chapter 3), (, pg. 100). Note that slightly differs from as the effective clipping threshold is instance-dependent. One may nonetheless arrive at Huber-style losses via a variant of gradient clipping; this connection will prove useful for our subsequent analysis. Per, gradient clipping involves using wherein the clipping considers the margin gradient. Consider now the following loss-based gradient clipping (L-gradient clipping), wherein we clip only the contribution arising from the loss: Compared to, we effectively treat the margin gradient norm ∇m θ (x, y) 2 as constant across instances, and focus on bounding the loss derivative. The latter may be a significant component in the gradient norm; e.g., for linear models, ∇m θ (x, y) 2 = x 2 is often bounded. Observe further that for linear models with x 2 ≡ R across instances, is a rescaled version of the clipped gradient: Interestingly, L-gradient clipping equivalently uses the following "Huberised" version of the loss. Lemma 2. Pick any admissible margin loss φ: R → R + with Fenchel conjugate φ *, and τ > 0. Then, the clipped gradient in is equivalent to employing a Huberised loss functionφ τ such that Evidently,φ τ linearises φ once its derivative is sufficiently large, akin to the Huber loss in. One may verify that for φ(z) = (1 − z) 2, one arrives exactly at. Example 3.1: For the logistic loss φ(z) = log(1 + e −z), the Huberised loss for τ ∈ is Per Figure 1a, this linearises φ beyond a fixed threshold. See Appendix C for further illustrations. The use of Huberised losses in classification is not new (; ;). However, we now provide a novel study of their label noise robustness. Before studying the effect of label noise on Huberised losses, it is apposite to consider whether they are suitable for use even in the absence of noise. One way to formalise this is to ask whether the losses maintain classification calibration, in the sense of §2. This is a minimal requirement on a loss to be useful for classification . One may further ask whether the losses preserve class-probabilities, i.e., are proper composite in the sense of §2. It is desirable to preserve this key trait of losses such as the logistic loss. The following clarifies both points. Lemma 3. Pick any admissible margin loss φ and τ > 0. Then, the Huberised lossφ τ in is classification calibrated. If φ is proper composite and τ ≥ −φ, thenφ τ is also proper composite. L-gradient clipping is thus benign for classification, generalising Rosset & Zhu (2007, Section 3.4), which was for the square-hinge loss. Interestingly, for composite φ and small τ, the proof reveals that φ τ has a non-invertible link function. Intuitively, suchφ τ are effectively linear, and linear losses are unsuited for estimating probabilities ). We now turn to our central object of inquiry: does gradient clipping endow robustness to label noise? To study this, we consider L-gradient clipping, which as noted above is a special case of gradient clipping under linear models with constant ∇m θ (x, y) 2. Since L-gradient clipping is in turn equivalent to using a Huberised loss, we may study the robustness properties of this loss. Surprisingly, when our loss is convex (e.g., softmax cross-entropy), Huberised losses are not robust to even very simple forms of label noise. Essentially, since these losses are still convex, they can still be affected by errant outlier observations. Formally, using a of (see Appendix F.1), we arrive at the following. Proposition 4. Pick any admissible margin loss φ and τ > 0. Then, ∃ a separable distribution for which the optimal linear classifier underφ τ is equivalent to random guessing under symmetric noise. To situate Proposition 4 in a broader context, we note that for regression problems, it is well known that the Huber loss is susceptible to "high leverage" outliers (, pg. 313), (, pg. 13), (, pg. 104), i.e., extremal instances which dominate the optimal solution. Proposition 4 complements these for the case of label noise in classification. Given that gradient clipping does not endow label noise robustness, how else might we proceed? In a regression context, the outlier vulnerability of the Huber loss can be addressed by using a trimmed average of the loss (; ;). Such ideas have been successfully explored for label noise problems . We will however demonstrate that a simple variant of clipping yields a loss that does possess label noise robustness. We now show that noise robustness can be achieved with CL-gradient clipping, a variant wherein for composite losses (e.g., softmax cross-entropy), we perform partial Huberisation of the base loss only. Consider a composite margin loss θ (x, y), where φ = ϕ • F for some base loss ϕ and invertible link F:R →; e.g., the logistic loss has ϕ(u) = − log u and F (z) = σ(z). We can interpret p θ (x, y). = F (m θ (x, y)) as a probability estimate; e.g., p θ (x, 1) = σ(m θ (x, 1)) is the probability of x being positive. Now, rewriting θ (x, y) = ϕ(p θ (x, y)), we may express as: L-gradient clipping in was defined as only clipping φ = (ϕ • F) above, which ensures that the ing Huberised loss is Lipschitz. Typically, however, F is already Lipschitz; e.g., this is the case for the commonly used sigmoid or softmax link. This suggests the following composite loss-based gradient clipping (CL-gradient clipping), wherein we only clip the derivative for the base loss ϕ: As before, CL-gradient clipping corresponds to optimising a new, "partially Huberised" loss. Lemma 5. Pick any admissible, composite margin loss φ = ϕ • F, and τ > 0. Then, the clipped gradient in is equivalent to employing a partially Huberised lossφ τ =φ τ • F, wherẽ Compared to the Huberised loss, the partially Huberised loss only linearises the base loss ϕ, while retaining the link F. Consequently, the composite lossφ τ behaves like the link beyond a certain threshold, and will thus be bounded: Example 4.1: For the logistic loss, φ(z) = ϕ(F (z)) with ϕ(u) = − log u and F (z) = σ(z), Note that partial Huberisation readily generalises to a multi-class setting. Indeed, suppose we have softmax probability estimates p θ (x, y) ∝ exp(m θ (x, y)). Then, whereas the softmax cross-entropy employs θ (x, y) = − log p θ (x, y), our partially Huberised softmax cross-entropy for τ > 1 is Following §3.2, we establish that CL-gradient clipping is always benign from a classification perspective, and provided τ is sufficiently large, from a probability estimation perspective as well. As before, we do this by exploiting the equivalence of CL-gradient clipping to a partially Huberised loss. Lemma 6. Pick any admissible composite margin loss φ = ϕ • F and τ > 0. Then, the lossφ τ in is classification calibrated. If further τ ≥ −ϕ (1 /2), thenφ τ is also proper composite. We now show that partially Huberised losses have an important advantage over Huberised losses: under symmetric label noise, the optimal solution on the clean distribution cannot be too far away from the optimal solution on the noisy distribution. This implies that label noise (such as that considered in Proposition 4) cannot have an excessively deleterious influence on the loss. Proposition 7. Pick any proper loss ϕ and τ > 0. Let f * be the risk minimiser ofφ τ on the clean distribution. For any non-trivial level of symmetric label noise, let reg τ (f *) denote the excess risk of f * with respect toφ τ on the noisy distribution. Then, there exists C > 0 such that Note that by van Rooyen et al. (2015, Proposition 4), it is impossible for the above bound to hold with C = 0 without using a linear loss. Nonetheless, by virtue of partially Huberised losses being partially linear, we are able to bound the degradation under label corruption. The saturating behaviour of the partially Huberised loss also implies robustness to outliers in feature space; see Appendix E. The partially Huberised log loss in, can be related to a family of losses studied in several works (; ; ; b; a): 3 for an illustration of ϕ α. There are two similarities between and ϕ α. First, both proposals interpolate between the log and linear losses: when α → 0 +, ϕ α approaches the log loss, and when α = 1, ϕ α equals the linear loss. Second, both proposals modify the base loss, allowing the link F to be chosen independently. In particular, one may use the heavy-tailed F of; Amid et al. (2019b) in conjunction with our partially Huberised loss. One difference between and ϕ α is that the partially Huberised loss is exactly linear for a suitable region; consequently, it is guaranteed to be Lipschitz, unlike ϕ α. This can be understood in terms of the loss gradients: for a class-probability estimate p θ (x, y), let θ (x, y). Both gradients thus take into account whether a sample is "informative", in the sense of being poorly-predicted (p θ (x, y) ∼ 0). Further, to guard against such samples being the of label noise, both ensure this influence is not overwhelming, but in different ways: the generalised cross-entropy softens the influence, while still allowing it to be unbounded as p θ (x, y) → 0. On the other hand, the partially Huberised loss enforces a hard cap of τ −1 on the influence. This is to be contrast with a truncated loss also considered in , which enforces a hard cap on the loss, thus completely discarding poorly-predicted instances. Table 1 summarises our , highlighting the perspective of gradient clipping as equivalently modifying the loss. Before proceeding, we make some qualifying comments. First, our analysis has assumed symmetric label noise. Often, one encounters asymmetric or instance-dependent noise. While corresponding guarantees for the linear loss may be ported over to the partially Huberised loss, they require stronger distributional assumptions . Second, Proposition 4 exhibits a specific distribution which defeats the Huberised loss under linear models. In practice, distributions may be more benign , and models are often nonlinear, meaning that Huberised losses (and thus gradient clipping) are thus unlikely to succumb as extremely to label noise as Proposition 4 suggests. The aim of §4 is however to establish that a simple modification of clipping avoids worst-case degradation, without adding significant complexity. Equivalent loss Reference Label noise robust? (Proposition 7) Table 1: Summary of types of gradient clipping considered in this paper. We consider binary classification problems involving a labelled example (x, y), parametrised scoring function s θ (x) with margin m(θ). = y · s θ (x), and differentiable composite margin loss φ(z). This loss internally converts scores to probabilities p(θ). = F (m(θ)) for link function F (·), which is evaluated with some base loss ϕ; i.e., φ = ϕ • F. Gradient clipping applies to the full loss, i.e., φ(m(θ)). L-gradient clipping applies only to the composite loss, leaving the score untouched; this is equivalent to using a Huberised loss. CL-gradient clipping applies only to the base loss, leaving the link untouched; this is equivalent to using a partially Huberised loss. Only the latter has robustness guarantee under symmetric label noise. Third, for minibatch size N > 1, the effect of clipping is not a simple loss modification, since the loss gradients for each sample will be modified by the entire minibatch loss gradient norm g(θ) 2. Since this minibatch is randomly drawn, one cannot mimic gradient clipping by a simple modification of the loss function. However, the for N = 1 suffice to establish that gradient clipping cannot in general endow robustness. One may use our partially Huberised loss in conjunction with minibatch gradient clipping, to potentially obtain both robustness and optimisation benefits. Finally, partially Huberised losses such as require setting a hyperparameter τ (e.g., by crossvalidation), similar to α in the generalised cross-entropy per §4.3. Intuitively, the optimal τ trades off the noise-robustness of the linear loss, and the gradient informativeness of the base loss (per the discussion in §4.3). Setting τ to be large tacitly assumes that one's samples are largely noise-free. We now present experiments illustrating that: (a) we may exhibit label noise scenarios that defeat a Huberised but not partially Huberised loss, confirming Propositions 4, 7, and (b) partially Huberised versions of existing losses perform well on real-world datasets subject to label noise. Synthetic datasets. Our first experiments involve two synthetic datasets, which control for confounding factors. We begin with a setting from , comprising a 2D linearly separable distribution. (See Appendix F.1 for an illustration.) We draw N = 1, 000 random samples from this distribution, and flip each label with ρ = 45% probability. We train a linear classifier to minimise one of several losses, and evaluate the classifier's accuracy on 500 clean test samples. We compare the logistic loss with its Huberised and partially Huberised versions, using τ = 1 and τ = 2 respectively. Figure 2a presents the over 500 independent trials. The logistic loss and its Huberised counterpart suffer significantly under noise, while the partially Huberised loss often achieves perfect near-discrimination. This confirms that in the worst case, L-gradient clipping may succumb to noise, while CL-gradient clipping performs well in the same scenario. We next consider a 1D setting based on Ding (2013, Section 3.2.3), comprising 10, 000 linearly separable "inliers" and 50 "outliers". Assuming the use of a linear model parameterised by scalar 91.6 ± 0.1 88.6 ± 0.0 83.6 ± 0.1 72.2 ± 0.0 PHuber-GCE τ = 10 92.0 ± 0.1 88.5 ± 0.1 80.8 ± 0.1 62.6 ± 0.2 CIFAR-100 CE 66.6 ± 1.4 49.7 ± 0.3 29.9 ± 0.9 11.4 ± 0.2 CE + clipping 28.8 ± 0.1 20.6 ± 0.4 14.7 ± 0.6 9.0 ± 0.4 Linear 12.1 ± 1.6 6.6 ± 1.2 5.7 ± 0.9 3.6 ± 0.1 GCE 70.1 ± 0.1 63.9 ± 0.1 52.0 ± 0.2 29.9 ± 0.5 PHuber-CE τ = 10 66.2 ± 1.5 56.2 ± 2.2 44.4 ± 0.7 18.5 ± 0.4 PHuber-GCE τ = 10 69.8 ± 0.2 64.4 ± 0.4 52.4 ± 0.2 31.5 ± 0.8 Table 2: Test set accuracy where the training labels are corrupted with probability ρ. We report the mean and standard error over 3 trials. The highlighted cells are the best performing loss at a given ρ. "PHuber" here refers to our partial Huberisation from §4, which is equivalent to a variant of gradient clipping. θ ∈ R, we plot the empirical risk of the samples with and without the outlier observations as θ is varied. Figure 2b shows that the logistic loss and its Huberised variant are strongly affected by the outliers: their optimal solution goes from θ * = +∞ to θ * = 0. However, the partially Huberised loss is largely immune to the outliers. Appendix F contains additional synthetic experiments. Real-world datasets. We now demonstrate that partially Huberised losses perform well with deep neural networks trained on MNIST, CIFAR-10 and CIFAR-100 . For MNIST, we train a LeNet using Adam with batch size N = 32, and weight decay of 10 −3. For CIFAR-10 and CIFAR-100, we train a ResNet-50 using SGD with momentum 0.1, weight decay of 5 × 10 −3, batch normalisation, and N = 64, 128 respectively. For each dataset, we corrupt the training labels with symmetric noise at flip probability ρ ∈ {0.0, 0.2, 0.4, 0.6}. We compare the test set accuracy of various losses combined with a softmax link. Our baseline is the cross-entropy loss (CE). As representative noise-robust losses, we consider the linear or unhinged loss ), and the generalised cross-entropy (GCE) with α = 0.7, following. We additionally assess global gradient clipping (with τ = 0.1) of the CE, which per §3 is akin to a Huberised loss. We apply our partial Huberisation of to the CE ("PHuber-CE"), and the GCE ("PHuber-GCE"). The latter highlights that partial Huberisation is not tied to the cross-entropy, and is applicable even on top of existing noise-robust losses. Recall that partial Huberisation offers a choice of tuning parameter τ, similar to the α parameter in GCE, and noise-rate estimates in loss-correction techniques more generally. For each dataset, we pick τ ∈ {2, 10} (equivalently corresponding to probability thresholds 0.5 and 0.1 respectively) so as to maximise accuracy on a validation set of noisy samples with the maximal noise rate ρ = 0.6; the chosen value of τ was then used for each noise level. Tuning τ separately for each setting of the noise rate ρ can be expected to help performance, at the expense of increased computational cost. Recall also that as τ → 1, partial Huberisation mimics using the base loss, while as τ → +∞, partial Huberisation mimics using the linear loss; our hypothesis is that an intermediate τ can attain a suitable balance between noise robustness, and gradient informativeness. Table 2 shows that in the noise-free case (ρ = 0.0), all methods perform comparably. However, when injecting noise, accuracy for the CE degrades dramatically. Further, gradient clipping sometimes offers improvements under high noise; however, the performance is far inferior to other losses, which is in keeping with their robustness guarantees. Indeed, the linear loss, which is provably robust to symmetric noise, generally performs well even when ρ = 0.6. However, optimisation under this loss is more challenging, since the gradient does not account for instances' importance (per §4.3). This is particularly reflected on the CIFAR-100 dataset, where this loss suffers to learn even under no noise. The GCE and partially Huberised losses do not suffer from this issue, even at high noise levels. Generally, the partially Huberised losses are competitive with or improve upon the counterpart losses they build upon. In particular, the partially Huberised CE performs much better than the CE under high noise, while the partially Huberised GCE slightly bumps up the GCE numbers on CIFAR-100. This indicates that partially Huberised may be useful in combining with generic base losses to cope with noise. We reiterate here that our partially Huberised loss may be used in conjunction with other ideas, e.g., pruning , consensus , or abstention . We leave such exploration for future work. We established that gradient clipping by itself does not suffice to endow even simple models with label noise robustness; however, a simple variant resolves this issue. Experiments confirm that our composite loss-based gradient clipping performs well on datasets corrupted with label noise. One interesting direction for future work is to analyse the behaviour of gradient-clipping inspired losses for the more general problem of distributionally robust learning (; ; ;). Similarly, for the linear loss function lin (x, y; θ). = −m θ (x, y), we have ∇ lin (x, y; θ) = −∇m θ (x, y). To compute the normalised gradient, we need since φ (z) < 0 by assumption that φ is admissible, and thus decreasing. Consequently, if N = 1, else. Assuming a linear scorer s(x; θ) = θ T x, the score gradient ∇s(x; θ) = x, which is independent of θ. The clipped gradient thus corresponds to the gradient under a "Huberised" loss function else. Proof of Lemma 2. Since φ is strictly convex and decreasing, it must be strictly decreasing. We have where (φ) −1 exists since φ is strictly inceasing by definition of strict convexity. Now, by definition, the Huberised loss is and soφ which exactly equals clip τ (φ (z)). We remark here thatφ τ is continuous, since for any convex conjugate, Plugging in u = τ, at the intersection point z 0. −1 (−τ) of the two pieces of the function, Proof of Lemma 3. For admissible φ, the Huberised lossφ τ of is trivially convex, differentiable everywhere, and decreasing. In particular, we must haveφ τ < 0, and soφ τ must be classification calibrated by Bartlett et al. (2006, Theorem 2). As an illustration, Figure 3 shows the minimiser of the conditional risk, Note that this quantity must be non-negative if and only if η > 1 /2 for a loss to be classification calibrated. This is easily verified to be true for the Huberised logistic loss, regardless of τ. is strictly monotone and continuous. Continuity is immediate; for monotonicity, observe that by definition, For brevity, let z 0. −1 (−τ), where we take Thus, the quantity of interest isφ Since φ is strictly convex, φ is strictly decreasing and thus invertible. Thus, the ratioφ is invertible, provided z 0 ≤ 0, i.e., τ ≥ −φ. Consequently,φ τ is proper composite when τ ≥ −φ. To intuit the need for the restriction on τ, observe that by (, Corollary 12) the tentative link function for the loss is When τ < −φ, the above is seen to be non-invertible. See also Appendix C for an illustration of this link function for the logistic loss. Proof of Proposition 4. In order to apply Long & Servedio (2010, Theorem 2), we simply need to check that the lossφ τ is a convex potential in the sense of Long & Servedio (2010, Definition 1). This requires thatφ τ is convex, non-increasing, continuously differentiable, and asymptotes to zero (or equally, is bounded from below). Each of these is satisfied by assumption of φ being admissible. Proof of Lemma 5. By Lemma 2, we may write clip τ (ϕ (u)) as the derivative of a partially Huberised base lossφ τ given bỹ This induces a composite margin lossφ τ, and definẽ τ (x, y; θ). =φ τ (m θ (x, y)). The gradient under this loss is Thus, CL-gradient clipping is equivalent to using the loss˜ τ. Proof of Lemma 6. We proceed in a similar manner to Lemma 3: to show that the loss is proper, we must establish thatφ is invertible. By Reid & Williamson (2010, Corollary 14), for the margin loss φ to be proper composite with link F, it must be true that F satisfies the symmetry condition F (−z) = 1 − F (z). We thus haveφ Since F is invertible by assumption, the above is invertible if and only ifφ This quantity is invertible provided u 0 ≤ 1 2, i.e., τ ≥ −ϕ (1 /2). A subtlety, however, is that the above does not necessarily span the entire range of [0, +∞]; consequently,φ τ itself is proper composite, with a link function of its own. Even when τ is small, one may verify that the lossφ τ is nonetheless classification calibrated: this is because for any η ∈, the minimiser z * (η) of the conditional risk must satisfy the stationarity condition We thus need to find a suitable z * such that the left hand side equates to a given constant. Now, for any C = 1 there is a unique u such thatφ = C. One may verify that this z * > 0 ⇐⇒ η > 1 2; for example, see Figure 4, which visualises the risk minimiser for various values of τ. Thus, the sign of the minimising score conveys whether or not the positive class-probability is dominant; thus, the loss is classification calibrated. Proof of Proposition 7. Let R τ (f) denote the risk on the clean distribution of a predictor f with respect to the partially Huberised lossφ τ with parameter τ. Similarly, letR τ (f) denote the risk on the noisy distribution.; van , we havē That is, the risk on the noisy distribution equals a scaled version of the risk on the clean distribution, plus an additional term. This term is a constant independent of f if and only ifφ τ satisfies the symmetry condition of , namely,φ τ (u) +φ τ (1 − u) = C for some constant C. Even when the symmetry condition does not hold, one may nonetheless aim to bound this additional term as follows. For simplicity, we restrict attention here to ϕ being the log or cross-entropy loss ϕ(u) = − log u. By definition, for any f ∈, τ. Evidently, all piecewise functions involved are bounded on the respective intervals. For example, By taking the maximum of each of the quantities on the right hand side -which are constants depending on τ -we may thus find constants C 1, C 2 such that Now let f * denote the minimiser of the clean risk R τ (f), andf * the minimiser of the noisy risk R τ (f). Then, using each of the above inequalities, where the last inequality is because R τ (f *) ≤ R τ (f *) by definition of f *. The claim follows. Beyond requiring classification-calibration, it is often desirable to use classifier outputs as valid probabilities. Proper losses (; ;) ϕ: {±1}× →R + are the core losses of such class-probability estimation tasks, for which Equation 13 stipulates that when using ϕ to distinguish positive and negative labels, it is optimal to predict the positive class-probability. Typically, it is more useful to work with losses that accept real-valued scores, e.g. as output by the pre-activation of the final layer of a neural network. To this end, proper composite losses Given a proper loss ϕ and "symmetric" link F with F (−v) = 1−F (v), the loss (y, v) = ϕ(y, F (v)) defines a margin loss (, Corollary 14). Proper composite losses may also be extended to multiclass settings in the natural way : one now defines a proper loss ϕ: where K is the number of classes and ∆ K denotes the K-simplex. A proper composite loss may be defined using a link F:. Combined with the log-loss ϕ(y, p) = − log p y, this yields the standard softmax cross-entropy loss. We illustrate the Huberised, partially Huberised, and generalised cross-entropy losses as their underlying tuning parameters are varied. Additionally, we illustrate the link functions that are implicit in each of the losses, which illustrates that they may be non-invertible if τ is too large. C.1 HUBERISED LOSS Figure 5 illustrates the Huberised version of the logistic loss, and its derivative. Following the proof of Lemma 3, for τ ∈ and z 0 = −σ −1 (τ), the Huberised lossφ τ has an implicit link function (see Figure 6) Compared to the standard sigmoid, the Huberised link saturates more slowly as τ is decreased. Note that when τ ≤ 1 2, the link function is not invertible everywhere: this in the loss not being proper composite per our definition. Figure 7 illustrates the partially Huberised version of the logistic loss, as well as the base log-loss. Following the proof of Lemma 6, the partially Huberised lossφ τ has implicit link functioñ Figure 9 illustrates the link function. The logistic loss has ϕ(u) = − log u, and so ϕ (1 /2) = −2. For τ = 1, the link function will be non-invertible everywhere, which is expected since the loss here is the linear loss, which is not suitable for class-probability estimation. For τ ∈, the link function will be invertible for p / ∈ 1 − 1 τ, 1 τ. Intuitively, the case τ ∈ corresponds to the linear regions of the losses on positive and negative instances crossing over. For τ ≥ 2, the link function will always be invertible. It may be observed that partial Huberisation causes the link function to saturate at values [1/(1 + τ), τ /(1 + τ)]: this does not affect classification calibration, but does imply that rescaling is necessary in order to intepret the output probabilities. Figure 8 illustrates the base ϕ α loss, and its composition with a sigmoid link function., the link function is not invertible everywhere. = φ(y · θ T x) with empirical risk minimiserθ N on a sample {(x n, y n)} N n=1, suppose that the sample is corrupted with an outlier (x, y). One would like to ensure thatθ N is not swayed by making x 2 arbitrarily large: that is, ∇ (x n, y n ;θ N) + ∇ (x, y ;θ N) = 0, or equivalently, lim x 2 →+∞ ∇ (x, y ;θ N) = 0. Fortunately, the saturating behaviour of the partially Huberised loss affords this, as we now show. Proposition 8. Pick any convex, differentiable, proper composite margin loss φ, whose link F satisfies lim z→−∞ z · F (z) = 0. For any τ > 0, let˜ τ (x, y; θ). =φ τ (y · θ T x) be the τ -partially Huberised loss under a linear model. Then, for any (x, y) and θ such that θ 2 < +∞ and θ T x = 0, Proof of Proposition 8. By definition, Thus, where ψ x,θ denotes the angle between x and θ. If θ T x = 0, then cos ψ x,θ = 0, and so the third term is finite. Thus, ∇˜ τ (x, y; θ) 2 → lim z→±∞ |z|·φ τ (z), depending on the sign of y ·θ T x. By definition ofφ τ, the derivative of the loss asymptotes to either F (v), or φ (v). Now, lim z→−∞ z · F (v) = 0 by assumption, and lim z→+∞ z · φ (v) = 0 since φ is convex, the claim is shown. We provide some more details regarding the synthetic data used in the body, as well as on an additional two-dimensional synthetic dataset. The problem considered in (see Figure 10) comprises a distribution concentrated on six atoms {±, ±(γ, 5γ), ±(γ, −γ)} ⊂ R 2 for some γ > 0; we chose γ = 1 24. An instance (x 1, x 2) is labelled as y = x 1 ≥ 0. The instances are weighted so that the first four atoms have probability mass 1 8, and the last two atoms mass 1 4. We modify this distribution slightly by treating the atoms as means of isotropic Gaussians, and treating the marginal distribution over instances to be a mixture of these Gaussians with mixing weights given by the corresponding probability masses of the atoms. For the experiment involving outliers in feature space, the data comprises points on the line. Positively labelled samples are drawn from a unit variance Gaussian centered at, with positively labeled outliers drawn from (−200, 1). Negatively labelled samples comprise the negation of all points. We learn an unregularised linear classifier from this data, which corresponds to a single scalar θ. We further illustrate the differences amongst methods on a 2D dataset inspired by Amid et al. (2019a). The data comprises 500 points, falling into two bands (Figure 11). The decision boundaries for various losses, when trained with a linear model using explicit quadratic features, are shown in Figure 12. We subject the data to 45% symmetric label noise. We see that the logistic and generalised cross-entropy losses see marked changes in their decision boundaries. By contrast, the partially Huberised loss maintains the correct classification boundary., which defeats any member of a broad family of convex losses. The data comprises six points, with the blue points labelled positive, and the red points labelled negative. The two "fat" points have twice as much probability mass as their "thin" counterparts. While the dataset is trivially linearly separable, minimising a broad range of convex losses with a linear model under any non-zero amount of symmetric label noise in a predictor that is tantamount to random guessing. On the clean version of the data, all losses yield roughly equitable decision boundaries. However, when adding 45% symmetric label noise, the logistic loss sees marked changes to its boundary. The partially Huberised loss maintains the correct classification boundary.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Gradient clipping doesn't endow robustness to label noise, but a simple loss-based variant does.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: Among deep generative models, flow-based models, simply referred as \emph{flow}s in this paper, differ from other models in that they provide tractable likelihood. Besides being an evaluation metric of synthesized data, flows are supposed to be robust against out-of-distribution~(OoD) inputs since they do not discard any information of the inputs. However, it has been observed that flows trained on FashionMNIST assign higher likelihoods to OoD samples from MNIST. This counter-intuitive observation raises the concern about the robustness of flows' likelihood. In this paper, we explore the correlation between flows' likelihood and image semantics. We choose two typical flows as the target models: Glow, based on coupling transformations, and pixelCNN, based on autoregressive transformations. Our experiments reveal surprisingly weak correlation between flows' likelihoods and image semantics: the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invariant transformations~(SITs). We explore three SITs~(all small pixel-level modifications): image pixel translation, random noise perturbation, latent factors zeroing~(limited to flows using multi-scale architecture, e.g. Glow). These findings, though counter-intuitive, resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels. So flows' likelihoods, modeling on pixel-level intensities, is not able to indicate the existence likelihood of the high-level image semantics. We call for attention that it may be \emph{abuse} if we use the predictive likelihoods of flows for OoD samples detection. Deep generative models have been very successful in image generation (; ;), natural language generation , audio synthesis and so on. Among them, generative adversarial networks (GANs) are implicit generative models that explicit likelihood function is not required, and are trained by playing a minimax game between the discriminator and the generator; Variational auto-encoders are latent variable generative models optimized by maximizing a lower bound, called evidence lower bound, of the data log-likelihood. Flow-based models (; differ from them in that they provide exact log-likelihood evaluation with change of variables theorem . A flow usually starts with a simple base probability distribution, e.g. diagonal Gaussian, then follows a chain of transformations in order to approximate complex distributions. Each transformation is parameterized by specially designed neural networks so that the log-determinant of its Jacobian can be efficiently computed. Most of the previous works focus on how to design more flexible transformations to achieve tighter log-likelihoods, and generate more realistic samples. It is also believed that flows can be used to detect out-of-distribution(OoD) samples by assigning low likelihoods on them. However, it has been observed that flows fail to do so. For example, flows trained on FashionMNIST surprisingly assign higher likelihoods on MNIST samples . Though analyses on pixel-level statistics are performed on this phenomenon , and density evaluation combined with uncertainty estimation is used to detect OoD samples , the reasons behind flows' counter-intuitive behaviours are still not clear. Humans easily discriminate MNIST images from FashionMNIST images, since their high-level image semantics are perceptually different. Accordingly, it takes some metrics that can reflect the high-level image semantics for OoD detection. In this paper, we empirically explore the correlation between flows' likelihoods and image semantics, and question the rationality and applicability of using predictive likelihoods of flows for OoD detection. We first introduce a concept of semanticinvariant transformation (SIT). An SIT transforms an input without changing its high-level semantics, e.g. a dog image through an SIT is still supposed to be recognized as a dog. We choose two typical flow-based models as target models: Glow , based on coupling transformations, and pixelCNN, based on autoregressive transformations. We evaluate on image datasets MNIST and FashionMNIST under three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (specific to invertible flows using multi-scale architectures, e.g. Glow). We demonstrate that the predictive likelihoods of the target models show weak correlation to the image semantics in the following ways: • Small pixel translations of test images could in obvious likelihood decreases of Glow. • Perturbing small random noises, unnoticeable to humans, to test images could lead to catastrophic likelihood decreases of target models. This also applies even if we keep the semantic object of a test image intact, and only add noises to the . • For an invertible flow using multi-scale architecture, e.g. Glow, the inferred latent variables of an image is a list of gaussianized and standardized factors. We find that the contributions of a flow's blocks to the log-likelihood are constant and independent of inputs. Thus, simply zeroing the preceding latent factors of a sample image, and feed them to flow's reverse function. We could obtain new samples with surprisingly higher likelihoods, yet with perceptually unnoticeable changes from the original image. We emphasize that all these SITs are small pixel-level modifications on test images, and undoubtedly have no influences on humans' recognition of the semantic objects in the images. However, they lead to obvious inconsistency of flows' likelihoods on test samples. Considering that the predictive likelihood of a flow is the joint probability of all the image pixels, it may not convincingly indicate the existence of a semantic object in an image. Thus it could be problematic to use flows for downstream tasks which require metrics that can reflect image semantics, e.g. OoD detection. 2.1 CHANGE OF VARIABLES THEOREM Given a random variable z with probability density function p(z), after applying an invertible function f: R D → R D on z, we get a new random variable z = f (z). Then probability density function of the changed variable z is given by: We can construct arbitrarily complex probability distributions by transforming a simple base distribution p(z 0) with a chain of mappings f k of length K. Then we have: Flow-based models are generative models designed by applying the above theorem, thus exact loglikelihood evaluation of data is feasible. Then the practical problem of building flow-based models applied on high-dimensional data, like images, becomes how to design invertible transformations whose Jacobian determinant can be efficiently computed. Research on flows is very active and rapidly evolving. In this paper, we particularly focus on flowbased generative models on images and the behaviours of their likelihoods. They can roughly be divided into two categories according to the granularity of the transformation layers: The affine coupling is proposed in Real NVP , whose Jacobian is a lower triangular matrix that can be efficiently computed. An earlier and simpler version is additive coupling proposed in NICE , which can be obtained by simply removing the scale item exp s(x 1:d) in affine coupling. Additive coupling layer is volume-preserving and the log-determinant of its Jacobian is always 0. Glow improves Real NVP by replacing the fixed shuffling permutation with 1 × 1 invertible convolution. Since forward and inverse operation of a coupling layer have the same computational efficiency, both likelihood evaluation and sampling(or generation) for coupling flows are equally efficient. Autoregressive Flow As the building blocks of autoregressive flow, autoregressive transformations model the joint probability p(x) as the product of one-dimensional conditionals: where the probability of observation x i conditions only on its previous observations x <i. The autoregressive property of an autoregressive layer is enforced by specially designed mechanism, e.g. masking. In PixelCNN (van den, this is implemented as masked convolutional layers, which are inherently easier to be parallelized than its counterpart PixelRNN . The likelihood evaluation of PixelCNN takes only one-forward pass, but its inference, i.e. generation, takes O(D), since we have to sample pixel-by-pixel. PixelCNN can be further parallelized (b) to accelerate its inference speed. Flows for generation Serving as powerful decoders, PixelCNN can also be combined with other generative models, e.g. PixelGAN and PixelVAE . Variants of PixelCNN are also used to model audio, video , and text. PixelCNN, combining with attention, is also applied to few-shot autoregressive density estimation Reed et al. (2017a). proposes several improvements to coupling flow, reducing its gap to autoregressive flow in terms of density estimation. Autoregressive models for density estimation Autoregressive models can be specially designed for general-purpose density estimation. Masked Autoencoder for Distribution Estimation is a pioneering work that use masked neural networks to model the autoregressive density. MADE constitutes the building block of two popular normalizing flows: Inverse Autoregressive Flow and Masked Autoregressive Flow . IAF and MAF are similar but with different computational trade-offs. IAF, providing efficient sampling, is designed to improve the expressiveness of the approximate posterior of VAE. MAF is a more powerful density estimator which stacks multiple MADEs. Let g be a flow-based generative model trained on dataset sampled from some unknown p(x), and p g (x) be the g's predictive probability density function of sample x. Semantic-Invariant Transformation (SIT) Roughly speaking, SIT can be any transformation that do not change humans' recognition of image semantics. For example, suppose x is a dog image. After applying SIT T to x, T (x) is supposed to be high recognized as a dog image. As a proof of concept evaluation, we limit our evaluations to three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (specific to Glow). We probe the correlation between the predictive likelihood of a flow g and image semantics by examining the influences of SITs on test samples' likelihoods. Specifically, a reasonable observation we should expect is that for SIT T: p g (T (x)) − p g (x) < δ holds for a small positive scalar δ. We report bits-per-dim (BPD), which is given by: where NLL is the negative log-likelihood of the test sample, h, w, c are height, width, number of channels. Lower BPD implies higher likelihood. Throughout this paper, we use BPD and likelihood interchangeably. We refer to Supp. A for setup and training details of the target models. Translation invariance is a fundamental property in learning image representations that are robust for downstream tasks. In this section, we evaluate the influences of image translations on flows' likelihoods. The in Fig. 1 and examples in Fig. 2 show that even 1 or 2-pixel left translation could lead to obvious increase of Glow's predictive BPDs, while the PixelCNN's predictive BPDs are robust to pixel translation. This surprising difference can be attributed to the difference of their architectures. Glow, like other flows based on coupling transformation layers , models the joint probability of the pixels in a coarse-grained way. They rely on multi-scale architecture modeling different level of abstractions in order to achieve competitive BPDs. At higher scale levels, the intermediate tensors have smaller spatial sizes and bigger channel sizes. This is performed at the starting point of each scale level with squeeze operation, which trades spatial sizes for channel sizes by transforming a h × w × c tensor into a h/2 × w/2 × 4c tensor. Note that squeeze operation actually destroys the spatial positions of adjacent pixels, and 1-pixel translation could lead to quite different spatial partitions. While for PixelCNN, the intermediate tensors are not reshaped, and the spatial positions of pixels are kept still; Furthermore, the prediction of each pixel conditions only on a neighborhood of (previous) pixels in masked convolution, so translation invariance is preserved. Problematic Likelihood Comparisons The foundation of using likelihood-based models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples x out than indistribution samples x in, i.e. p g (x in) p g (x out). However, it has been observed that flows assign higher likelihoods on OoD samples than even training samples . Analyses of pixel-level datasets statistics in show that this may be due to OoD datasets just "sit inside of" in-distribution datasets with roughly the same mean, smaller variance. Surprisingly, similar counter-intuitive likelihood assignment also occurs in in-distribution samples. For example, in Fig. 1, images with class label 1 are consistently have significantly lower BPDs, i.e. higher likelihoods than samples of other classes. In OoD detection, we assume that a sample with a higher likelihood indicates that it is more likely to be an in-distribution sample. Following the same logic, this tells that all images of class label 1 are more likely to be in-distribution samples than samples from other classes, which contradicts the fact that they are all in-distribution samples for sure. We may reasonably suspect that flows' counter-intuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics, e.g. different numbers. This kind of counter-intuitive likelihood comparisons exist not only between in-distribution and OoD samples, but also within in-distribution samples from different classes. find that 1-pixel shift of the images could lead to quite different nearest neighbours from the training set, measured in Euclidean distance, which also demonstrate the gap between pixel-level metrics and humans' perception. Image pixels are discrete integers; and in practice, right amount of real-valued uniform noise is added to dequantize the pixels. For images x ∈ [0 Humans can robustly recognize semantic objects in images regardless of the s. So we also evaluate influences of adding random perturbations to only the s of test images. This can be simply implemented using proper mask: where m is the mask, and is a small scaling factor ensuring the noise is small enough. In our evaluations, we use unit Gaussian noises, and set the scaling factor = 0.001. Examples in Fig. 3 manifest that adding small noises catastrophically lower samples' likelihoods. Compared to Glow, PixelCNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the influences of the added noise. We get similar even if we keep the semantic objects of test images intact, and add noises only to the s. Note that the Gaussian noises · N we added are out of the coverage (with < 0 elements) of the uniform noises added during training, so theoretically this is expected since models are not optimized in that areas. However, it does reveal that flows are not aware of the image semantics, and treat the pixels of objects and the pixels of s with no discrimination. Other tested noises include ·(1/256+ N) and · [−1/256, 0], and similar were obtained. Algorithm 1 Generate x * by zeroing the latent factors Infer the latent factors. 3: z * ← zero(z, k) Zero the preceding k factors. 4: x * ← g.reverse(z *, y) Reverse the zeroed latent factors. 5: return x * Let us first decompose the Glow architecture into blocks and review their contributions to the final log-likelihood. A Glow consists of a sequence of modules at different scale levels. At each scale level, it starts with a squeeze operation, which reshapes the intermediate tensor without contribution to the log-likelihood. Following each squeeze operation is a stack of step flow blocks. A step flow block consists of three layers: actnorm, invertible 1 × 1 convolutional layer, and coupling layer. The log-determinants of actnorm and 1×1 convolutional layer are input-independent, and depend only on their inner weights (see Table 1 in ). Additive coupling layer is volumepreserving whose log-determinant is 0, thus is also input-independent. For affine coupling layer, its log-determinant depends on the affined half, but is quantitatively small. Compared to Glow with additive coupling layers, affine layers bring only a small improvement of < 0.05 BPD . Then, the intermediary tensor is split into two halves along the channel dimension. One halve is gaussianized with a convolutional block, then the other halve is factored out after being standardized. This procedure significantly reduces the amount of computation and memory. We refer to for more details due to limited space. So for a Glow using additive couplings, the cumulative log-determinant of the flow blocks within a particular scale-level is constant regardless of different samples. Only the log-determinants of gaussianized factorings (between the transitions of different scale-levels) depend on the individual inputs. Denote z = {z 1, . . ., z L} as the latent variables of input image x. Each z i is the standardized vector of i-th scale level. This simply means that a sample x whose latent variables z are close to center 0 will have a higher log-likelihood. Empirically, it also applies to Glow using affine coupling layers, since the influence of varying the latent variables on the log-determinants of affine coupling layers is quantitatively small compared to its gain. We could make use of this property to generate samples with higher likelihoods via the invertibility of Glow for free (see Algorithm 1, label y is optional). We find that the semantic object of a test image depends heavily on the last factored latent z L, rather than the preceding factors. Examples in Fig. 5 show that zeroing the preceding 1 and 2 latent factors could give us samples with obviously lower BPDs but without obvious changes (slightly faded pixel intensities) of the semantic objects. Results in Fig. 4 show that zeroing the first latent factor give the maximum increment of likelihoods. We also evaluate the influences of these SITs on the performance of discriminative classifiers. In contrast to the obvious change of flows' likelihoods, these small perturbations could decrease the testing accuracies of classifiers to some insignificant, or negligible (on MNIST) extent (see Tab 1). The discriminative classifier used here is a shallow residual network of 8 layers on 32 × 32 images in the structure specified in (Difference to Adversarial Examples Both the SITs we use in this paper and adversarial perturbations are small perturbations, but they are inherently different. Adversarial perturbations are intentionally crafted to cause misbehaviors, which are usually specific to individual images, and easily fool a classifier with almost 0% accuracy. While three SITs above are universal transformations over all images that take no additional computations and basically come for free. What is the problem of likelihood-based generative models? Discriminative classifiers, trained to extract class-relevant features, are known to be vulnerable to adversarial examples, and give over-confident predictions even for OoD samples. Generative models are supposed to be more robust since they model every pixel information of an image. However, likelihood modeling in high-dimensional space can be hard and lead to counter-intuitive observations. It was observed that likelihood-based generative models can assign even higher likelihoods on OoD samples . observe this phenomenon on both flows and VAEs. They decompose the change-of-variable theorem and investigate the influences of different transformation layers, find that the phenomenon still exists regardless of whether the transformation is volume-preserving or not. Their second-order analysis on pixel statistics suggests that OoD datasets, e.g. MNIST, just sit inside of in-distribution datasets, e.g. FashinMNIST, with roughly the same mean, smaller variance. They suspect that flows may simply fit the pixel intensities without really capture the high-level semantics. find that the likelihood of an image is mostly dominated by the irrelevant pixels, and propose a remedy to correct the original likelihood with a likelihood ratio. Though significantly improves the accuracy of OoD detection, but still fail to answer the question: whether the likelihood ratio shows high correlation to high-level semantics. This paper differs from previous works and step further to explore the correlations between the likelihood of flow-based generative models and image semantics. Theoretical analyses in (; van den) point out an important argument that generative models' ability to produce plausible samples is neither sufficient nor necessary for high likelihood. Results in this paper provide more experimental evidences for this simple argument that even for powerful exact likelihood-based generative models-flows, the likelihoods of samples can be largely weakly correlated to the high-level image semantics. Thus, special attention should be paid to this argument before we apply likelihood-based generative models to downstream tasks. For example, considering the weak correlation between flows' likelihoods and image semantics, it may be inappropriate to use them for OoD samples detection. On the other hand, these counter-intuitive behaviours of flows raise our awareness of the gap between the predictive likelihoods of flows and the expectation that these likelihoods can closely relate to the semantics for OoD detection. What is exactly the likelihood of a image? We should keep in mind that the predictive likelihood of a flow is the joint probability of all the image pixels. There is no doubt that flows, trained by maximizing its likelihood, could generate impressive synthesized data. There seem to be no problem that in terms of image generation, we expect that every single generated pixel in a image is the most likely one (hinging on its contextual pixels). However, the likelihood is explicitly modeled on pixels, so can be easily influenced by pixel-level modifications. Images' likelihoods significantly decrease even small noises are added to the pixels of s. For downstream tasks that need some "likelihood" to indicate the object in an image is a cat, rather than a car, the pixels of s are almost irrelevant. This drive us to think that we may need to model likelihood in some kind of semantic space or with some "perceptual" metrics, rather than on raw pixels. One promising direction is to define likelihood of an images on its high-level representation, and successful examples are (; Nilesh A.). The target models Glow and PixelCNN are implemented in Pytorch 1.0, and the implementation of PixelCNN is based on https://github.com/pclucas14/pixel-cnn-pp. Though several modifications proposed in PixenCNN++ are used in our implementation, we still use PixelCNN to annotate our model. We implement both unconditional and conditional versions of Glow. We report the of conditional Glow in our experiments. For Glow, using multi-scale architecture with more levels is critical to achieve lower BPDs. We resize the FashionMNIST, MNIST images from 28 × 28 to 32 × 32 and set the number of levels to 5. So the last factored the latent variables will be 1×1 in the spatial size. We provide pretrained models in code package as supplementary material for reproducing the (We only provide models on MNIST due to the upload size limit). For PixelCNN, both 28 × 28 and 32 versions are provided. In image translation experiments, we report the BPDs of PixelCNN on MNIST images of size 28×28 in the paper. Unlike Glow, PixelCNN treat the pixels as discretized intensity levels, and resizing the image size to 32 × 32 could lead to dangling intensity levels. We also find that if performing image translation experiment on PixelCNN(32 × 32), 1-pixel or 2-pixel translation will also lead to considerably BPD decreases.

show experimental evidences about the weak correlation between flows' likelihoods and image semantics.

represent the tldr for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.