Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN. Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters. CrescendoNet provides a new way to construct high performance deep convolutional neural networks without residual connections. Moreover, through investigating the behavior and performance of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior, which differs from the FractalNet that is also a deep convolutional neural network without residual connections. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training. Deep convolutional neural networks (CNNs) have significantly improved the performance of image classification BID3 BID25. However, training a CNN also becomes increasingly difficult with the network deepening. One of important research efforts to overcome this difficulty is to develop new neural network architectures BID6 BID14.Recently, residual network BID3 and its variants BID8 have used residual connections among layers to train very deep CNN. The residual connections promote the feature reuse, help the gradient flow, and reduce the need for massive parameters. The ResNet BID3 and DenseNet BID8 achieved state-of-the-art accuracy on benchmark datasets. Alternatively, FractalNet BID14 expanded the convolutional layers in a fractal form to generate deep CNNs. Without residual connections BID3 and manual deep supervision BID15, FractalNet achieved high performance on image classification based on network structural design only. Many studies tried to understand reasons behind the representation view of deep CNNs. BID27 showed that residual network can be seen as an ensemble of relatively shallow effective paths. However, BID2 argued that ensembles of shallow networks cannot explain the experimental of lesioning, layer dropout, and layer reshuffling on ResNet. They proposed that residual connections have led to unrolled iterative estimation in ResNet. Meanwhile, BID14 speculated that the high performance of FractalNet was due to the unrolled iterative estimation of features of the longest path using features of shorter paths. Although unrolled iterative estimation model can explain many experimental , it is unclear how it helps improve the classification performance of ResNet and FractalNet. On the other hand, the ensemble model can explain the performance improvement easily. In this work, we propose CrescendoNet, a new deep convolutional neural network with ensemble behavior. Same as other deep CNNs, CrescendoNet is created by stacking simple building blocks, called Crescendo blocks FIG0 ). Each Crescendo block comprises a set of independent feed-forward paths with increased number of convolution and batch-norm layers (a). We only use the identical size, 3 × 3, for all convolutional filters in the entire network. Despite its simplicity, CrescendoNet shows competitive performance on benchmark CIFAR10, CI-FAR100, and SVHN datasets. Similar to FractalNet, CrescendoNet does not include residual connections. The high performance of CrescendoNet also comes completely from its network structural design. Unlike the FractalNet, in which the numbers of convolutional layers and associated parameters are increased exponentially, the numbers of convolutional layers and parameters in Crescendo blocks are increased linearly. CrescendoNet shows clear ensemble behavior (Section 3.4). In CrescendoNet, although the longer paths have better performances than those of shorter paths, the combination of different length paths have even better performance. A set of paths generally outperform its subsets. This is different from FractalNet, in which the longest path alone achieves the similar performance as the entire network does, far better than other paths do. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new pathwise training procedure, in which paths in each building block are trained independently and sequentially. The path-wise procedure can reduce the memory needed for training. Especially, we can reduce the amortized memory used for training CrescendoNet to about one fourth. We summarize our contribution as follows:• We propose the Crescendo block with linearly increased convolutional and batch-norm layers. The CrescendoNet generated by stacking Crescendo blocks further demonstrates that the high performance of deep CNNs can be achieved without explicit residual learning.• Through our analysis and experiments, we discovered an emergent behavior which is significantly different from which of FractalNet. The entire CrescendoNet outperforms any subset of it can provide an insight of improving the model performance by increasing the number of paths by a pattern.• We introduce a path-wise training approach for CrescendoNet, which can lower the memory requirements without significant loss of accuracy given sufficient data. Crescendo Block The Crescendo block is built by two layers, the convolution layer with the activation function and the following batch normalization layer BID10. The convolutional layers have the identical size, 3 × 3. The Conv-Activation-BatchNorm unit f 1, defined in the Eq.1 is the base branch of the Crescendo block. We use ReLU as the activation function to avoid the problem of vanishing gradients BID17. DISPLAYFORM0 The variable z denotes the input feature maps. We use two hyper-parameters, the scale S and the interval I to define the structure of the Crescendo block H S. The interval I specifies the depth difference between every two adjacent branches and the scale S sets the number of branches per block. The structure of the n th branch is defined by the following equation: DISPLAYFORM1 where the superscript nI is the number of recursion time of the function f 1. The structure of Crescendo block H S can be obtained below: DISPLAYFORM2 where ⊕ denotes an element-wise averaging operation. Note that the feature maps from each path are averaged element-wise, leaving the width of the channel unchanged. A Crescendo block with S = 4 and I = 1 is shown in FIG0.The structure of Crescendo block is designed for exploiting more feature expressiveness. The different depths of parallel paths lead to different receptive fields and therefore generate features in different abstract levels. In addition, such an incremental and parallel form explicitly supports the ensemble effects, which shows excellent characteristics for efficient training and anytime classification. We will explain and demonstrate this in the following sections. CrescendoNet Architecture The main body of CrescendoNet is composed of stacked Crescendo blocks with max-pooling layers between adjacent blocks (FIG0). Following the main body, like most deep CNNs, we use two fully connected layers and a soft-max layer as the classifier. In all experiments, the two fully connected layers have 384 hidden units and 192 hidden units respectively. The overall structure of CrescendoNet is simple and we only need to tune the Crescendo block to modify the entire network. To reduce the memory consumption during training CrescendoNet, we propose a path-wise training procedure, leveraging the independent multi-path structure of our model. We denote stacked Conv-BatchNorm layers in one Crescendo block as one path. We train each path individually, from the shortest to the longest repetitively. When we are training one path, we freeze the parameters of other paths. In other words, these frozen layers only provide learned features to support the training. FIG1 illustrates the procedure of path-wise training within a CrescendoNet block containing four paths. There are two advantages of path-wise training. First, path-wise training procedure significantly reduces the memory requirements for convolutional layers, which constitutes the major memory cost for training CNNs. For example, the higher bound of the memory required for computation and storage of gradients using momentum stochastic gradient descent algorithms can be reduced to about 40% for a Crescendo block with 4 paths where interval = 1. Second, path-wise training works well with various optimizers and regularizations. Even dropout and drop-path can be applied to the model during the training. Dropout and drop-connect BID28, which randomly set a selected subset of activations or weights to zero respectively, are effective regularization techniques for deep neural networks. Their variant, drop-path BID14, shows further performance improvement by dropping paths when training FractalNet. We use both dropout and drop-path for regularizing the Crescendo block. We drop the branches in each block with a predefined probability. For example, given drop-path rate, p = 0.3, the expectation of the number of dropped branches is 1.2 for a Crescendo block with four branches. For the fully connected layers, we use L2 norm of their weights as an additional term to the loss. We evaluate our models with three benchmark datasets: CIFAR10, CIFAR100, and Street View House Numbers (SVHN) BID19. CIFAR10 and CIFAR100 each have 50,000 training images and 10,000 test images, belonging to 10 and 100 classes respectively. All the images are in RGB format with the size of 32 × 32-pixel. SVHN are color images, with the same size of 32 × 32-pixel, containing 604,388 and 26,032 images for training and testing respectively. Note that these digits are cropped from a series of numbers. Thus, there may be more than one digit in an image, but only the one in the center is used as the label. For data augmentation, we use a widely adopted scheme BID16 BID14 BID6 b; BID24 BID21 BID3. We first pad images with 4 zero pixels on each side, then crop padded images to 32 × 32-pixel randomly and horizontally flipping with a 50% probability. We preprocess each image in all three datasets by subtracting off the mean and dividing the variance of the pixels. We use Mini-batch gradient descent to train all our models. We implement our models using TensorFlow distributed computation framework BID0 and run them on NVidia P100 GPU. We also optimize our models by adaptive momentum estimation (Adam) optimization BID11 and Nesterov Momentum optimization BID18 respectively. For Adam optimization, we set the learning rate hyper-parameter to 0.001 and let Adam adaptively tune the learning rate during the training. We choose the momentum decay hyper-parameter β 1 = 0.9 and β 2 = 0.999. And we set the smoothing term = 10 −8. This configuration is the default setting for the AdamOptimizer class in TensorFlow. For Nesterov Momentum optimization, we set the hyper-parameter momentum = 0.9. We decay the learning rate from 0.1 to 0.01 after 512 epochs for CIFAR and from 0.05 to 0.005, then to 0.0005, after 42 epochs and 63 epochs respectively for SVHN. We use truncated normal distribution for parameter initialization. The standard deviation of hyper-parameters is 0.05 for convolutional weights and 0.04 for fully connected layer weights. For all datasets, we use the batch size of 128 on each training replica. For the whole net training, we run 700 epochs on CIFAR and 70 epochs on SVHN. For the path-wise training, we run 1400 epochs on CIFAR and 100 epochs on SVHN.Using a CrescendoNet model with three blocks each contains four branches as illustrated in FIG0, we investigate the following preliminary aspects: the model performance under different block widths, the ensemble effect, and the path-wise training performance. We study the Crescendo block with three different width configurations: equal width globally, equal width within the block, and increasing width. All the three configurations have the same fully connected layers. For the first one, we set the number of feature maps to 128 for all the convolutional layers. For the second, the numbers of feature maps are for convolutional layers in each block. For the last, we gradually increase the feature maps for each branch in three blocks to correspondingly. For example, the number of feature maps for the second and fourth branches in the second block is and. The exact number of maps for each layer is defined by the following equation: DISPLAYFORM0 where n maps denotes the number of feature maps for a layer, n inmaps and n outmaps are number of input and output maps respectively, n layers is the number of layers in the block, and i layer is the index of the layer in the branch, starting from 1.To inspect the ensemble behavior of CrescendoNet, we compare the performance of models with and without drop-path technique and subnets composed by different combinations of branches in each block. For the simplicity, we denote the branch combination as a set P containing the index of the branch. For example, P = {1, 3} means the blocks in the subnet only contains the first and third branches. The same notation is used in TAB0 and FIG2. Table 1 gives a comparison among CrescendoNet and other representative models on CIFAR and SVHN benchmark datasets. For five datasets, CrescendoNet with only 15 layers outperforms almost all networks without residual connections, plus original ResNet and ResNet with Stochastic Depth. For CIFAR10 and CIFAR100 without data augmentation, CrescendoNet also performs better than all the given models except DenseNet with bottleneck layers and compression (DenseNet-BC) with 250 layers. However, CrescendoNet's error rate 1.76% matches the 1.74% error rate of given DenseNet-BC, on SVHN dataset which has plentiful data for each class. Comparing with FractalNet, another outstanding model without residual connection, CrescendoNet has a simpler structure, fewer parameters, but higher accuracies. The lower rows in Table 1 compare the performance of our model given different configuration. In three different widths, the performance simultaneously grows with the number of feature maps. In other words, there is no over-fitting when we increase the capacity of CrescendoNet in an appropriate scope. Thus, CrescendoNet demonstrates a potential to further improve its performance by scaling up. In addition, the drop-path technique shows its benefits to our models on all the datasets, just as it does to FractalNet. Another interesting from Table 1 is the performance comparison between Adam and Nesterov Momentum optimization methods. Comparing with Nesterov Momentum method, Adam performs similarly on CIFAR10 and SVHN, but worse on CIFAR100. Note that there are roughly 60000, 5000, and 500 training images for each class in SVHN, CIFAR10, and CIFAR100 respectively. This implies that Adam may be a better option for training CrescendoNet when the training data is abundant, due to the convenience of its adaptive learning rate scheduling. The last row of Table 1 gives the from path-wise training. Training the model with less memory requirement can be achieved at the cost of some performance degradation. However, Pathwise trained CrescendoNet still outperform many of networks without residual connections on given datasets.. The show the ensemble behavior of our model. Specifically, the more paths contained in the network, the better the Table 1: Whole net classification error (%) on CIFAR10/CIFAR100/SVHN. We highlight the top three accuracies in each column with the bold font. The three numbers in the parentheses denote the number of output feature maps of each block. The plus sign (+) denotes the data augmentation. The sign (-W) means that the feature maps of layers in each branch increase as explained in the model configuration section. The compared models include: Network in Network BID24, ALL-CNN , Deeply Supervised Net BID15, Highway Network BID24, FractalNet BID14, ResNet BID3, ResNet with Stochastic Depth BID7, Wide ResNet BID30, and DenseNet BID6 performance. And the whole net outperforms any single path network with a large margin. For example, the whole net and the net based on the longest path show the inference error rate of 6.90% and 10.69% respectively, for CIFAR10 without data augmentation. This implicit ensemble behavior differentiates CrescendoNet from FractalNet, which shows a student-teacher effect. Specifically, the longest path in FractalNet can achieve a similar or even lower error rate compared to the whole net. To investigate the dynamic behavior of subnets, we test the error rate changes of subnets during the training. We use Adam to train the CrescendoNet with the structure shown in FIG0 on CIFAR10 for 450 epochs. FIG2 illustrates the behavior of different path combinations during the training. It shows that the inference accuracy of the whole net grows simultaneously with all the subnets, which demonstrates the ensemble effect. Second, for any single path network, the performance grows with the depth. This behavior of the anytime classifier is also shown by FractalNet. In other words, we could use the short path network to give a rough but quick inference, then use more paths to gradually increase the accuracy. This may be useful for time-critical applications, like integrated recognition system for autonomous driving tasks. Conventional deep CNNs, such as AlexNet VGG-19 (Simonyan & BID20, directly stacked the convolutional layers. However, the vanishing gradient problem makes it difficult to train and tune very deep CNN of conventional structures. Recently, stacking small convolutional blocks has become an important method to build deep CNNs. Introducing new building blocks becomes the key to improve the performance of deep CNN. BID16 first introduced the NetworkInNetwork module which is a micro neural network using a multiple layer perceptron (MLP) for local modeling. Then, they piled the micro neural networks into a deep macro neural network. BID25 introduced a new building block called Inception, based on which they built GoogLeNet. Each Inception block has four branches of shallow CNNs, building by convolutional kernels with size 1 × 1, 3 × 3, 5 × 5, and max-pooling with kernel size 3 × 3. Such a multiple-branch scheme is used to extract diversified features while reducing the need for tuning the convolutional sizes. The main body of GoogLeNet has 9 Inception blocks stacked each other. Stacking multiplebranch blocks can create an exponential combination of feed-forward paths. Such a structure com-bined with the dropout technique can show an implicit ensemble effect BID27 BID22. GoogLeNet was further improved with new blocks to more powerful models, such as Xception BID1 and Inception-v4 BID26. To improve the scalability of GoogLeNet, BID26 used convolution factorization and label-smoothing regularization in Inception-v4. In addition, BID1 explicitly defined a depth-wise separable convolution module replacing Inception module. Recently, BID14 introduced FractalNet built by stacked Fractal blocks, which are the combination of identical convolutional layers in a fractal expansion fashion. FractalNet showed that it is possible to train very deep neural network through the network architecture design. FractalNet implicitly also achieved deep supervision and student-teacher learning by the fractal architecture. However, the fractal expansion form increases the number of convolution layers and associated parameters exponentially. For example, the original FractalNet model with 21 layers has 38.6 million parameters, while a ResNet of depth 1001 with similar accuracy has only 10.2 million parameters BID6. Thus, the exponential expansion reduced the scalability of FractalNet. Another successful idea in network architecture design is the use of skip-connections BID3 b; BID6 BID30 BID29. ResNet BID3 used the identity mapping to short connect stacked convolutional layers, which allows the data to pass from a layer to its subsequent layers. With the identity mapping, it is possible to train a 1000-layer convolutional neural network. BID6 recently proposed DenseNet with extremely residual connections. They connected each layer in the Dense block to every subsequent layer. DenseNet achieved the best performance on benchmark datasets so far. On the other hand, Highway networks BID23 used skip-connections to adaptively infuse the input and output of traditional stacked neural network layers. Highway networks have helped to achieve high performance in language modeling and translation. CNN has shown excellent performance on image recognition tasks. However, it is still challenging to tune, modify, and design an CNN. We propose CrescendoNet, which has a simple convolutional neural network architecture without residual connections BID3. Crescendo block uses convolutional layers with same size 3 × 3 and joins feature maps from each branch by the averaging operation. The number of convolutional layers grows linearly in CrescendoNet while exponentially in FractalNet BID14. This leads to a significant reduction of computational complexity. Even with much fewer layers and a simpler structure, CrescendoNet matches the performance of the original and most of the variants of ResNet on CIFAR10 and CIFAR100 classification tasks. Like FractalNet BID14, we use dropout and drop-path as regularization mechanisms, which can train CrescendoNet to be an anytime classifier, namely, CrescendoNet can perform inference with any combination of the branches according to the latency requirements. Our experiments also demonstrated that CrescendoNet synergized well with Adam optimization, especially when the training data is sufficient. In other words, we can avoid scheduling the learning rate which is usually performed empirically for training existing CNN architectures. CrescendoNet shows a different behavior from FractalNet in experiments on CIFAR10/100 and SVHN. In FractalNet BID14, the longest path alone achieves the similar performance as the entire network, far better than other paths, which shows the student-teacher effect. The whole FractalNet except the longest path acts as a scaffold for the training and becomes dispensable later. On the other hand, CrescendoNet shows that the whole network significantly outperforms any set of it. This fact sheds the light on exploring the mechanism which can improve the performance of deep CNNs by increasing the number of paths.

We introduce CrescendoNet, a deep CNN architecture by stacking simple building blocks without residual connections.

represent the short summary for finding the corresponding scientific article (long)

------

For instance, <<We use the non-negative rank of ReLU activation matrices as a complexity measure and show it (negatively) correlates with good generalization.>>  to <<We propose a new notion of'non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our demonstrate that as an indicator for memorization, our technique can be used to perform early stopping. A fundamental challenge in machine learning is balancing the bias-variance tradeoff, where overly simple learning models underfit the data (suboptimal performance on the training data) and overly complex models are expected to overfit or memorize the data (perfect training set performance, but suboptimal test set performance). The latter direction of this tradeoff has come into question with the observation that deep neural networks do not memorize their training data despite having sufficient capacity to do so BID38, the explanation of which is a matter of much interest. Due to their convenient gradient properties and excellent performance in practice, rectified-linear units (ReLU) have been widely adopted and are now ubiquitous in the field of deep learning. In addition, the relative simplicity of this function (max(·, 0)) makes the analysis of ReLU networks more straight-forward than networks with other activation functions. We propose a new notion of'non-linearity' of a ReLU layer with respect to an input batch. We show that networks that generalize well have deep layers that are approximately linear with respect to batches of similar inputs. In contrast, networks that memorize their training data are highly nonlinear with respect to similar inputs, even in deep layers. Our method is based on the fact that the main source of non-linearity in ReLU networks is the threshold at zero. This thresholding determines the support of the ing activation matrix, which plays an important role in the analysis of non-negative matrices. As we discuss in Section 3, the non-negative rank of a matrix is constrained by the shape of the support, and is therefore indicative of the degree of non-linearity in a ReLU activation matrix with respect to the input. Although computing the non-negative rank is NP-hard , we can restrict it with approximate non-negative matrix factorization (NMF) BID20. Consequently, we propose to estimate the'non-linearity' of a ReLU layer with respect to an input batch by performing NMF on a grid over the approximation rank k, and measuring the impact on network performance. This procedure can be seen as measuring the robustness of a neural network to increasing compression of its activations. We therefore compare our NMF-based approach to two additional dimensionality reduction techniques, namely principal component analysis (PCA) and random ablations. We informally define memorization as the implicit learning of a rule that associates a specific sample (i.e., with index i) to a particular label (e.g., with index j). Such a rule does not benefit the network in terms of improving its performance on new data. We show that our NMF-based approach is extremely sensitive to memorization in neural networks. We report for a variety of neural network architectures trained on several image and audio datasets. We conduct a layer-by-layer analysis and our reveal interesting details on the internal mechanism of memorization in neural networks. Finally, as an indicator for memorization, we use our proposed measure to perform early stopping. The study of factors involved in the bias-variance tradeoff in learning models goes back several decades. Classical in statistical learning consider properties of learning models such as the VC-dimension BID33 and Rademacher complexity BID4. These properties give generalization bounds in terms of the capacity model to (over)fit data. When considering the vast capacity of deep neural networks, such bounds become irrelevant and fail to explain their ability to generalize well in practice BID38 BID5.More direct analyses have been done with respect to a specific setting of model parameters. For instance, BID3 showed that the number of weights in a network is less important compared to their scalar value (e.g. 2 -norm), and more recently BID5 presented a bound for deep neural networks based on the product of spectral norms of the network's weight matrices. BID0 showed that memorizing networks contain more information in their weights. Methods to explain generalization have been proposed that examine a network's robustness to perturbations BID13 BID6 BID14 BID26 BID22. These methods propose the notion of flatness of minima on the loss surface, assuming that perturbing the parameters without dramatically changing performance is an indicator of the generalization of a network. However, any reversible transformation, such as simple scaling, can arbitrarily manipulate the local flatness without affecting generalization BID8. The procedure we propose can be viewed as applying perturbations, albeit to activations and not parameters, and must address this concern. The perturbations we apply to activations account for magnitude, since they depend on a change of rank or non-negative rank of the activation matrix, a property which is robust to rescaling and similar reversible transformations. In contrast to the methods described thus far, which deal exclusively with the parameters of the model, methods have been developed that account for the role of the data distribution. BID23 proposed to use the Fisher-Rao norm, which uses the geometry of the data distribution to weigh the contribution of different model parameters. The empirical studies of BID24 and BID27 explore robustness to specific types of noise. The former uses Gaussian noise and masking noise injected into hidden activations, while the latter interpolates between input samples to study network behavior on and off the data manifold. In both cases, robustness to noise proved a reliable indicator for good generalization. Additionally, BID1 derive generalization bounds in terms of robustness to noise. Our experimental setup is reminiscent of BID24 in that both methods apply a form of compression to hidden activations and test for robustness to this type of noise. Specifically, they set random axis-aligned directions in feature space to zero which can be viewed as a crude form of dimensionality reduction, i.e., by simply removing canonical dimensions. In our experiments we refer to this method as random ablations. Our show that robustness to NMF compression is much more correlated with low memorization/high generalization than robustness to random ablations. BID2 have also studied various empirical aspect of memorization. As a dimensionality reduction technique, NMF has gained popularity due to its producing meaningful factorizations that lend themselves to qualitative interpretation across various domains such as document clustering BID37, audio source separation BID11, and face recognition BID12. In the context of deep convolutional neural networks, BID7 applied NMF to the activations of an image classifier and showed that the gives a decomposition into semantic parts, which benefits from the transformation invariance learned by the neural network. Consider a ReLU layer of a neural network, parameterized by a weight matrix W ∈ R m×q. For a batch of n inputs X ∈ R n×m, we compute the layer activation matrix A as follows: DISPLAYFORM0 where R + are the non-negative reals. We omit the bias term for notational convenience. The processing of a single input x by a ReLU network is equivalent to sampling a sub-network that is linear with respect to the sample BID35. This could be accomplished by simply setting to zero the columns of each W whose dot product with the input is negative (and would thus be set to zero by ReLU), and then removing the thresholding. Extending this notion to a batch of several input samples to a ReLU layer, suppose the samples are sufficiently close to each other such that they all share the same ReLU mask m ∈ {0, 1} q. In this case, we may say that the layer is linear with respect to its input batch. This is because, for the entire batch, instead of using ReLU, we could zero out a subset of columns and obtain a linear system, i.e., A = XW diag(m).For an activation matrix A (Equation 1), we consider the support M = supp(A), which we describe as a binary 0/1 matrix where M i,j = 1 where A i,j > 0. Because A is a ReLU activation matrix, the structure of M is mainly determined by the thresholding at zero.2 Because thresholding is the main source of non-linearity in a ReLU network, the support takes on a special meaning in this case. We want to characterize how close to being linear a layer is with respect to its input X by examining the support of the ing activations M. If all the rows of M are identical to a unique vector m, we can say the layer is completely linear with respect to X. In general, the'simpler' the support M, the closer to linearity the layer. One measure that captures this idea is the rectangle cover number of a matrix, rc(M), an important quantity in the study of communication complexity BID17. Also known as the Boolean rank, rc(M) is the smallest number r for which there exist binary matrices U B ∈ {0, 1} n×r, V B ∈ {0, 1} r×q such that their Boolean matrix multiplication satisfies M = U B V B. As a complexity measure for ReLU activations, rc(M) = 1 means the layer is linear with respect to its input, and higher values rc(M) imply increasing non-linearity. This is visualized in FIG0.Intuitively, imagine having to fit a layer with'ReLU switches', each of which controls a subset of weight matrix columns. In the linear case, one switch would suffice to describe the data. In the most non-linear case, we would require a switch for every column, which is also the maximal value of rc(M).Because computing the rectangle cover number rc(M) is complex, several approximations and bounds to it have been studied BID9. For the support of a non-negative matrix, a well-known upper-bound is: DISPLAYFORM0 where rank + (A) is the non-negative rank of A BID10 that is defined as the smallest number k such that there exist non-negative matrices DISPLAYFORM1 Similar to the rectangle cover number, the non-negative rank is hard-constrained by the combinatorial arrangement of supp(A), but additionally accounts for the actual value in the non-zero entries of A.While computing rank + (A) is not easier than computing rc(supp(A)), we can restrict it by performing approximate non-negative matrix factorization (NMF). For a given non-negative rank constraint k, NMF solves for: DISPLAYFORM0 with U +, V + as defined above. The U + V + =Ã k ≈ A is the closest matrix to A under the Frobenius norm that has rank + at most k. Consequently, we propose to estimate the'linearity' of a ReLU layer with respect to a batch of similar inputs by performing NMF on a grid over the non-negative rank k, and measuring the impact on network performance by observing the change in the prediction (output layer) as we change k. This procedure also addresses the fact that in practice network activations tend to be noisy, whereas supp(A) is not robust to noise, i.e., A i,j = > 0 → M i,j = 1 even for very small.Concretely, if we let A i be the activation matrix at layer i, during the forward pass, we replace the feature activations of one or several layers with their rank k NMF approximations: DISPLAYFORM1 For convolutional networks, we first reshape the tensor of feature maps from n × q × h × w to (n · h · w) × q, i.e., we flatten the batch (n) and spatial dimensions (h, w) to form a matrix with q columns, where q is the number of channels in that layer. We then inversely reshape the approximated features to continue forward propagation through the network. We now characterize the input batch, with respect to which we would like to measure layer linearity. Informally, the goal of training is to cluster together input samples that have similar (or identical) output labels,while separating them from samples of other labels. In the context of classification then, we expect therefore that from a certain depth and onward, samples of the same class will have similar activations, and thus a simpler support. In other words, while a network may exploit flexible non-linear structure to separate different classes, we expect that with respect to a single class, deep layers are approximately linear.. We analyze each layer by applying NMF compression to its activation matrix with increasing rank k, while observing the impact on classification performance. In (a) and (b) we show the k vs. accuracy curves at an layers of different depth. We can immediately see that in deep layers, networks with high memorization are significantly less robust to NMF compression, indicating higher degrees of non-linearity. Furthermore, networks trained on fully randomized labels (p = 1) behave differently than networks with partial or no randomization. By summarizing each curve in (a) and (b) by its area under the curve (AuC), we show in (c) a birds-eye view over all layers. All networks with p < 1 pass through distinct phases consisting of a feature extraction phase until conv3_1, followed by a memorization phase until conv4_2, followed by a final clustering phase. Interestingly, the case p = 1 shifts the process into earlier layers, explaining why layer-by-layer it appears as an outlier. When single-class batches are not approximately linear even in deep layers, i.e., activations are not clustered within a few linear regions, we empirically show in the next section that this behavior is indicative of memorization. We start by studying networks that have been forced into different levels of memorization due to label randomization applied to their training set BID38. The level of induced memorization is controlled by setting a probability p for a training label to be randomized, i.e., p = 0 is the unmodified dataset and p = 1 gives fully random labels. Note that the capacity of these networks is sufficiently large such that the training accuracy is 1 in all cases, regardless of the value of p. As such, we use batches of training data and observe how accuracy drop from 1 to constant prediction as we increase the level of compression. In all experiments, sampling single-class batches is done with respect to the label used for training (i.e., the random label if p > 0). We sample batches stochastically (up to the label), have found all methods discussed below to be robust to the batch size (e.g., 20-100). In all our experiments we set the batch size to 50.We perform experimental evaluations on several image datasets, namely CIFAR-10 (BID18), Fashion-MNIST BID36, SVHN BID25 ImageNet , as well as on the Urban Sounds audio classification dataset BID30. We use a fully-connected network for Fashion-MNIST and various CNN architectures for the others, which we describe in more detail the appendix. We start by analyzing the layers of an 11-layer CNN trained on CIFAR-10. We sampled 10 batches (one batch per class) of 50 images, and compressed the activation matrix at each layer individually down to various values of the non-negative rank. We then measured classification accuracy of the prediction. In this analysis we report average for 60 neural networks, ten networks (with different random initializations) trained per randomization level p. In FIG1 (a) and (b) we show k vs. accuracy curves of networks trained with increasing levels of label randomization, at an early layer (conv2_1) and a deep layer (conv4_1) respectively. We can immediately see that networks trained on fully randomized labels (p = 1) behave differently than networks with partial or no randomization. Furthermore, note that in deep layers, memorizing networks are significantly less robust to NMF compression, i.e., their activations posses a high nonnegative rank, which indicates high non-linearity with respect to the input, as discussed in Section 3.2. We can characterize each curve in (a) and (b) with a single number, its area under the curve (AuC). This allows us in FIG1 (c) to generate a single figure for all layers. Networks with p < 1 display a similar feed-forward trend up until layer conv3_1. Since these networks differ from each other in no way other than the level of label randomization on the training data, we hypothesize this to be a generic feature extraction phase common to all of them. In the next phase, until conv4_2, we see a big difference between networks, such that more memorization (higher p) is correlated with lower AuC, i.e., higher non-negative rank and hence non-linearity of those layers with respect to single-class batches. We therefore localize memorization to these layers. Lastly, the phase only of conv4_3 is where samples of the same class are clustered together, right before the final 10-dimensional classification layer (which is not shown). This final phase is in accordance with the premise that regardless of randomization level p, all of these networks achieve perfect training set accuracy. Interestingly, setting p = 1 shifts the process to earlier layers, explaining why layer-bylayer this case appears as an outlier. We compare different compression techniques with regards to detecting memorization. Given our choice of solving NMF under the Frobenius norm (Equation 3), a natural method to compare against is principal component analysis (PCA), which optimizes under the same norm but without the nonnegativity constraint on its factors. We also consider random ablations, i.e., setting a random subset of columns in the activation matrix to zero, since this technique has been used previously to detect memorization BID24.Rather than choosing a single layer, we sequentially apply compression to several layers. We target the final convolutional blocks of our CNNs, all of which contain three layers, each of which consists of 512 channels. In fully-connected networks, we applied compression to all layers. In FIG2 we give for the CIFAR-10 dataset, which confirm that NMF compression is indeed more sensitive to memorization, due to the properties of the non-negative rank discussed in Section 3.2. PCA, which is less constrained, is more'efficient' at compressing the activations, but is in turn less discriminative with respect to the level of memorization. Finally, we confirm that robustness to random ablations correlates with less memorization, however less so than NMF. It should be noted that NMF does show more variance than the other two methods, and incurs a higher computational cost, as discussed in section 6.6 in the appendix. In FIG3 we show additional for single-class NMF on three additional datasets and network architectures (described in the appendix), including a fully-connected network for Fashion-MNIST. The in (d) and (e), of applying PCA and NMF to multi-class batches, show that such batches produce activations with higher rank or non-negative rank compared to single-class batches. This is a of the network trying to separate samples of different labels. We have shown for networks forced into memorization due to label randomization. In this section we show our technique is useful for predicting good generalization in a more realistic setting, without artificial noise. In addition to the experiments below, we refer the reader to Section 6.3 in the appendix, where predict per-class generalization of a pre-trained VGG-19 network on a ImageNet classes. We trained 96 CNN classifiers on CIFAR-10, over a grid of hyper-parameter values for the batch size, weight decay and optimization algorithm, SGD vs. ADAM BID16. Following the same procedure as above, for each of the three methods, NMF, PCA, or random ablations, we computed the k vs. accuracy curves for each network, targeting its final convolutional block. In Figure 5 we compare the area under the curve (AuC) of each curve with the average generalization error on the test set. While all three methods show correlation with generalization error, NMF is most correlated with a Pearson correlation of -0.82, followed by PCA with -.064 and random ablation with -0.61. We test whether our method can detect memorization during training. Doing so would allow us to perform early stopping, i.e., stop training as memorization begins to decrease generalization. We trained CNNs on CIFAR-10 with the original labels. Each network was trained for 10K batches with a batch size of 100. We recorded the test set error every 250 batches, and tested the non-linearity of the deepest three convolutional layers using our NMF-based approach with a coarse grid on k. As before, we compute the area under each k vs. accuracy curve as in FIG2. Finally, we also computed the area under the curve produced by random ablations. Results of two instances are shown in Figure 6 (a) and (b). In these figures we compare the test loss against our single-class NMF method and random ablations. We smooth the plots using a radius of two epochs to reduce the noise. The matching-color dashed lines mark the local minima of the test loss in as well as the location of the first local maxima of the NMF and random ablation AuC curves after smoothing has been applied. We notice that the test loss minima align almost precisely with the maximum NMF AuC. We further confirm this behavior in Figure 6 (c), where we compare the stopping times of NMF and the random ablations method against the best test loss over 10 different runs. We have introduced a notion of a ReLU layer's non-linearity with respect to an input batch, which is based on its proximity to a linear system. We measure this property indirectly via NMF applied to deep activations of single-class batches. While more analysis is required before definite guarantees could be given, we find that our approach is successful in detecting memorization and generalization across a variety of neural network architectures and datasets. The exact architectures we used for each dataset are given in Table 1. We denote a linear or convolutional layer followed by a ReLU as Linear + and Conv +, respectively. It is interesting to study the impact of ablating the activation in the directions found by NMF and PCA by forward propagating the residual, i.e., DISPLAYFORM0 This is interesting because in the case of PCA, for instance, the top k directions are those that capture most of the variance in the activation matrix, and presumably the k directions found by NMF are of similar importance. This is not true for the random ablations, where the ablated directions are of no special importance. In Figure 7 we see that networks with no induced memorization that are most vulnerable to ablation of NMF and PCA direction. In other words, while non-memorizing networks are more robust to random ablations, they are not robust to ablations of specific important directions. This is in contrast to the interpretation of Morcos et al. FORMULA0 The VGG-19 model BID31, trained on ImageNet , is known for its good generalization ability, as evident by its widespread use as a general feature extractor. We use a pre-trained model here as an example of a well-generalizing network and analyze it with our method. We apply NMF compression to the three deepest convolutional layer, on activations of both singleclass batches and multi-class batches. We select 50 random classes from ImageNet and gather batches of 50 training samples from each class. In Figure 8, shown in blue, NMF applied to single-class batches, has a denoising effect and improves over the baseline accuracy of the batch, shown as a dashed line. As the constraint on k is relaxed, that accuracy drops back to its baseline level. We contrast this behavior with the one shown in green when using multi-class batches. Here, only when k is large do we regain baseline accuracy, and sensitivity to ablation is similarly diminished. This is due to the critical role non-linearity plays in separating the different classes. Ablating the NMF directions dramatically reduces classification accuracy. Finally, in Figure 8 (c) we show there is a significant per-class correlation (Pearson r = 0.78) between NMF AuC and accuracy on test accuracy on batches from the ImageNet test set. In this experiment we use NMF-based early stopping to improve neural network accuracy in the context of few-shot learning, i.e., learning with very few samples per output label. We choose this setting since it is representative of data scarcity, where one would like to use all available data for training, rather than holding some out for validation. We demonstrate this on the case of MNIST BID19 digits with only 2 samples per class, which in a training set of 20 samples. For early stopping with NMF, we set a very simple grid over k, a single point at k = 1. Thau NMf AuC thus simply becomes the training accuracy when compressed with NMF k = 1.In FIG6 it is evident that training set accuracy with NMF k = 1 shows similar gradients as the test set accuracy. Based on this observation, as before, we extract the first peak of the smoothed NMF curve, and stop there. Results are shown in Table 2 for 10 runs with randomly sampled training sets. We compare accuracy at our early stopping point to the best test set accuracy detected throughout the run (Best case), the average test set accuracy where the training set accuracy is 1 (Average case), and similarly the lowest test set accuracy where the set accuracy is 1 (Worst case).: NMF early stopping for few-shot learning of MNIST digits with only 20 samples. By observing the training set accuracy under NMF k = 1 compression, we are able to correctly guess the gradient of the test set accuracy. We use this to perform early stopping with the simple heuristic of stopping at the first peak, which leads to improved accuracy as shown in Tabel 2. Table 2: NMF early stopping for few-shot learning of MNIST digits with only 20 samples. By observing the training set accuracy under NMF k = 1 compression, we are able to correctly guess the gradient of the test set accuracy. Early stopping at the first peak consistently improves accuracy. In the first row of Table 2 we see that on average our method significantly improves over not using early stopping, and is on par with a recently proposed method specifically deigned for few-short learning. Furthermore, in the last two rows we show that per sampled training set, early stopping consistently improves accuracy. In FIG1 we show for every layer the area under the curve (AuC) of its k vs. classification accuracy curve. However in addition to the accuracy, the NMF reconstruction itself is also a quantity of interest. There main difficulty involved with interpreting the NMF error is scale. The error depends on the magnitude of the activations, which varies across networks, layers and even channels. In FIG0 Layer-by-layer view of (a) raw and (b) normalized NMF reconstruction errors, which NMF is trying to minimize. As we again notice the outlier behavior of networks trained with label randomization p = 1, in (c) we localize the transition between the two regimes to around p = 0.9. approximate with depth, with an interesting interaction between the memorization level and depth. The error in absolute terms echo the accuracy curve, with p = 1 again presenting outlier behavior. Returning to accuracy measurement as in 2 (c), sampling p more densely reveals in FIG0 (c) that a phase shift occurs around p = 0.9, where networks "shift" their memorization to earlier layers. Applying NMF compression to large matrices naturally incurs certain overhead. We find, however, that our implementation of the multiplicative update algorithm BID21 runs in reasonable time thanks to GPU acceleration. We report timing for typical batch used for VGG-19, i.e., 100 samples of 224×224 color images. At layer conv5_4 activations form a tensor of size 100 × 14 × 14, which we flatten to a matrix of size 19600 × 512. In FIG0 we show the timing curve for this batch as we increase k, using an NVIDIA Titan X card. As can be seen, at k = 500 processing of the batch to convergence requires 197 milliseconds on average. The final runtime depends on the number of classes sampled, and the granularity of the grid over k. We found our measurements to be robust to heavy subsampling of both.>>

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the scientific article to find a short TLDR

Question: Gaussian processes are the leading class of distributions on random functions, but they suffer from well known issues including difficulty scaling and inflexibility with respect to certain shape constraints (such as nonnegativity). Here we propose Deep Random Splines, a flexible class of random functions obtained by transforming Gaussian noise through a deep neural network whose output are the parameters of a spline. Unlike Gaussian processes, Deep Random Splines allow us to readily enforce shape constraints while inheriting the richness and tractability of deep generative models. We also present an observational model for point process data which uses Deep Random Splines to model the intensity function of each point process and apply it to neuroscience data to obtain a low-dimensional representation of spiking activity. Inference is performed via a variational autoencoder that uses a novel recurrent encoder architecture that can handle multiple point processes as input. Gaussian Processes (GPs) are one of the main tools for modeling random functions BID24. They allow control of the smoothness of the function by choosing an appropriate kernel but have the disadvantage that, except in special cases (for example BID11 ; BID9), inference in GP models scales poorly in both memory and runtime. Furthermore, GPs cannot easily handle shape constraints. It can often be of interest to model a function under some shape constraint, for example nonnegativity, monotonicity or convexity/concavity BID22 BID26 BID23 BID20. While some shape constraints can be enforced by transforming the GP or by enforcing them at a finite number of points, doing so cannot always be done and usually makes inference harder, see for example BID18.Splines are another popular tool for modeling unknown functions BID29. When there are no shape constraints, frequentist inference is straightforward and can be performed using linear regression, by writing the spline as a linear combination of basis functions. Under shape constraints, the basis function expansion usually no longer applies, since the space of shape constrained splines is not typically a vector space. However, the problem can usually still be written down as a tractable constrained optimization problem BID26. Furthermore, when using splines to model a random function, a distribution must be placed on the spline's parameters, so the inference problem becomes Bayesian. BID7 proposed a method to perform Bayesian inference in a setting without shape constraints, but the method relies on the basis function expansion and cannot be used in a shape constrained setting. Furthermore, fairly simple distributions have to be placed on the spline parameters for their approximate posterior sampling algorithm to work adequately, which in the splines having a restrictive and oversimplified distribution. On the other hand, deep probabilistic models take advantage of the major progress in neural networks to fit rich, complex distributions to data in a tractable way BID25 BID21 BID15 BID10 BID14. However, their goal is not usually to model random functions. In this paper, we introduce Deep Random Splines (DRS), an alternative to GPs for modeling random functions. DRS are a deep probabilistic model in which standard Gaussian noise is transformed through a neural network to obtain the parameters of a spline, and the random function is then the corresponding spline. This combines the complexity of deep generative models and the ability to enforce shape constraints of splines. We use DRS to model the nonnegative intensity functions of Poisson processes BID16. In order to ensure that the splines are nonnegative, we use a parametrization of nonnegative splines that can be written as an intersection of convex sets, and then use the method of alternating projections BID28 to obtain a point in that intersection (and differentiate through that during learning). To perform scalable inference, we use a variational autoencoder BID15 with a novel encoder architecture that takes multiple, truly continuous point processes as input (not discretized in bins, as is common).Our contributions are: (i) Introducing DRS, (ii) using the method of alternating projections to constrain splines, (iii) proposing a variational autoencoder model whith a novel encoder architecture for point process data which uses DRS, and (iv) showing that our model outperforms commonly used alternatives in both simulated and real data. The rest of the paper is organized as follows: we first explain DRS, how to parametrize them and how constraints can be enforced in section 2. We then present our model and how to do inference in section 3. We then compare our model against competing alternatives in simulated data and in two real spiking activity datasets in section 4, and observe that our method outperforms the alternatives. Finally, we summarize our work in section 5. Throughout the paper we will consider functions on the interval [T 1, T 2) and will select I + 1 fixed knots T 1 = t 0 < · · · < t I = T 2. We will refer to a function as a spline of degree d and smoothness s < d if the function is a d-degree polynomial in each interval [t i, t i+1) for i = 0,..., I − 1, is continuous, and s times differentiable. We will denote the set of splines of degree d and smoothness s by G d,s = {g ψ : ψ ∈ Ψ d,s}, where Ψ d,s is the set of parameters of each polynomial in each interval. That is, every ψ ∈ Ψ d,s contains the parameters of each of the I polynomial pieces (it does not contain the locations of the knots as we take them to be fixed since we observed overfitting when not doing so). While the most natural ways to parametrize splines of degree d are a linear combination of basis functions or with the d + 1 polynomial coefficients of each interval, these parametrizations do not lend themselves to easily enforce constraints such as nonnegativity BID26 ). We will thus use a different parametrization which we will explain in detail in the next section. We will denote by Ψ ⊆ Ψ d,s the subset of spline parameters that in the splines having the shape constraint of interest, for example, nonnegativity. DRS are a distribution over G d,s. To sample from a DRS, a standard Gaussian random variable Z ∈ R m is transformed through a neural network parametrized by θ, f θ: R m → Ψ. The DRS is then given by g f θ (Z) and inference on θ can be performed through a variational autoencoder BID15. Note that f maps to Ψ, thus ensuring that the spline has the relevant shape constraint. We now explain how we can enforce piecewise polynomials to form a nonnegative spline. We add the nonnegativity constraint to the spline as we will use it for our model in section 3, but constraints such as monotonicity and convexity/concavity can be enforced in an analogous way. In order to achieve this, we use a parametrization of nonnegative splines that might seem overly complicated at first. However, it has the critical advantage that it decomposes into the intersection of convex sets that are easily characterized in terms of the parameters, which is not the case for the naive parametrization which only includes the d + 1 coefficients of every polynomial. We will see how to take advantage of this fact in the next section. A beautiful but perhaps lesser known spline (see BID17) gives that a polynomial p(t) of degree d, where d = 2k + 1 for some k ∈ N, is nonnegative in the interval [l, u) if and only if it can be written down as follows: DISPLAYFORM0where [t] = (1, t, t 2, . . ., t k) and Q 1 and Q 2 are (k + 1) × (k + 1) symmetric positive semidefinite matrices. It follows that a piecewise polynomial of degree d with knots t 0,..., t I defined as p (i) (t) for t ∈ [t i−1, t i) for i = 1,..., I is nonnegative if and only if it can be written as: DISPLAYFORM1 for i = 1,..., I, where each Q1 and Q2 are (k + 1) × (k + 1) symmetric positive semidefinite matrices. We can thus parametrize every piecewise nonnegative polynomial on our I intervals with (Q DISPLAYFORM2 . If no constraints are added on these parameters, the ing piecewise polynomial might not be smooth, so certain constraints have to be added in order to guarantee that we are parametrizing a nonnegative spline and not just a nonnegative piecewise polynomial. To that end, we define C 1 as the set of (Q DISPLAYFORM3 DISPLAYFORM4 That is, C 1 is the set of parameters whose ing piecewise polynomial as in equation 2 is continuous. Analogously, let C j for j = 2, 3, . . . be the set of (Q DISPLAYFORM5 such that: DISPLAYFORM6 So that C j is the set of parameters whose corresponding piecewise polynomials have matching left and right (j − 1)-th derivatives. Let C 0 be the set of (Q DISPLAYFORM7 which are symmetric positive semidefinite. We can then parametrize the set of nonnegative splines on DISPLAYFORM8 Note that the case where d is even can be treated analogously (see supplementary material). In order to use a DRS, f θ has to map to Ψ, that is, we need to have a way for a neural network to map to the parameter set corresponding to nonnegative splines. We achieve this by taking f θ (z) = h(f θ (z)), wheref θ is an arbitrary neural network and h is a surjective function onto Ψ. The most natural choice for h is the projection onto Ψ. However, while computing the projection onto Ψ (for Ψ as in section 2.1) can be done by solving a convex optimization problem, it cannot be done analytically. This is an issue because when we train the model, we will need to differentiate f θ with respect to θ. Note that BID2 propose a method to have an optimization problem as a layer in a neural network. One might hope to use their method for our problem, but it cannot be applied due to the semidefinite constraint on our matrices. The method of alternating projections BID28 BID3 allows us to approximately compute such a function h analytically. If C 0,..., C s+1 are closed, convex sets in DISPLAYFORM0, where P j is the projection onto C j for j = 0,..., s + 1. The method of alternating projections then consists on iteratively projecting onto each set in a cyclic fashion. We call computing ψ (k) from ψ (k−1) the k-th iteration of the method of alternating projections. This method can be useful to obtain a point in the intersection if each P j can be easily computed. In our case, projecting onto C 0 can be done by doing eigenvalue decompositions of Q and zeroing out negative elements in the diagonal matrices containing the eigenvalues. While this might seem computationally expensive, the matrices are small and this can be done efficiently. For example, for cubic splines (d = 3), there are 2I matrices each one of size 2 × 2. Projecting onto C j for j = 1,... s + 1 can be done analytically as it can be formulated as a quadratic optimization problem with linear constraints. Furthermore, because of the local nature of the constraints where every interval is only constrained by its neighboring intervals, this quadratic optimization problem can be reduced to solving a tridiagonal system of linear equations of size I − 1 which can be solved efficiently in O(I) time with simplified Gaussian elimination. While the derivation of this fact is a straightforward application of the KKT conditions, the algebra is cumbersome, so we omit it here to include it in the supplementary material. By letting h be the first M iterations of the method of alternating projections, we can ensure that f θ maps (approximately) to Ψ, while still being able to compute ∇ θ f θ (z). Note that we could find such an h function using Dykstra's algorithm (not to be confused with Dijkstra's shortest path algorithm), which is a modification of the method of alternating projections that converges to the projection of BID8 BID4 BID27 ), but we found that the method of alternating projections was faster to differentiate when using reverse mode automatic differentiation packages BID0. DISPLAYFORM1 Another way of finding such an h would be unrolling any iterative optimization method that solves the projection onto Ψ, such as gradient-based methods or Newton methods. We found the alternating projections method more convenient as it does not involve additional hyperparameters such as learning rate that drastically affect performance. Furthermore, the method of alternating projections is known to have a linear convergence rate (as fast as gradient-based methods) that is independent of the starting point BID3. This last observation is important, as the starting point in our case is determined by the output off θ, so that the convergence rate being independent of the starting point ensures thatf θ cannot learn to ignore h, which is not the case for gradient-based and Newton methods (for a fixed number of iterations and learning rate, there might exist an initial point that is too far away to actually reach the projection). Finally, note that if we wanted to enforce, for example, that the spline be monotonic, we could parametrize its derivative and force it to be nonnegative or nonpositive. Convexity or concavity can be enforced analogously. Since we will use DRS as intensity functions for Poisson processes, we begin this section with a brief review of these processes. An inhomogeneous Poisson process in a set S is a random subset of S. The process can (for our purposes) be parametrized by an intensity function g: S → R + and in our case, S = [T 1, T 2). We write S ∼ PP S (g) to denote that the random set S, whose elements we call events, follows a Poisson process on S with intensity g. If S = {x k} K k=1 ∼ PP S (g), then |S ∩ A|, the number of events in any A ⊆ S, follows a Poisson distribution with parameter A g(t)dt and the log likelihood of S is given by: DISPLAYFORM0 Splines have the very important property that they can be analytically integrated (as the integral of polynomials can be computed in closed form), which allows to exactly evaluate the log likelihood in equation 5 when g is a spline. As a consequence, fitting a DRS to observed events is more tractable than fitting models that use GPs to represent g, such as log-Gaussian Cox processes BID22. Inference in the latter type of models is very challenging, despite some efforts by Cunningham et al. FORMULA1; BID1 BID19. Splines also vary smoothly, which incorporates the reasonable assumption that the expected number of events changes smoothly over time. These properties were our main motivations for choosing splines to model intensity functions. Suppose we observe N simultaneous point processes in [T 1, T 2) a total of R repetitions (we will call each one of these repetitions/samples a trial). Let X r,n denote the n-th point process of the r-th trial. Looking ahead to an application we study in the , data of this type is a standard setup for microelectrode array data, where N neurons are measured from time T 1 to time T 2 for R repetitions, and each event in the point processes corresponds to a spike (the time at which the neurons "fired"). Each X r,n is also called a spike train. The model we propose, which we call DRS-VAE, is as follows: DISPLAYFORM0 where each f (n) θ: R m → Ψ is obtained as described in section 2.2. The hidden state Z r for the r-th trial X r:= (X r,1, . . ., X r,N) can be thought as a low-dimensional representation of X r. Note that while the intensity function of every point process and every trial is a DRS, the latent state Z r of each trial is shared among the N point processes. Once again, one might think that our parametrization of nonnegative splines is unnecessarily complicated and that having f (n) θ in equation 6 be a simpler parametrization of an arbitrary spline (e.g. basis coefficients) and using τ (g ψr,n) instead of g ψr,n, where τ is a nonnegative function, might be a better solution to enforcing nonnegativity constraints. The function τ would have to be chosen in such a way that the integral of equation 5 can still be computed analytically, making τ (t) = t 2 a natural choice. While this would avoid having to use the method of alternating projections, we found that squared splines perform very poorly as they oscillate too much. Autoencoding variational Bayes BID15 ) is a technique to perform inference in the following type of model: DISPLAYFORM0 where each Z r ∈ R m is a local hidden variable which we do not observe, θ are the model parameters and X r is the data that we actually observe, whose distribution depends on Z r. A variational autoencoder estimates θ and approximates the posterior p(z|x) by a distribution q φ (z|x) parametrized by φ. Further simplifying assumptions are made and q φ (z|x) is taken such that it respects conditional independence: DISPLAYFORM1 where each q φ (z r |x r) is taken to be normal with mean and variance depending on R m valued nonlinear functions (usually taken to be neural networks) of x r: DISPLAYFORM2 where diag σ 2 φ (x r) is a diagonal matrix whose diagonal elements are given by σ 2 φ (x r). Performing (approximate) Bayesian inference becomes finding values of φ that adequately approximate the true posterior. To achieve this task, the ELBO L, which is given by the following expression, is jointly maximized over (θ, φ): DISPLAYFORM3 Maximizing the ELBO over φ is equivalent to minimizing the KL of the approximate posterior to the true posterior (for a fixed θ), while maximizing it over θ is equivalent to maximizing a lower bound on the log likelihood. This lower bound is close to the actual log likelihood when the true posterior is correctly approximated. Furthermore, the first term in the sum in equation 10 can be written down in closed form as it is just the KL divergence between two normal random variables, while the second term in the sum can be written using the reparametrization trick: DISPLAYFORM4 where refers to coordinate-wise multiplication. This allows for straightforward differentiation with respect to φ, and thus stochastic gradient methods can be used. In order to perform inference, we use autoencoding variational Bayes. Because of the point process nature of the data, µ φ and σ φ require a recurrent architecture, since their input x r = (x r,1, x r,2, . . ., x r,N) consists of N point processes. This is challenging because the input is not just a sequence, but N sequences of different lengths (numbers of events). In order to deal with this, we use N separate LSTMs BID13, one per point process. Each LSTM takes as input the events of the corresponding point process. The final states of each LSTM are then concatenated and transformed through a dense layer (followed by an exponential activation in the case of σ φ to ensure positivity) in order to map to the hidden space R m. We also tried bidirectional LSTMs BID12 but found regular LSTMs to be faster while having similar performance. The architecture is depicted in figure 1. Combining equations 10 and 11 for our model of equation 6, we approximate the ELBO at each stochastic gradient step by: DISPLAYFORM5 DISPLAYFORM6 where B is a randomly selected subset of trials, 1,... L are iid N (0, I m), µ φ,j (x r) and σ 2 φ,j (x r) are, respectively, the j-th coordinates of µ φ (x r) and σ 2 φ (x r), K r,n is the number of events in the n-th point process of the r-th trial, ψ r,n,l = f DISPLAYFORM7 l ) and x r,n,k is the k-th event of the n-th point process of the r-th trial. BID10 have a similar model, where a hidden Markov model is transformed through a neural network to obtain event counts on time bins. The hidden state for a trial in their model is then an entire hidden Markov chain, which will have significantly higher dimension than our hidden state. Also, their model can be recovered from ours if we change the standard Gaussian distribution of Z r in equation 6 to reflect their Markovian structure and choose G to be piecewise constant, nonnegative functions. We also emphasize the fact that our model is very easy to extend: for example, it would be straightforward to extend it to multi-dimensional point processes (not neural data any more) by changing G and its parametrization. It is also straightforward to use a more complicated point process than the Poisson one by allowing the intensity to depend on previous event history. Furthermore, DRS can be used in settings that require random functions, even if no point process is involved. We simulated data with the following procedure: First, we set 2 different types of trials. For each type of trial, we sampled one true intensity function on for each of the N = 2 point processes by sampling from a GP and exponentiating the . We then sampled 600 times from each type of trial, ing in 1200 trials. We randomly selected 1000 trials for training and set aside the rest for testing. We then fit the model described in section 3.2 and compared it against the PfLDS model of BID10 and the GPFA model of BID30. Both of these methods discretize time into B time bins and have a latent variable per time bin and per trial (as opposed to our model which is only per trial). They do this as a way of enforcing temporal smoothness by placing an appropriate prior over their latent trajectories, which we do not have to do as we implicitly enforce temporal smoothness by using splines to model intensity functions. PfLDS uses Gaussian linear dynamics for their latent space and a Poisson distribution on the number of events per time bin, while GPFA places a GP distribution on the latent space and a Gaussian distribution on the square-rooted number of events per time bin. We compare these methods against DRS-VAE since both were designed to analyze the same type of point process data and, inference wise, PfLDS uses a similar autoencoding variational Bayes algorithm whereas GFPA uses GPs to model random functions. We used a uniform grid with 11 knots (ing in I = 10 intervals), picked d = 3, s = 2, used a mini-batch size of 2 and used L = 2. The state of each LSTM has 100 units, andf is a feed-forward neural network with ReLU activations and with 3 hidden layers, each one with 100 units. We apply 102 iterations of the method of alternating projections. Since a twice-differentiable cubic spline on I intervals has I + 3 degrees of freedom, when discretizing time for PfLDS and GPFA we use B = I + 3 = 13 time bins. This way the distribution recovered by PfLDS also has B = 13 degrees of freedom, while the distribution recovered by GPFA has even more, as each Gaussian has a covariance in addition to the mean. We set the latent dimension m in our model to 2 and we also set the latent dimension per time bin in PfLDS and GPFA to 2, meaning that the overall latent dimension for an entire trial was 2B = 26. These two choices make the comparison conservative as they allow more flexibility for the two competing methods than for ours. For the feed-forward architecture in PfLDS, we also used 3 hidden layers, each with 100 units. The left panel of FIG3 shows the posterior means of the hidden variables in our model for each of the 200 test trials. Each posterior mean is colored according to its type of trial. We can see that different types of trials form separate clusters, meaning that our model successfully obtains low-dimensional representations of the trials. Note that the model is trained without having access to the type of each trial; colors are assigned in the figure post hoc. The right panel of FIG3 shows the events (in red) for a particular point process on a particular trial, along with the true intensity (in green) that generated the events and posterior samples from our model (in purple) and from PfLDS (in blue) of the corresponding intensities. Note that since PfLDS parametrizes the number of counts on each time bin, it does not have a corresponding intensity. We plot instead a piecewise constant intensity on each time bin in such a way that the ing Poisson process has the same count distribution as the distribution that is parametrized by PfLDS. We can see that our method recovers a smoother function that is closer to the truth than PfLDS. TAB0 shows performance from our model compared against PfLDS and GPFA. The second column shows the per-trial ELBO on test data, and we can see that our model has a larger ELBO than PfLDS. While having a better ELBO does not imply that our log likelihood is better, it does suggest that it is. Since both PfLDS and GPFA put a distribution on event counts on time bins instead of a distribution on event times as our models does, the log likelihoods are not directly comparable. However, in the case of PfLDS, we can easily convert from the Poisson likelihood on time bins to the piecewise constant intensity Poisson process likelihood, so that the numbers become comparable. In order to get a quantitative comparison between our model and GPFA, we take advantage of the fact that we know the true intensity that generated the data and compare average L 2 distance, across point processes and trials, between posterior intensity samples and actual intensity function. Once again, we can see that our method outperforms the alternatives. Table 1 also includes the standard deviation of these L 2 distances. Since the standard deviations are somewhat large in comparison to the means, for each of the two competing alternatives, we carry out a two sample t-test comparing the L 2 distance means obtained with our method against the alternative. The p-values indicate that our method recovers intensity functions that are closer to the truth in a statistically significant way. We also fit our model to the dataset collected by BID5. The dataset, after preprocessing (see supplementary material for details), consists of measurements of 20 neurons for 3590 trials on the interval [−100, 300) (in ms) of a primate. In each trial, the primate reaches with its arm to a specific location, which changes from trial to trial (we can think of the 40 locations as types of trials), where time 0 corresponds to the beginning of the movement. We randomly split the data into a training set with 3000 trials and a test set with the rest of the trials. We chose d = 3, s = 2, L = 2, applied 102 iterations of the method of alternating projections, set the state of each LSTM to have 25 units andf is a feed-forward network with ReLU activations and with 3 hidden layers, each one with 10 units (we tried more complicated architectures but saw no improvement). We used 18 uniformly spaced knots (that is, 17 intervals). For the comparison against PfLDS, we split time into 20 bins, ing in time bins of 20ms (which is a standard length), once again making sure that the degrees of freedom are comparable. Since we do not have access to the ground truth, we do not compare against GPFA as the L 2 metric computed in the previous section cannot be used here. Again, we used a hidden dimension m = 2 for our model, ing in hidden trajectories of dimension 40 for PfLDS. We experimented with larger values of m but did not observe significant improvements in either model. FIG4 shows the spike train (red) for a particular neuron on a particular trial, along with posterior samples from our model (in purple) and from PfLDS (in blue) of the corresponding intensities. We can see that the posterior samples look like plausible intensities to have generated the corresponding spike trains and that our posterior intensities look smoother than the ones obtained by PfLDS. TAB1 shows the per-trial ELBO on test data for our model and for PfLDS. Again, our model has a larger ELBO than PfLDS, even when PfLDS has access to 20 times more hidden dimensions: our method is more successful at producing low-dimensional representations of trials than PfLDS. The table also shows the percentage of correctly predicted test trial types when using 15-nearest neighbors on the posterior means of train data (the entire trajectories are used for PfLDS). While 23.73% might seem small, it should be noted that it is significantly better than random guessing (which would have 2.5% accuracy) and that the model was not trained to minimize this objective. Regardless, we can see that our method also outperforms PfLDS in this metric, even when using a much lower-dimensional representation of each trial. The last entry in the table shows the percentage of explained variation when doing ANOVA on the test posterior means (denoted SSG/SST), using trial type as groups. Once again, we can see that our model recovers a more meaningful representation of the trials. We also fit our model to a not yet published dataset collected by our collaborators from the Churchland lab at Columbia University. After preprocessing (see supplementary material), it consists of 1300 and 188 train and test trials, respectively. During each trial, 20 neurons of a primate are measured as it pedals for approximately 8s. There are 8 types of trials, based on whether the primate is pedaling forwards or backwards and at which speed. We use the same hyperparameter settings as for the reaching data, except we use 26 uniformly spaced knots (25 intervals) and 28 bins for PfLDS, as well as a hidden dimension m = 10, ing in hidden trajectories of dimension 280 for PfLDS. TAB2 quantitatively compares our method against PfLDS. The ELBO is actually higher for PfLDS, which appears (in preliminary analysis not shown) to be caused by an artifact of preprocessing the data rather than any essential performance loss. While the ELBO was better for PfLDS, the quality of our latent representations is significantly better, as shown by the accuracy of 15-nearest neighbors to predict test trial types (random guessing would have 12.5% accuracy) and the ANOVA percentage of explained variation of the test posterior means. This is particularly impressive as our latent representations have 28 times fewer dimensions. We did experiment with different hyperparameter settings, and found that the ELBO of PfLDS increased slightly when using more time bins (at the cost of even higher-dimensional latent representations), whereas our ELBO remained the same when increasing the number of intervals. However, even in this setting the accuracy of 15-nearest neighbors and the percentage of explained variation did not improve for PfLDS. In this paper we introduced Deep Random Splines, an alternative to Gaussian processes to model random functions. Owing to our key modeling choices and use of from the spline and optimization literatures, fitting DRS is tractable and allows one to enforce shape constraints on the random functions. While we only enforced nonnegativity and smoothness in this paper, it is straightforward to enforce constraints such as monotonicity (or convexity/concavity). We also proposed a variational autoencoder that takes advantage of DRS to accurately model and produce meaningful low-dimensional representations of neural activity. Future work includes using DRS-VAE for multi-dimensional point processes, for example spatial point processes. While splines would become harder to use in such a setting, they could be replaced by any family of easily-integrable nonnegative functions, such as, for example, conic combinations of Gaussian kernels. Another line of future work involves using a more complicated point process than the Poisson, for example a Hawkes process, by allowing the parameters of the spline in a certain interval to depend on the previous spiking history of previous intervals. Finally, DRS can be applied in more general settings than the one explored in this paper since they can be used in any setting where a random function is involved, having many potential applications beyond what we analyzed here. Parametrization for nonnegative splines of even degreeAs mentioned on section 2.1, there is an alternative characterization of nonnegative polynomials of even degree d = 2k on an interval [l, u) that is analogous to equation 1 of the manuscript, which says that the polynomial p is nonnegative on the interval if and only if it can be written as: DISPLAYFORM0 where again, [t] = (1, t, t 2, . . ., t k) and Q 1 is a (k + 1) × (k + 1) symmetric positive semidefinite matrix. In this case Q 2 is now a k × k symmetric positive semidefinite matrix and[t] = (1, t, t 2, . . ., t k−1). Again, it follows that a piecewise polynomial of degree d with knots t 0,..., t I defined as p (i) (t) for t ∈ [t i−1, t i) for i = 1,..., I is nonnegative if and only if it can be written as: DISPLAYFORM1 for i = 1,..., I, where each Q Projecting onto the space of smooth splines As mentioned in section 2.2, mapping to Ψ = ∩ s+1 j=0 C j can be achieved through the method of alternating projections. As mentioned previously, projecting onto C 0 can be easily done through eigen-decomposition. We now go through the details on how to project onto the other C j sets. We will only cover C 1, C 2 and C 3 for odd-degree splines as we used splines of degree 3 and smoothness 2, but projecting onto C j for j ≥ 4 for higher degree splines can be done in an analogous way. Projections for even degree splines can also be derived in an analogous way. Suppose we are given (Q DISPLAYFORM0, which are (k +1)×(k +1) matrices (not necessarily in Ψ), defining a piecewise polynomial as in equation 2 of the manuscript. Computing the projection (X DISPLAYFORM1 onto C 1 can be done by solving the following optimization problem: DISPLAYFORM2 where || · || F denotes the Frobenius norm and each constraint is merely forcing the piecewise function to be continuous at knot i for i = 1, . . ., I − 1. Note that this is a quadratic optimization problem with linear constraints, and can be solved analytically. The corresponding Lagrangian is: DISPLAYFORM3 where λ = (λ 1, . . ., λ I−1) ∈ R I−1. By solving the KKT conditions, it can be verified that: DISPLAYFORM4 Analogously, computing the projection (X DISPLAYFORM0 onto C 2 can be done by solving the following optimization problem: DISPLAYFORM1 where [t] = (0, 0, 2, 6t, . . ., k(k − 1)t k−2 ) and each constraint is now forcing the values of the left and right second derivatives of the piecewise function to match at knot i for i = 1,..., I − 1. Again, this is a quadratic optimization problem with linear constraints. By writing the Lagrangian and solving the KKT conditions, we get: where B i−1 = 2M DISPLAYFORM2 We include only successful trials (i.e. when the primate reaches to the correct location) and use only spikes occurring in a window of −100ms and 300ms from the time that movement starts. We also reduce the total number of neurons as inference with our method requires one LSTM per neuron and having too many neurons renders training slow. In order to do so, we use the following GLM: DISPLAYFORM0 where y r is the trial type of trial r, C = 40 is the number of trial types,K r,· ∈ R N is a vector containing the (centered and standardized) number of spikes in trial r for each of the N = 223 neurons, and β ∈ R N ×C are the GLM parameters. We train the GLM using group lasso BID31, where the groups are defined by neurons. That is, the GLM is trained through maximum likelihood with an added penalty: DISPLAYFORM1 where β n,· is the n th row of β. This makes it so that the coefficients in each group hit zero simultaneously. A neuron n is removed if ||β n,· || = 0. We use a regularization parameter λ such that all but 20 neurons are removed. This provides a principled way of reducing the number of neurons while making sure that the kept neurons are useful. As PfLDS does not require the use of LSTMs, it can be run on the data without removing neurons. While doing this did increase performance of PfLDS, it did so very marginally and our model still heavily outperformed PfLDS. Once again, we only keep successful trials (i.e. when the primate pedals in the correct direction and speed) and reduce the total number of neurons N = 256 to 20 by using group lasso. Since each trial has a different length, we extend every trial to have the same length as the longest trial. We add no spikes to these extended time periods. We also tried running PfLDS with all the neurons and saw only a very marginal improvement, like we did with reaching data.

Represent the TLDR for finding the corresponding scientific article

We combine splines with neural networks to obtain a novel distribution over functions and use it to model intensity functions of point processes.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.