Question: Represent the scientific article to find a short TLDR

Question: Recent work suggests goal-driven training of neural networks can be used to model neural activity in the brain. While response properties of neurons in artificial neural networks bear similarities to those in the brain, the network architectures are often constrained to be different. Here we ask if a neural network can recover both neural representations and, if the architecture is unconstrained and optimized, also the anatomical properties of neural circuits. We demonstrate this in a system where the connectivity and the functional organization have been characterized, namely, the head direction circuit of the rodent and fruit fly. We trained recurrent neural networks (RNNs) to estimate head direction through integration of angular velocity. We found that the two distinct classes of neurons observed in the head direction system, the Ring neurons and the Shifter neurons, emerged naturally in artificial neural networks as a of training. Furthermore, connectivity analysis and in-silico neurophysiology revealed structural and mechanistic similarities between artificial networks and the head direction system. Overall, our show that optimization of RNNs in a goal-driven task can recapitulate the structure and function of biological circuits, suggesting that artificial neural networks can be used to study the brain at the level of both neural activity and anatomical organization. Artificial neural networks have been increasingly used to study biological neural circuits. In particular, recent work in vision demonstrated that convolutional neural networks (CNNs) trained to perform visual object classification provide state-of-the-art models that match neural responses along various stages of visual processing;; Güçlü & van; ). Recurrent neural networks (RNNs) trained on cognitive tasks have also been used to account for neural response characteristics in various domains (; ; ; ; ; ; ; ;). While these provide important insights on how information is processed in neural circuits, it is unclear whether artificial neural networks have converged upon similar architectures as the brain to perform either visual or cognitive tasks. Answering this question requires understanding the functional, structural, and mechanistic properties of artificial neural networks and of relevant neural circuits. We address these challenges using the brain's internal compass -the head direction system, a system that has accumulated substantial amounts of functional and structural data over the past few decades in rodents and fruit flies (a; ; ; ; ; ; ; ;). We trained RNNs to perform a simple angular velocity (AV) integration task and asked whether the anatomical and functional features that have emerged as a of stochastic gradient descent bear similarities to biological networks sculpted by long evolutionary time. By leveraging existing knowledge of the biological head direction (HD) systems, we demonstrate that RNNs exhibit striking similarities in both structure and function. Our suggest that goal-driven training of artificial neural networks provide a framework to study neural systems at the level of both neural activity and anatomical organization.). e) The brain structures in the fly central complex that are crucial for maintaining and updating heading direction, including the protocerebral bridge (PB) and the ellipsoid body (EB). f) The RNN model. All connections within the RNN are randomly initialized. g) After training, the output of the RNN accurately tracks the current head direction. We trained our networks to estimate the agent's current head direction by integrating angular velocity over time (Fig. 1f). Our network model consists of a set of recurrently connected units (N = 100), which are initialized to be randomly connected, with no self-connections allowed during training. The dynamics of each unit in the network r i (t) is governed by the standard continuous-time RNN equation: for i = 1,..., N. The firing rate of each unit, r i (t), is related to its total input x i (t) through a rectified tanh nonlinearity, r i (t) = max(0, tanh(x i (t))). Every unit in the RNN receives input from all other units through the recurrent weight matrix W rec and also receives external input, I(t), through the weight matrix W in. These weight matrices are randomly initialized so no structure is a priori introduced into the network. Each unit has an associated bias, b i which is learned and an associated noise term, ξ i (t), sampled at every timestep from a Gaussian with zero mean and constant variance. The network was simulated using the Euler method for T = 500 timesteps of duration τ /10 (τ is set to be 250ms throughout the paper). Let θ be the current head direction. Input to the RNN is composed of three terms: two inputs encode the initial head direction in the form of sin(θ 0) and cos(θ 0), and a scalar input encodes both clockwise (CW, negative) and counterclockwise, (CCW, positive) angular velocity at every timestep. The RNN is connected to two linear readout neurons, y 1 (t) and y 2 (t), which are trained to track current head direction in the form of sin(θ) and cos(θ). The activities of y 1 (t) and y 2 (t) are given by: Velocity at every timestep (assumed to be 25 ms) is sampled from a zero-inflated Gaussian distribution (see Fig. 5). Momentum is incorporated for smooth movement trajectories, consistent with the observed animal behavior in flies and rodents. More specifically, we updated the angular velocity as AV(t) = σX + momentum * AV(t−1), where X is a zero mean Gaussian random variable with standard deviation of one. In the Main condition, we set σ = 0.03 radians/timestep and the momentum to be 0.8, corresponding to a mean absolute AV of ∼100 deg/s. These parameters are set to roughly match the angular velocity distribution of the rat and fly (; ; ;). In Sec. 4, we manipulate the magnitude of AV by changing σ to see how the trained RNN may solve the integration task differently. We optimized the network parameters W to minimize the mean-squared error in equation between the target head direction and the network outputs generated according to equation, plus a metabolic cost for large firing rates (L 2 regularization on r). Parameters were updated with the Hessian-free algorithm . Similar were also obtained using Adam . We found that the trained network could accurately track the angular velocity (Fig. 1g). We first examined the functional and structural properties of model units in the trained RNN and compared them to the experimental data from the head direction system in rodents and flies. We first plotted the neural activity of each unit as a function of HD and AV (Fig. 2a). This revealed two distinct classes of units based on the strength of their HD and AV tuning (see Appendix Fig. 6a, b, c). Units with essentially zero activity are excluded from further analyses. The first class of neurons exhibited HD tuning with minimal AV tuning (Fig. 2f). The second class of neurons were tuned to both HD and AV and can be further subdivided into two populations -one with high firing rate when animal performs CCW rotation (positive AV), the other favoring CW rotation (negative AV) (CW tuned cell shown in Fig. 2g). Moreover, the preferred head direction of each sub-population of neurons tile the complete angular space (Fig. 2b). Embedding the model units into 3D space using t-SNE reveals a clear ring-like structure, with the three classes of units being separated (Fig. 2c). Neurons with HD tuning but not AV tuning have been widely reported in rodents (a; ;), although the HD*AV tuning profiles of neurons are rarely shown (but see). By re-analyzing the data from , we find that neurons in the anterodorsal thalamic nucleus (ADN) of the rat brain are selectively tuned to HD but not AV (Fig. 2d, also see), with HD*AV tuning profile similar to what our model predicts. Preliminary evidence suggests that this might also be true for ellipsoid body (EB) ring neurons in the fruit fly HD system . Previous studies have shown that the temporal relationship of cell firing to the rat's head direction differs across H D cells recorded from the PoS and ADN (, , . Specifically, H D cells in the ADN appear to encode the rat's f uture directional heading by ϳ25 msec, whereas H D cells in the PoS encode the rat's present or recent past directional heading. Of the 20 H D cells recorded from the L M N, a second H D cell was recorded simultaneously on the same electrode wire for three cells. Because the tuning curves of these simultaneously recorded cells partially overlapped one another, the accuracy of the time shift analysis is likely to be compromised for these cells, and they were therefore excluded from the following analyses. Time shift analyses were conducted on a representative session for each of the remaining 17 L M N H D cells. Figure 5A -C illustrates an example of the optimal time shifts of an L M N H D cell for each of the three parameters. For this cell, the optimal time shifts for peak firing rate, range width, and information Previous studies have shown that the temporal relationship of cell firing to the rat's head direction differs across HD cells recorded from the PoS and ADN (, , . Specifically, HD cells in the ADN appear to encode the rat's future directional heading by ϳ25 msec, whereas HD cells in the PoS encode the rat's present or recent past directional heading. Of the 20 HD cells recorded from the LMN, a second HD cell was recorded simultaneously on the same electrode wire for three cells. Because the tuning curves of these simultaneously recorded cells partially overlapped one another, the accuracy of the time shift analysis is likely to be compromised for these cells, and they were therefore excluded from the following analyses. Time shift analyses were conducted on a representative session for each of the remaining 17 LMN HD cells. Figure 5A -C illustrates an example of the optimal time shifts of an LMN HD cell for each of the three parameters. For this cell, the optimal time shifts for peak firing rate, range width, and information Previous studies have shown that the temporal relationship of cell firing to the rat's head direction differs across H D cells recorded from the PoS and ADN (, , . Specifically, H D cells in the ADN appear to encode the rat's f uture directional heading by ϳ25 msec, whereas H D cells in the PoS encode the rat's present or recent past directional heading. Of the 20 H D cells recorded from the L M N, a second H D cell was recorded simultaneously on the same electrode wire for three cells. Because the tuning curves of these simultaneously recorded cells partially overlapped one another, the accuracy of the time shift analysis is likely to be compromised for these cells, and they were therefore excluded from the following analyses. Time shift analyses were conducted on a representative session for each of the remaining 17 L M N H D cells. Figure 5A -C illustrates an example of the optimal time shifts of an L M N H D cell for each of the three parameters. For this cell, the optimal time shifts for peak firing rate, range width, and information et al., 1994). The head turn modulation of HD cell activity in these distinct brain areas suggests that LMN HD cells are head turn-modulated HD cells, whereas those in the cortical regions represent a dual code, signaling both head turn and directional heading. Previous studies have shown that the temporal relationship of cell firing to the rat's head direction differs across HD cells recorded from the PoS and ADN (, , . Specifically, HD cells in the ADN appear to encode the rat's future directional heading by ϳ25 msec, whereas HD cells in the PoS encode the rat's present or recent past directional heading. Of the 20 HD cells recorded from the LMN, a second HD cell was recorded simultaneously on the same electrode wire for three cells. Because the tuning curves of these simultaneously recorded cells partially overlapped one another, the accuracy of the time shift analysis is likely to be compromised for these cells, and they were therefore excluded from the following analyses. Time shift analyses were conducted on a representative session for each of the remaining 17 LMN HD cells. Figure 5A -C illustrates an example of the optimal time shifts of an LMN HD cell for each of the three parameters. For this cell, the optimal time shifts for peak firing rate, range width, and information Neurons tuned to both HD and AV tuning have also been reported previously in rodents and fruit flies (; ;), although the joint HD*AV tuning profiles of neurons have only been documented anecdotally with a few cells . In rodents, certain cells are also observed to display HD and AV tuning (Fig. 2e). In addition, in the fruit fly heading system, neurons on the two sides of the protocerebral bridge (PB) are also tuned to CW and CCW rotation, respectively, and tile the complete angular space, much like what has been observed in our trained network . These observations collectively suggest that neurons that are HD but not AV selective in our model can be tentatively mapped to "Ring" units in the EB, and the two sub-populations of neurons tuned to both HD and AV map to "Shifter" neurons on the left PB and right PB, respectively. We will correspondingly refer to our model neurons as either'Ring' units or'CW/CCW Shifters'(Further justification of the terminology will be given in Sec. 3.2 & 3.3). We next sought to examine the tuning properties of both Ring units and Shifters of our network in greater detail. First, we observe that for both Ring units and Shifters, the HD tuning curve varies as a function of AV (see example Ring unit in Fig. 2f and example Shifter in Fig. 2g). Population summary statistics concerning the amount of tuning shift are shown in Appendix Fig. 7a. The preferred HD tuning is biased towards a more CW angle at CW angular velocities, and vice versa for CCW angular velocities. Consistent with this observation, the HD tuning curves in rodents were also dependent upon AV (see example neurons in Fig. 2h,i) (; ; ; ; . Second, the AV tuning curves for the Shifters exhibit graded response profiles, consistent with the measured AV tuning curves in flies and rodents (see Fig. 1b,d). Across neurons, the angular velocity tuning curves show substantial diversity (see Appendix Fig. 6b), also consistent with experimental reports . In summary, the majority of units in the trained RNN could be mapped onto the biological head direction system in both general functional architecture and also in detailed tuning properties. Our model unifies a diverse set of experimental observations, suggesting that these neural response properties are the consequence of a network solving an angular integration task optimally. vity of the trained network are structured, and exhibit similarity to the connectivity lex. a) the connectivity of the network trained with mid-speed. We sort the neuron nctional classes, and further arrange them according to the preferred HD with each ible structural connectivity within and across different cell types. b) mapping the ture onto fly central complex. Inserted panels represent the average connectivity one s.d.) as a function of the difference between preferred HD for within and class of neurons. ly structured connectivity have been proposed to perform integration. It's shown igure 3: Connectivity of the trained network is structured and exhibits similarities with the connectivity in the fly central complex. a) Pixels represent connections from the units in each column to the units in each row. Excitatory connections are in red, and inhibitory connections are in blue. Units are first sorted by functional classes, and then are further sorted by their preferred HD within each class. The black box highlights recurrent connections to the ring units from ring units, from CCW Shifters, and from CW Shifters. b) Ensemble connectivity from each functional cell type to the Ring units as highlighted in a), in relation to the architecture of the PB & EB in the fly central complex. Plots show the average connectivity (shaded area indicates one s.d.) as a function of the difference between the preferred HD of the cell and the Ring unit it is connecting to. Ring units connect strongly to units with similar HD tuning and inhibit units with dissimilar HD tuning. CCW shifters connect strongly to ring units with preferred head directions that are slightly CCW-shifted to its own, and CW shifters connect strongly to Ring units with preferred head directions that are slightly CW-shifted to its own. Refer to Appendix Fig. 8b for the full set of ensemble connectivity between different classes. Previous experiments have detailed a subset of connections between EB and PB neurons in the fruit fly. We next analyzed the connectivity of Ring units and Shifters in the trained RNN to ask whether it recapitulates these connectivity patterns -a test which has never been done to our knowledge in any system between artificial and biological neural networks (see Fig. 3). We ordered Ring units, CCW Shifters, and CW Shifters by their preferred head direction tuning and plotted their connection strengths (Fig. 3a). This revealed highly structured connectivity patterns within and between each class of units. We first focused on the connections between individual Ring units and observed a pattern of local excitation and global inhibition. Neurons that have similar preferred head directions are connected through positive weights and neurons whose preferred head directions are anti-phase are connected through negative weights (Fig. 3b). This pattern is consistent with the connectivity patterns inferred in recent work based on detailed calcium imaging and optogenetic perturbation experiments , with one caveat that the connectivity pattern inferred in this study is based on the effective connectivity rather than anatomical connectivity. We conjecture that Ring units in the trained RNN serve to maintain a stable activity bump in the absence of inputs (see section 3.3), as proposed in previous theoretical models (; ;). We then analyzed the connectivity between Ring units and Shifters. We found that CW shifters excite Ring units with preferred head directions that are clockwise to its own, and inhibit Ring units with preferred head directions counterclockwise to its own (Fig. 3b). The opposite pattern is observed for CCW shifters. Such asymmetric connections from Shifters to the Ring units are consistent with the connectivity pattern observed between the PB and the EB in the fruit fly central complex (; ;), and also in agreement with previously proposed mechanisms of angular integration (; ; ;) (Fig. 3b). We note that while the connectivity between PB Shifters and EB Ring units are one-to-one (; ;), the connectivity profile in our model is broad, with a single CW Shifter exciting multiple Ring units with preferred HDs that are clockwise to its own, and vice versa for CCW shifters. In summary, the RNN developed several anatomical features that are consistent with structures reported or hypothesized in previous experimental . A few novel predictions are worth mentioning. First, in our model the connectivity between CW and CCW Shifters exhibit specific recurrent connectivity (Fig. 8). Second, the connections from Shifters to Ring units exhibit not only excitation in the direction of heading motion, but also inhibition that is lagging in the opposite direction. This inhibitory connection has not been observed in experiments yet but may facilitate the rotation of the neural bump in the ring units during turning (; ; ;). In the future, EM reconstructions together with functional imaging and optogenetics should allow direct tests of these predictions. We have segregated neurons into Ring and Shifter populations according to their HD and AV tuning, and have shown that they exhibit different connectivity patterns that are suggestive of different functions. Ring units putatively maintain the current heading direction and shifter units putatively rotate activity on the ring according to the direction of angular velocity. To substantiate these functional properties, we performed a series of perturbation experiments by lesioning specific subsets of connections. We first lesioned connections when there is zero angular velocity input. Normally, the network maintains a stable bump of activity within each class of neurons, i.e., Ring units, CW Shifters, and CCW Shifters (see Fig. 4a,b). We first lesioned connections from Ring units to all units and found that the activity bumps in all three classes disappeared and were replaced by diffuse activity in a large proportion of units. As a consequence, the network could not report an accurate estimate of its current heading direction. Furthermore, when the connections were restored, a bump formed again We next lesioned connections during either constant CW or CCW angular velocity. Normally, the network can integrate AV accurately (Fig. 4k-n). As expected, during CCW rotation, we observe a corresponding rotation of the activity bump in Ring units and in CCW Shifters, but CW Shifters display low levels of activity. The converse is true during CW rotation. We first lesioned connections from CW Shifters to all units, and found that it significantly impaired rotation in the CW direction, and also increased the rotation speed in the CCW direction. Lesioning of CCW Shifters to all units had the opposite effect, significantly impairing rotation in the CCW direction. These are consistent with the hypothesis that CW/CCW Shifters are responsible for shifting the bump in a CW and CCW direction, respectively, and are consistent with the data in , which shows that inhibition of Shifter units in the PB of the fruit fly heading system impairs the integration of HD. Our lesion experiments further support the segregation of units into modular components that function to separately maintain and update heading during angular motion. Optimal computation requires the system to adapt to the statistical structure of the inputs . In order to understand how the statistical properties of the input trajectories without any external input (Fig. 4d), suggesting the network can spontaneously generate an activity bump through recurrent connections mediated by Ring units. We then lesioned connections from CW Shifters to all units and found that all three bumps exhibit a CCW rotation, and the read-out units correspondingly reported a CCW rotation of heading direction (Fig. 4e,f). Analogous were obtained with lesions of CCW Shifters, which ed in a CW drifting bump of activity (Fig. 4g,h). These are consistent with the hypothesis that CW and CCW Shifters simultaneously activate the ring, with mutually cancelling signals, even when the heading direction is stationary. When connections are lesioned from both CW and CCW Shifters to all units, we observe that Ring units are still capable of holding a stable HD activity bump (Fig. 4i,j), consistent with the predictions that while CW/CCW shifters are necessary for updating heading during motion, Ring units are responsible for maintaining heading. We next lesioned connections during either constant CW or CCW angular velocity. Normally, the network can integrate AV accurately (Fig. 4k-n). As expected, during CCW rotation, we observe a corresponding rotation of the activity bump in Ring units and in CCW Shifters, but CW Shifters display low levels of activity. The converse is true during CW rotation. We first lesioned connections from CW Shifters to all units, and found that it significantly impaired rotation in the CW direction, and also increased the rotation speed in the CCW direction. Lesioning of CCW Shifters to all units had the opposite effect, significantly impairing rotation in the CCW direction. These are consistent with the hypothesis that CW/CCW Shifters are responsible for shifting the bump in a CW and CCW direction, respectively, and are consistent with the data in , which shows that inhibition of Shifter units in the PB of the fruit fly heading system impairs the integration of HD. Our lesion experiments further support the segregation of units into modular components that function to separately maintain and update heading during angular motion. Optimal computation requires the system to adapt to the statistical structure of the inputs . In order to understand how the statistical properties of the input trajectories affect how a network solves the task, we trained RNNs to integrate inputs generated from low and high AV distributions. When networks are trained with small angular velocities, we observe the presence of more units with strong head direction tuning but minimal angular velocity tuning. Conversely, when networks are trained with large AV inputs, fewer ring units emerge and more units become Shifter-like and exhibit both HD and AV tuning (Fig. 5c,f,i). We sought to quantify the overall AV tuning under each velocity regime by computing the slope of each neuron's AV tuning curve at its preferred HD angle. We found that by increasing the magnitude of AV inputs, more neurons developed strong AV tuning (Fig. 5b,e,h). In summary, with a slowly changing head direction trajectory, it is advantageous to allocate more resources to hold a stable activity bump, and this requires more ring units. In contrast, with quickly changing inputs, the system must rapidly update the activity bump to integrate head direction, requiring more shifter units. This prediction may be relevant for understanding the diversity of the HD systems across different animal species, as different species exhibit different overall head turning behavior depending on the ecological demand (; ; ;). Previous work in the sensory systems have mainly focused on obtaining an optimal representation (; ; ; ; ;) with feedforward models. Several recent studies have probed the importance of recurrent connections in understanding neural computation by training RNNs to perform tasks (e.g., ; ;), but the relation of these trained networks to the anatomy and function of brain circuits are not mapped. Using the head direction system, we demonstrate that goal-driven optimization of recurrent neural networks can be used to understand the functional, structural and mechanistic properties of neural circuits. While we have mainly used perturbation analysis to reveal the dynamics of the trained RNN, other methods could also be applied to analyze the network. For example, in Appendix Fig. 10, using fixed point analysis , we found evidence consistent with attractor dynamics. Due to the limited amount of experimental data available, comparisons regarding tuning properties and connectivity are largely qualitative. In the future, studies of the relevant brain areas using Neuropixel probes and calcium imaging will provide a more in-depth characterization of the properties of HD circuits, and will facilitate a more quantitative comparison between model and experiment. In the current work, we did not impose any additional structural constraint on the RNNs during training. We have chosen to do so in order to see what structural properties would emerge as a consequence of optimizing the network to solve the task. It is interesting to consider how additional structural constraints affect the representation and computation in the trained RNNs. One possibility would to be to have the input or output units only connect to a subset of the RNN units. Another possibility would be to freeze a subset of connections during training. Future work should systematically explore these issues. Recent work suggests it is possible to obtain tuning properties in RNNs with random connections . We found that training was necessary for the joint HD*AV tuning (see Appendix Fig. 9) to emerge. consider a simple binary classification task, our integration task is computationally more complicated. Stable HD tuning requires the system to keep track of HD by accurate integration of AV, and to stably store these values over time. This computation might be difficult for a random network to perform . Our approach contrasts with previous network models for the HD system, which are based on hand-crafted connectivity (; ; ; ; ; ; ; Kakaria & de ;). Our modeling approach optimizes for task performance through stochastic gradient descent. We found that different input statistics lead to different heading representations in an RNN, suggesting that the optimal architecture of a neural network varies depending on the task demandan insight that would be difficult to obtain using the traditional approach of hand-crafting network solutions. Although we have focused on a simple integration task, this framework should be of general relevance to other neural systems as well, providing a new approach to understand neural computation at multiple levels. Our model may be used as a building block for AI systems to perform general navigation . In order to effectively navigate in complex environments, the agent would need to construct a cognitive map of the surrounding environment and update its own position during motion. A circuit that performs heading integration will likely be combined with another circuit to integrate the magnitude of motion (speed) to perform dead reckoning. Training RNNs to perform more challenging navigation tasks such as these, along with multiple sources of inputs, i.e., vestibular, visual, auditory, will be useful for building robust navigational systems and for improving our understanding of the computational mechanisms of navigation in the brain . Figure 9: Joint HD × AV tuning of the initial, randomly connected network and the final trained network. a) Before training, the 100 units in the network do not have pronounced joint HD × AV tuning. The color scale is different for each unit (blue = minimum activity, yellow = maximum activity) to maximally highlight any potential variation in the untrained network. b) After training, the units are tuned to HD × AV, with the exception of 12 units (shown at the bottom) which are not active and do not influence the network.

Artificial neural networks trained with gradient descent are capable of recapitulating both realistic neural activity and the anatomical organization of a biological circuit.

represent the tldr for finding the corresponding scientific article.

We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.


Question: Represent the scientific article to find a short TLDR

Question: Convolutional Neural Networks (CNNs) are computationally intensive, which limits their application on mobile devices. Their energy is dominated by the number of multiplies needed to perform the convolutions. Winograd’s minimal filtering algorithm and network pruning can reduce the operation count, but these two methods cannot be straightforwardly combined — applying the Winograd transform fills in the sparsity in both the weights and the activations. We propose two modifications to Winograd-based CNNs to enable these methods to exploit sparsity. First, we move the ReLU operation into the Winograd domain to increase the sparsity of the transformed activations. Second, we prune the weights in the Winograd domain to exploit static weight sparsity. For models on CIFAR-10, CIFAR-100 and ImageNet datasets, our method reduces the number of multiplications by 10.4x, 6.8x and 10.8x respectively with loss of accuracy less than 0.1%, outperforming previous baselines by 2.0x-3.0x. We also show that moving ReLU to the Winograd domain allows more aggressive pruning. Deep Convolutional Neural Networks (CNNs) have shown significant improvement in many machine learning applications. However, CNNs are compute-limited. Their performance is dominated by the number of multiplies needed to perform the convolutions. Moreover, the computational workload of CNNs continues to grow over time. BID16 proposed a CNN model with less than 2.3 × 10 7 multiplies for handwritten digit classification. Later, BID13 developed AlexNet, an ImageNet-winning CNN with more than 1.1 × 10 9 multiplies. In 2014, ImageNetwinning and runner up CNNs increased the number of multiplies to 1.4 × 10 9 BID24 ) and 1.6 × 10 10 BID22 respectively. Despite the powerful representational ability of large scale CNNs, their computational workload prohibits deployment on mobile devices. Two research directions have been explored to address the problem. BID14 proposed using Winograd's minimal filtering algorithm BID25 to reduce the number of multiplies needed to perform 3 × 3 kernel convolutions. On the other end, pruning the model BID5 and exploiting the dynamic sparsity of activations due to ReLU also reduces the required multiplies. Unfortunately, the above two directions are not compatible: the Winograd transformation fills in the zeros in both the weights and the activations FIG0 ) -eliminating the gain from exploiting sparsity. Thus, for a pruned network, Winograd's algorithm actually increases the number of multiplies; the loss of sparsity more than offsets the reduced operation count. In this paper, we introduce two modifications to the original Winograd-based convolution algorithm to eliminate this problem. First, we move the ReLU operation to be after the Winograd transform to also make the activations sparse at the point where the multiplies are performed. Second, we prune the weights after (rather than before) they are transformed. Thus, the weights are sparse when the elementwise multiply is performed -reducing the operation count. Together, these two modifications enable the gains of Winograd's algorithm and of exploiting sparsity to be combined. We open-source our code and models at https://github.com/xingyul/Sparse-Winograd-CNN. Linear Algebra property in Convolution: Previous research proposes using the linear algebra property of convolution to reduce the number of multiplies by trading additions for multiplies. BID3 convert convolution into matrix multiplies and utilize the linear algebra property at the sub-matrix block level. This approach achieves a 47% saving in multiplies. BID14 exploits the element-level linear algebra property of convolution, i.e. Winograd's minimal filtering algorithm BID25. This approach reduces the number of multiplies by 2.25× to 4×, depending on the image patch size used in the algorithm. Winograd's algorithm is also used in a state-of-the-art deep learning library, cuDNN BID2, to improve computation efficiency. Model Compression: Model compression reduces the number of multiplies of CNNs by pruning network parameters BID15 BID8 and exploiting weight sparsity. BID5 proposed learning the sparsity pattern of network weights by eliminating weights whose absolute value is less than an empirical threshold. This approach can prune the convolutional layers of the model to only 30% − 50% of the original size and reduce the number of multiplies required. first proposed pruning and re-training the weights in Winograd domain for conventional Winograd convolution. later showed promising on large datasets and reported 90% sparsity in the Winograd parameters of AlexNet with less than 0.1% accuracy loss. Dynamic Activation Sparsity: The ReLU non-linearity sets activations whose values are negative to zero, causing dynamic sparsity in activations. Model compression can work in tandem with dynamic activation sparsity and reduce multiplication workload. BID5 showed that exploiting sparsity of both weights and activations can reduce the number of multiplies by 4 − 11×. BID11 further proposed to manually set a small positive ReLU threshold at test time to exploit greater sparsity in activation without losing testing accuracy. Research in novel architectures also led to optimizations for deep learning accelerators to exploit the sparsity in activations. BID6 proposed using a Leading Non-zero Detection unit (LNZD) for their fully-connected layer accelerator to efficiently skip zeros in input activations. BID1 proposed a similar mechanism for a convolution layer accelerator. We first introduce the conventional Winograd convolution and show how sparsity of weights or activations is lost during the dataflow of the algorithm. We then present the novel Winograd-ReLU CNN architecture. It preserves sparsity in both weights and activations before multiplies are performed and significantly reduces the computational workload. The basic block of the conventional Winograd convolution algorithm works on an p×p patch (denoted by d) extracted with stride of (p − 2) × (p − 2) from an H × W input feature map. With "valid" padding, the p×p patch is convolved with a 3×3 kernel (denoted by g) to produce an (p−2)×(p−2) output patch (denoted by S). The output patches are assembled into an output feature map. Input activation patch d and kernel g (spatial-domain activation and weights) are transformed using matrices B and G to be B T dB and GgG T (Winograd-domain activation and weights) respectively, both with shape p × p. After element-wise product in Winograd-domain, the output activation S is obtained using matrix A (equation ). Matrices B, G and A are p-specific. When p = 4, B and A consists of 1, −1 and 0, so the multiplication with B and A only requires addition. It reduces the number of multiplies from 9(p − 2) 2 to p 2. Lavin FORMULA1 gives details of the algorithm. DISPLAYFORM0 Spatial Baseline Network: When using a "vanilla" pruned network, as introduced by BID5, a ReLU non-linear operation is performed by the previous layer on spatial-domain input d and spatial-domain weight g is pruned. The output activation patch S is obtained from equation FORMULA1. This is illustrated in FIG0 (a) for p = 4. Though g and d may both be sparse due to pruning and ReLU respectively, the element-wise multiply is dense due to G(·)G T and B(·)B T transformations filling the spatial-domain zeros. Sparsity does not reduce the number of multiplies in Winograd's algorithm. DISPLAYFORM1 Winograd Native Pruned Network: When using the Winograd-domain pruned network introduced by and BID17, the spatial-domain input d is ReLU-ed by the previous layer while the Winograd-domain weight GgG T is pruned. The output activation patch S is obtained from equation. The algorithm when p = 4 is also illustrated in FIG0 (b). Though Winograd-domain weights are sparse due to pruning, Winograd-domain activations are still dense due to B(·)B T transforms. The sparsity in spatial activations due to ReLU does not reduce the number of multiplies. DISPLAYFORM2 To address the above problems, we introduce the Winograd-ReLU Network. Instead of applying ReLU to the activations in the spatial domain, we apply ReLU to the activations in the Winograd domain, as in equation FORMULA3 and FIG0 (c). The ReLU operation zeros all negative transformed activations, reducing the number of multiplies in the Winograd domain. DISPLAYFORM0 In the Winograd-ReLU CNN, we eliminate the spatial-domain kernel entirely. Because this ReLU is really associated with the previous layer, we perform this transformed ReLU starting with the second layer. We point out that the proposed new CNN architecture is not mathematically equivalent to the vanilla CNN nor the conventional Winograd CNN. Due to the change of network architecture, the training and pruning should also be changed. Our method operates in three phases: dense training, pruning, and retraining. Dense training: we train a dense p × p kernel directly in the transform domain. The transformed kernel is initialized and trained directly by back-propagation through the inverse transform -eliminating the need to maintain a kernel in the spatial domain or to transform a spatial kernel. Pruning: we prune the transformed kernel by computing the threshold t required to achieve a desired pruning rate r and setting all weights whose absolute value less than t to zero. In our experiments, we used the same r for all Winograd-ReLU layers. Because sensitivity varies from layer to layer, we expect that better performance could be achieved by varying the pruning rate r i for each layer i. Re-training: we re-train the model using a "sparsity mask" to force the weights that were pruned to remain zero. The sparsity mask is computed during the pruning step and is kept constant during re-training. The gradient of the network's loss, L, with respect to the input activation and Winograd weights can be derived using the chain rule. Equation FORMULA4 shows the calculation of input activation gradient ∇ d L and Winograd weight gradient ∇ GgG T L using the loss gradient passed from upstream layers DISPLAYFORM1 4 EXPERIMENTS We applied the methodology described above to several different CNNs on different datasets. The original network models are chosen such that the majority of the convolution layers have 3 × 3 kernels. This ensures the largest portion of layers can be converted to Winograd convolution layers and ReLU be put in Winograd domain. We used image classification datasets of different scales: CIFAR-10, CIFAR-100 BID12 ) and ImageNet 2012 BID21. For network architectures, we chose VGG-nagadomi , ConvPool-CNN-C model BID23 and a variation of ResNet-18 BID9 ) respectively on three datasets. Using the Tensorflow BID0 ) framework, we trained the spatial baseline CNN, corresponding conventional Winograd CNN, and Winograd-ReLU CNN models from scratch. Then the three models are iteratively pruned and re-trained. For a specific dataset, we used the same data augmentation for the training of all models on the dataset. We used VGG-nagadomi on the CIFAR-10 dataset. VGG-nagadomi is a lightweight version of VGGNet BID22. It contains 8 convolution layers with 3×3 kernels. The best reported validation set accuracy it achieves on CIFAR-10 is 93.31% . We trained three models from scratch. The corresponding conventional Winograd CNN model and Winograd-ReLU CNN model can achieve validation set accuracy of 93.30% and 93.43% respectively. The first convolution layer is most sensitive to pruning and we set its density to a constant of 80%. We iteratively pruned and re-trained other convolution layers with density from 80% down to 20%. Figure 2: Test accuracy vs density for the three models in FIG0 on VGG-nagadomi. Figure 2 shows test accuracy as a function of weight density for the three models. The two baseline models can only be pruned to 60% density before accuracy falls significantly (> 0.1%). Our Winograd-ReLU CNN model can be pruned to 40% density before falling to the same accuracy. TAB1 shows the input activation density and compares the workloads for each pruned convolution layer in three models. Pruning two baseline models reduces the convolution layer workload by 5.1× and 3.7× 1 respectively. Pruning the Winograd-ReLU model reduces the convolution layer workload by 13.3×, a 2.6× and 3.6× improvement respectively over the two baselines. The improvement of overall network workload reduction is 2.2× and 3.0× respectively over two baselines.1 All Winograd CNN model workload reduction include the intrinsic 2.25× reduction. We used the ConvPool-CNN-C model on on the CIFAR-100 dataset. ConvPool-CNN-C contains 9 convolution layers, out of which 7 have 3 × 3 kernels. We trained three models from scratch. The spatial baseline CNN model and conventional Winograd CNN model can achieve single model validation accuracy of 69.34% and 69.32% respectively. The corresponding Winograd-ReLU network model can achieve validation set accuracy of 69.75%. We pruned the first convolution layer to a constant density of 80%. We iteratively pruned and re-trained the other layers to densities from 80% down to 20%. Figure 3: Test accuracy vs density for the three models in FIG0 on ConvPool-CNN-C. Figure 3 shows the accuracy as a function of density for spatial baseline and Winograd-ReLU models. The spatial-baseline and Winograd-ReLU models can be pruned to 60% density without significant (> 0.1%) loss of accuracy. In contrast, the conventional Winograd CNN model can only be pruned to 70% density. At a given density, the Winograd-ReLU model has the highest accuracy. TAB3 shows the input activation density and compares the workloads for each pruned convolution layer in three models. Pruning two baseline models reduces the convolution layer workload by 3.5× and 3.2× respectively. Pruning the Winograd-ReLU model reduces the workload by 7.1×, a 2.1× and 2.2× improvement respectively over the two baselines. The improvement of overall network workload reduction is 2.0× and 2.2× respectively over two baselines. DISPLAYFORM0 We used a variation of the full pre-activation version BID10 of ResNet-18 BID9 on the ImageNet 2012 dataset. We used this version because it performs the best among various ResNet versions and its structure suits our Winograd-ReLU approach -its ReLU units are located before convolutions in the residual modules. The variation is different from original ResNet-18 by replacing all 2 × 2-stride 3 × 3 convolution layers with a 2 × 2 max-pooling layer followed by a 1 × 1-stride 3 × 3 convolution layer. Such difference ensure most of convolution layers can be converted to Winograd convolution layer. Another difference is that it doesn't have the last max pooling layer so the last group of residual modules has spatial size of 14 × 14, in order to keep the spatial size even instead of odd. This setting suits Winograd convolution with p = 4 best in that even spatial size is required for even p values. We trained three models from scratch. Figure 4: Top-1 and top-5 validation accuracy vs density for three models on a variation of ResNet-18. Figure 4 shows the accuracy as a function of density for three models. The spatial baseline CNN model and conventional Winograd CNN model can be pruned to 60% and 50% respectively without significant (> 0.1%) loss of top-1 or top-5 accuracy. The Winograd-ReLU model can be pruned much further, to 30%/35% density without significant (> 0.1%) loss of top-1/top-5 accuracy. At these densities, top-1 accuracies are 66.53%, 66.45% and 66.61% for three models respectively, with a dense spatial baseline of 66.67%; top-5 accuracies are 87.29%, 87.30% and 87.35% for three models respectively, with a dense spatial baseline of 87.42%. TAB5 shows the input activation density and compares the workloads for each pruned convolution layer in three models. Pruning the two baseline models reduces the convolution layer workload by 5.1× and 4.5× respectively. Pruning the Winograd-ReLU model reduces the workload by 13.2×, a 2.6× and 2.9× improvement respectively over the two baselines. The improvement of overall network workload reduction is 2.3× and 2.6× respectively over two baselines. In this section, we summarize the experiment and compare the three models in terms of a) weight and activation dimensions and b) the dynamic density of activations. We then visualize the kernels to illustrate the pattern of the proposed Winograd-ReLU model kernel. DISPLAYFORM0 In a convolutional neural network, a convolution-ReLU pair acts as a classifier on a spatial patch of an input feature. The dimension of the space being classified is the total number of elements passing through the ReLU layer. The decision boundaries of the classifier are determined by the weights. Insufficient non-zero weights or insufficient activations in too simple a decision boundary and causes accuracy loss. Experimental have shown that Winograd-ReLU CNN can reach the same accuracy as both vanilla spatial baseline CNN and conventional Winograd CNN without pruning, and that WinogradReLU CNN is more robust to aggressive pruning. In this subsection we provide an explanation for the latter observation from the aspect of activation and weight dimensions. We provide a summary on dimensions in Table 4. Table 4: Comparison of ReLU dimension and weight dimension in three types of networks. Assume the convolution-ReLU pair operates on input activation of spatial size of H × W and the number of input and output channels are C and K respectively. Spatial Baseline CNN BID5 Winograd native pruned CNN We can see that our Winograd-ReLU architecture has an advantage on the dimensions of weights and activations over other two models. This means Winograd-ReLU CNNs classify on a higher dimension with more complex decision boundaries, which forms a stronger representational ability in high dimensional image feature space. DISPLAYFORM0 As is shown in the ImageNet in the previous section, dynamic activation density of spatial baseline CNN model varies significantly among layers. Layers at earlier stages typically have higher density in activation than later stages. In Winograd-ReLU CNN model, the dynamic activation densities vary little among layers and are all close to 50%.An explanation is that the nature of image convolution ensures activations d to be spatially smooth. Thus, due to the structure of matrix B BID14, 15 of 16 elements in the 4 × 4 matrix of Winograd-domain activation patch B T · d · B have a mean close to zero. This benefits classification within a patch since ReLU layer is most powerful when half of activations are positive. We visualize the kernels of the proposed Winograd-ReLU model. We selected the first 6 input and output channels of layer res2a_2a of ResNet-18 at three different pruning densities. Unlike spatial domain kernels, Winograd-ReLU kernels do not show clear physical meanings such as edge or corner detectors. However, we observe that values of the elements (from top-left, 1-based indices) in each kernel are typically distinct in a kernel and are most likely kept during aggressive pruning. A possible reason for this is that the elements of Winograd-domain activation in a 4 × 4 patch are special: interested readers can calculate B T · d · B symbolically and will realize that elements are the only elements that are transformed with a linear combination of only adding and no subtraction. In a spatially smooth activation patch, this means the elements are the ones and the only ones with a non-zero mean. We have shown that we can combine the computational savings of sparse weights and activations with the savings of the Winograd transform by making two modifcations to conventional CNNs. To make the weights sparse at the point of multiplication, we train and prune the weights in the transform domain. This simple approach does not reduce the workload with respect to spatial pruning, though, so we move the ReLU non-linear operation after the Winograd transform to make the activations sparse at the point of multiplication. Moving ReLU to the Winograd domain also allows the weights to be more aggressively pruned without losing accuracy. With a 2 × 2 output patch (p = 4), the net is a reduction of 10.4×, 6.8× and 10.8× in computation on three datasets: CIFAR-10, CIFAR-100 and ImageNet. We plan to extend this work in the following directions. First, we expect that even greater savings on computation can be realized by using larger patch sizes (e.g., p = 6), and there may be benefit in exploring different Winograd transformation matrices (B,G and A). Second, we expect that using different pruning rates r i for each network layer will help maintain accuracy and improve overall workload reduction. Finally, we expect that combining our Winograd-ReLU network with other network simplification techniques, e.g. quantization of weights and/or activations BID4 BID18 BID20, will reduce the energy of computation even further.

Represent the short summary for finding the corresponding scientific article (long).

Prune and ReLU in Winograd domain for efficient convolutional neural network

Represent the TLDR for finding the corresponding scientific article.

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.