Question: Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance. However, the differences between the learned solutions of networks which generalize and those which do not remain unclear. Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated. Here, we connect these lines of inquiry to demonstrate that a network’s reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyper- parameters, and over the course of training. While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units. Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance. Recent work has demonstrated that deep neural networks (DNNs) are capable of memorizing extremely large datasets such as ImageNet BID39. Despite this capability, DNNs in practice achieve low generalization error on tasks ranging from image classification BID17 to language translation BID37. These observations raise a key question: why do some networks generalize while others do not?Answers to these questions have taken a variety of forms. A variety of studies have related generalization performance to the flatness of minima and PAC-Bayes bounds BID18 BID20 BID27 BID14, though recent work has demonstrated that sharp minima can also generalize BID13. Others have focused on the information content stored in network weights BID0, while still others have demonstrated that stochastic gradient descent itself encourages generalization BID8 BID34 BID36.Here, we use ablation analyses to measure the reliance of trained networks on single directions. We define a single direction in activation space as the activation of a single unit or feature map or some linear combination of units in response to some input. We find that networks which memorize the training set are substantially more dependent on single directions than those which do not, and that this difference is preserved even across sets of networks with identical topology trained on identical data, but with different generalization performance. Moreover, we found that as networks begin to overfit, they become more reliant on single directions, suggesting that this metric could be used as a signal for early stopping. We also show that networks trained with batch normalization are more robust to cumulative ablations than networks trained without batch normalization and that batch normalization decreases the class selectivity of individual feature maps, suggesting an alternative mechanism by which batch normalization may encourage good generalization performance. Finally, we show that, despite the focus on selective single units in the analysis of DNNs (and in neuroscience; BID21 BID40 BID28 BID9, the class selectivity of single units is a poor predictor of their importance to the network's output. In this study, we will use a set of perturbation analyses to examine the relationship between a network's generalization performance and its reliance upon single directions in activation space. We will then use a neuroscience-inspired measure of class selectivity to compare the selectivity of individual directions across networks with variable generalization performance and examine the relationship between class selectivity and importance. We analyzed three models: a 2-hidden layer MLP trained on MNIST, an 11-layer convolutional network trained on CIFAR-10, and a 50-layer residual network trained on ImageNet. In all experiments, ReLU nonlinearities were applied to all layers but the output. Unless otherwise noted, batch normalization was used for all convolutional networks BID19 . For the ImageNet ResNet, top-5 accuracy was used in all cases. Partially corrupted labels As in BID39, we used datasets with differing fractions of randomized labels to ensure varying degrees of memorization. To create these datasets, a given fraction of labels was randomly shuffled and assigned to images, such that the distribution of labels was maintained, but any true patterns were broken. Ablations We measured the importance of a single direction to the network's computation by asking how the network's performance degrades once the influence of that direction was removed. To remove a coordinate-aligned single direction, we clamped the activity of that direction to a fixed value (i.e., ablating the direction). Ablations were performed either on single units in MLPs or an entire feature map in convolutional networks. For brevity, we will refer to both of these as'units.' Critically, all ablations were performed in activation space, rather than weight space. More generally, to evaluate a network's reliance upon sets of single directions, we asked how the network's performance degrades as the influence of increasing subsets of single directions was removed by clamping them to a fixed value (analogous to removing increasingly large subspaces within activation space). This analysis generates curves of accuracy as a function of the number of directions ablated: the more reliant a network is on low-dimensional activation subspaces, the more quickly the accuracy will drop as single directions are ablated. Interestingly, we found that clamping the activation of a unit to the empirical mean activation across the training or testing set was more damaging to the network's performance than clamping the activation to zero (see Appendix A.1). We therefore clamped activity to zero for all ablation experiments. Addition of noise As the above analyses perturb units individually, they only measure the influence of coordinate-aligned single directions. To test networks' reliance upon random single directions, we added Gaussian noise to all units with zero mean and progressively increasing variance. To scale the variance appropriately for each unit, the variance of the noise added was normalized by the empirical variance of the unit's activations across the training set. To quantify the class selectivity of individual units, we used a metric inspired by the selectivity indices commonly used in systems neuroscience BID12 BID9 Figure 1: Memorizing networks are more sensitive to cumulative ablations. Networks were trained on MNIST (2-hidden layer MLP, a), CIFAR-10 (11-layer convolutional network, b), and ImageNet (50-layer ResNet, c). In a, all units in all layers were ablated, while in b and c, only feature maps in the last three layers were ablated. Error bars represent standard deviation across 10 random orderings of units to ablate. & ). The class-conditional mean activity was first calculated across the test set, and the selectivity index was then calculated as follows: DISPLAYFORM0 with µ max representing the highest class-conditional mean activity and µ −max representing the mean activity across all other classes. For convolutional feature maps, activity was first averaged across all elements of the feature map. This metric varies from 0 to 1, with 0 meaning that a unit's average activity was identical for all classes, and 1 meaning that a unit was only active for inputs of a single class. We note that this metric is not a perfect measure of information content in single units; for example, a unit with a little information about every class would have a low class selectivity index. However, it does measure the discriminability of classes along a given direction. The selectivity index also identifies units with the same class tuning properties which have been highlighted in the analysis of DNNs BID21 BID38 BID11 BID40 BID28. However, in addition to class selectivity, we replicate all of our using mutual information, which, in contrast to class selectivity, should highlight units with information about multiple classes, and we find qualitively similar outcomes (Appendix A.5). We also note that while a class can be viewed as a highly abstract feature, implying that our may generalize to feature selectivity, we do not examine feature selectivity in this work. Here, we provide a rough intuition for why a network's reliance upon single directions might be related to generalization performance. Consider two networks trained on a large, labeled dataset with some underlying structure. One of the networks simply memorizes the labels for each input example and will, by definition, generalize poorly ('memorizing network') while the other learns the structure present in the data and generalizes well ('structure-finding network'). The minimal description length of the model should be larger for the memorizing network than for the structurefinding network. As a , the memorizing network should use more of its capacity than the structure-finding network, and by extension, more single directions. Therefore, if a random single direction is perturbed, the probability that this perturbation will interfere with the representation of the data should be higher for the memorizing network than for the structure-finding network 2. To test whether memorization leads to greater reliance on single directions, we trained a variety of network types on datasets with differing fractions of randomized labels and evaluated their performance as progressively larger fractions of units were ablated (see Sections 2.2 and 2.1). By definition, these curves must begin at the network's training accuracy (approximately 1 for all networks tested) and fall to chance levels when all directions have been ablated. To rule out variance due to the specific order of unit ablation, all experiments were performed with mutliple random ablation orderings of units. As many of the models were trained on datasets with corrupted labels and, by definition, cannot generalize, training accuracy was used to evaluate model performance. Consistent with our intuition, we found that networks trained on varying fractions of corrupted labels were significantly more sensitive to cumulative ablations than those trained on datasets comprised of true labels, though curves were not always perfectly ordered by the fraction of corrupted labels (Fig. 1).We next asked whether this effect was present if networks were perturbed along random bases. To test this, we added noise to each unit (see Section 2.2). Again, we found that networks trained on corrupted labels were substantially and consistently more sensitive to noise added along random bases than those trained on true labels (Fig. 2).The above apply to networks which are forced to memorize at least a portion of the training set -there is no other way to solve the task. However, it is unclear whether these would apply to networks trained on uncorrupted data. In other words, do the solutions found by networks with the same topology and data, but different generalization performance exhibit differing reliance upon single directions? To test this, we trained 200 networks on CIFAR-10, and evaluated their generalization error and reliance on single directions. All networks had the same topology and were trained on the same dataset (unmodified CIFAR-10). Individual networks only differed in their random initialization (drawn from identical distributions), the data order used during training, and their learning rate 3. We found that the 5 networks with the best generalization performance were more robust to the ablation of single directions than the 5 networks with the worst generalization performance (Fig. 3a). To quantify this further, we measured the area under the ablation curve for each of the 200 networks and plotted it as a function of generalization error (Fig. 3b). Interestingly, networks appeared to undergo a discrete regime shift in their reliance upon single directions; however, this effect might have been caused by degeneracy in the set of solutions found by the optimization procedure, and we note that there was also a negative correlation present within clusters (e.g., top left cluster). These demonstrate that the relationship between generalization performance and single direction reliance is not merely a side-effect of training with corrupted labels, but is instead present even among sets networks with identical training data. This relationship raises an intriguing question: can single direction reliance be used to estimate generalization performance without the need for a held-out test set? And if so, might it be used as a signal for early stopping or hyperpameter selection? As a proof-of-principle experiment for early stopping, we trained an MLP on MNIST and measured the area under the cumulative ablation curve (AUC) over the course of training along with the train and test loss. Interestingly, we found that the point in training at which the AUC began to drop was the same point that the train and test loss started to diverge (Fig. 4a). Furthermore, we found that AUC and test loss were negatively correlated (Spearman's correlation: -0.728; Fig. 4b).As a proof-of-principle experiment for hyperparameter selection, we trained 192 CIFAR-10 models with different hyperparemeter settings (96 hyperparameters with 2 repeats each; see Appendix A.2). We found that AUC and test accuracy were highly correlated (Spearman's correlation: 0.914; Fig. 4c), and by performing random subselections of 48 hyperparameter settings, AUC selected one of the top 1, 5, and 10 settings 13%, 83%, and 98% of the time, respectively, with an average difference in test accuracy between the best model selected by AUC and the optimal model of only 1 ± 1.1% (mean ± std). These suggest that single direction reliance may serve as a good proxy for hyperparameter selection and early stopping, but further work will be necessary to evaluate whether these hold in more complicated datasets. Dropout Our experiments are reminiscent of using dropout at training time, and upon first inspection, dropout may appear to discourage networks' reliance on single directions BID35. However, while dropout encourages networks to be robust to cumulative ablations up until the dropout fraction used in training, it should not discourage reliance on single directions past that point. Given enough capacity, a memorizing network could effectively guard against dropout by merely copying the information stored in a given direction to several other directions. However, the network will only be encouraged to make the minimum number of copies necessary to guard against the dropout fraction used in training, and no more. In such a case, the network would be robust to dropout so long as all redundant directions were not simultaneously removed, yet still be highly reliant on single directions past the dropout fraction used in training. To test whether this intuition holds, we trained MLPs on MNIST with dropout probabilities ∈ {0.1, 0.2, 0.3} on both corrupted and unmodified labels. Consistent with the observation in BID4, we found that networks with dropout trained on randomized labels required more epochs to converge and converged to worse solutions at higher dropout probabilities, suggesting that dropout does indeed discourage memorization. However, while networks trained on both corrupted and unmodified labels exhibited minimal loss in training accuracy as single directions were removed up to the dropout fraction used in training, past this point, networks trained on randomized labels were much more sensitive to cumulative ablations than those trained on unmodified labels (Fig. 5a). Interestingly, networks trained on unmodified labels with different dropout fractions were all similarly robust to cumulative ablations. These suggest that while dropout may serve as an effective regularizer to prevent memorization of randomized labels, it does not prevent over-reliance on single directions past the dropout fraction used in training. Batch normalization In contrast to dropout, batch normalization does appear to discourage reliance upon single directions. To test this, we trained convolutional networks on CIFAR-10 with and without batch normalization and measured their robustness to cumulative ablation of single directions. Networks trained with batch normalization were consistently and substantially more robust to these ablations than those trained without batch normalization (Fig. 5b). This suggests that in a bFigure 6: Batch normalization decreases class selectivity and increases mutual information. Distributions of class selectivity (a) and mutual information (b) for networks trained with (blue) and without batch normalization (purple). Each distribution comprises 4 model instances trained on uncorrupted CIFAR-10.addition to reducing covariate shift, as has been proposed previously BID19, batch normalization also implicitly discourages reliance upon single directions. Our thus far suggest that networks which are less reliant on single directions exhibit better generalization performance. This may appear counter-intuitive in light of extensive past work in both neuroscience and deep learning which highlights single units or feature maps which are selective for particular features or classes BID21 BID38 BID11 BID40 BID28. Here, we will test whether the class selectivity of single directions is related to the importance of these directions to the network's output. First, we asked whether batch normalization, which we found to discourage reliance on single directions, also influences the distribution of information about class across single directions. We used the selectivity index described above (see Section 2.3) to quantify the discriminability between classes based on the activations of single feature maps across networks trained with and without batch normalization. Interestingly, we found that while networks trained without batch normalization exhibited a large fraction of feature maps with high class selectivity 4, the class selectivity of feature maps in networks trained with batch normalization was substantially lower (Fig. 6a). In contrast, we found that batch normalization increases the mutual information present in feature maps (Fig. 6b). These suggest that batch normalization actually discourages the presence of feature maps with concentrated class information and rather encourages the presence of feature maps with information about multiple classes, raising the question of whether or not such highly selective feature maps are actually beneficial. We next asked whether the class selectivity of a given unit was predictive of the impact on the network's loss of ablating said unit. Since these experiments were performed on networks trained on unmodified labels, test loss was used to measure network impact. For MLPs trained on MNIST, we found that there was a slight, but minor correlation (Spearman's correlation: 0.095) between a unit's class selectivity and the impact of its ablation, and that many highly selective units had minimal impact when ablated FIG2. By analyzing convolutional networks trained on CIFAR-10 and ImageNet, we again found that, across layers, the ablation of highly selective feature maps was no more impactful than the ablation of non-selective feature maps FIG2. In fact, in the CIFAR-10 networks, there was actually a negative correlation between class selectivity and feature map importance (Spearman's correlation: -0.428, FIG2). To test whether this relationship was depth-dependent, we calculated the correlation between class selectivity and importance separately for each layer, and found that the vast majority of the negative correlation was driven by early layers, while later layers exhibited no relationship between class selectivity and importance FIG2. Interestingly, in all three networks, ablations in early layers were more impactful than ablations in later layers, consistent with theoretical observations BID29. Additionally, we performed all of the above experiments with mutual information in place of class selectivity, and found qualitatively similar (Appendix A.5). As a final test, we compared the class selectivity to the L 1 -norm of the filter weights, a metric which has been found to be a successful predictor of feature map importance in the model pruning literature BID22. Consistent with our previous observations, we found that class selectivity was largely unrelated to the L 1 -norm of the filter weights, and if anything, the two were negatively correlated (Fig. A3, see Appendix A.4 for details). Taken together, these suggest that class selectivity is not a good predictor of importance, and imply that class selectivity may actually be detrimental to network performance. Further work will be necessary to examine whether class and/or feature selectivity is harmful or helpful to network performance. Much of this work was directly inspired by BID39, and we replicate their using partially corrupted labels on CIFAR-10 and ImageNet. By demonstrating that memorizing networks are more reliant on single directions, we also provide an answer to one of the questions they posed: is there an empirical difference between networks which memorize and those which generalize?Our work is also related to work linking generalization and the sharpness of minima BID18 BID20 BID27. These studies argue that flat minima generalize better than sharp minima (though BID13 recently found that sharp minima can also generalize well). This is consistent with our work, as flat minima should correspond to solutions in which perturbations along single directions have little impact on the network output. Another approach to generalization has been to contextualize it in information theory. For example, BID0 demonstrated that networks trained on randomized labels store more information in their weights than those trained on unmodfied labels. This notion is also related to BID33, which argues that during training, networks proceed first through a loss minimization phase followed by a compression phase. Here again, our work is consistent, as networks with more information stored in their weights (i.e., less compressed networks) should be more reliant upon single directions than compressed networks. More recently, BID4 analyzed a variety of properties of networks trained on partially corrupted labels, relating performance and time-to-convergence to capacity. They also demonstrated that dropout, when properly tuned, can serve as an effective regularizer to prevent memorization. However, we found that while dropout may discourage memorization, it does not discourage reliance on single directions past the dropout probability. We found that class selectivity is a poor predictor of unit importance. This observation is consistent with a variety of recent studies in neuroscience. In one line of work, the benefits of neural systems which are robust to coordinate-aligned noise have been explored BID6 Perturbation analyses have been performed for a variety of purposes. In the model pruning literature, many studies have removed units with the goal of generating smaller models with similar performance BID22 BID3 BID24, and recent work has explored methods for discovering maximally important directions BID30 ). Recently, BID10 used cumulative ablations to measure network robustness, though the relationship to generalization was not explored. A variety of studies within deep learning have highlighted single units which are selective for features or classes BID21 BID38 BID11 BID40 BID28 BID1. Additionally, BID1 analyzed the minimum number of sufficient feature maps (sorted by a measure of selectivity) to achieve a given accuracy. However, none of the above studies has tested the relationship between a unit's class selectivity or information content and its necessity to the network's output. BID7 have quantified a related metric, concept selectivity, across layers and networks, finding that units get more concept-selective with depth, which is consistent with our own observations regarding class selectivity (see Appendix A.3). However, they also observed a correlation between the number of concept-selective units and performance on the action40 dataset across networks and architectures. It is difficult to compare these directly, as the data used are substantially different as is the method of evaluating selectivity. Nevertheless, we note that BID7 measured the absolute number of concept-selective units across networks with different total numbers of units and depths. The relationship between the number of concept-selective units and network performance may therefore arise as a of a larger number of total units (if a fixed fraction of units is concept-selective) and increased depth (we both observed that selectivity increases with depth). In this work, we have taken an empirical approach to understand what differentiates neural networks which generalize from those which do not. Our experiments demonstrate that generalization capability is related to a network's reliance on single directions, both in networks trained on corrupted and uncorrupted data, and over the course of training for a single network. They also show that batch normalization, a highly successful regularizer, seems to implicitly discourage reliance on single directions. One clear extension of this work is to use these observations to construct a regularizer which more directly penalizes reliance on single directions. As it happens, the most obvious candidate to regularize single direction reliance is dropout (or its variants), which, as we have shown, does not appear to regularize for single direction reliance past the dropout fraction used in training (Section 3.3). Interestingly, these suggest that one is able to predict a network's generalization performance without inspecting a held-out validation or test set. This observation could be used in several interesting ways. First, in situations where labeled training data is sparse, testing networks' reliance on single directions may provide a mechanism to assess generalization performance without sacrificing training data to be used as a validation set. Second, by using computationally cheap empirical measures of single direction reliance, such as evaluating performance at a single ablation point or sparsely sampling the ablation curve, this metric could be used as a signal for early-stopping or hyperparameter selection. We have shown that this metric is viable in simple datasets (Section 3.2), but further work will be necessary to evaluate viability in more complicated datasets. Another interesting direction for further research would be to evaluate the relationship between single direction reliance and generalization performance across different generalization regimes. In this work, we evaluate generalization in which train and test data are drawn from the same distribution, but a more stringent form of generalization is one in which the test set is drawn from a unique, but overlapping distribution with the train set. The extent to which single direction reliance depends on the overlap between the train and test distributions is also worth exploring in future research. This work makes a potentially surprising observation about the role of individually selective units in DNNs. We found not only that the class selectivity of single directions is largely uncorrelated with their ultimate importance to the network's output, but also that batch normalization decreases the class selectivity of individual feature maps. This suggests that highly class selective units may actually be harmful to network performance. In addition, it implies than methods for understanding neural networks based on analyzing highly selective single units, or finding optimal inputs for single units, such as activation maximization BID15 ) may be misleading. Importantly, as we have not measured feature selectivity, it is unclear whether these will generalize to featureselective directions. Further work will be necessary to clarify all of these points. To remove the influence of a given direction, its value should be fixed or otherwise modified such that it is no longer dependent on the input. However, the choice of such a fixed value can have a substantial impact. For example, if its value were clamped to one which is highly unlikely given its distribution of activations across the training set, network performance would likely suffer drastically. Here, we compare two methods for ablating directions: ablating to zero and ablating to the empirical mean over the training set. Using convolutional networks trained on CIFAR-10, we performed cumulative ablations, either ablating to zero or to the feature map's mean (means were calculated independently for each element of the feature map), and found that ablations to zero were significantly less damaging than ablations to the feature map's mean FIG4. Interestingly, this corresponds to the ablation strategies generally used in the model pruning literature BID22 BID3 BID24. MNIST MLPs For class selectivity, generalization, early stopping, and dropout experiments, each layer contained 128, 512, 2048 and 2048 units, respectively. All networks were trained for 640 epochs, with the exception of dropout networks which were trained for 5000 epochs. CIFAR-10 ConvNets Convolutional networks were all trained on CIFAR-10 for 100 epochs. Layer sizes were: 64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512, with strides of 1, 1, 2, 1, 1, 2, 1, 1, 2, 1, 1, respectively. All kernels were 3x3. For the hyperparameter sweep used in Section 3.2, learning rate and batch size were evaluated using a grid search. ImageNet ResNet 50-layer residual networks BID17 were trained on ImageNet using distributed training with 32 workers and a batch size of 32 for 200,000 steps. Blocks were structured as follows (stride, filter sizes, output channels): (1x1, 64, 64, 256) x 2, (2x2, 64, 64, 256), (1x1, 128, 128, 512) x 3, (2x2, 128, 128, 512), (1x1, 256, 256, 1024) x 5, (2x2, 256, 256, 1024), (1x1, 512, 512, 2048) x 3. For training with partially corrupted labels, we did not use any data augmen- tation, as it would have dramatically increasing the effective training set size, and hence prevented memorization. Here, we evaluate the distribution of class selectivity as a function of depth. In both networks trained on CIFAR-10 (Fig. A2a) and ImageNet (Fig. A2b), selectivity increased as a function of depth. This is consistent with BID7, who show that concept-selectivity increases with depth. It is also consistent with BID2, who show depth increases the linear decodability of class information (though they evaluate linear decodability based on an entire layer rather than a single unit).A.4 RELATIONSHIP BETWEEN CLASS SELECTIVITY AND THE FILTER WEIGHT L 1 -NORM Importantly, our on the lack of relationship between class selectivity and importance do not suggest that there are not directions which are more or less important to the network's output, nor do they suggest that these directions are not predictable; they merely suggest that class selectivity is not a good predictor of importance. As a final test of this, we compared class selectivity to the L 1 -norm of the filter weights, a metric which has been found to be a strongly correlated with the impact of removing a filter in the model pruning literature BID22. Since the L 1 -norm of the filter weights is predictive of impact of a feature map's removal, if class selectivity is also a good predictor, the two metrics should be correlated. In the ImageNet network, we found that there was no correlation between the L 1 -norm of the filter weights and the class selectivity (Fig. A3a), while in the CIFAR-10 network, we found there was actually a negative correlation (Fig. A3b). To examine whether mutual information, which, in contrast to class selectivity, highlights units with information about multiple classes, is a good predictor of importance, we performed the same experiments as in Section 3.4 with mutual information in place of class selectivity. We found, that while the were a little less consistent (e.g., there appears to be some relationship in very early and very late layers in CIFAR-10), mutual information was generally a poor predictor of unit importance FIG6.

Represent the short summary for finding the corresponding scientific article (long)

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Typical amortized inference in variational autoencoders is specialized for a single probabilistic query. Here we propose an inference network architecture that generalizes to unseen probabilistic queries. Instead of an encoder-decoder pair, we can train a single inference network directly from data, using a cost function that is stochastic not only over samples, but also over queries. We can use this network to perform the same inference tasks as we would in an undirected graphical model with hidden variables, without having to deal with the intractable partition function. The can be mapped to the learning of an actual undirected model, which is a notoriously hard problem. Our network also marginalizes nuisance variables as required. We show that our approach generalizes to unseen probabilistic queries on also unseen test data, providing fast and flexible inference. Experiments show that this approach outperforms or matches PCD and AdVIL on 9 benchmark datasets. Learning the parameters of an undirected probabilistic graphical model (PGM) with hidden variables using maximum likelihood (ML) is a notably difficult problem (; ;). When all variables are observed, the range of applicable techniques is broadened (; ; ;), but the problem remains intractable in general. When hidden variables are present, the intractability is twofold: (a) integrating out the hidden variables (also a challenge in directed models) and (b) computing the partition function. The second problem is generally deemed to be harder . After learning, the probabilistic queries are in most cases not tractable either, so one has to resort to approximations such as belief propagation or variational inference. These approximations operate in the same way regardless of whether the model is directed, and do not need to compute the partition function. In general, ML learning is harder than inference both in directed and undirected models, but even more so in the latter case. Approximate inference via belief propagation (BP) or variational inference (VI) can be cast as an optimization problem. As such, it rarely has a closed-form solution and is instead solved iteratively, which is computationally intensive. To address this problem, one can use amortized inference. A prime example of this are variational autoencoders : a learned function (typically a neural network) is combined with the reparameterization trick (; Titsias and Lázaro-) to compute the posterior over the hidden variables given the visible ones. Although a variational autoencoder (VAE) performs inference much faster than actual VI optimization, this is not without limitations: they are specialized to answer a single predefined query. In contrast, BP and VI answer arbitrary queries, albeit usually need more computation time. The end goal of learning the parameters of a PGM is to obtain a model that can answer arbitrary probabilistic queries. A probabilistic query requests the distribution of a subset of the variables of the model given some (possibly soft) evidence about another subset of variables. This allows, for instance, to train a model on full images and then perform inpainting in a test image in an arbitrary region that was not known at training time. Since the end goal is to be able to perform arbitrary inference, in this work we suggest to learn a system that is able to answer arbitrary probabilistic queries and avoid ML learning altogether, which completely sidesteps the difficulties associated to the partition function. This puts directed and undirected models on equal footing in terms of usability. To this end, we first unroll inference (we will use BP, but other options are possible) over iterations into a neural network (NN) that outputs the of an arbitrary query, and then we train said NN to increase its prediction accuracy. At training time we randomize the queries, looking for a consistent parameterization of the NN that generalizes to new queries. The hope for existence of such a parameterization comes from BP actually working for arbitrary queries in a graphical model with a single parameterization. We call this approach query training (QT). The starting point is an unnormalized PGM parameterized by θ. Its probability density can be expressed as p(x; θ) = p(v, h; θ) ∝ exp(φ(v, h; θ)), where v are the visible variables available in our data and h are the hidden variables. A query is a binary vector q of the same dimension as v that partitions the visible variables in two subsets: One for which (soft) evidence is available (inputs) and another whose conditional probability we want to estimate (outputs). Learning undirected models via query training this is not without limitations: they are specialized to answer a single predefined query. In contrast, BP and VI answer arbitrary queries, albeit usually need more computation time. The end goal of learning the parameters of a PGM is to obtain a model that can answer arbitrary probabilistic queries. A probabilistic query requests the distribution of a subset of the variables of the model given some (possibly soft) evidence about another subset of variables. This allows, for instance, to train a model on full images and then perform inpainting in a test image in an arbitrary region that was not known at training time. Since the end goal is to be able to perform arbitrary inference, in this work we suggest to learn a system that is able to answer arbitrary probabilistic queries and avoid ML learning altogether, which completely sidesteps the di culties associated to the partition function. This puts directed and undirected models on equal footing in terms of usability. To this end, we first unroll inference (we will use BP, but other options are possible) over iterations into a neural network (NN) that outputs the of an arbitrary query, and then we train said NN to increase its prediction accuracy. At training time we randomize the queries, looking for a consistent parameterization of the NN that generalizes to new queries. The hope for existence of such a parameterization comes from BP actually working for arbitrary queries in a graphical model with a single parameterization. We call this approach query training (QT). The starting point is an unnormalized PGM parameterized by ✓. Its probability density can be expressed as p(x; ✓) = p(v, h; ✓) / exp((v, h; ✓)), where v are the visible variables available in our data and h are the hidden variables. A query is a binary vector q of the same dimension as v that partitions the visible variables in two subsets: One for which (soft) evidence is available (inputs) and another whose conditional probability we want to estimate (outputs). Figure 1: One step of query training. A random sample from the training data is split according to a random query mask in input and output dimensions. The input is processed inside the QT-NN by N identical stages, producing an estimation of the sample. The cross-entropy between the true and estimated outputs is computed. A random sample from the training data is split according to a random query mask in input and output dimensions. The input is processed inside the QT-NN by N identical stages, producing an estimation of the sample. The cross-entropy between the true and estimated outputs is computed. The query-trained neural network (QT-NN) follows from specifying a graphical model φ(v, h; θ), a temperature T and a number of inference timesteps N over which to run parallel BP. The general equations of the QT-NN are given next in Section 2.2, and the equations for the simple case in which the PGM is an RBM is provided in Appendix A. As depicted in Fig. 1, a QT-NN takes as input a sample v from the dataset and a query mask q. The query q blocks the network from accessing the "output" variables, and instead only offers access to the "input" variables. Which variables are inputs and which ones are outputs is precisely the information that q contains. Then the QT-NN produces as output an estimationv of the whole input sample. Obviously, we only care about how well the network estimates the variables that it did not see at the input. So we measure how well v matches the correct v in terms of cross-entropy (CE), but only for the variables that q regards as "output". Taking expectation wrt v and q, we get the loss function that we use to train the QT-NN We minimize this loss wrt θ, T via stochastic gradient descent, sampling from the training data and some query distribution. The number of QT-NN layers N is fixed a priori. One can think of the QT-NN as a more flexible version of the encoder in a VAE: instead of hardcoding inference for a single query (normally, hidden variables given visible variables), the QT-NN also takes as input a mask q specifying which variables are observed, and provides inference for unobserved ones. Note that h is never observed. For a given set of graphical model parameters θ and temperature T we can write a feedforward function that approximately resolves arbitrary inference queries by unrolling the parallel BP equations for N iterations. First, we combine the available evidence v and the query q into a set of unary factors. Unary factors specify a probability density function over a variable. Therefore, for each dimension inside v that q labels as "input", we provide a (Dirac or Kronecker) delta centered at the value of that dimension. For the "output" dimensions and hidden variables h we set the unary factor to an uninformative, uniform density. Finally, soft evidence, if present, can be incorporated through the appropriate density function. The of this process is a unary vector of factors u that contains an informative density exclusively about the inputs and whose dimensionality is the sum of the dimensionalities of v and h. Each dimension of u will be a real number for binary variables, and a full distribution in the general case. Once v and the query q are encoded in u, we can write down the equations of parallel BP over iterations as an NN with N layers, i.e., the QT-NN. To simplify notation, let us consider a factor graph that contains only pairwise factors. Then the probabilistic predictions of the QT-NN and the messages from each layer to the next can be written as: Here m (n) collects all the messages 1 that exit layer n − 1 and enter layer n. Messages have direction, so m (n) ij is different from m (n) ji. Observe how the input term u is re-fed at every layer. The output of the network is a beliefv i for each variable i, which is obtained by a softmax in the last layer. All these equations follow simply from unrolling BP over iterations, with its messages encoded in log-space. The portion of the parameters θ relevant to the factor between variables i and j is represented by θ ij = θ ji, and the portion that only affects variable i is contained in θ i. Observe that all layers share the same parameters. The functions f θ ij (·) are directly derived from φ(x; θ) using the BP equations, and therefore inherit its parameters. Finally, parameter T is the "temperature" of the message passing, and can be set to T = 1 to retrieve the standard sum-product belief propagation or to 0 to recover max-product belief revision. Values in-between interpolate between sum-product and max-product and increase the flexibility of the NN. See Appendix A for the precise equations obtained when the PGM is an RBM. If the distribution over queries only contains queries with a single variable assigned as output (and the rest as input), and there are no hidden variables, the above cost function reduces to pseudo-likelihood training . Query training is superior to pseudo-likelihood (PL) in two ways: Firstly, it provides an explicit mechanism for handling hidden variables, and secondly and more importantly, it preserves learning in the face of high correlations in the input data, which in catastrophic failure when using PL. If two variables a and b are highly correlated, PL will fail to learn the weaker correlation between a and z, since b will always be available during training to predict a, rendering any correlation with z useless at training time. If at test time we want to predict a from z because b is not available, the prediction will fail. In contrast, query training removes multiple variables from the input, driving the model to better leverage all available sources of information. Early works in learning undirected PGMs relied on contrastive energies . More recent approaches are NVIL and AdVIL , with the latter being regarded as superior. We will use an RBM in our experiments and compare QT with PCD which is very competitive in this setting . We also show for AdVIL, although it is not necessarily expected to be superior to PCD for this model. We use exactly the same datasets and preprocessing used in the AdVIL paper, with the same RBM sizes, check for further details. The random queries are generated by assigning each variable to input or output with 0.5 chance. We report the normalized cross-entropy (NCE), which is the aggregated cross-entropy over the test data, divided by the cross-entropy of a uniform model under the same query (i.e., values below 1.0 mean a better-than-trivial model). 1. For a fully connected graph, the number of messages is quadratic in the number of variables, showing the advantage of a sparse connectivity pattern, which can be encoded in the PGM choice. Computing the NCE for QT is as simple as running the trained QT-NN. PCD and AdVIL, however, cannot solve arbitrary inference queries directly and one has to resort to slow Gibbs sampling in the learned model. Alternatively, one can turn the RBM weights learned by this methods into a QT-NN with T = 1 (essentially, running BP for a fixed number of iterations). We also provide those as PCD-BP and AdVIL-BP. For PCD we train for 1000 epochs and cross-validate the learning parameter. For AdVIL we use the code provided by the authors. For QT we unfold BP in N = 10 layers and use ADAM to learn the weights. The validation set is used to choose the learning rate and for early stopping. We use minibatches of size 500. The T parameter is learned during training. The are shown in Table 3. QT-NN produces significantly better for most datasets (marked in boldface), showing that it has learned to generalize to new probabilistic queries on unseen data. Query training is a general approach to learn to infer when the inference target is unknown at training time. It offers the following advantages: 1) no need to estimate the partition function or its gradient (the "sleep" phase of other common algorithms); 2) produces an inference network, which can be faster and more accurate than iterative VI or BP because its weights are trained to compensate for the imperfections of approximate inference run for a small number of iterations; 3) arbitrary queries can be solved. In contrast, a VAE is only trained to infer the posterior over the hidden variables, or some other constant query. Why would QT-NNs generalize to new queries or scale well? The worry is that only a small fraction of the exponential number of potential queries is seen during training. The existence of a single inference network that works reasonably well for many different queries follows from the existence of a single PGM in which BP can approximate inference. The discoverability of such a network from limited training data is not guaranteed. However, there is hope for it, since the amount of training data required to adjust the model parameters should scale with the number of these, and not with the number of potential queries. Just like training data should come from the same distribution as test data, the training queries must come from the same distribution the test queries to avoid "query overfitting". In future work we will show how QT can be used in more complex undirected models, such as grid MRFs. Other interesting research avenues are modifications to allow sample generation and unroll other inference mechanisms, such as VI. • We use 0 HV to represent a matrix of zeros of size H × V. • Similarly 1 V represents a matrix of ones of size V × 1. • When any of the above defined scalar functions is used with matrix arguments, the function is applied elementwise. Some observations: • The Hadamard product with q effectively removes the information from the elements of v not present in the query mask, replacing them with 0, which corresponds to a uniform binary distribution in logit space. • The output of the network isv andĥ, the inferred probability of 1 for both the visible and hidden units. The outputĥ is inferred but actually not used during training. • The computation of f w (x) as specified above is designed to be numerically robust. It starts by computing f MP w (x), which would be the value of f w (x) for a temperature T = 0, i.e., max-product message passing, and then performs a correction on top for positive temperatures.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Instead of learning the parameters of a graphical model from data, learn an inference network that can answer the same probabilistic queries.

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the scientific article to find a short TLDR

Question: A plethora of computer vision tasks, such as optical flow and image alignment, can be formulated as non-linear optimization problems. Before the resurgence of deep learning, the dominant family for solving such optimization problems was numerical optimization, e.g, Gauss-Newton (GN). More recently, several attempts were made to formulate learnable GN steps as cascade regression architectures. In this paper, we investigate recent machine learning architectures, such as deep neural networks with residual connections, under the above perspective. To this end, we first demonstrate how residual blocks (when considered as discretization of ODEs) can be viewed as GN steps. Then, we go a step further and propose a new residual block, that is reminiscent of Newton's method in numerical optimization and exhibits faster convergence. We thoroughly evaluate the proposed Newton-ResNet by conducting experiments on image and speech classification and image generation, using 4 datasets. All the experiments demonstrate that Newton-ResNet requires less parameters to achieve the same performance with the original ResNet. A wealth of computer vision problems (e.g., structure from motion , stereo , image alignment , optical flow (; ;) ) are posed as nonlinear optimization problems. Before the resurgence of the machine learning era, the dominant family for solving such optimization problems 1 was numerical optimization, e.g, Gauss-Newton (GN). Recently, it was proposed that the GN steps, called descent directions, can be learned and represented as a cascade regression to solve non-linear least square problems (Xiong & De la). With the advent of deep learning, the aforementioned ideas were combined with learnable feature representations using deep convolutional neural networks for solving problems such as alignment and stereo . In this paper, we first try to draw similarities between learning descent directions and the structure of the popular residual networks. Motivated by that, we further extend residual learning by adopting ideas from Newton's numerical optimization method, which exhibits faster convergence rate than Gauss-Newton based methods (both theoretically and empirically as we show in our experiments). ResNet is among the most popular architectures for approximating non-linear functions through learning. The core component of ResNet is the residual block which can be seen as a linear difference equation. That is, the t th residual block is expressed as x t`1 " x t`C x t for input x t. By considering the residual block as a discretization of Euler ODEs , each residual block expresses a learnable, first order descent direction. We propose to accelerate the convergence (i.e., employ fewer residual blocks) in approximation of non-linear functions by introducing a novel residual block that exploits second-order information in analogy to Newton's method in non-linear optimization. Since the (second order) derivative is not analytically accessible, we rely on the idea of Xiong & De la to learn the descent directions by exploiting second order information of the input. We build a deep model, called Newton-ResNet, that involves the proposed residual block. Newton-ResNet requires less residual blocks to achieve the same accuracy compared to original ResNet. This is depicted in Fig. 1; the contour 2 shows the loss landscape near the minimum of each method and indeed the proposed method requires fewer steps. Our contributions are as follows: • We first establish a conceptual link between residual blocks in deep networks and standard optimization techniques, such as Gauss-Newton. This motivates us to design a novel residual block that learns the descent directions with second order information (akin to Newton steps in nonlinear optimization). A deep network composed of the proposed residual blocks is coined as Newton-ResNet. • We show that Newton-ResNet can effectively approximate non-linear functions, by demonstrating that it requires less residual blocks and hence significantly less parameters to achieve the performance of the original ResNet. We experimentally verify our claim on four different datasets of images and speech in classification tasks. Additionally, we conduct experiments on image generation with Newton-ResNet-based GAN . • We empirically demonstrate that Newton-ResNet is a good function approximator even in the absence of activation functions, where the corresponding ResNet performs poorly. The literature on resnets is vast; we focus below on the perspectives of a) theoretical understanding, b) alternative architectures and c) modifications of the transformation path. A significant line of research is the theoretical understanding behind the performance of residual connections. The work of proves that arbitrarily deep linear residual networks have no spurious local optima; all critical points correspond to a global minimum. attribute the success of resnet to the norm presentation. More recently, proves that a network with residual connections is provably better than the corresponding network without the residuals. focus on the gradients in residual connections; they study the correlations during initialization and introduce an appropriate initialization. These works are orthogonal to ours; they methodologically study the theoretical properties of deep learning, while we focus on reducing the number of residual blocks required. same topology with different feature fusion method (addition and concatenation respectively). They propose a framework that generalizes both residual and dense connections. The work that is most closely related to ours is that of; they define a topology that includes residual connections and higher order correlations. However, we offer a new perspective on the higher order correlations. In addition, we experiment with a) large scale problems, b) with linear blocks that highway networks have not used. A popular line of research modifies the transformation path of each residual block. In ResNeXt and Inception the authors add group convolutions in the transformation path. the transformation path is a (set of) residual blocks itself, i.e. they obfuscate one residual block inside another. In wide residual networks they advocate for increased width of each block. All related works are complementary to ours, since we do not modify the transformation path modules. The applications of residual networks are diverse and often with impressive . Such applications include object detection/recognition , face recognition , generative models . However, these networks have tens or hundreds of residual blocks. A line of research that reduces the number of parameters is that of pruning the network . propose to prune the weights with small magnitude, while propose a meta-learning technique to improve heuristic pruning techniques. However, pruning does not reduce the training resources (it even increases the time because for a single model, we train the network at least twice), and it is largely based on hand-engineered heuristics. That is, there is no solid understanding of the theoretical properties of pruning methods. To develop our intuition, we explore the linear residual block in sec. 3.1, i.e. the residual block without any activation functions. In sec. 3.2, we extend the proposed formulation in the presence of activation functions as typically used in ResNet. Figure 2: Schematic of the (a) original, (b) our residual block for the t th layer. The path that includes C is referred to as the transformation path; while the other (with the identity transformation) is referred to as the shortcut path. The symbol C denotes the operations in the transformation path, e.g. convolutions in. The symbols N 1, N 2 are normalization layers, e.g. batch normalization or 1ˆ1 convolutions. The symbol˚denotes an element-wise product. Before the introduction of the residual block, all the neural networks were a composition of linear layers, e.g. fully-connected or convolutional layers, and activation functions. The (input) representation was transformed in each layer through a linear operation if we ignore the activation functions. The residual block of enables the input representation to pass through unchanged by introducing a two-pathway block consisting of a shortcut path and a transformation path. The t th residual block (in a network) is x t`1 " x t`C x t for input x t 3. That is, the residual block expresses a linear difference equation. We propose instead a new residual block that captures second order information. The new residual block is: for input x t with S the same dimensions as C and˚an element-wise product. The scalar parameter α P R is learnable and plays the role of scaling the significance of the second order interactions. To reduce the number of parameters, we can share the parameters of S and C; we introduce some normalization in the quadratic term. The proposed residual block then is expressed as: with N 1, N 2 two normalization operators. The proposed residual block is depicted in Fig. 2. Frequently activation functions are used in conjunction with the residual block. We consider a residual block with activation functions and two convolutions in the transformation path. If we define the function f t pxq " C 2 pφpC 1 xqq, the residual block is x t`1 " φpx t`ft px t qq with φ denoting an activation function, such as RELU. To avoid cluttering the notation, batch normalization is ignored in the last equation. The proposed residual block in the presence of activation functions becomes: The proposed residual block can be used with different architectures, e.g. three convolutions or with group convolutions. Implementation details: All the optimization-related hyper-parameters, e.g. the optimizer, the learning rate, the initializations, the number of epochs etc., remain the same as in the original ResNet. Further improvement can be obtained by tuning those values for our residual block, but this is out of our scope. Unless mentioned otherwise, each experiment is conducted 5 times and the average and the standard deviation are reported. The following four datasets are used in this work: 1. CIFAR10 : This is a widely used dataset that contains 60, 000 images of natural scenes. Each image is of resolution 32ˆ32ˆ3 and is classified in one of the 10 classes. 2. CIFAR100 (Krizhevsky et al.): This is an extension over CIFAR10; it includes the same amount of images but there are 100 classes. 3. ImageNet : The ImageNet 2012 dataset comprises 1.28 million training images and 50K validation images from 1000 different classes. We train networks on the training set and report the top-1 and top-5 error on the validation set. 4. Speech Commands : This newly released dataset contains 60, 000 audio files; each audio contains a single word of a duration of one second. There are 35 different words (classes) with each word having 1, 500´4, 100 recordings. Every audio file is converted into a mel-spectrogram of resolution 32ˆ32. Below, we conduct an experiment on image classification on CIFAR10 in sec. 4.1; we modify the experiment by removing the activation functions, i.e. have networks linear with respect to the weights, in sec. 4.2. Sequentially, image classification experiments on CIFAR100 and ImageNet are conducted in sec. 4.3 and sec. 4.4 respectively. In addition to the image classification experiments, we exhibit how the proposed residual block can be used on image generation in sec. 4.5. Furthermore, an experiment on audio classification is conducted in sec. 4.6. We utilize CIFAR10 as a popular dataset for classification. We train each method for 120 epochs with batch size 128. The SGD optimizer is used with initial learning rate of 0.1. The learning rate is multiplied with a factor of 0.1 in epochs 40, 60, 80, 100. We use two ResNet architectures, i.e. ResNet18 and ResNet34, as baselines. Our model, called Newton-ResNet, is built with the proposed residual blocks; we add enough blocks to match the performance of the respective baseline. In Table 1 the two different ResNet baselines are compared against Newton-ResNet; the respective Newton-ResNet models have the same accuracy. However, each Newton-ResNet has " 40% less parameters than the respective baseline. In addition, we visualize the test accuracy for ResNet18 and the respective Newton-ResNet in Fig. 3. The test error of the two models is similar throughout the training; a similar phenomenon is observed for ResNet34 in Fig. 6. We remove all the activation functions, both from the transformation path and the output activation functions. The rest of the settings remain the same as in sec. 4.1. As can be noticed in Table 2, Newton-ResNet outperforms ResNet18 by a significant margin when removing the activation functions. It is worth noting that the performance of Newton-ResNet with/without activation functions differs by 7%, i.e. decent performance can be obtained without any activation functions. o Ð φ(BN(conv(o))); end 5: for i=1:lin_proj do 8: s Ð φ2(norm(conv1ˆ1(s))); end 9: xt`1 Ð xt`o`s 10: return xt`1 11: end function Table 3: The differences of the proposed method with the original residual block are highlighted in blue. The x_proj, lin_proj are 1ˆ1 convolutions added for normalization purposes in the proposed residual block. We verify the aforementioned classification on CIFAR100. The settings remain the same as in sec. 4.1. As can be noticed in Table 4 the test accuracy of ResNet34 and Newton-ResNet is similar, however Newton-ResNet has " 44% less parameters. The experiment of sec. 4.2 with the linear blocks is repeated on CIFAR100. That is, we remove all the activation functions and train the networks. The accuracy of each method is reported on Table 5. The difference observed in sec. 4.1 becomes even more pronounced. That is, ResNet performs poorly in this case and is substantially outperformed by Newton-ResNet. We perform a large-scale classification experiment on ImageNet; due to the computational resources required, this experiment is conducted only once. Following standard practices, we utilize the following data augmentation techniques: normalization through mean RGB-channel subtraction, random resized crop to 224ˆ224, scale from 5% to 100%, aspect ratio from random horizontal flip. During inference, we perform the following augmentation techniques: normalization through mean RGB-channel subtraction, scale to 256ˆ256, and single center crop to 224ˆ224. All models are trained on a DGX station with four Tesla V100 (32GB) GPUs. We use Mxnet 4 and choose float16 instead of float32 to achieve 3.5ˆacceleration and half the GPU memory consumption. In our preliminary experiments, we noticed that the second order might cause numeric overflow in float16; this was not observed in the rest of the experiments that use float32. Hence, we use a tanh as a normalization for the second order term, i.e. the last term of. Optimization is performed using SGD with momentum 0.9, weight decay 1e´4 and a mini-batch size of 1024. The initial learning rate is set to 0.4 and decreased by a factor of 10 at 30, 60, and 80 epochs. Models are trained for 90 epochs from scratch, using linear warm-up of the learning rate during first five epochs according to. For other batch sizes due to the limitation of GPU memory, we linearly scale the learning rate (e.g. learning rate 0.1 for batch size 256). We report both the Top-1 and Top-5 single-crop validation error in Table 6. For a fair comparison, we report the from our training in both the original ResNet and Newton-ResNet 5. Newton-ResNet consistently improves the performance with an extremely small increase in computational complexity and model size. Remarkably, Newton-ResNet50 achieves a single-crop Top-5 validation error of 6.358%, exceeding ResNet50 (6.838%) by 0.48% and approaching the performance achieved by the much deeper ResNet101 network (6.068% Top-5 error). The loss and Top-1 error throughout the training are visualized in Fig. 4, which demonstrates that the proposed method performs favourably to the baseline ResNetwhen the same amount of residual blocks are used. Table 7, we remove all the "relu" activation functions for both baseline and the proposed method. Without "relu", Newton-ResNet50 achieves a single-crop Top-5 validation error of 9.114%, significantly exceeding ResNet50 (71.562%) and approaching the performance achieved by the "relu" version (6.068% Top-5 error). Deep discriminative networks include hundreds of layers, while their generative counterparts' depth is critical and restricted (mainly because they are hard to train and fit in existing hardware). We explore whether we can reduce the number of residual blocks in generative models. Generative Adversarial Networks (GAN) of have dominated the related literature due to their impressive visual . GANs include two modules, a generator and a discriminator, which are both implemented with resnet-based neural networks. The generator samples z from a prior distribution, e.g. uniform, and tries to model the target distribution; the discriminator tries to distinguish between the samples synthesized from the generator and the target distribution. GAN is typically optimized with an alternating gradient descent method. We select the architecture of (SNGAN) as a strong baseline on CIFAR10. The baseline includes 3 resnet blocks in the generator and 3 in the discriminator. We replace the original residual blocks with the proposed residual blocks; two such blocks in each module suffice to achieve the same performance. That is, we reduce by 1 the blocks in both the generator and the discriminator. In Table 8 the experimental is summarized. Note that the experiment is conducted 10 times and the mean and variance are reported. In Fig. 5 some random samples synthesized by the two methods are depicted; visually the generated samples are similar. The quantitative are added in Table 9. The two models share the same accuracy, however Newton-ResNet includes 38% less parameters. This is consistent with the experiments on classical image datasets, i.e. sec. 4.1. In this work, we establish a link between the residual blocks of ResNet architectures and learning decent directions in solving non-linear least squares (e.g., each block can be considered as a decent direction). We exploit this link and we propose a novel residual block that uses second order interactions as reminiscent of Newton's numerical optimization method (i.e., learning Newton-like descent directions). Newton-type methods are likely to converge faster than first order methods (e.g., Gauss-Newton). We demonstrate that in the proposed architecture this translates to less residual blocks (i.e., less decent directions) in the network for achieving the same performance. We conduct validation experiments on image and audio classification with residual networks and verify our intuition. Furthermore, we illustrate that with our block it is possible to remove the non-linear activation functions and still achieve competitive performance.

We demonstrate how residual blocks can be viewed as Gauss-Newton steps; we propose a new residual block that exploits second order information.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.