Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The adversarial training procedure proposed by is one of the most effective methods to defend against adversarial examples in deep neural net- works (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks. Consequentially, an adversarial training based defense is susceptible to a new class of attacks, the “blind-spot attack”, where the input images reside in “blind-spots” (low density regions) of the empirical distri- bution of training data but is still on the ground-truth data manifold. For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values. Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes defending on any valid test examples difficult due to the curse of dimensionality and the scarcity of training data. Additionally, we find that blind-spots also exist on provable defenses including and because these trainable robustness certificates can only be practically optimized on a limited set of training data. Since the discovery of adversarial examples in deep neural networks (DNNs) BID28, adversarial training under the robustness optimization framework BID24 has become one of the most effective methods to defend against adversarial examples. A recent study by BID1 showed that adversarial training does not rely on obfuscated gradients and delivers promising for defending adversarial examples on small datasets. Adversarial training approximately solves the following min-max optimization problem: where X is the set of training data, L is the loss function, θ is the parameter of the network, and S is usually a norm constrained p ball centered at 0. propose to use projected gradient descent (PGD) to approximately solve the maximization problem within S = {δ | δ ∞ ≤}, where = 0.3 for MNIST dataset on a 0-1 pixel scale, and = 8 for CIFAR-10 dataset on a 0-255 pixel scale. This approach achieves impressive defending on the MNIST test set: so far the best available white-box attacks by BID37 can only decrease the test accuracy from approximately 98% to 88% 1. However, on CIFAR-10 dataset, a simple 20-step PGD can decrease the test accuracy from 87% to less than 50% 2.The effectiveness of adversarial training is measured by the robustness on the test set. However, the adversarial training process itself is done on the training set. Suppose we can optimize perfectly, then certified robustness may be obtained on those training data points. However, if the empirical distribution of training dataset differs from the true data distribution, a test point drawn from the true data distribution might lie in a low probability region in the empirical distribution of training dataset and is not "covered" by the adversarial training procedure. For datasets that are relatively simple and have low intrinsic dimensions (MNIST, Fashion MNIST, etc), we can obtain enough training examples to make sure adversarial training covers most part of the data distribution. For high dimensional datasets (CIFAR, ImageNet), adversarial training have been shown difficult (; BID29 and only limited success was obtained. A recent attack proposed by shows that adversarial training can be defeated when the input image is produced by a generative model (for example, a generative adversarial network) rather than selected directly from the test examples. The generated images are well recognized by humans and thus valid images in the ground-truth data distribution. In our interpretation, this attack effective finds the "blind-spots" in the input space that the training data do not well cover. For higher dimensional datasets, we hypothesize that many test images already fall into these blindspots of training data and thus adversarial training only obtains a moderate level of robustness. It is interesting to see that for those test images that adversarial training fails to defend, if their distances (in some metrics) to the training dataset are indeed larger. In our paper, we try to explain the success of robust optimization based adversarial training and show the limitations of this approach when the test points are slightly off the empirical distribution of training data. Our main contributions are:• We show that on the original set of test images, the effectiveness of adversarial training is highly correlated with the distance (in some distance metrics) from the test image to the manifold of training images. For MNIST and Fashion MNIST datasets, most test images are close to the training data and very good robustness is observed on these points. For CIFAR, there is a clear trend that the adversarially trained network gradually loses its robustness property when the test images are further away from training data.• We identify a new class of attacks, "blind-spot attacks", where the input image resides in a "blind-spot" of the empirical distribution of training data (far enough from any training examples in some embedding space) but is still in the ground-truth data distribution (well recognized by humans and correctly classified by the model). Adversarial training cannot provide good robustness on these blind-spots and their adversarial examples have small distortions.• We show that blind-spots can be easily found on a few strong defense models including, BID32 and BID24. We propose a few simple transformations (slightly changing contrast and ), that do not noticeably affect the accuracy of adversarially trained MNIST and Fashion MNIST models, but these models become vulnerable to adversarial attacks on these sets of transformed input images. These transformations effectively move the test images slightly out of the manifold of training images, which does not affect generalization but poses a challenge for robust learning. Our imply that current adversarial training procedures cannot scale to datasets with a large (intrinsic) dimension, where any practical amount of training data cannot cover all the blind-spots. This explains the limited success for applying adversarial training on ImageNet dataset, where many test images can be sufficiently far away from the empirical distribution of training dataset. Adversarial examples in DNNs have brought great threats to the deep learning-based AI applications such as autonomous driving and face recognition. Therefore, defending against adversarial examples is an urgent task before we can safely deploy deep learning models to a wider range of applications. Following the emergence of adversarial examples, various defense methods have been proposed, such as defensive distillation by BID18 and feature squeezing by BID35. Some of these defense methods have been proven vulnerable or ineffective under strong attack methods such as C&W in BID5. Another category of recent defense methods is based on gradient masking or obfuscated gradient BID4;; BID10 BID25; BID20 ), but these methods are also successfully evaded by the stronger BPDA attack BID1 ). Randomization in DNNs BID34 BID33 is also used to reduce the success rate of adversarial attacks, however, it usually incurs additional computational costs and still cannot fully defend against an adaptive attacker BID1 BID0 ).An effective defense method is adversarial training, which trains the model with adversarial examples freshly generated during the entire training process. First introduced by Goodfellow et al., adversarial training demonstrates the state-of-the-art defending performance. formulated the adversarial training procedure into a min-max robust optimization problem and has achieved state-of-the-art defending performance on MNIST and CIFAR datasets. Several attacks have been proposed to attack the model release by. On the MNIST testset, so far the best attack by BID37 can only reduce the test accuracy from 98% to 88%. Analysis by BID1 shows that this adversarial training framework does not rely on obfuscated gradient and truly increases model robustness; gradient based attacks with random starts can only achieve less than 10% success rate with given distortion constraints and are unable to penetrate this defense. On the other hand, attacking adversarial training using generative models have also been investigated; both BID33 and propose to use GANs to produce adversarial examples in black-box and white-box settings, respectively. Finally, a few certified defense methods BID19 BID24 BID32 were proposed, which are able to provably increase model robustness. Besides adversarial training, in our paper we also consider several certified defenses which can achieve relatively good performance (i.e., test accuracy on natural images does not drop significantly and training is computationally feasible), and can be applied to medium-sized networks with multiple layers. Notably, BID24 analyzes adversarial training using distributional robust optimization techniques. BID32 and BID32 proposed a robustness certificate based on the dual of a convex relaxation for ReLU networks, and used it for training to provably increase robustness. During training, certified defense methods can provably guarantee that the model is robust on training examples; however, on unseen test examples a non-vacuous robustness generalization guarantee is hard to obtain. Along with the attack-defense arms race, some insightful findings have been discovered to understand the natural of adversarial examples, both theoretically and experimentally. show that even for a simple data distribution of two class-conditional Gaussians, robust generalization requires significantly larger number of samples than standard generalization. BID6 extend the well-known PAC learning theory to the case with adversaries, and derive the adversarial VC-dimension which can be either larger or smaller than the standard VC-dimension. BID3 conjecture that a robust classifier can be computationally intractable to find, and give a proof for the computation hardness under statistical query (SQ) model. Recently, BID2 prove a computational hardness under a standard cryptographic assumption. Additionally, finding the safe area approximately is computationally hard according to and. BID16 explain the prevalence of adversarial examples by making a connection to the "concentration of measure" phenomenon in metric measure spaces. BID27 conduct large scale experiments on ImageNet and find a negative correlation between robustness and accuracy. BID30 discover that data examples consist of robust and non-robust features and adversarial training tends to find robust features that have strongly-correlations with the labels. Both adversarial training and certified defenses significantly improve robustness on training data, but it is still unknown if the trained model has good robust generalization property. Typically, we evaluate the robustness of a model by computing an upper bound of error on the test set; specifically, given a norm bounded distortion, we verify if each image in test set has a robustness certificate BID8 BID23. There might exist test images that are still within the capability of standard generalization (i.e., correctly classified by DNNs with high confidence, and well recognized by humans), but behaves badly in robust generalization (i.e., adversarial examples can be easily found with small distortions). Our paper complements those existing findings by showing the strong correlation between the effectiveness of adversarial defenses (both adversarial training and some certified defenses) and the distance between training data and test points. Additionally, we show that a tiny shift in input distribution (which may or may not be detectable in embedding space) can easily destroy the robustness property of an robust model. To verify the correlation between the effectiveness of adversarial training and how close a test point is to the manifold of training dataset, we need to propose a reasonable distance metric between a test example and a set of training examples. However, defining a meaningful distance metric for high dimensional image data is a challenging problem. Naively using an Euclidean distance metric in the input space of images works poorly as it does not reflect the true distance between the images on their ground-truth manifold. One strategy is to use (kernel-)PCA, t-SNE BID14, or UMAP BID17 to reduce the dimension of training data to a low dimensional space, and then define distance in that space. These methods are sufficient for small and simple datasets like MNIST, but for more general and complicated dataset like CIFAR, extracting a meaningful low-dimensional manifold directly on the input space can be really challenging. On the other hand, using a DNN to extract features of input images and measuring the distance in the deep feature embedding space has demonstrated better performance in many applications BID13 BID22, since DNN models can capture the manifold of image data much better than simple methods such as PCA or t-SNE. Although we can form an empirical distribution using kernel density estimation (KDE) on the deep feature embedding space and then obtain probability densities for test points, our experience showed that KDE work poorly in this case because the features extracted by DNNs are still high dimensional (hundreds or thousands dimensions).Taking the above considerations into account, we propose a simple and intuitive distance metric using deep feature embeddings and k-nearest neighbour. Given a feature extraction neural network h(x), a set of n training data points X train = {x DISPLAYFORM0 where DISPLAYFORM1} is an ascending ordering of training data based on the p distance between x j test and x i train in the deep embedding space, i.e., DISPLAYFORM2 In other words, we average the embedding space distance of k nearest neighbors of x j in the training dataset. This simple metric is non-parametric and we found that the are not sensitive to the selection of k; also, for naturally trained and adversarially trained feature extractors, the distance metrics obtained by different feature extractors reveal very similar correlations with the effectiveness of adversarial training. We are also interested to investigate the "distance" between the training dataset and the test dataset to gain some insights on how adversarial training performs on the entire test set. Unlike the setting in Section 3.1, this requires to compute a divergence between two empirical data distributions. Given n training data points X train = {x DISPLAYFORM0 test}, we first apply a neural feature extractor h to them, which is the same as in Section 3.1. Then, we apply a non-linear projection (in our case, we use t-SNE) to project both h(x i train) and h(x j test) to a low dimensional space, and obtainx DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 test; H) are the KDE density functions. K is the kernel function (specifically, we use the Gaussian kernel) and H is the bandwidth parameter automatically selected by Scott's rule BID22. V is chosen as a box bounding all training and test data points. For a multi-class dataset, we compute the aforementioned KDE and K-L divergence for each class separately. Inspired by our findings of the negative correlation between the effectiveness of adversarial training and the distance between a test image and training dataset, we identify a new class of adversarial attacks called "blind-spot attacks", where we find input images that are "far enough" from any existing training examples such that:• They are still drawn from the ground-truth data distribution (i.e. well recognized by humans) and classified correctly by the model (within the generalization capability of the model); • Adversarial training cannot provide good robustness properties on these images, and we can easily find their adversarial examples with small distortions using a simple gradient based attack. Importantly, blind-spot images are not adversarial images themselves. However, after performing adversarial attacks, we can find their adversarial examples with small distortions, despite adversarial training. In other words, we exploit the weakness in a model's robust generalization capability. We find that these blind-spots are prevalent and can be easily found without resorting to complex generative models like in. For the MNIST dataset which, BID32 and BID24 demonstrate the strongest defense so far, we propose a simple transformation to find the blind-spots in these models. We simply scale and shift each pixel value. Suppose the input image x ∈ [−0.5, 0.5] d, we scale and shift each test data example x element-wise to form a new example x: DISPLAYFORM0 where α is a constant close to 1 and β is a constant close to 0. We make sure that the selection of α and β will in a x that is still in the valid input range [−0.5, 0.5] d. This transformation effectively adjusts the contrast of the image, and/or adds a gray to the image. We then perform Carlini & Wagner's attacks on these transformed images x to find their adversarial examples x adv. It is important that the blind-spot images x are still undoubtedly valid images; for example, a digit that is slightly darker than the one in test set is still considered as a valid digit and can be well recognized by humans. Also, we found that with appropriate α and β the accuracy of MNIST and Fashion-MNIST models barely decreases; the model has enough generalization capability for this set of slightly transformed images, yet their adversarial examples can be easily found. Although the blind-spot attack is beyond the threat model considered in adversarial training (e.g. ∞ norm constrained perturbations), our argument is that adversarial training (and some other defense methods with certifications only on training examples such as BID32) are unlikely to scale well to datasets that lie in a high dimensional manifold, as the limited training data only guarantees robustness near these training examples. The blind-spots are almost inevitable in high dimensional case. For example, in CIFAR-10, about 50% of test images are already in blind-spots and their adversarial examples with small distortions can be trivially found despite adversarial training. Using data augmentation may eliminate some blind-spots, however for high dimensional data it is impossible to enumerate all possible inputs due to the curse of dimensionality. In this section we present our experimental on adversarially trained models by. Results on certified defense models by BID32; BID32 and BID24 are very similar and are demonstrated in Section 6.4 in the Appendix. We conduct experiments on adversarially trained models by on four datasets: MNIST, Fashion MNIST, and CIFAR-10. For MNIST, we use the "secret" model release for the MNIST attack challenge 3. For CIFAR-10, we use the public "adversarially trained" model 4. For Fashion MNIST, we train our own model with the same model structure and parameters as the robust MNIST model, except that the iterative adversary is allowed to perturb each pixel by at most = 0.1 as a larger will significantly reduce model accuracy. We use our presented simple blind-spot attack in Section 3.3 to find blind-spot images, and use Carlini & Wagner's (C&W's) ∞ attack BID5 ) to find their adversarial examples. We found that C&W's attacks generally find adversarial examples with smaller perturbations than projected gradient descent (PGD). To avoid gradient masking, we initial our attacks using two schemes: from the original image plus a random Gaussian noise with a standard deviation of 0.2; from a blank gray image where all pixels are initialized as 0. A successful attack is defined as finding an perturbed example that changes the model's classification and the ∞ distortion is less than a given used for robust training. For MNIST, = 0.3; for Fashion-MNIST, = 0.1; and for CIFAR, = 8/255. All input images are normalized to [−0.5, 0.5]. In this set of experiments, we build a connection between attack success rate on adversarially trained models and the distance between a test example and the whole training set. We use the metric defined in Section 3.1 to measure this distance. For MNIST and Fashion-MNIST, the outputs of the first fully connected layer (after all convolutional layers) are used as the neural feature extractor h(x); for CIFAR, we use the outputs of the last average pooling layer. We consider both naturally and adversarially trained networks as the neural feature extractor, with p = 2 and k = 5. The are shown in FIG2, 2 and 3. For each test set, after obtaining the distance of each test point, we bin the test data points based on their distances to the training set and show them in the histogram at the bottom half of each figure (red). The top half of each figure (blue) represents the attack success rates for the test images in the corresponding bins. Some bars on the right are missing because there are too few points in the corresponding bins. We only attack correctly classified images and only calculate success rate on those images. Note that we should not compare the distances shown between the left and right columns of Figures 1, 2 and 3 because they are obtained using different embeddings, however the overall trends are very similar. The distribution of the average 2 (embedding space) distance between the images in test set and the top-5 nearest images in training set. As we can observe in all three figures, most successful attacks in test sets for adversarially trained networks concentrate on the right hand side of the distance distribution, and the success rates tend to grow when the distance is increasing. The trend is independent of the feature extractor being used (naturally or adversarially trained). The strong correlation between attack success rates and the distance from a test point to the training dataset supports our hypothesis that adversarial training tends to fail on test points that are far enough from the training data distribution. To quantify the overall distance between the training and the test set, we calculate the K-L divergence between the KDE distributions of training set and test set for each class according to Eq.. Then, for each dataset we take the average K-L divergence across all classes, as shown in TAB1. We use both adversarially trained networks and naturally trained networks as our feature extractors h(x). Additionally, we also calculate the average normalized distance by calculating the 2 distance between each test point and the training set as in Section 4.2, and taking the average over all test points. To compare between different datasets, we normalize each element of the feature representation h(x) to mean 0 and variance 1. We average this distance among all test points and divide it by √ d t to normalize the dimension, where d t is the dimension of the feature representation h(·).Clearly, Fashion-MNIST is the dataset with the strongest defense as measured by the attack success rates on test set, and its K-L divergence is also the smallest. For CIFAR, the divergence between training and test sets is significantly larger, and adversarial training only has limited success. The hardness of training a robust model for MNIST is in between Fashion-MNIST and CIFAR. Another important observation is that the effectiveness of adversarial training does not depend on the accuracy; for Fashion-MNIST, classification is harder as the data is more complicated than MNIST, but training a robust Fashion-MNIST model is easier as the data distribution is more concentrated and adversarial training has less "blind-spots". In this section we focus on applying the proposed blind-spot attack to MNIST and Fashion MNIST. As mentioned in Section 3.3, for an image x from the test set, the blind-spot image x = αx + β obtained by scaling and shifting is considered as a new natural image, and we use the C&W ∞ attack to craft an adversarial image x adv for x. The attack distortion is calculated as the ∞ distance between x and x adv. For MNIST, = 0.3 so we set the scaling factor to α = {1.0, 0.9, 0.8, 0.7}. For Fashion-MNIST, = 0.1 so we set the scaling factor to α = {1.0, 0.95, 0.9}. We set β to either 0 or a small constant. The case α = 1.0, β = 0.0 represents the original test set images. We report the model's accuracy and attack success rates for each choice of α and β in Table 2 and Table 3. Because we scale the image by a factor of α, we also set a stricter criterion of success -the ∞ perturbation must be less than α to be counted as a successful attack. For MNIST, = 0.3 and for Fashion-MNIST, = 0.1. We report both success criterion, and α in Tables 2 and 3. (a) α = 1.0 β = 0.0 dist= 0.218 DISPLAYFORM0 Figure 4: Blind-spot attacks on Fashion-MNIST and MNIST data with scaling and shifting on adversarially trained models. First row contains input images after scaling and shifting and the second row contains the found adversarial examples. "dist" represents the ∞ distortion of adversarial perturbations. The first rows of figures (a) and (d) represent the original test set images (α = 1.0, β = 0.0); first rows of figures (b), (c), (e), and (f) illustrate the images after transformation. Adversarial examples for these transformed images have small distortions. We first observe that for all pairs of α and β the transformation does not affect the models' test accuracy at all. The adversarially trained model classifies these slightly scaled and shifted images very well, with test accuracy equivalent to the original test set. Visual comparisons in Figure 4 show that when α is close to 1 and β is close to 0, it is hard to distinguish the transformed images from the original images. On the other hand, according to Tables 2 and 3, the attack success rates for those transformed test images are significantly higher than the original test images, for both the original criterion and the stricter criterion α. In Figure 4, we can see that the ∞ adversarial perturbation Table 2: Attack success rate (suc. rate) and test accuracy (acc) of scaled and shifted MNIST. An attack is considered successful if its ∞ distortion is less than thresholds (th.) 0.3 or 0.3α. Table 3: Attack success rate (suc. rate) and test accuracy (acc) of scaled and shifted Fashion-MNIST. An attack is considered as successful if its ∞ distortion is less than threshold (th.) 0.1 or 0.1α. required is much smaller than the original image after the transformation. Thus, the proposed scale and shift transformations indeed move test images into blind-spots. More figures are in Appendix. DISPLAYFORM1 DISPLAYFORM2 One might think that we can generally detect blind-spot attacks by observing their distances to the training dataset, using a metric similar to Eq.. Thus, we plot histograms for the distances between tests points and training dataset, for both original test images and those slightly transformed ones in FIG4. We set α = 0.7, β = 0 for MNIST and α = 0.9, β = 0 for Fashion-MNIST. Unfortunately, the differences in distance histograms for these blind-spot images are so tiny that we cannot reliably detect the change, yet the robustness property drastically changes on these transformed images. In this paper, we observe that the effectiveness of adversarial training is highly correlated with the characteristics of the dataset, and data points that are far enough from the distribution of training data are prone to adversarial attacks despite adversarial training. Following this observation, we defined a new class of attacks called "blind-spot attack" and proposed a simple scale-and-shift scheme for conducting blind-spot attacks on adversarially trained MNIST and Fashion MNIST datasets with high success rates. Our findings suggest that adversarial training can be challenging due to the prevalence of blind-spots in high dimensional datasets. As discussed in Section 3.1, we use k-nearest neighbour in embedding space to measure the distance between a test example and the training set. In Section 4.2 we use k = 5. In this section we show that the choice of k does not have much influence on our . We use the adversarially trained model on the CIFAR dataset as an example. In Figures 6, 7 and 8 we choose k = 10, 100, 1000, respectively. The are similar to those we have shown in Figure 3: a strong correlation between attack success rates and the distance from a test point to the training dataset. Figure 6: Attack success rates and distance distribution of the adversarially trained CIFAR model by. Upper: C&W ∞ attack success rate, = 8/255. Lower: distribution of the average 2 (embedding space) distance between the images in test set and the top-10 (k = 10) nearest images in training set. Figure 7: Attack success rates and distance distribution of the adversarially trained CIFAR model by. Upper: C&W ∞ attack success rate, = 8/255. Lower: distribution of the average 2 (embedding space) distance between the images in test set and the top-100 (k = 100) nearest images in training set. We also studied the German Traffic Sign (GTS) BID12 dataset. For GTS, we train our own model with the same model structure and parameters as the adversarially trained CIFAR model. We set = 8/255 for adversarial training with PGD, and also use the same as the threshold of success. The are shown in Figure 9. The GTS model behaves similarly to the CIFAR model: attack success rates are much higher when the distances between the test example and the training dataset are larger. We demonstrate more MNIST and Fashion-MNIST visualizations in FIG2. Figure 9: Attack success rate and distance distribution of GTS in. Upper: C&W ∞ attack success rate, = 8/255. Lower: distribution of the average 2 (embedding space) distance between the images in test set and the top-5 nearest images in training set. In this section we demonstrate our experimental on two other state-of-the-art certified defense methods, including convex adversarial polytope by BID32 and BID32, and distributional robust optimization based adversarial training by BID24. Different from the adversarial training by, these two methods can provide a formal certification on the robustness of the model and provably improve robustness on the training dataset. However, they cannot practically guarantee non-trivial robustness on test data. We did not include other certified defenses like BID19 and BID11 because they are not applicable to multi-layer networks. For all defenses, we use their official implementations and pretrained models (if available). FIG2 shows the on CIFAR using the defenses in BID32. TAB5 show the blind-spot attack on MNIST and Fashion-MNIST for robust models in BID32 and BID24, respectively. FIG2 shows the blind-spot attack examples on, BID32 and BID24.(a) α = 1.0 β = 0.0 dist= 0.363 Table 5: Blind-spot attack on MNIST and Fashion-MNIST for robust models by BID24. Note that we use 2 distortion for this model as it is the threat model under study in their work. For each group, the first row contains input images transformed with different scaling and shifting parameter α, β (α = 1.0, β = 0.0 is the original image) and the second row contains the found adversarial examples. d represents the distortion of adversarial perturbations. For models from and BID32 we use ∞ norm and for models from BID24 we use 2 norm. Adversarial examples for these transformed images can be found with small distortions d. DISPLAYFORM0

represent the tldr for finding the corresponding scientific article

For example, Sensory deficits in early training phases can lead to irreversible performance loss in both artificial and neuronal networks, suggesting information phenomena as the common cause, and point to the importance of the initial transient and forgetting. should be similar to Similar to humans and animals, deep artificial neural networks exhibit critical periods during which a temporary stimulus deficit can impair the development of a skill. The extent of the impairment depends on the onset and length of the deficit window, as in animal models, and on the size of the neural network. Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training. To better understand this phenomenon, we use the Fisher Information of the weights to measure the effective connectivity between layers of a network during training. Counterintuitively, information rises rapidly in the early phases of training, and then decreases, preventing redistribution of information resources in a phenomenon we refer to as a loss of "Information Plasticity". Our analysis suggests that the first few epochs are critical for the creation of strong connections that are optimal relative to the input data distribution. Once such strong connections are created, they do not appear to change during additional training. These findings suggest that the initial learning transient, under-scrutinized compared to asymptotic behavior, plays a key role in determining the outcome of the training process. Our findings, combined with recent theoretical in the literature, also suggest that forgetting (decrease of information in the weights) is critical to achieving invariance and disentanglement in representation learning. Finally, critical periods are not restricted to biological systems, but can emerge naturally in learning systems, whether biological or artificial, due to fundamental constrains arising from learning dynamics and information processing. Critical periods are time windows of early post-natal development during which sensory deficits can lead to permanent skill impairment BID12. Researchers have documented critical periods affecting a range of species and systems, from visual acuity in kittens BID35 BID33 to song learning in birds BID18. Uncorrected eye defects (e.g., strabismus, cataracts) during the critical period for visual development lead to amblyopia in one in fifty adults. The cause of critical periods is ascribed to the biochemical modulation of windows of neuronal plasticity BID10. In this paper, however, we show that deep neural networks (DNNs), while completely devoid of such regulations, respond to sensory deficits in ways similar to those observed in humans and animal models. This surprising suggests that critical periods may arise from information processing, rather than biochemical, phenomena. We propose using the information in the weights, measured by an efficient approximation of the Fisher Information, to study critical period phenomena in DNNs. We show that, counterintuitively, the information in the weights does not increase monotonically during training. Instead, a rapid growth in information ("memorization phase") is followed by a reduction of information ("reorganization" or "forgetting" phase), even as classification performance keeps increasing. This behavior is consistent across different tasks and network architectures. Critical periods are centered in the memorization phase. Under review as a conference paper at ICLR 2019 Performance is permanently impaired if the deficit is not corrected early enough, regardless of how much additional training is performed. As in animal models, critical periods coincide with the early learning phase during which test accuracy would rapidly increase in the absence of deficits (dashed). (B) For comparison, we report acuity for kittens monocularly deprived since birth and tested at the time of eye-opening (solid), and normal development visual acuity in kittens as a function of age (dashed) BID7 BID23.artificial neural networks (ANNs) are only loosely inspired by biological systems .Most studies to date have focused either on the behavior of networks at convergence (Representation Learning) or on the asymptotic properties of the numerical scheme used to get there (Optimization). The role of the initial transient, especially its effect in biasing the network towards "good" regions of the complex and high-dimensional optimization problem, is rarely addressed. To study this initial learning phase of ANNs, we replicate experiments performed in animal models and find that the responses to early deficits are remarkably similar, despite the large underlying differences between the two systems. In particular, we show that the quality of the solution depends only minimally on the final, relatively well-understood, phase of the training process or on its very first epochs; instead, it depends critically on the period prior to initial convergence. In animals, sensory deficits introduced during critical periods induce changes in the architecture of the corresponding areas BID4 BID34 BID9. To determine whether a similar phenomenon exists in ANNs, we compute the Fisher Information of the weights of the network as a proxy to measure its "effective connectivity", that is, the density of connections that are effectively used by the network in order to solve the task. Like others before us BID28, we observe two distinct phases during the training, first a "learning phase" in which the Fisher Information of the weights increases as the network learns from the data, followed by a "consolidation" or "compression" phase in which the Fisher Information decreases and stabilizes. Sensitivity to critical-period-inducing deficits is maximal exactly when the Fisher Information peaks. A layer-wise analysis of the network's effective connectivity shows that, in the tasks and deficits we consider, the hierarchy of low-level and high-level features in the training data is a key aspect behind the observed phenomena. In particular, our experiments suggest that the existence of critical periods in deep neural networks depends on the inability of the network to change its effective connectivity pattern in order to process different information (in response to deficit removal). We call this phenomenon, which is not mediated by any external factors, a loss of the "Information Plasticity" of the network. A notable example of critical period-inducing deficit, which also commonly affects humans, is amblyopia (reduced visual acuity in one eye) caused unilateral cataracts during infancy or childhood (; BID32 : Even after surgical correction of the cataracts, the Figure 2 : Sensitivity of learning phase: (C) Final test accuracy of a DNN as a function of the onset of a short 40-epoch deficit. The decrease in the final performance can be used to measure the sensitivity to deficits. The most sensitive epochs corresponds to the early rapid learning phase, before the test error (dashed line) begins to plateau. Afterwards, the network is largely unaffected by the temporary deficit. (D) This can be compared with changes in the degree of functional disconnection (normalized numbers of V1 monocular cells disconnected from the contralateral eye) as a function of the kittens' age at the onset of a 10-12-day deficit window BID26. Dashed lines are as in A and B respectively. ability of the patients to regain normal acuity in the affected eye depends both on the duration of the deficit and on its age of onset, with earlier and longer deficits causing more severe effects. In order to replicate this experimental setup in ANNs, we train a standard convolutional network (CNN) to classify objects in small 32 ⇥ 32 RGB images from the CIFAR-10 dataset BID19 ) in 10 classes. To simulate the effect of cataracts, for the first t 0 epochs the images in the dataset are downsampled to 8⇥8 and then upsampled back to 32⇥32 using bilinear interpolation, in practice blurring the image and destroying small-scale details.1 After that, the training continues for 300 more epochs, giving the network enough time to converge and ensuring it is exposed to the same number of uncorrupted images as in the control (t 0 = 0) experiment. In FIG0, we graph the final performance of the network (described in Materials and Methods) as a function of the epoch at which the deficit is corrected (t 0).We clearly observe the existence of a critical period for this deficit in the ANN: if the blur is not removed within the first 60 epochs, the final performance is severely decreased when compared to the baseline (from a test error of ⇠6.4%, [In this plot it is 8%, it is 6.4% for the resnet later. We can swap the plots or change the text] in the absence of a deficit, to more than 18% when the blur is present over 140 epochs, a ⇠300% increase). The profile of the curve is also strikingly similar to the one obtained in kittens monocularly deprived from near birth and whose visual acuity upon eye-opening was tested and plotted against the length of the deficit window BID23. Just like in humans and animal models (where critical periods are characteristic of early development), the critical period in the DNN also arises during the initial rapid learning phase. At this stage, the network is quickly learning a solution before the test error plateaus and the longer asymptotic convergence phase begins. Sensitivity to deficit. To quantify more accurately the sensitivity of the ANN to image blurring throughout its early learning phase, we introduced the deficit in a short constant window (40 epochs), starting at different epochs, and then measured the decrease in the ANN's final performance compared to the baseline. In Figure 2, we plot the final testing error of the network against the epoch of onset of the deficit. We observe that the network's sensitivity to blurring peaks in the central part of the early rapid learning phase (around 30 epochs), while later deficits produce little or no effect. A similar experiment was also performed on kittens by Olson and Freeman, using a window of 10-12 days during which the animals were monocularly deprived and using it to "scan" the first 4 months after birth to obtain a sensitivity profile BID26. We subsequently evaluated the effect of other training data modifications: a more drastic deprivation where the input is substituted with random noise, simulating complete sensory deprivation, and two "high-level" modifications of the training data: vertical flipping of the input image and permutation. Performance is permanently impaired if the deficit is not corrected early enough, regardless of how much additional training is performed. As in animal models, critical periods coincide with the early learning phase during which, in the absence of deficits, test accuracy would rapidly increase (dashed). (B) For comparison, we report acuity for kittens monocularly deprived since birth and tested at the time of eye-opening (solid), and normal visual acuity development (in kittens) as a function of their age (dashed) BID7 BID23. Sensitivity during learning: (C) Final test accuracy of a DNN as a function of the onset of a short 40-epoch deficit. The decrease in the final performance can be used to measure the sensitivity to deficits. The most sensitive epochs corresponds to the early rapid learning phase, before the test error (dashed line) begins to plateau. Afterwards, the network is largely unaffected by the temporary deficit. (D) This can be compared with changes in the degree of functional disconnection (normalized numbers of V1 monocular cells disconnected from the contralateral eye) as a function of the kittens' age at the onset of a 10-12-day deficit window BID26. Dashed lines are as in A and B respectively, up to a re-scaling of the y-axis. Our findings, described in Section 2, indicate that the early transient is critical in determining the final solution of the optimization associated with training an artificial neural network. In particular, the effects of sensory deficits during a critical period cannot be overcome, no matter how much additional training is performed. Yet most theoretical studies have focused on the network behavior after convergence (Representation Learning) or on the asymptotic properties of the optimization scheme used for training (SGD).To study this early phase, in Section 3, we use the Fisher Information to quantify the effective connectivity of a network during training, and introduce the notion of Information Plasticity in learning. Information Plasticity is maximal during the memorization phase, and decreases in the reorganization phase. We show that deficit sensitivity during critical periods correlates strongly with the effective connectivity. In Section 4 we discuss our contribution in relation to previous work. When considered in conjunction with recent on representation learning BID0, our findings indicate that forgetting (reducing information in the weights) is critical to achieving invariance to nuisance variability as well as independence of the components of the representation, but comes at the price of reduced adaptability later in the training. We also hypothesize that the loss of physical connectivity in biology (neural plasticity) could be a consequence, rather than a cause, of the loss of Information Plasticity, which depends on how the information is distributed throughout a network during the early stages of learning. These also shed light on the common practice of pre-training a model on a task and then fine-tune it for another, one of the most rudimentary forms of transfer learning. Our experiments show that, rather than helpful, pre-training can be detrimental, even if the tasks are similar (e.g., same labels, slightly blurred images). A notable example of critical period-related deficit, commonly affecting humans, is amblyopia (reduced visual acuity in one eye) caused by cataracts during infancy or childhood BID31 Published as a conference paper at ICLR 2019 DISPLAYFORM0 Figure 2: (Left) High-level perturbations do not induce a critical period. When the deficit only affects high-level features (vertical flip of the image) or the last layer of the CNN (label permutation), the network does not exhibit critical periods (test accuracy remains largely flat). On the other hand, a sensory deprivation-like deficit (image is replaced by random noise) does cause a deficit, but the effect is less severe than in the case of image blur. (Right) Dependence of the critical period profile on the network's depth. Adding more convolutional layers increases the effect of the deficit during its critical period (shown here is the decrease in test accuracy due to the deficit with respect to the test accuracy reached without deficits).von BID32. Even after surgical correction of cataracts, the ability of the patients to regain normal acuity in the affected eye depends both on the duration of the deficit and on its age of onset, with earlier and longer deficits causing more severe effects. In this section, we aim to study the effects of similar deficits in DNNs. To do so, we train a standard All-CNN architecture based on (see Appendix A) to classify objects in small 32 × 32 images from the CIFAR-10 dataset BID19 ). We train with SGD using an exponential annealing schedule for the learning rate. To simulate the effect of cataracts, for the first t 0 epochs the images in the dataset are downsampled to 8 × 8 and then upsampled back to 32 × 32 using bilinear interpolation, in practice blurring the image and destroying small-scale details. After that, the training continues for 160 more epochs, giving the network time to converge and ensuring it is exposed to the same number of uncorrupted images as in the control (t 0 = 0) experiment. In FIG0, we plot the final performance of a network affected by the deficit as a function of the epoch t 0 at which the deficit is corrected. We can readily observe the existence of a critical period: If the blur is not removed within the first 40-60 epochs, the final performance is severely decreased when compared to the baseline (up to a threefold increase in error). The decrease in performance follows trends commonly observed in animals, and may be qualitatively compared, for example, to the loss of visual acuity observed in kittens monocularly deprived from birth as a function of the length of the deficit BID23. 2 We can measure more accurately the sensitivity to a blur deficit during learning by introducing the deficit in a short window of constant length (40 epochs), starting at different epochs, and then measure the decrease in the DNN's final performance compared to the baseline FIG0 ). Doing this, we observe that the sensitivity to the deficit peaks in the central part of the early rapid learning phase (at around 30 epochs), while introducing the deficit later produces little or no effect. A similar experiment performed on kittens, using a window of 10-12 days during which the animals are monocularly deprived, again shows a remarkable similarity between the profiles of the sensitivity curves BID26. High-level deficits are not associated with a critical period: A natural question is whether any change in the input data distribution will have a corresponding critical period for learning. This is not the case for neuronal networks, which remain plastic enough to adapt to high-level changes in sensory processing BID4. For example, it is well-reported that even adult humans can rapidly adapt to certain drastic changes, such as the inversion of the visual field BID30 BID17. In Figure 2, we observe that DNNs are also largely unaffected by high-level deficits -such as vertical flipping of the image, or random permutation of the output labels: After deficit correction, the network quickly recovers its baseline performance. This hints at a finer interplay between the structure of the data distribution and the optimization algorithm, ing in the existence of a critical period. We now apply to the network a more drastic deficit, where each image is replaced by white noise. Figure 2 shows hows this extreme deficit exhibits a remarkably less severe effect than the one obtained by only blurring images: Training the network with white noise does not provide any information on the natural images, and in milder effects than those caused by a deficit (e.g., image blur), which instead conveys some information, but leads the network to (incorrectly) learn that no fine structure is present in the images. A similar effect has been observed in animals, where a period of early sensory deprivation (dark-rearing) can lengthen the critical period and thus cause less severe effects than those documented in light-reared animals . We refer the reader to Appendix C for a more detailed comparison between sensory deprivation and training on white noise. Architecture, depth, and learning rate annealing: FIG2 shows that a fully-connected network trained on the MNIST digit classification dataset also shows a critical period for the image blur deficit. Therefore, the convolutional structure is not necessary, nor is the use of natural images. Similarly, a ResNet-18 trained on CIFAR-10 also has a critical period, which is also remarkably sharper than the one found in a standard convolutional network FIG0 ). This is especially interesting, since ResNets allow for easier backpropagation of gradients to the lower layers, thus suggesting that the critical period is not caused by vanishing gradients. However, Figure 2 (Right) shows that the presence of a critical period does indeed depend critically on the depth of the network. In FIG2, we confirm that a critical period exists even when the network is trained with a constant learning rate, and therefore cannot be explained by an annealed learning rate in later epochs. Optimization method and weight decay: FIG2 (Bottom Right) shows that when using Adam as the optimization scheme, which renormalizes the gradients using a running mean of their first two moments, we still observe a critical period similar to that of standard SGD. However, changing the, showing two distinct phases of training: First, information sharply increases, but once test performance starts to plateau (green line), the information in the weights decreases during a "consolidation" phase. Eventually less information is stored, yet test accuracy improves slightly (green line). The weights' Fisher Information correlates strongly with the networks sensitivity to critical periods, computed as in FIG0 using both a window size of 40 and 60, and fitted here to the Fisher Information using a simple exponential fit. (Center) Recalling the connection between FIM ad connectivity, we may compare it to synaptic density during development in the visual cortex of macaques BID27. Here too, a rapid increase in connectivity is followed by elimination of synapses (pruning) continuing throughout life. (Right) Effects of critical period-inducing blurring on the Fisher Information: The impaired network uses more information to solve the task, compared to training in the absence of a deficit, since it is forced to memorize the labels case by case.hyperparameters of the optimization can change the shape of the critical period: In FIG2 (Bottom Left) we show that increasing weight decay makes critical periods longer and less sharp. This can be explained as it both slows the convergence of the network, and it limits the ability of higher layers to change to overcome the deficit, thus encouraging lower layers to also learn new features. We have established empirically that, in animals and DNNs alike, the initial phases of training are critical to the outcome of the training process. In animals, this strongly relates to changes in the brain architecture of the areas associated with the deficit BID4. This is inevitably different in artificial networks, since their connectivity is formally fixed at all times during training. However, not all the connections are equally useful to the network: Consider a network encoding the approximate posterior distribution p w (y|x), parameterized by the weights w, of the task variable y given an input image x. The dependency of the final output from a specific connection can be estimated by perturbing the corresponding weight and looking at the magnitude of the change in the final distribution. Specifically, given a perturbation w = w + δw of the weights, the discrepancy between the p w (y|x) and the perturbed network output p w (y|x) can be measured by their KullbackLeibler divergence, which, to second-order approximation, is given by: DISPLAYFORM0 where the expectation over x is computed using the empirical data distributionQ(x) given by the dataset, and DISPLAYFORM1 is the Fisher Information Matrix (FIM). The FIM can thus be considered a local metric measuring how much the perturbation of a single weight (or a combination of weights) affects the output of the network BID1. In particular, weights with low Fisher Information can be changed or "pruned" with little effect on the network's performance. This suggests that the Fisher Information can be used as a measure of the effective connectivity of a DNN, or, more generally, of the "synaptic strength" of a connection BID15. Finally, the FIM is also a semidefinite approximation of the Hessian of the loss function BID21 and hence of the curvature of the loss landscape at a particular point w during training, providing an elegant connection between the FIM and the optimization procedure BID1, which we will also employ later. Published as a conference paper at ICLR 2019Unfortunately, the full FIM is too large to compute. Rather, we use its trace to measure the global or layer-wise connection strength, which we can compute efficiently using (Appendix A): DISPLAYFORM2 In order to capture the behavior of the off-diagonal terms, we also tried computing the logdeterminant of the full matrix using the Kronecker-Factorized approximation of BID22, but we observed the same qualitative trend as the trace. Since the FIM is a local measure, it is very sensitive to the irregularities of the loss landscape. Therefore, in this section we mainly use ResNets, which have a relatively smooth landscape BID20. For other architectures we use instead a more robust estimator of the FIM based on the injection of noise in the weights BID0, also described in Appendix A.Two phases of learning: As its name suggests, the FIM can be thought as a measure of the quantity of information about the training data that is contained in the model BID6. Based on this, one would expect the overall strength of the connections to increase monotonically as we acquire information from experience. However, this is not the case: While during an initial phase the network acquires information about the data, which in a large increase in the strength of the connections, once the performance in the task begins to plateau, the network starts decreasing the overall strength of its connections. However, this does not correspond to a reduction in performance, rather, performance keeps slowly improving. This can be seen as a "forgetting, or "compression" phase, during which redundant connections are eliminated and non-relevant variability in the data is discarded. It is well-established how the elimination ("pruning") of unnecessary synapses is a fundamental process during learning and brain development BID27 ) (FIG3, Center); in FIG3 (Left) an analogous phenomenon is clearly and quantitatively shown for DNNs. Strikingly, these changes in the connection strength are closely related to the sensitivity to criticalperiod-inducing deficits such as image blur, computed using the "sliding window" method as in FIG0. In FIG3 we see that the sensitivity closely follows the trend of the FIM. This is remarkable since the FIM is a local quantity computed at a single point during the training of a network in the absence of deficit, while sensitivity during a critical period is computed, using test data, at the end of the impaired network training. FIG3 (Right) further emphasizes the effect of deficits on the FIM: in the presence of a deficit, the FIM grows and remains substantially higher even after the deficit is removed. This may be attributed to the fact that, when the data are so corrupted that classification is impossible, the network is forced to memorize the labels, therefore increasing the quantity of information needed to perform the same task. Layer-wise effects of deficits: A layer-wise analysis of the FIM sheds further light on how the deficit affects the network. When the network (in this case All-CNN, which has a clearer division among layers than ResNet) is trained without deficits, the most important connections are in the intermediate layers (FIG5, Left), which can process the input CIFAR-10 image at the most informative intermediate scale. However, if the network is initially trained on blurred data (FIG5, top right), the strength of the connections is dominated by the top layer (Layer 6). This is to be expected, since the low-level and mid-level structures of the images are destroyed, making the lower layers ineffective. However, if the deficit is removed early in the training (FIG5, top center), the network manages to "reorganize", reducing the information contained in the last layer, and, at the same time, increasing the information in the intermediate layers. We refer to these phenomena as changes in "Information Plasticity". If, however, the data change occurs after the consolidation phase, the network is unable to change its effective connectivity: The connection strength of each layer remains substantially constant. The network has lost its Information Plasticity and is past its critical period. The analysis of the FIM also sheds light on the geometry of the loss function and the learning dynamics. Since the FIM can be interpreted as the local curvature of the residual landscape, FIG3 shows that learning entails crossing bottlenecks: In the initial phase the network enters regions of high curvature (high Fisher Information), and once consolidation begins, the curvature decreases, allowing it to cross the bottleneck and enter the valley below. If the statistics change after crossing the bottleneck, the network is trapped. In this interpretation, the early phases of convergence are critical in leading the network towards the "right" final valley. The end of critical periods comes after the network has crossed all bottlenecks (and thus learned the features) and entered a wide valley (region of the weight space with low curvature, or low Fisher Information). Blur deficit until epoch 100 In the presence of an image blur deficit until epoch 100, more resources are allocated to the higher layers rather than to the middle layers. The blur deficit destroys low-and mid-level features processed by those layers, leaving only the global features of the image, which are processed by the higher layers. Even if the deficit is removed, the middle layers remain underdeveloped. (Top Center) When the deficit is removed at an earlier epoch, the layers can partially reconfigure (notice, e.g., the fast loss of information of layer 6), ing in less severe long-term consequences. We refer to the redistribution of information and the relative changes in effective connectivity as "Information Plasticity". (Bottom row) Same plots, but using a vertical flip deficit, which does not induce a critical period. As expected, the quantity of information in the layers is not affected. Critical periods have thus far been considered an exclusively biological phenomenon. At the same time, the analysis of DNNs has focused on asymptotic properties and neglected the initial transient behavior. To the best of our knowledge, we are the first to show that artificial neural networks exhibit critical period phenomena, and to highlight the critical role of the transient in determining the asymptotic performance of the network. Inspired by the role of synaptic connectivity in modulating critical periods, we introduce the use of Fisher Information to study this initial phase. We show that the initial sensitivity to deficits closely follows changes in the FIM, both global, as the network first rapidly increases and then decreases the amount of stored information, and layer-wise, as the network "reorganizes" its effective connectivity in order to optimally process information. Our work naturally relates to the extensive literature on critical periods in biology. Despite artificial networks being an extremely reductionist approximation of neuronal networks, they exhibit behaviors that are qualitatively similar to the critical periods observed in human and animal models. Our information analysis shows that the initial rapid memorization phase is followed by a loss of Information Plasticity which, counterintuitively, further improves the performance. On the other hand, when combined with the analysis of BID0 this suggests that a "forgetting" phase may be desirable, or even necessary, in order to learn robust, nuisance-invariant representations. The existence of two distinct phases of training has been observed and discussed by BID28, although their analysis builds on the (Shannon) information of the activations, rather than the (Fisher) information in the weights. On a multi-layer perceptron (MLP), BID28 empirically link the two phases to a sudden increase in the gradients' covariance. It may be tempting to compare these with our Fisher Information analysis. However, it must be noted that the FIM is computed using the gradients with respect to the model prediction, not to the ground truth label, leading to important qualitative differences. In Figure 6, we show that the covariance and norm of the gradients exhibit no clear trends during training with and without deficits, and, therefore, unlike the FIM, do not correlate with the sensitivity to critical periods. However, Published as a conference paper at ICLR 2019 a connection between our FIM analysis and the information in the activations can be established based on the work of BID0, which shows that the FIM of the weights can be used to bound the information in the activations. In fact, we may intuitively expect that pruning of connections naturally leads to loss of information in the corresponding activations. Thus, our analysis corroborates and expands on some of the claims of BID28, while using an independent framework. Aside from being more closely related to the deficit sensitivity during critical periods, Fisher's Information also has a number of technical advantages: Its diagonal is simple to estimate, even on modern state-of-the-art architectures and compelling datasets, and it is less sensitive to the choice estimator of mutual information, avoiding some of the common criticisms to the use of information quantities in the analysis of deep learning models. Finally, the FIM allows us to probe fine changes in the effective connectivity across the layers of the network FIG5 ), which are not visible in BID28.A complete analysis of the activations should account not only for the amount of information (both task-and nuisance-related), but also for its accessibility, e.g., how easily task-related information can be extracted by a linear classifier. Following a similar idea, BID24 aim to study the layer-wise, or "spatial" (but not temporal) evolution of the simplicity of the representation by performing a principal component analysis (PCA) of a radial basis function (RBF) kernel embedding of each layer representation. They show that, on a multi-layer perceptron, task-relevant information increasingly concentrate on the first principal components of the representation's embedding, implying that they become more easily "accessible" layer after layer, while nuisance information (when it is codified at all) is encoded in the remaining components. In our work we instead focus on the temporal evolution of the weights. However, it's important to notice that a network with simpler weights (as measured by the FIM) also requires a simpler smooth representation (as measured, e.g., by the RBF embedding) in order to operate properly, since it needs to be resistant to perturbations of the weights. Thus our analysis is wholly compatible with the intuitions of BID24. It would also be interesting to study the joint spatio-temporal evolution of the network using both frameworks at once. One advantage of focusing on the information of the weights rather than on the activations, or behavior of the network, is to have a readout of the "effective connectivity" during critical periods, which can be compared to similar readouts in animals. In fact, "behavioral" readouts upon deficit removal, both in artificial and neuronal networks, can potentially be confounded by deficit-coping changes at different levels of the visual pathways BID4 BID16. On the other hand, deficits in deprived animals are mirrored by abnormalities in the circuitry of the visual pathways, which we characterize in DNNs using the FIM to study its "effective connectivity", i.e., the connections that are actually employed by the network to solve the task. Sensitivity to critical periods and the trace of the Fisher Information peak at the same epochs, in accord with the evidence that skill development and critical periods in neuronal networks are modulated by changes (generally experience-dependent) in synaptic plasticity BID16 BID10. Our layer-wise analysis of the Fisher Information FIG5 ) also shows that visual deficits reinforce higher layers to the detriment of intermediate layers, leaving low-level layers virtually untouched. If the deficit is removed after the critical period ends, the network is not able to reverse these effects. Although the two systems are radically different, a similar response can be found in the visual pathways of animal models: Lower levels (e.g., retina, lateral geniculate nucleus) and higher-level visual areas (e.g., V2 and post-V2) show little remodeling upon deprivation, while most changes happen in different layers of V1 BID34 BID9 ).An insightful interpretation of critical periods in animal models was proposed by BID16: The initial connections of neuronal networks are unstable and easily modified (highly plastic), but as more "samples" are observed, they change and reach a more stable configuration which is difficult to modify. Learning can, however, still happen within the newly created connectivity pattern. This is largely compatible with our findings: Sensitivity to critical-period-inducing deficits peaks when connections are remodeled (Figure 4, Left), and different connectivity profiles are observed in networks trained with and without a deficit (FIG5). Moreover, high-level deficits such as imageflipping and label permutation, which do not require restructuring of the network's connections in order to be corrected, do not exhibit a critical period. Applying a deficit at the beginning of the training may be compared to the common practice of pretraining, which is generally found to improve the performance of the network. BID5 study the somewhat related, but now seldom used, practice of layer-wise unsupervised pre-training, and suggest that it may act as a regularizer by moving the weights of the network towards an area of the loss landscape closer to the attractors for good solutions, and that early examples have a stronger effect in steering the network towards particular solutions. Here, we have shown that pre-training on blurred data can have the opposite effect; i.e., it can severely decrease the final performance of the network. However, in our case, interpreting the deficits effect as moving the network close to a bad attractor is difficult to reconcile with the smooth transition observed in the critical periods, since the network would either converge to this attractor, and thus have low accuracy, or escape completely. Instead, we reconcile our experiments with the geometry of the loss function by introducing a different explanation based on the interpretation of the FIM as an approximation of the local curvature. FIG3 suggests that SGD encounters two different phases during the network training: At first, the network moves towards high-curvature regions of the loss landscape, while in the second phase the curvature decreases and the network eventually converges to a flat minimum (as observed in BID13). We can interpret these as the network crossing narrow bottlenecks during its training in order to learn useful features, before eventually entering a flat region of the loss surface once learning is completed and ending up trapped there. When combining this assumption with our deficit sensitivity analysis, we can hypothesize that the critical period occurs precisely upon crossing of this bottleneck. It is also worth noticing how there is evidence that convergence to flat minima (minima with low curvature) in a DNN correlates with a good generalization performance BID11 BID20 BID3 BID13. Indeed, using this interpretation, FIG3 (Right) tells us that networks more affected by the deficit converge to sharper minima. However, we have also found that the performance of the network is already mostly determined during the early "sensitive" phase. The final sharpness at convergence may therefore be an epiphenomenon, rather than the cause of good generalization. Our goal in this paper is not so much to investigate the human (or animal) brain through artificial networks, as to understand fundamental information processing phenomena, both in their biological or artificial implementations. It is also not our goal to suggest that, since they both exhibit critical periods, DNNs are necessarily a valid model of neurobiological information processing, although recent work has emphasized this aspect. We engage in an "Artificial Neuroscience" exercise in part to address a technological need to develop "explainable" artificial intelligence systems whose behavior can be understood and predicted. While traditionally well-understood mathematical models were used by neuroscientists to study biological phenomena, information processing in modern artificial networks is often just as poorly understood as in biology, so we chose to exploit well-known biological phenomena as probes to study information processing in artificial networks. Conversely, it would also be interesting to explore ways to test whether biological networks prune connections as a consequences of a loss of Information Plasticity, rather than as a cause. The mechanisms underlying network reconfiguration during learning and development might be an evolutionary outcome obtained under the pressure of fundamental information processing phenomena. DISPLAYFORM0 In order to avoid interferences between the annealing scheme and the architecture, in these experiments we fix the learning rate to 0.001.The Fully Connected network used for the MNIST experiments has hidden layers of size. All hidden layers use batch normalization followed by ReLU activations. We fix the learning rate to 0.005. Weight decay is not used. We use data augmentation with random translations up to 4 pixels and random horizontal flipping. For MNIST, we pad the images with zeros to bring them to size 32 × 32. To compute the trace of the Fisher Information Matrix, we use the following expression derived directly from the definition: DISPLAYFORM0 where the input image x is sampled from the dataset, while the label y is sampled from the output posterior. Expectations are approximated by Monte-Carlo sampling. Notice, however, that this expression depends only on the local gradients of the loss with respect to the weights at a point w = w 0, so it can be noisy when the loss landscape is highly irregular. This is not a problem for ResNets BID20, but for other architectures we use instead a different technique, proposed in BID0. More in detail, let L(w) be the standard cross-entropy loss. Given the current weights w 0 of the network, we find the diagonal matrix Σ that minimizes: DISPLAYFORM1 where β is a parameter that controls the smoothness of the approximation. Notice that L can be minimized efficiently using the method in BID14. To see how this relates to the Fisher Information Matrix, assume that L(w) can be approximated locally in DISPLAYFORM2 Taking the derivative with respect to Σ, and setting it to zero, we obtain Σ ii = β/H ii. We can then use Σ to estimate the trace of the Hessian, and hence of the Fisher information. Published as a conference paper at ICLR 2019 DISPLAYFORM3 Fitting of sensitivity curves and synaptic density profiles from the literature was performed using: DISPLAYFORM4 as the fitting equation, where t is the age at the time of sampling and τ 1, τ 2, k and d are unconstrained parameters BID2.The exponential fit of the sensitivity to the Fisher Information trace uses the expression DISPLAYFORM5 where a, b and c are unconstrained parameters, F (t) is the Fisher Information trace at epoch t of the training of a network without deficits and S k is the sensitivity computed using a window of size k. That is, S k (t) is the increase in the final test error over a baseline when the network is trained in the presence of a deficit between epochs t and t + k. B ADDITIONAL PLOTS Figure 6: Log of the norm of the gradient means (solid line) and standard deviation (dashed line) during training when: (Left) No deficit is present, (Center) A blur deficit is present until epoch 70, and (Right) a deficit is present until the last epoch. Notice that the presence of a deficit does not decrease the magnitude of the gradients propagated to the first layers during the last epochs, rather it seems to increase it, suggesting that vanishing gradients are not the cause of the critical period for the blurring deficit. Noise deficit until epoch 100 Figure 7: Same plot as in FIG5, but for a noise deficit. Unlike with blur, much more resources are allocated to the lower-layers rather than higher-layers. This may explain why it is easier for the network to reconfigure to solve the task after the deficit is removed. No deficit Blurred up to epoch 100 Figure 8: Visualization of the filters of the first layer of the network used for the experiment in FIG0. In absence of a deficit, the network learns high-frequency filters, as seen by the fact that many filters are not smooth (first picture). However, when a blurring deficit is present, the network learns only smooth filters corresponding to low-frequencies of the input (third picture). If the deficit is removed after the end of the critical period, the network does not manage to learn high-frequency filters (second picture). Critical periods are task-and deficit-specific. The specific task we address is visual acuity, but the performance is necessarily measured through different mechanisms in animals and Artificial Neural Networks. In animals, visual acuity is traditionally measured by testing the ability to discriminate between black-and-white contrast gratings (with varying spatial frequency) and a uniform gray field. The outcome of such tests generally correlates well with the ability of the animal to use the eye to solve other visual tasks relying on acuity. Convolutional Neural Networks, on the other hand, have a very different sensory processing mechanism (based on heavily quantized data), which may trivialize such a test. Rather, we directly measure the performance of the network on an high-level task, specifically image classification, for which CNNs are optimized. We chose to simulate cataracts in our DNN experiments, a deficit which allows us to explore its complex interactions with the structure of the data and the architecture of the network. Unfortunately, while the overall trends of cataract-induced critical periods have been studied and understood in animal models, there is not enough data to confidently regress sensibility curves comparable to those obtained in DNNs. For this reason, in FIG0 we compare the performance loss in a DNN trained in the presence of a cataract-like deficit with the obtained from monocularly deprived kittens, which exhibit similar trends and are one of the most common experimental paradigms in the visual neurosciences. Simulating complete visual deprivation in a neural network is not as simple as feeding a constant stimulus: a network presented with a constant blank input will rapidly become trivial and thus unable to train on new data. This is to be expected, since a blank input is a perfectly predictable stimulus and thus the network can quickly learn the (trivial) solution to the task. We instead wanted to model an uninformative stimulus, akin to noise. Moreover, even when the eyes are sutured or maintained in the darkness, there will be excitation of photoreceptors that is best modeled as noise. To account for this, we simulate sensory deprivation by replacing the input images with a dataset composed of (uninformative) random Gaussian noise. This way the network is trained on solving the highly non-trivial task of memorizing the association between the finitely-many noise patterns and their corresponding labels.

We show that even the strongest adversarial training methods cannot defend against adversarial examples crafted on slightly scaled and shifted test images.

Represent the TLDR for finding the corresponding scientific article

Batch normalization reduces adversarial robustness, as well as general robustness in many cases, particularly to noise corruptions.


Question: Represent the scientific article to find a short TLDR

Question: Edge intelligence especially binary neural network (BNN) has attracted considerable attention of the artificial intelligence community recently. BNNs significantly reduce the computational cost, model size, and memory footprint. However, there is still a performance gap between the successful full-precision neural network with ReLU activation and BNNs. We argue that the accuracy drop of BNNs is due to their geometry. We analyze the behaviour of the full-precision neural network with ReLU activation and compare it with its binarized counterpart. This comparison suggests random bias initialization as a remedy to activation saturation in full-precision networks and leads us towards an improved BNN training. Our numerical experiments confirm our geometric intuition. Convolutional neural network has become one of the most powerful tools for solving computer vision, natural language processing, speech recognition, machine translation, and many other complex tasks. The most successful and widely-used recipe for deep neural network is ReLU-style activation function with MSRA style weight initialization . The standard sigmoid and the hyperbolic tangent were the most common activation functions, before the introductio of ReLU. ReLU-like activation functions are widely proved to be superior in terms of accuracy and convergence speed. It is more common to use low-bit quantized networks such as Binary Neural Networks (BNNs) to implement such deep neural networks on edge devices such as cell phones, smart wearables, etc. BNNs only keeps the sign of weights and compute the sign of activations {−1, +1} by applying Sign function in the forward pass. In backward propagation, BNN uses Straight-Through-Estimator (STE) to estimate the backward gradient through the Sign function and update on full-precision weights. The forward and backward loop of a BNN, therefore, becomes similar to the full-precision neural network with hard hyperbolic tangent htanh activation. The htanh function is a piece-wise linear version of the nonlinear hyper-bolic tangent, and is known to be inferior in terms of accuracy compared to ReLU-like activation function. We examine a full-precision network with htanh activation to provide a new look in improving BNN performance. We conclude that the bias initialization is the key to mimic ReLU geometric behavior in networks with htanh activation. This challenges the common practice of deterministic bias initialization for neural networks. Although the analysis is based on htanh function, this equally applies to BNNs that use STE, a htanh-like, back propagation scheme. Other saturating activations like hyperbolic tangent and sigmoid commonly applied in recurrent neural networks may benefit from this resolution as well. Our novelties can be summarized in four items i) we analyze the geometric properties of ReLU and htanh activation. This provides an insight into the training efficiency of the unbounded asymmetric activation functions such as ReLU. ii) we propose bias initialization strategy as a remedy to the bounded activations such as htanh. iii) We back up our findings with experiments on full-precision to reduce the performance gap between htanh and ReLU activations. iv) We show this strategy also improves BNNs, whose geometric behavior is similar to the full-precision neural network with htanh activation. There are very few works that focus on the initialization strategy of the bias term of the neural network. To the best of our knowledge, we are the first to propose random bias initialization as a remedy to the saturating full-precision neural network, also as a method to improve BNN training. 2 RELATED WORKS proposed training deep neurals network with ReLU activation, and argued that ReLU activation alleviates the vanishing gradient problem and encourages sparsity in the model. The hyperbolic tangent only allowed training of shallow neural networks. Since AlexNet , almost every successful neural network architectures use ReLU activation or its variants, such as adaptive ReLU, leaky ReLU, etc. Although many works reported that ReLU activation outperforms the traditional saturating activation functions, the reason for its superior performance remains an open question. utilized automatic search techniques on searching different activation functions. Most top novel activation functions found by the searches have an asymmetric saturating regime, which is similar to ReLU. adapts ReLU and sigmoid while training. To improve the performance of saturating activations, proposed penalized tanh activation, which introduces asymmetric saturating regime to tanh by inserting leaky ReLU before tanh. The penalized tanh could achieve the same level of performance as ReLU activating CNN. It is worth to mention that similar ideas also appear in the related works of binarized neural network. improved the performance of saturating activations by adding random noise when the neuron is saturated, so the backward signal can easily pass through the whole model, and the model becomes easier to optimize. In this works, we proposed to randomize the non-saturated regime by using random bias initialization. This initialization can guarantee all backward signals can pass through the whole model equally. The initial work on BNN appeared in , which limits both weights and activations to −1 and +1, so the weighted sum can be computed by bit-wise XNOR and PopCount instructions. This solution reduces memory usage and computational cost up to 32X compared with its full-precision counterpart. In the original paper, BNN was tested on VGG-7 architecture. Although it is an over-parameterized architecture for CIFAR 10 dataset, there is a performance gap between BNN and full-precision with ReLU activation. We believe the different between the two activations, BNN using the sign and full-precision using ReLU, is partially responsible for this gap. XNOR-Net developed the idea of BNN and proposed to approximate the full-precision neural network by using scaling factors. They suggest inserting non-Binary activation (like ReLU) after the binary convolution layer. This modification helps training considerably. Later, Tang et al. replaced replacing ReLU activation with PReLU in XNOR-Net to improve the accuracy. Note that XNOR-Net and many relaated works require to store the full-precision activation map during the inference stage, therefore their memory occupation is significantly larger than the pure 1-bit solution like the vanilla BNN. A typical full-precision neural network block can be described by Neural networks are trained using the back-propagation algorithm. Back propagation is composed of two components i) forward pass and ii) backward propagation. In the forward pass, the loss function L is evaluated on the current weights, and in backward propagation, the weights are updated sequentially. To simplify the analysis, we assume that all weight vectors W normalized, as the magnitude of the weight vectors does not affect the layer output. The j th neuron response in the (i + 1) th layer are computed as First, the input data points x i are projected to the j th row vector of the weight matrix. The dot product of W During backward propagation, the backward gradient update on W i j and x i are computed using For the case of ReLU activation The activation function only allows the gradients from data point on the activated region to backward propagate and update the hyper-plane (equation 4). From the hyper-plane analysis, we realize that ReLU activation has three ideal properties that are distinguishing it from the others i) the diversity of activated regions at initialization, ii) The equality of data points at initialization, iii) The equality of hyper-planes at initialization. These may explain why ReLU activation outperforms the traditional Hyperbolic tangent or sigmoid activations. To argue each property, let us suppose that the distribution of the dot products is zero-centered. This assumption is automatically preserved in the batch normalization. Weight initialization techniques, like Xavier and MSRA, randomly initialize the weights to maintain the output variance. i) Region diversity: the activated regions of hyper-planes solely depend on the direction of the weight vector, which is randomly initialized. This allows different hyper-planes to learn from a different subset of data points, and ultimately diversifies the backward gradient signal. ii) Data equality: an arbitrary data point x i, is located on the activated regions of approximately half of the total hyper-planes in layer i. In other words, the backward gradients from all data points can pass through the approximately same amount of activation function, update hyper-planes, and propagate the gradient. iii) Hyperplane equality: an arbitrary hyper-plane W i j, is affected by the backward gradients from approximately 50% of the total data points. All hyper-planes on average receive the same amount of backward gradients. Hyper-plane equality speeds up the convergence and facilitates model optimization, see Figure 1 (right panel). Similar to the ReLU activation, only the backward gradients from data points located in the activated region of a hyper-plane can backward propagate through the activation function and update this hyper-plane. This analysis also applies to htanh activation. The performance gap between ReLU activation and htanh activation is caused by their different activated region distribution, see Figure 3. Clearly, htanh activation is not as good as ReLU in defining balanced and fair activated regions. However, we analyze each property for htanh as well. i) Region diversity: activated regions of htanh are not as diverse as ReLU. Activated regions of htanh cover only the area close to the origin. Assuming Gaussian data, this is a dense area that the majority of data points are located in. ii) Data equality: data points are not treated fairly htanh activation function. Data points that closer to the origin can activate more hyper-planes than the data points far from the origin. If the magnitude of a data point x i is small enough, it can activate all hyper-planes in the same layer, see the deepred region of Figure 3 (right panel). As a consequence, in backward gradients, few data instances affect all hyper-planes. In other words, the backward gradients from a part of the training data points have a larger impact on the model than the others. We considered this imbalance ultimately affects model generalization problem since the model training focuses on a subset of the training data points close to the origin. iii) Hyperplane equality: The initial activated regions should cover a similar-sized subset of the training data points overall, and this property is shared in both ReLU and htanh activations. Similar analysis also applies to other activation functions with the zero-centered activated region, like Sigmoid or Hyperbolic Tangent. Here we proposed a simple initialization strategy to alleviate the data inequality issue and improve activated region diversity for the htanh activation relying on our geometric insight. We argue bias initialization with a uniform distribution between [−λ, λ], where λ is a hyper-parameter is a solution to region diversity. With random bias initialization, the data points that far from the origin can activate more hyper-planes. If λ > max(x) + 1, all data points activate approximately the same number of hyper-planes during backward propagation, so data equality can be achieved. Also, with the diverse initial activated region, different hyper-planes learn from different subset of training data points. However, this initialization strategy comes with a drawback. Hyper-plane equality no longer holds when the biases are not set to zero. Hyper-planes with larger initial bias have less activated data. Therefore, choosing the optimal value of λ is a trade-off between the hyper-plane equality and the data equality. Experiments below shows that the validation curve becomes unsteady if λ value set to too high. Empirically, with a batch normalization layer, λ ≈ 2 provide a good initial estimate. In this case, the activated regions covering from −3 to +3, so it allows the gradients from almost all data points to propagate. It's worth to mention that the experiments showed that small λ also helps to improve the performance of ResNet architecture. The proposed bias initialization method is evaluated on the CIFAR-10. The network architectures are based on the original implementation of the BNN . We choose the VGG-7 architecture and the ResNet architecture. The VGG-7 architecture, is a simple and over-parameterized model for CIFAR 10. This is an ideal architecture to compare the performance between different activations. In the original implementation, this full-precision architecture is designed to compare with BNN, so the BatchNorm layers are Table 2: Validation error rate % for Binary training, λ = 0 coincides with common deterministic initialization. inserted after each ReLU activation to match the pattern of BNN. This arrangement has a negative impact on the performance of the full-precision neural network. In our full-precision experiments, we put the BatchNorm layers back to their original position, so the error rate of the baseline model is improved from the original reported 9.0% to 6.98%. This is close to 6.50%, which is the best CIFAR 10 error rate reported on VGG-9 architecture that includes BatchNorm. Considering VGG-9 has much more parameters than this architecture, we keep it as the baseline for full-precision VGG. We also followed the training recipe from the original implementation, SGD optimizer with 0.9 momentum and weight decay set to 5 × 10 − 4. We use more training epochs to compensate for the slow convergence caused by htanh activation. Figure 4 confirms that the random bias initialization strategy helps to reduce the performance gap between htanh and ReLU activation. A similar effect is observed for ResNet type architectures. We also tested the proposed bias initialization on the ResNet-like architecture. The are depicted in Figure 4 re-assures that bias initialization improves htanh and pushes it toward ReLU accuracy, see Table 1. Binary training that use STE is similar to htanh activation. We expect to observe a similar effect in BNN training with STE gradient approximator. The validation error rate is summarized in Table 2. In the Binary VGG-7 experiments, we reduced the accuracy gap between full-precision network with ReLU activation and BNN from 4% to 1.5%. The bias initialization strategy is effective to close the gap on binary ResNet architecture by almost 1%, even while the full-precision model even under-fits on CIFAR10 data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Improve saturating activations (sigmoid, tanh, htanh etc.) and Binarized Neural Network with Bias Initialization

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.