Question: Represent the scientific article to find a short TLDR.

Question: Modern deep artificial neural networks have achieved impressive through models with orders of magnitude more parameters than training examples which control overfitting with the help of regularization. Regularization can be implicit, as is the case of stochastic gradient descent and parameter sharing in convolutional layers, or explicit. Explicit regularization techniques, most common forms are weight decay and dropout, have proven successful in terms of improved generalization, but they blindly reduce the effective capacity of the model, introduce sensitive hyper-parameters and require deeper and wider architectures to compensate for the reduced capacity. In contrast, data augmentation techniques exploit domain knowledge to increase the number of training examples and improve generalization without reducing the effective capacity and without introducing model-dependent parameters, since it is applied on the training data. In this paper we systematically contrast data augmentation and explicit regularization on three popular architectures and three data sets. Our demonstrate that data augmentation alone can achieve the same performance or higher as regularized models and exhibits much higher adaptability to changes in the architecture and the amount of training data. One of the central issues in machine learning research and application is finding ways of improving generalization. Regularization, loosely defined as any modification applied to a learning algorithm that helps prevent overfitting, plays therefore a key role in machine learning (; Müller, 2012). In the case of deep learning, where neural networks tend to have several orders of magnitude more parameters than training examples, statistical learning theory indicates that regularization becomes even more crucial. Accordingly, a myriad of techniques have been proposed as regularizers: weight decay and other L p penalties; dropout and stochastic depth , to name a few examples. Moreover, whereas in simpler machine learning algorithms the regularizers can be easily identified as explicit terms in the objective function, in modern deep neural networks the sources of regularization are not only explicit, but implicit . In this regard, many techniques have been studied for their regularization effect, despite not being explicitly intended as such. That is the case of unsupervised pre-training , multi-task learning , convolutional layers , batch normalization or adversarial training . In sum, there are multiple elements in deep learning that contribute to reduce overfitting and thus improve generalization. Driven by the success of such techniques and the efficient use of GPUs, considerable research effort has been devoted to finding ways of training deeper and wider networks with larger capacity (; ;). Ironically, the increased representational capacity is eventually reduced in practice by the use of explicit regularization, most commonly weight decay and dropout. It is known, for instance, that the gain in generalization provided by dropout comes at the cost of using larger models and training for longer . Hence, it seems that with these standard regularization methods deep networks are wasting capacity . Unlike explicit regularization, data augmentation improves generalization without reducing the capacity of the model. Data augmentation, that is synthetically expanding a data set by apply-ing transformations on the available examples, has been long used in machine learning and identified as a critical component of many recent successful models, like AlexNet , All-CNN or ResNet , among others. Although it is most popular in computer vision, data augmentation has also proven effective in speech recognition , music source separation or text categorization . Today, data augmentation is an almost ubiquitous technique in deep learning, which can also be regarded as an implicit regularizer for it improves generalization. Recently, the deep learning community has become more aware of the importance of data augmentation (Hernández-García & König, 2018b) and new techniques, such as cutout (a) or augmentation in the feature space (b), have been proposed. Very interestingly, a promising avenue for future research has been set by recently proposed models that automatically learn the data transformations (; ; ;). Nonetheless, another study by analyzed the performance of different techniques for object recognition and concluded that one of the most successful techniques so far is still the traditional data augmentation carried out in most studies. However, despite its popularity, the literature lacks, to our knowledge, a systematic analysis of the impact of data augmentation on convolutional neural networks compared to explicit regularization. It is a common practice to train the models with both explicit regularization, typically weight decay and dropout, and data augmentation, assuming they all complement each other. included data augmentation in their analysis of generalization of deep networks, but it was questionably considered an explicit regularizer similar to weight decay and dropout. To our knowledge, the first time data augmentation and explicit regularization were systematically contrasted was the preliminary study by Hernández-García & König (2018b). The present work aims at largely extending that work both with more empirical and a theoretical discussion. Our specific contributions are the following: • Propose definitions of explicit and implicit regularization that aim at solving the ambiguity in the literature (Section 2). • A theoretical discussion based on statistical learning theory about the differences between explicit regularization and data augmentation, highlighting the advantages of the latter (Section 3). • An empirical analysis of the performance of models trained with and without explicit regularization, and different levels of data augmentation on several benchmarks (Sections 4 and 5). Further, we study their adaptability to learning from fewer examples (Section 5.2) and to changes in the architecture (Section 5.3). • A discussion on why encouraging data augmentation instead of explicit regularization can benefit both theory and practice in deep learning (Section 6). 2 raised the thought-provoking idea that "explicit regularization may improve generalization performance, but is neither necessary nor by itself sufficient for controlling generalization error." The authors came to this from the observation that turning off the explicit regularizers of a model does not prevent the model from generalizing reasonably well. This contrasts with traditional machine learning involving convex optimization, where regularization is necessary to avoid overfitting and generalize . Such observation led the authors to suggest the need for "rethinking generalization" in order to understand deep learning. We argue it is not necessary to rethink generalization if we instead rethink regularization and, in particular, data augmentation. Despite their thorough analysis and relevant , arguably underestimated the role of implicit regularization and considered data augmentation an explicit form of regularization much like weight decay and dropout. This illustrates that the terms explicit and implicit regularization have been used subjectively and inconsistently in the literature before. In order to avoid the ambiguity and facilitate the discussion, we propose the following definitions of explicit and implicit regularization 1: • Explicit regularization techniques are those which reduce the representational capacity of the model they are applied on. That is, given a model class H 0, for instance a neural network architecture, the introduction of explicit regularization will span a new hypothesis set H 1, which is a proper subset of the original set, i.e. H 1 H 0. • Implicit regularization is the reduction of the generalization error or overfitting provided by means other than explicit regularization techniques. Elements that provide implicit regularization do not reduce the representational capacity, but may affect the effective capacity of the model, that is the achievable set of hypotheses given the model, the optimization algorithm, hyperparameters, etc. One of the most common explicit regularization techniques in machine learning is L p -norm regularization, of which weight decay is a particular case, widely used in deep learning. Weight decay sets a penalty on the L 2 norm of the learnable parameters, thus constraining the representational capacity of the model. Dropout is another common example of explicit regularization, where the hypothesis set is reduced by stochastically deactivating a number of neurons during training. Similar to dropout, stochastic depth, which drops whole layers instead of neurons, is also an explicit regularization technique. There are multiple elements in deep neural networks that implicitly regularize the models. Note, in this regard, that the above definition, contrary to explicit regularization, does not refer to techniques, but to a regularization effect, as it can be provided by elements of very different nature. For instance, stochastic gradient descent (SGD) is known to have an implicit regularization effect without constraining the representational capacity. Batch normalization does not either reduce the capacity, but it improves generalization by smoothing the optimization landscape. Of quite a different nature, but still implicit, is the regularization effect provided by early stopping, which does not reduce the representational, but the effective capacity. By analyzing the literature, we identified some previous pieces of work which, lacking a definition of explicit and implicit regularization, made a distinction apparently based on the mere intention of the practitioner. Under such notion, data augmentation has been considered in some cases an explicit regularization technique, as in. Here, we have provided definitions for explicit and implicit regularization based on their effect on the representational capacity and argue that data augmentation is not explicit, but implicit regularization, since it does not affect the representational capacity of the model. The generalization of a model class H can be analyzed through complexity measures such as the VC-dimension or, more generally, the Rademacher complexity R n (H) = E S∼D n R S (H), where: is the empirical Rademacher complexity, defined with respect to a set of data samples S = (x i, ..., x n). Then, in the case of binary classification and the class of linear separators, the generalization error of a hypothesis,ˆ S (h), can be bounded using the Rademacher complexity: with probability 1 − δ. Tighter bounds for some model classes, such as fully connected neural networks, can be obtained , but it is not trivial to formally analyze the influence on generalization of specific architectures or techniques. Nonetheless, we can use these theoretical insights to discuss the differences between explicit regularization-particularly weight decay and dropout-and implicit regularization-particularly data augmentation. A straightforward yet very relevant from the analysis of any generalization bound is the strong dependence on the number of training examples n. Increasing n drastically improves the generalization guarantees, as reflected by the second term in RHS of Equation 1 and the dependence of the Rademacher complexity (LHS) on the sample size as well. Data augmentation exploits prior knowledge of the data domain D to create new examples and its impact on generalization is related to an increment in n, since stochastic data augmentation can generate virtually infinite different samples. Admittedly, the augmented samples are not independent and identically distributed and thus, the effective increment of samples does not strictly correspond to the increment in n. This is why formally analyzing the impact of data augmentation on generalization is complex and out of the scope of this paper. Recently, some studies have taken steps in this direction by analyzing the effect of simplified data transformations on generalization from a theoretical point of view;. In contrast, explicit regularization methods aim, in general, at improving the generalization error by constraining the hypothesis class H, which hopefully should reduce its complexity, R n (H), and, in turn, the generalization errorˆ S (h). Crucially, while data augmentation exploits domain knowledge, most explicit regularization methods only naively constrain the hypothesis class. For instance, weight decay constrains the learnable models H by setting a penalty on the weights norm. have recently shown that weight decay has little impact on the generalization bounds and confidence margins. Dropout has been extensively used and studied as a regularization method for neural networks , but the exact way in which dropout may improve generalization is still an open question and it has been concluded that the effects of dropout on neural networks are somewhat mysterious, complicated and its penalty highly non-convex . have established new generalization bounds on the variance induced by a particular type of dropout on feedforward neural network. Nevertheless, dropout can also be analyzed as a random form of data augmentation without domain knowledge , that is data-dependent regularization. Therefore, any generalization bound derived for dropout can be regarded as a pessimistic bound for domain-specific, standard data augmentation. A similar argument applies for weight decay, which, as first shown by , is equivalent to training with noisy examples if the noise amplitude is small and the objective is the sum-of-squares error function. In sum, many forms of explicit regularization are at least approximately equivalent to adding random noise to the training examples, which is the simplest form of data augmentation 2. Thus, it is reasonable to argue that more sophisticated data augmentation can overshadow the benefits provided by explicit regularization. In general, we argue that the reason why explicit regularization may not be necessary is that neural networks are already implicitly regularized by many elements-stochastic gradient descent (SGD), convolutional layers, normalization and data augmentation, to name a few-that provide a more successful inductive bias . For instance, it has been shown that linear models optimized with SGD converge to solutions with small norm, without any explicit regularization . In the remainder of the paper, we present a set of experiments that shed more light on the advantages of data augmentation over weight decay and dropout. This section describes the experimental setup for systematically analyzing the role of data augmentation in deep neural networks compared to weight decay and dropout and builds upon the methods used in preliminary studies (Hernández-García & König, 2018a; b;). We perform our experiments on three distinct, popular architectures that have achieved successful in object recognition tasks: the all convolutional network, All-CNN ; the wide residual network, WRN ; and the densely connected network, DenseNet . Importantly, we keep the same training hyper-parameters (learning rate, training epochs, batch size, optimizer, etc.) as in the original papers in the cases they are reported. Below we present the main features of each network and more details can be found in the supplementary material. • All-CNN: it consists only of convolutional layers with ReLU activation , it is relatively shallow and has few parameters. For ImageNet, the network has 16 layers and 9.4 million parameters; for CIFAR, it has 12 layers and 1.3 million parameters. In our experiments to compare the adaptability of data augmentation and explicit regularization to changes in the architecture, we also test a shallower version, with 9 layers and 374,000 parameters, and a deeper version, with 15 layers and 2.4 million parameters. • WRN: a residual network, ResNet , that achieves better performance with fewer layers, but more units per layer. Here, we choose for our experiments the WRN-28-10 version (28 layers and about 36.5 M parameters), which is reported to achieve the best on CIFAR. • DenseNet: a network architecture arranged in blocks whose layers are connected to all previous layers, allowing for very deep architectures with few parameters. Specifically, for our experiments we use a DenseNet-BC with growth rate k = 12 and 16 layers in each block, which has a total of 0.8 million parameters. We perform the experiments on the highly benchmarked data sets ImageNet (So as to analyze the role of data augmentation, we train every network architecture with two different augmentation schemes as well as with no data augmentation at all: • Light augmentation: This scheme is common in the literature, for example , and performs only horizontal flips and horizontal and vertical translations of 10% of the image size. • Heavier augmentation: This scheme performs a larger range of affine transformations such as scaling, rotations and shear mappings, as well as contrast and brightness adjustment. On ImageNet we additionally perform a random crop of 128 × 128 pixels. The choice of the allowed transformations is arbitrary and the only criterion was that the objects are still recognizable in general. We deliberately avoid designing a particularly successful scheme. The details of the heavier scheme can be consulted in the supplementary material. Every architecture is trained on each data set both with explicit regularization-weight decay and dropout as specified in the original papers-and with no explicit regularization. Furthermore, we train each model with the three data augmentation schemes. The performance of the models is computed on the held out test tests. As in previous works , we average the softmax posteriors over 10 random light augmentations, since slightly better are obtained. All the experiments are performed on Keras on top of TensorFlow and on a single GPU NVIDIA GeForce GTX 1080 Ti. This section presents the most relevant of the experiments comparing the roles of data augmentation and explicit regularization on convolutional neural networks. First, we present the experiments with the original architectures in section 5.1. Then, Sections 5.2 and 5.3 show the of training the models with fewer training examples and with shallower and deeper versions of the All-CNN architecture. The figures aim at facilitating the comparison between the models trained with and without explicit regularization, as well as between the different levels of data augmentation. The purple bars (top of each pair) correspond to the models trained without explicit regularization-weight decay and dropout-and the red bars (bottom) to the models trained with it. The different color shades correspond to the three augmentation schemes. The figures show the relative performance of each model with respect to a particular baseline in order to highlight the relevant comparisons. A detailed and complete report of all the can be found in the supplementary material. The on CIFAR refer to the top-1 test accuracy while on ImageNet we report the top-5. Figure 1: Relative improvement of adding data augmentation and explicit regularization to the baseline models, (accuracy − baseline)/accuracy * 100. The baseline accuracy is shown on the left. The suggest that data augmentation alone (purple bars) can achieve even better performance than the models trained with both weight decay and dropout (red bars). First, we contrast the regularization effect of data augmentation and weight decay and dropout on the original networks trained with the complete data sets. For that purpose, in Figure 1 we show the relative improvement in test performance achieved by adding each technique or combination of techniques to the baseline model, that is the model trained with neither explicit regularization nor data augmentation (see the left of the bars). Several can be extracted from Figure 1 and Table 1. Most importantly, training with data augmentation alone (top, purple bars) improves the performance in most cases as much as or even more than training with both augmentation and explicit regularization (bottom, red bars), on average Figure 2: Fraction of the baseline performance when the amount of available training data is reduced, accuracy/baseline * 100. The models trained wit explicit regularization present a significant drop in performance as compared to the models trained with only data augmentation. The differences become larger as the amount of training data decreases. 8.57 and 7.90 % respectively. This is quite a surprising and remarkable : note that the studied architectures achieved state-of-the-art at the moment of their publication and the models included both light augmentation and weight decay and dropout, whose parameters were presumably finely tuned to achieve higher accuracy. The replication of these corresponds to the middle red bars in Figure 1. We show here that simply removing weight decay and dropout-while even keeping all other hyperparameters intact, see Section 4.1-improves the formerly state-of-the-art accuracy in 4 of the 8 studied cases. Second, it can also be observed that the regularization effect of weight decay and dropout, an average improvement of 3.02 % with respect to the baseline, 1 is much smaller than that of data augmentation. Simply applying light augmentation increases the accuracy in 8.46 % on average. Finally, note that even though the heavier augmentation scheme was deliberately not designed to optimize the performance, in both CIFAR-10 and CIFAR-100 it improves the test performance with respect to the light augmentation scheme. This is not the case on ImageNet, probably due to the increased complexity of the data set. It can be observed though that the effects are in general more consistent in the models trained without explicit regularization. In sum, it seems that the performance gain achieved by weight decay and dropout can be achieved and often improved by data augmentation alone. We argue that one of the main drawbacks of explicit regularization techniques is their poor adaptability to changes in the conditions with which the hyperparameters were tuned. To test this hypothesis and contrast it with the adaptability of data augmentation, here we extend the analysis by training the Figure 3: Fraction of the original performance when the depth of the All-CNN architecture is increased or reduced in 3 layers. In the explicitly regularized models, the change of architecture implies a dramatic drop in the performance, while the models trained without explicit regularization present only slight variations with respect to the original architecture. same networks with fewer examples. The models are trained with the same random subset of data and evaluated in the same test set as the previous experiments. In order to better visualize how well each technique resists the reduction of training data, in Figure 2 we show the fraction of baseline accuracy achieved by each model when trained with 50 % and 10 % of the available data. In this case, the baseline is thus each corresponding model trained with the complete data set. Table 2 summarizes the mean and standard deviation of each combination. An extended report of , including additional experiments with 80 % and 1 % of the data, is provided in the supplementary material. One of the main of this set of experiments is that if no data augmentation is applied, explicit regularization hardly resist the reduction of training data by itself. On average, with 50 % of the available data, these models only achieve 83.20 % of the original accuracy, which, remarkably, is worse than the models trained without any explicit regularization (88.11 %). On 10 % of the data, the average fraction is the same (58.75 and 58.72 %, respectively). This implies that training with explicit regularization is even detrimental for the performance. When combined with data augmentation, the models trained with explicit regularization (bottom, red bars) also perform worse (88.78 and 61.16 % with 50 and 10 % of the data, respectively), than the models with just data augmentation (top, purple bars, 91.64 and 68.12 % on average). Note that the difference becomes larger as the amount of available data decreases. Importantly, it seems that the combination of explicit regularization and data augmentation is only slightly better than training without data augmentation. We can think of two reasons that could explain this: first, the original regularization hyperparameters seem to adapt poorly to the new conditions. The hyperparameters are specifically tuned for the original setup and one would have to re-tune them to achieve comparable . Second, since explicit regularization reduces the representational capacity, this might prevent the models from taking advantage of the augmented data. In contrast, the models trained without explicit regularization more naturally adapt to the reduced availability of data. With 50 % of the data, these models, trained with data augmentation achieve about 91.5 % of the performance with respect to training with the complete data sets. With only 10 % of the data, they achieve nearly 70 % of the baseline performance, on average. This highlights the suitability of data augmentation to serve, to a great extent, as true, useful data . Finally, in this section we test the adaptability of data augmentation and explicit regularization to changes in the depth of the All-CNN architecture (see Section 4.1). We show the fraction of the performance with respect to the original architecture in Figure 3. A noticeable from Figure 3 is that all the models trained with weight decay and dropout (bottom, red bars) suffer a dramatic drop in performance when the architecture changes, regardless of whether it becomes deeper or shallower and of the amount of data augmentation. As in the case of reduced training data, this may be explained by the poor adaptability of the regularization hyperparameters, which highly depend on the architecture. This highly contrasts with the performance of the models trained without explicit regularization (top, purple bars). With a deeper architecture, these models achieve slightly better performance, effectively exploiting the increased capacity. With a shallower architecture, they achieve only slightly worse performance 4. Thus, these models seem to more naturally adapt to the new architecture and data augmentation becomes beneficial. It is worth commenting on the particular case of the CIFAR-100 benchmark, where the difference between the models with and without explicit regularization is even more pronounced, in general. It is a common practice in object recognition papers to tune the parameters for CIFAR-10 and then test the performance on CIFAR-100 with the same hyperparameters. Therefore, these are typically less suitable for CIFAR-100. We believe this is the reason why the benefits of data augmentation seem even more pronounced on CIFAR-100 in our experiments. In sum, these highlight another crucial advantage of data augmentation: the effectiveness of its hyperparameters, that is the type of image transformations, depend mostly on the type of data, rather than on the particular architecture or amount of available training data, unlike explicit regularization hyperparameters. Therefore, removing explicit regularization and training with data augmentation increases the flexibility of the models. We have presented a systematic analysis of the role of data augmentation in deep convolutional neural networks for object recognition, focusing on the comparison with popular explicit regularization techniques-weight decay and dropout. In order to facilitate the discussion and the analysis, we first proposed in Section 2 definitions of explicit and implicit regularization, which have been ambiguously used in the literature. Accordingly, we have argued that data augmentation should not be considered an explicit regularizer, such as weight decay and dropout. Then, we provided some theoretical insights in Section 3 that highlight some advantages of data augmentation over explicit regularization. Finally, we have empirically shown that explicit regularization is not only unnecessary , but also that its generalization gain can be achieved by data augmentation alone. Moreover, we have demonstrated that, unlike data augmentation, weight decay and dropout exhibit poor adaptability to changes in the architecture and the amount of training data. Despite the limitations of our empirical study, we have chosen three significantly distinct network architectures and three data sets in order to increase the generality of our , which should ideally be confirmed by future work on a wider range of models, data sets and even other domains such text or speech. It is important to note, however, that we have taken a conservative approach in our experimentation: all the hyperparameters have been kept as in the original models, which included both weight decay and dropout, as well as light augmentation. This setup is clearly suboptimal for models trained without explicit regularization. Besides, the heavier data augmentation scheme was deliberately not optimized to improve the performance and it was not the scope of this work to propose a specific data augmentation technique. As future work, we plan to propose data augmentation schemes that can more successfully be exploited by any deep model. The relevance of our findings lies in the fact that explicit regularization is currently the standard tool to enable the generalization of most machine learning methods and is included in most convolutional neural networks. However, we have empirically shown that simply removing the explicit regularizers often improves the performance or only marginally reduces it, if some data augmentation is applied. These are supported by the theoretical insights provided in in Section 3. suggested that regularization might play a different role in deep learning, not fully explained by statistical learning theory . We have argued instead that the theory still naturally holds in deep learning, as long as one considers the crucial role of implicit regularization: explicit regularization seems to be no longer necessary because its contribution is already provided by the many elements that implicitly and successfully regularize the models: to name a few, stochastic gradient descent, convolutional layers and data augmentation. Data augmentation is often regarded by authors of machine learning papers as cheating, something that should not be used in order to test the potential of a newly proposed architecture (; ;). In contrast, weight decay and dropout are almost ubiquitous and considered intrinsic elements of the algorithms. In view of the presented here, we believe that the deep learning community would benefit if we rethink data augmentation and switch roles with explicit regularization: a good model should generalize well without the need for explicit regularization and successful methods should effectively exploit data augmentation. In this regard it is worth highlighting some of the advantages of data augmentation: Not only does it not reduce the representational capacity of the model, unlike explicit regularization, but also, since the transformations reflect plausible variations of the real objects, it increases the robustness of the model and it can be seen as a data-dependent prior, similarly to unsupervised pre-training . have shown that data augmentation consistently yields models with smaller sensitivity to perturbations. Interestingly, recent work has found that models trained with heavier data augmentation learn representations that are more similar to the inferior temporal (IT) cortex, highlighting the biological plausibility of data augmentation (Hernández-García et al., 2018). Deep neural networks are especially well suited for data augmentation because they do not rely on pre-computed features and because the large number of parameters allows them to shatter the augmented training set. Moreover, unlike explicit regularization, data augmentation can be performed on the CPU, in parallel to the gradient updates. Finally, an important from Sections 5.2 and 5.3 is that data augmentation naturally adapts to architectures of different depth and amounts of available training data, whereas explicitly regularized models are highly sensitive to such changes and need specific fine-tuning of their hyperparameters. In sum, data augmentation seems to be a strong alternative to explicit regularization techniques. Some argue that despite these advantages, data augmentation is a limited approach because it depends on some prior expert knowledge and it cannot be applied to all domains. However, we argue instead that expert knowledge should not be disregarded but exploited. A single data augmentation scheme can be designed for a broad family of data (for example, natural images) and effectively applied to a broad set of tasks (for example, object recognition, segmentation, localization, etc.). Besides, interesting recent works have shown that it is possible to automatically learn the data augmentation strategies . We hope that these insights encourage more research attention on data augmentation and that future work brings more sophisticated and effective data augmentation techniques, potentially applicable to different data modalities. This appendix presents the details of the network architectures used in the main experiments: All-CNN, Wide Residual Network (WRN) and DenseNet. All-CNN is a relatively simple, small network with a few number of layers and parameters, WRN is deeper, has residual connections and many more parameters and DenseNet is densely connected and is much deeper, but parameter effective. A.1 ALL CONVOLUTIONAL NETWORK All-CNN consists exclusively of convolutional layers with ReLU activation , it is relatively shallow and has few parameters. For ImageNet, the network has 16 layers and 9.4 million parameters; for CIFAR, it has 12 layers and about 1.3 million parameters. In our experiments to compare the adaptability of data augmentation and explicit regularization to changes in the architecture, we also test a shallower version, with 9 layers and 374,000 parameters, and a deeper version, with 15 layers and 2.4 million parameters. The four architectures can be described as in Table 3, where KCD(S) is a D × D convolutional layer with K channels and stride S, followed by batch normalization and a ReLU non-linearity. N.Cl. is the number of classes and Gl. Avg. refers to global average pooling. The CIFAR network is identical to the All-CNN-C architecture in the original paper, except for the introduction of the batch normalization layers. The ImageNet version also includes batch normalization layers and a stride of 2 instead of 4 in the first layer to compensate for the reduced input size (see below). Importantly, we keep the same training parameters as in the original paper in the cases they are reported. Specifically, the All-CNN networks are trained using stochastic gradient descent, with fixed Nesterov momentum 0.9, learning rate of 0.01 and decay factor of 0.1. The batch size for the experiments on ImageNet is 64 and we train during 25 epochs decaying the learning rate at epochs 10 and 20. On CIFAR, the batch size is 128, we train for 350 epochs and decay the learning rate at epochs 200, 250 and 300. The kernel parameters are initialized according to the Xavier uniform initialization . WRN is a modification of ResNet that achieves better performance with fewer layers, but more units per layer. Here we choose for our experiments the WRN-28-10 version (28 layers and about 36.5 M parameters), which is reported to achieve the best on CIFAR. It has the following architecture: where KR is a residual block with residual function BN-ReLU-KC3-BN-ReLU-KC 3. BN is batch normalization, Avg. is spatial average pooling of size 8 and FC is a fully connected layer. On ImageNet, the stride of the first convolution is 2. The stride of the first convolution within the residual blocks is 1 except in the first block of the series of 4, where it is set to 2 in order to subsample the feature maps. Similarly, we keep the training parameters of the original paper: we train with SGD, with fixed Nesterov momentum 0.9 and learning rate of 0.1. On ImageNet, the learning rate is decayed by 0.2 at epochs 8 and 15 and we train for a total of 20 epochs with batch size 32. On CIFAR, we train with a batch size of 128 during 200 epochs and decay the learning rate at epochs 60, 120 and 160. The kernel parameters are initialized according to the He normal initialization . The main characteristic of DenseNet is that the architecture is arranged into blocks whose layers are connected to all the layers below, forming a dense graph of connections, which permits training very deep architectures with fewer parameters than, for instance, ResNet. Here, we use a network with bottleneck compression rate θ = 0.5 (DenseNet-BC), growth rate k = 12 and 16 layers in each of the three blocks. The model has nearly 0.8 million parameters. The specific architecture can be descried as follows: where DB(c) is a dense block, that is a concatenation of c convolutional blocks. Each convolutional block is of a set of layers whose output is concatenated with the input to form the input of the next convolutional block. A convolutional block with bottleneck structure has the following layers: TB is a transition block, which downsamples the size of the feature maps, formed by the following layers: Like with All-CNN and WRN, we keep the training hyper-parameters of the original paper. On the CIFAR data sets, we train with SGD, with fixed Nesterov momentum 0.9 and learning rate of 0.1, decayed by 0.1 on epochs 150 and 200 and training for a total of 300 epochs. The batch size is 64 and the are initialized with He initialization. In this appendix we present the details of the heavier data augmentation scheme, introduced in Section 3.2: • Affine transformations: This appendix details the of the main experiments shown in Figures 1, 2 and 3 and provides the of many other experiments not presented above in order not to clutter the visualization. Some of these are the top-1 accuracy on ImageNet, the of the models trained with dropout, but without weight decay; and the of training with 80 % and 1 % of the data. Additionally, for many experiments we also train a version of the network without batch normalization. These are provided within brackets in the tables. Note that the original All-CNN published by did not include batch normalization. In the case of WRN, we remove all batch normalization layers except the top-most one, before the spatial average pooling, since otherwise many models would not converge. An important observation from Table 5 is that the interaction of weight decay and dropout is not always consistent, since in some cases better can be obtained with both explicit regularizers active and in other cases, only dropout achieves better generalization. In contrast, the effect of data augmentation seems to be consistent: just some light augmentation achieves much better than training only with the original data set and performing heavier augmentation almost always further improves the test accuracy, without the need for explicit regularization. Not surprisingly, batch normalization also contributes to improve the generalization of All-CNN and it seems to combine well with data augmentation. On the contrary, when combined with explicit regularization the are interestingly not consistent in the case of All-CNN: it seems to improve the generalization of the model trained with both weight decay and dropout, but it drastically reduces the performance with only dropout, in the case of CIFAR-10 and CIFAR-100 without augmentation. A probable explanation is, again, that the regularization hyperparameters would need to be readjusted with a change of the architecture. Furthermore, it seems that the gap between the performance of the models trained with and without batch normalization is smaller when they are trained without explicit regularization and when they include heavier data augmentation. This can be observed in Table 5, as well as in Table 6, which contains the of the models trained with fewer examples. It is important to note as well the benefits of batch normalization for obtaining better when training with fewer examples. However, it is surprising that there is only a small drop in the performance of WRN-95.47 % to 94.95 % without regularization-from removing the batch normalization layers of the residual blocks, given that they were identified as key components of ResNet . The in Table 6 clearly support the presented in Section 4.2: data augmentation alone better resists the lack of training data compared to explicit regularizers. Already with 80% and 50% of the data better are obtained in some cases, but the differences become much bigger when training with only 10% and 1% of the available data. It seems that explicit regularization prevents the model from both fitting the data and generalizing well, whereas data augmentation provides useful transformed examples. Interestingly, with only 1% of the data, even without data augmentation the models without explicit regularization perform better. The same effect can be observed in Table 7, where both the shallower and deeper versions of All-CNN perform much worse when trained with explicit regularization, even when trained without data augmentation. This is another piece of evidence that explicit regularization needs to be used very carefully, it requires a proper tuning of the hyperparameters and is not always beneficial. In this appendix we provide the computations of the Frobenius norm of the weight matrices of the models trained with different levels of explicit regularization and data augmentation, as a rough estimation of the complexity of the learned models. Table 8 shows the Frobenius norm of the weight matrices of the models trained with different levels of explicit regularization and data augmentation. The clearest is that heavier data augmentation seems to yield solutions with larger norm. This is always true except in some All-CNN models trained without batch normalization. Another observation is that, as expected, weight decay constrains the norm of the learned function. Besides, the models trained without batch normalization exhibit smaller differences between different levels of regularization and augmentation and, in the case of All-CNN, less consistency. One of the relevant presented in this paper is the poor performance of the regularized models on the shallower and deeper versions of All-CNN, compared to the models without explicit regularization (see Table 7). One hypothesis is that the amount of regularization is not properly adjusted through the hyperparameters. This could be reflected in the norm of the learned weights, shown in Table 9. However, the norm alone does not seem to fully explain the large performance differences between the different models. Finding the exact reasons why the regularized models not able to generalize well might require a much thorough analysis and we leave it as future work. Although it is out of the scope of this paper to elaborated on the taxonomy of regularization techniques for deep neural networks, an important contribution of this work is providing definitions of explicit and implicit regularization, which have been used ambiguously in the literature before. It is therefore worth mentioning here some of the previous works that have used these terms and to point to literature that has specifically elaborated on the regularization taxonomy or proposed other related terms. et al. observed that the size of neural networks could not explain and control by itself the effective capacity of neural networks and proposed that other elements should implicitly regularize the models. However, no definitions or clear distinction between explicit and implicit regularization was provided. compared different regularization techniques and mentioned the role of implicit regularization, but did not provide definitions either, and, importantly, they considered data augmentation an explicit form of regularization. We have argued against that view throughout this paper, especially in Sections 2 and 6.1. An extensive review of the taxonomy of regularization techniques was carried out by Kukačka et al.. Although no distinction is made between explicit and implicit regularization, they define the class regularization via optimization, which is somehow related to implicit regularization. However, regularization via optimization is more specific than our definition and data augmentation, among others, would not fall into that category. provided a distinction between data-independent and data-dependent regularization. They define data-independent regularization as those techniques that impose certain constraint on the hypothesis set, thus constraining the optimization problem. Examples are weight decay and dropout. We believe this is closely related to our definition of explicit regularization. Then, they define data-dependent regularization as those techniques that make assumptions on the hypothesis set with respect to the training data, as is the case of data augmentation. While we acknowledge the usefulness of such taxonomy, we believe the division between dataindependent and dependent regularization leaves some ambiguity about other techniques, such as batch-normalization, which neither imposes an explicit constraint on H nor on the training data. The taxonomy of explicit vs. implicit regularization is however complete, since implicit regularization refers to any regularization effect that does not come from explicit (or data-independent) techniques. Finally, we argue it would be useful to distinguish between domain-specific, perceptually-motivated data augmentation and other kinds of data-dependent regularization. Data augmentation ultimately aims at creating new examples that could be plausible transformations of the real-world objects. In other words, the augmented samples should be no different in nature than the available data. In statistical terms, they should belong to the same underlying probability distribution. In contrast, one can think of data manipulations that would not mimic any plausible transformation of the data, which still can improve generalization and thus fall into the category of data-dependent regularization (and implicit regularization). One example is mixup, which is the subject of study of.

Represent the short summary for finding the corresponding scientific article (long)

Deep neural networks trained with data augmentation do not require any other explicit regularization (such as weight decay and dropout) and exhibit greater adaptaibility to changes in the architecture and the amount of training data.

Represent the TLDR for finding the corresponding scientific article.


Question: Represent the scientific article to find a short TLDR

Question: Adversarial feature learning (AFL) is one of the promising ways for explicitly constrains neural networks to learn desired representations; for example, AFL could help to learn anonymized representations so as to avoid privacy issues. AFL learn such a representations by training the networks to deceive the adversary that predict the sensitive information from the network, and therefore, the success of the AFL heavily relies on the choice of the adversary. This paper proposes a novel design of the adversary, {\em multiple adversaries over random subspaces} (MARS) that instantiate the concept of the {\em volunerableness}. The proposed method is motivated by an assumption that deceiving an adversary could fail to give meaningful information if the adversary is easily fooled, and adversary rely on single classifier suffer from this issues. In contrast, the proposed method is designed to be less vulnerable, by utilizing the ensemble of independent classifiers where each classifier tries to predict sensitive variables from a different {\em subset} of the representations. The empirical validations on three user-anonymization tasks show that our proposed method achieves state-of-the-art performances in all three datasets without significantly harming the utility of data. This is significant because it gives new implications about designing the adversary, which is important to improve the performance of AFL. Since its invention over ten years ago BID4, deep neural networks (DNN) have shown significant performance improvements in various fields. When we apply DNN or more general machine learning techniques to real-world data, one of the key challenges is how to systematically incorporate the desired constraints into the learned representations in a controllable manner. For example, when practitioners apply these techniques to the data that contain a lot of user information (such as images with username BID1 or data of wearables BID6), the desired representations should not contain user-information that may in privacy issues. Moreover, for legal and ethical reasons, machine learning algorithms have to make fair decisions, which do not rely on sensitive variables such as gender, age, or race BID8 BID1. Such a requires removal of information related to specific factors (such as user ID, race, etc.) from the representation; this is called censoring representations in this paper. One of the recently proposed approaches for censoring representation is adversarial feature learning (AFL) BID1 BID6 BID13, which employs the adversarial training framework to constrain the representations FIG0. Specifically, AFL considers an adversarial classifier who attempts to predict sensitive variables from the representations of a DNN and simultaneously trains the DNN to deceive the classifier. By alternatively or jointly (using gradient reversal layer proposed by BID2) training the adversary and DNN in such a manner, AFL ensures that there is little or no information about the sensitive variables in the representations. Although some previous studies report significant performance improvements of the AFL in the context of censoring representations, the success of the AFL depends on the choice of the adversarial classifier. For example, if we use a logistic regression as the adversarial classifier, AFL can only eliminate the information that is linearly separated in the representation spaces and cannot remove any non-linear dependency. It is also possible that deceiving some classifier might be too easy, ing in poor performance improvements of AFL. As such, the design of adversary is crucial for the performance of AFL; however, existing studies fail to address how to design the adversary for improving the quality of AFL.In this paper, we propose a novel design of adversary for improving the performance of AFL, multiple-adversaries over random subspace (MARS), which consider the vulnerableness of the adversary. The proposed design is motivated by the recent report BID6 that is just increasing the capacity of adversary did not successfully improves the performance of AFL BID6, and assumptions that deceiving an adversary fail to give meaningful information if the adversary is easily fooled, and adversary relies on single classifier suffer from this issues. The proposed method incorporates multiple adversaries where each adversary tries to predict sensitive variables from a different subset of the representations. This design makes adversary less vulnerable to the update of the encoder since the encoder needs to in a set of diverse adversaries. In this paper, we validate the effectiveness of the proposed design by empirically showing that MARS archives better performance compared to baselines (that uses a single adversary and multiple adversaries over the entire representation spaces), and MARS is less vulnerable compared to the baselines. The primary contributions of this paper are as follows:• This is the first study verifying the importance of the design of adversary in AFL and proposes the novel design for improving AFL. This is significant because the suggest that design of adversary is vital for the performance of adversary, and gives new implications about designing the adversary in AFL, which is important to improve the performance of AFL. It is worth mentioning that, except our paper, all existing studies focus only on the accuracy/capacity for designing adversaries, which is not enough for improving the performance of AFL as shown in this paper.• The proposed method achieved state-of-the-art performance in the task of censoring representations, which is essential to extend the applicability of DNN to many real-world applications. The empirical validation using three user-anonymization tasks shows that the proposed method allows the learning of significantly more anonymized representations with negligible performance degradation. Specifically, the probability of correctly predicting the user ID from learned representations is more than 0.07 points better on average than that of a single adversary and multiple adversaries over entire representation spaces.2 PROBLEM DEFINITION AND RELATED WORKS 2.1 PROBLEM DEFINITION: CENSORING REPRESENTATIONS Censoring representation is a task to obtaining unbiased features. Here, unbiased features are features that are less affected by S, where S is a random variable that we want to remove from the data for some reason. One typical reason is related to fairness or privacy, which requires the output of neural networks not to be affected by unfair information or not contain user information. It should be noted that poor design of the censoring procedure significantly reduces the utility of data. For example, the output of random mapping f rand apparently has no information about S, but it also gives no information about target Y. Alternatively, as a more realistic example, a neural network with limited capacity possibly acquires less information about S, but it may also in poorer performance. Therefore, the primary goal of censoring representation is to obtain an encoder E that reduces information about S, while maintaining information about Y. Formally, the task can be written as a joint optimization problem of the loss: DISPLAYFORM0 where X indicates the input random variable, E is an encoder that transforms X to representation R, λ is the weighting parameter, and V and L are loss functions that represent how much information about S and Y is present, respectively. Note that S can be any form of variables such as binary variable, categorical variable, or continuous variable. In this paper, we primarily consider a particular variant of censoring representation tasks, where we learn E with deep neural networks and S is the user ID (anonymization tasks). A recently proposed approach for the censoring representation task is adversarial feature learning (AFL). In AFL, V (E(X), S) is measured by an external neural network D. In other words, if the external networks can accurately predict S from R, AFL regards that R has too much information about S and if it is difficult to predict S accurately, AFL regards that R has little or no information about S. The external network D is called discriminator or adversary in this context. The information is used to update the weights of the encoder E so that the updated representations have less information about S. Formally, AFL solves the joint optimization problems: min To the best of the authors' knowledge, the adversarial training framework was first introduced by BID12, and later re-invented by BID3 in the context of an imagegeneration task. In the context of censoring representations, BID1 first proposed the use of adversarial training to remove sensitive information from the representations. They show its efficacy for fair classification tasks (S is binary) compared to an LFR (Learned Fair Representation) proposed by BID14 that regularizes the l 1 distance between the distributions for data with different S. BID6 first applied the adversarial training to learn anonymized representations on the data of wearables, where S is categorical. More recently, BID13 first introduced the notion of adversarial feature learning and showed superior performances compared to variational fair auto-encoder BID8. DISPLAYFORM0 Although AFL already shows state-of-the-art performance to learn unbiased representations, how to improve its performance is still a challenge, which is tackled in this paper. This work is motivated by previous studies conducted by BID6. They reported that simply using highcapacity networks as an adversary (such as deep neural networks) sometimes fails to improve the performance; on the other hand, the learned representations are still highly biased if the adversary does not have enough capacity. Our work can be regarded as an introduction of the new design consideration (i.e., vulnerableness) of adversary for improving the quality of AFL and validation of it by proposing a method that instantiates the concept, which is not done in the previous works. It is worth mentioning that the concept of multiple adversaries itself is already proposed and verified in the context of the image generation with adversarial training BID0, and this paper does not insist that using multiple adversaries is the primal novelty. From the methodological perspective, the proposed method (MARS) can be seen as an extension of the concept of multiple adversaries by introducing the concept of diversity, and we verify that this extension is essential in the context of censoring representations. Though the primary focus of this paper is limited to the context of adversarial feature learning, we further discuss the applicability of the proposed method on the other application of adversarial training, such as image/text generation BID3 BID16 or domain adaptation BID2; this is done at the end of this paper. The proposed method, multiple-adversaries over random subspaces (MARS), considers multiple adversaries where each adversary is in charge of different subsets of features. The development of MARS is motivated by the assumption that the adversary should not be vulnerable and an ensemble of diverse classifiers make the adversary less vulnerable, ing in the improved performance of AFL. The experiment parts of this paper empirically validate the importance of the design by showing that MARS gives superior performances compared to the method with single adversary or that of simple ensemble of adversaries without enforcing adversaries to have diverse view, and using ensemble of independent classifiers as adversary actually makes the adversary robust. It is worth mentioning that the proposed method could be benefited from the variance reduction because the ensemble of diverse classifiers typically reduces the variance. More generally, this technique is so-called random subspace method in the context of ensemble researches and widely used in a various methods, such as random forest BID5.To make the what parts of the proposed method contribute to the performance gain clearer, we also verify the effect of variance reductions later in the experimental parts.3.2 ALTHORIGHM FIG0 shows the comparison of overall architectures of between AFL with an adversary, and with multiple adversaries over random subspaces. Since the primary difference between naive AFL and MARS is how to build the adversary (training adversaries) and how to use the information of adversary (training encoders), we focus on the explanation about the part. There are many possible options to select a subspace; however, this paper considers the case where each subspace is randomly selected. Suppose that n is the number of dimensions of representation R ∈ R n, and K is the number of adversaries. In MARS, each adversary D k is trained to predict S over a randomly selected subset of features R k, whose dimension is m k and m k < n. Each adversary is trained to maximize the expected log-likelihood. Formally, the optimization problem for D k is as follows max DISPLAYFORM0 where Sub k is a function that return the subset of R, which is fixed before the training. Precisely, each Sub k determine whether to remove the i-th dimension of R with the probability of α. The θ D k is the parameter of D k. The optimization is done by stochastic gradient descent as usual. The adversaries are then integrated, and a single prediction is made to train the encoder. Formally, the probability distribution of q D (s = s i |h = E(x)) is parameterized by an ensembled adversary D: DISPLAYFORM1 which means, the predictions of adversaries D k are integrated by averaging the predictions. Note that integration operator F is not essential to be the averaging, it may be the max operator or the soften version of the max operator proposed in BID0 ).The encoder is trained for minimizing the log-likelihood of Eq.2 and negative log-likelihood q M (y|h). Formally, DISPLAYFORM2 where θ E and θ M are the parameter of E and M respectively. Algorithm 1 show overall algorithm. We used two datasets to demonstrate the efficacy of MARS. Both datasets are related to human activity recognition using the data of wearables, where privacy issues have recently been pointed Algorithm 1 Optimization of the proposed model DISPLAYFORM0 Update weights of E and M (eq. 3) end while out by BID6. In both datasets, the task is to learn anonymized representations (R that does not contain user-identifiable information), while maintaining classification performance. The opportunity recognition dataset BID10 consists of sensory data regarding human activity in a breakfast scenario, and the task is to predict the activity performed by a user. A wide variety of body-worn, object-based, and ambient sensors are used (see FIG0 in BID10 for more details). Each record consists of 113 real-value sensory readings, excluding time information. We considered two recognition tasks: gesture recognition (Opp-G) and locomotion recognition (Opp-L). Opp-G requires the recognition of 18 class activities 1, while Opp-L requires the recognition of 4 class locomotion-activities, i.e., stand, walk, sit, and lie. Given a sampling frequency of 30 Hz, the sliding window procedure with 30 frames and a 50% overlap produced 57,790 samples. We used data from subjects 2-4 as training/validation sets (90% for training and 10% for validation) and that of subject 1 as test sets. The USC-HAD dataset BID15 ) is a relatively new benchmark dataset, which contains a relatively large number of subjects (14 subjects: 7 males and 7 females). The data include 12 activity classes that are corresponding to the most basic and everyday activities of people's daily lives 2. MotionNode, which is a 6-DOF inertial measurement unit specially designed for human motion sensing applications, is used to record the outputs from accelerometers that record 6 real sensory values. The sliding window procedure, using 30 frames and a 50% overlap, produced 172,169 samples. We used data from subjects 1-10 as training/validation sets (90% for training and 10% for validation) and that of subjects 11-14 as test sets. In all experiments, we parameterized the encoder E by convolutional neural networks (CNN) with three convolution-ReLU-pooling repeats followed by one fully connected layer and M by logistic regression, following a previous study BID6. Every component of the network was trained with the Adam algorithm (learning rate is set to 0.0001) BID7 (150 epochs). In the optimization, we used the annealing heuristics for weighting parameter λ following BID6; we linearly increased λ during 15 epochs to 135 epochs of training. Following previous works, we evaluated the level of anonymization by training a classifier f eva that tries to predict S over learned representations. This means that we regarded that a learned representation is anonymized well if we could not build an accurate classifier. To be more specific, we trained the evaluator with the data that is used for training encoder E and a part of the data whose data did not use for training the encoder E. The rest of the data is used for the evaluations. To demonstrate the efficacy of the proposed method, we compare the following four methods and its variants: None: w/o adversary (correspond to standard CNN), Adv: w/ a single adversary, MA: w/ multiple adversaries where each adversary tries to predict from the entire space, and MARS: w/ multiple adversaries where each adversary tries to predict from a different subspace of the entire space. Each adversary is parametrized by multi-layer perceptron (MLP) with 800 hidden units. If we need to express the type of adversary, we denote it by a suffix. For example, Adv-LR means logistic regression parametrizes an adversary. Without mentioning otherwise, we set the num- In addition to None, Adv, MA, MARS, we compared a logistic repression (Adv-LR), DNNs with 400-200 hidden units, the ensemble of DNNs (MA-DNN), and the ensemble of DNNs over random subspaces (MARS-DNN) for E. Adv 2 and Adv 5 correspond to the case where we train an adversary with MLP 2 iterations or 5 iterations against one iteration for training encoder E. For evaluator f eva, we tested LR, multi-layer perceptron with 50 hidden units (MLP 1), multi-layer perceptron with 800 hidden units (MLP 2), and deep neural networks with 400-200 hidden units. The best performance in each combination of datasets and f eva is highlighted in underline and bold. The second best performance is highlighted in bold. We can make the following observations. For small-capacity evaluator (LR), a single adversary (Adv and its variants) marks superior performances to MA and MARS. However, the method with single adversary tends to give poor performance for the stronger adversary (MLP 2 or DNN). For high-capacity evaluator (MLP 2 or DNN), the proposed method outperforms baseline methods. Specifically, in the case when the evaluator is DNN, MARS and MARS-DNN marks 0.806 and 0.789 on average of user accuracy, which is significantly better than the best baseline Adv-DNN (0.863 on average of user accuracy). Increasing the iterations of training adversary does not pay off, as shown in the weak performance improvements of Adv 2 or Adv 5 against Adv. In addition to the quantitative evaluation, we also give qualitative evaluation by visualizing the learned representations using t-SNE BID9 (Figure 2). Precisely, we randomly sample 1,000 examples from the validation sets and reduce the dimension of data by using principal component analysis over the output of the E to speed up the computations and suppress some noise. The number of components is selected such that the amount of variance that needs to be explained The interesting thing here is, one method fails to achieve good final performance and the other achieve it even if both methods give the similar accuracy of D during training. For example, in the Opp-L dataset (center of the Figure), MA-DNN and MARS show similar performance curve during training, but MARS gives more than 0.10 better score at final (0.85 to 0.96). This implies that accuracy of the discriminator is not an only definitive factor that determines the success of AFL.Figure 3-c shows how the update of E affect the − log q D k. Specifically, each line represents the average effect of the loss by the update E, and the shaded area represents the minimum and maximum effect of the loss, which means that larger shaded area indicates the higher variety of the D k. The number listed next to each method represents the standard deviation of the classifiers. The demonstrates that the influences of the update to each adversary are more varied for MARS compared to MA, which implies the adversaries of higher α have more diversity, and it makes the D less vulnerable against the update. Figures 4-a and b compares the performance between different K and α. Note that α = 0.0 means the method using all feature spaces, which is MA. The shows that although there is no clear winner among the α, using subspaces gives lower accuracy on predicting S, the number of discriminator K effect accuracy on S while the accuracy on Y is robust to the K especially if α ̸ = 0, indicating the possibility that the performance of AFL could be improved by incorporating multiple adversaries. This study proposed MARS, which incorporates multiple adversaries where each adversary has a different role and conducted empirical validations on the efficacy of the proposed method for censoring representations, specifically user-anonymization for the data of wearables. TAB0 compares the proposed method and several baselines and shows the efficacy of the proposed method against various evaluators. Figure 2 qualitatively shows that the proposed method provides wellanonymized representations. FIG4 -c shows that each adversary in MARS has the diverse role, ing MARS more robust to the update of E as a whole. All these support that the proposed method is more effective in removing the influence of a specific factor (user in experiments) compared to the previous methods. One of the reasons why MARS works well is that the adversary is designed to have diverse-views by incorporating random subspace methods, ing the encoder need to be stronger to deceive the adversary. It is worth mentioning that the capacity or accuracy of the adversary is not the only a definitive factor that determines the success of the adversarial feature learning, as shown by the superior performance of MARS over MA that has 1 1−α times the larger capacity of MARS. Moreover, the final performance of AFL is significantly different even if the accuracy of D is reasonably similar during training, as shown in FIG4 -b. As mentioned in the related work section, such knowledge is essential to design the adversary in practice, and prior studies of adversarial feature learning did not address this issues. Although this paper focused on the case where the subsets are randomly selected and fixed, this might not be necessary. One of the possible extensions is to determine subsets with more sophisticated ways (e.g., by performing clustering or soft-clustering on the representation spaces after few training iterations), or to learn how to select the subset itself by adding the criterion regarding the diversity of adversaries. Also, it might be possible to realize the diversity of adversaries by methods other than subspace selection. One possible way is to constrain weights of two adversaries so that they are an orthogonal view, which is used in semi-supervised learning using co-training BID11, or it might be worth a try to add different noises for each adversary. It might be worth mentioning about applicability and implications of MARS for other applications of adversarial training, such as image generation. From the perspective of the applicability, the MARS itself does not rely on any domain-specific settings and is therefore general enough for many applications based on adversarial training. For example, we can build multiple-adversaries upon the subset of feature spaces (maybe not on the image spaces). This makes discriminator have diverse-view, so it might be useful for preventing mode collapse that is one of the well-known problems in imagegeneration with adversarial training. In the context of image-generation, Generative Multi Adversarial Networks proposed by BID0, which also use multiple adversaries, shows that multiple adversaries are useful for generating better images, and for avoiding mode collapse. It might be interesting to see if enhancing the diversity of discriminators by preparing asymmetric adversaries as with this paper helps to generate a better image or to avoid mode collapse better. Table 2 shows the selected λ for each combination of datasets and baselines. Although the best hyper-parameter might be determined by the balance between log q M and log q D, here we cannot see the obvious relationships between the best λ and the easiness of tasks.

Represent the TLDR for finding the corresponding scientific article

This paper improves the quality of the recently proposed adversarial feature leaning (AFL) approach for incorporating explicit constrains to representations, by introducing the concept of the {\em vulnerableness} of the adversary.

A method to binarize both weights and activations of a deep neural network that is efficient in computation and memory usage and performs better than the state-of-the-art.


Question: Represent the article for finding a single-sentence TLDR.

Question: A key problem in neuroscience, and life sciences more generally, is that data is generated by a hierarchy of dynamical systems. One example of this is in \textit{in-vivo} calcium imaging data, where data is generated by a lower-order dynamical system governing calcium flux in neurons, which itself is driven by a higher-order dynamical system of neural computation. Ideally, life scientists would be able to infer the dynamics of both the lower-order systems and the higher-order systems, but this is difficult in high-dimensional regimes. A recent approach using sequential variational auto-encoders demonstrated it was possible to learn the latent dynamics of a single dynamical system for computations during reaching behaviour in the brain, using spiking data modelled as a Poisson process. Here we extend this approach using a ladder method to infer a hierarchy of dynamical systems, allowing us to capture calcium dynamics as well as neural computation. In this approach, spiking events drive lower-order calcium dynamics, and are themselves controlled by a higher-order latent dynamical system. We generate synthetic data by generating firing rates, sampling spike trains, and converting spike trains to fluorescence transients, from two dynamical systems that have been used as key benchmarks in recent literature: a Lorenz attractor, and a chaotic recurrent neural network. We show that our model is better able to reconstruct Lorenz dynamics from fluorescence data than competing methods. However, though our model can reconstruct underlying spike rates and calcium transients from the chaotic neural network well, it does not perform as well at reconstructing firing rates as basic techniques for inferring spikes from calcium data. These demonstrate that VLAEs are a promising approach for modelling hierarchical dynamical systems data in the life sciences, but that inferring the dynamics of lower-order systems can potentially be better achieved with simpler methods. Many datasets in the life sciences are generated by a hierarchy of dynamical systems, wherein lower-order dynamical systems that directly generate the data are driven by higher-order dynamical systems that are not observable. This problem is outlined in figure 1A, in which noisy observations x depend on the state z 1 of a low-order dynamical system that is perturbed by inputs u 1. The state of this dynamical system is also coupled to the state z 2 of a higher-order dynamical system, which can be perturbed independently by inputs u 2. One example of such a system in in-vivo two-photon calcium imaging from neuroscience. Calcium imaging provides systems neuroscientists with the ability to observe the activity of hundreds of neurons simultaneously during behavioural experiments. Such experiments have allowed neuroscientists to ask questions about the underlying computations and algorithms that neural circuits are implementing in perception, decision-making, memory, and many other processes. Such experiments can be characterized as observing a hierarchical dynamical system (Fig 1B) in which measurable calcium fluorescence is primarily determined by dynamics based on voltage-gated calcium channels and calcium binding to fluorescence dyes, and the rate of fluorescence transients controlled by the underlying computation. Recent applications of sequential variational autoencoders to neural data analysis has seen great success in inferring underlying computations in populations of cells in macaque and human motor cortex. By characterizing neural computation as low-dimensional dynamic factors in a non-hierarchical dynamical systems, showed that these dynamic factors trained to generate the inhomogeneous intensity functions explaining the rate of spikes assumed to follow a Poisson process. Crucially, these low-dimensional factors could also decode reaching behaviour of macaques and humans with much higher fidelity than any other dimensionality reduction method. Although this is a significant advance in our ability to analyze neural data in the form of spikes trains, two-photon calcium imaging poses the additional problem of identifying latent spike trains in fluorescence traces. This problem has been independently addressed in a number of different ways, including deconvolution and variational inference. If we continue to model the frequency of events as being generated by a Poisson process, this can be seen as hierarchy of dynamical systems (Fig 1A), in which low dimensional dynamics generate spike probabilities that in turn drive fluctuations in biophysical dynamics of calcium activity (Fig 1B. Here we propose a method that extends LFADS to accommodate calcium activity using this hierarchical dynamical systems approach, in which we can infer both the latent dynamics and the latent spike trains from the observed calcium fluorescence signal. The model is a variational ladder autoencoder (VLAE) with recurrent neural networks (RNNs) that supports uncovering latent dynamical systems (Fig 1C, full directed acyclic graph in Fig A1). It can be seen as a unification of two recent applications of variational autoencoders (VAEs) in neuroscience: 1) Latent Factor Analysis for Dynamical Systems (LFADS) and 2) DeepSpike, a VAE approach to inferring spike counts from calcium imaging data . We choose the VLAE approach since it has been shown to avoid the problem of trying to reconstruct the data solely from the lower-order features by separating latent variables via divergent deterministic pathways in the encoder, and convergent deterministic pathways in the generator (Sønderby et al., 2016; . Model hyperparameters are shown in table A1. The inferred dynamical system underlying the frequency of calcium events in the data is identical to that of LFADS (Fig 1C, blue modules). The prior distribution of initial conditions g 0 and external inputs u t are modelled as Gaussian distributions P (g 0) = N (µ g0, σ 2 g0), and P (u t) = N (µ ut, σ 2 ut). The underlying dynamical systemġ = G(g t, u t) is modelled by a Gated Recurrent Unit (GRU) taking the initial hidden state g 0 and inputs u t. Low dimensional factors f t are calculated as a linear transformation of the generator hidden state f t = W f ac g t. These factors are used to reconstruct the Poisson process intensity function with a fully connected layer and exponential non-linearity Inferred spike counts s t are generated by sampling z t from Gaussian distributions P (z t) = N (µ zt, σ 2 zt) and projecting these through an affine transformation and non-linearity along with the factors from the deeper layer, i.e., We assume a simple model of calcium dynamics:ẏ = −y t /τ y + α y s t + β y where the parameters τ y, α y, β y are measured from the data, however it is a topic for future research to fit the calcium dynamics simultaneously. In our synthetic data, these are valued at 0.4 s, 1, and 0 respectively. The value of τ y is chosen as it is the known decay time constant of GCamP6, a commonly used calcium fluorescence indicator used in calcium imaging experiments. The calcium fluorescence signal x t is mapped onto the parameters of the variational posterior distributions Q(z t |x), Q(g 0 |x), Q(u t |x). These distributions are all modelled as Gaussians, with the mean and standard deviations parameterized by a stack of bidirectional GRUs, con. The final hidden states of E 1 gen and E 2 gen are mapped onto the parameters of Q(z 0 |x) and Q(g 0 |x) respectively with fully connected layers. The hidden states E 1 con and E 2 con are concatenated and passed as inputs to single direction GRUs C 1 and C 2. The hidden states of C 1 and C 2 are concatenated at each time step t with s t−1 and f t−1. Subsequently these concatenated activities are mapped onto the parameters of Q(z t |x) and Q(u t |x) with fully connected layers. One of the advantages of using VLAEs is that the evidence lower bound (ELBO) formulation is the same as for VAEs despite the hierarchical latent space. As such, our cost function remains very similar to that of LFADS. g0,u∼Q(z,g0,u|x) [log(s|λ)] Where W gg is the weight matrix in the GRU that take g as input. The likelihood function P (x t |y t) is modelled as a Gaussian distribution x t ∼ N (y t, σ 2 y), where σ 2 y is learned. Although s t is not discrete, P (s t |λ t) is treated as an approximate Poisson process s t ∼ P oisson(λ t) = s λt t exp(−λ t)/Γ(s t + 1). This is an inductive bias to ensure that the inferred spike trains in the lower-order dynamic system is constrained to follow the dynamics of the higher-order dynamic system. The latent variables are treated as independent, with hierarchy imposed in the deterministic paths, i.e., Q(z, g 0, u|x) = Q(z|x)Q(g 0 |x)Q(u|x) Parameters of our model were optimized with ADAM, with an initial learning rate of 0.01, which decreased by a factor of 0.95 whenever plateaus in training error were detected. As in LFADS training, KL and L2 terms in the cost function were'warmed up', i.e., had a scaling factor being 0 and 1 applied that gradually increased. Warm-up for the deeper parameters (blue modules in Figure 1) was delayed until warm-up for shallower parameters was completed (red modules in Figure 1). The model was tested on synthetic data with Lorenz dynamics embedded in the frequency of calcium fluorescence transients, as described by , where generated spikes were convolved with an exponential kernel with a time constant of 0.4 ms, and white noise added to the ing traces. We measure the performance of the model in three ways: 1) uncovering the underlying Lorenz dynamics, 2) reconstructing the rates of calcium transients an inhomogeneous Poisson intensity functions, 3) reconstructing the spike counts contributing to increases in the calcium fluorescence signal. The model was compared against a ground-truth where the spike counts are known, and LFADS is used to reconstruct the latent dynamics and intensity function, and against a model where spike counts are extracted using a deconvolution algorithm before using LFADS to reconstruct the rates and intensity function (OASIS + LFADS). It was also tested against a model that used a 1-D convolution of the intensity function to reconstruct either the first two (Gaussian-LFADS) or four (Edgeworth-LFADS) time-varying moments of fluorescence, as used previously in estimating the intensity functions of filtered Poisson processes in neuroscience . Figure 2 shows examples of performance of our model in reconstructing the fluorescence traces (Fig 2A), Poisson intensity functions (Fig 2B), spikes (Fig 2C) and Lorenz dynamics (Fig 2D). Visually, the model provides very close fit to the fluorescence traces, intensity functions, and Lorenz dynamics. The model also captures spike-timing, although these spike trains appear smoothed. Table 1 compares the R 2 goodness-of-fit on reconstructing held-out validation data with ground-truth latent dynamic structure. Of all approaches, our model easily performs best at reconstructing fluorescence traces. It should be noted that the reason the Gaussian and Edgeworth LFADS models perform so poorly at reconstructing fluorescence is that it only attempts to reconstruct the time-varying moments of fluorescence as a 1-D convolution of the intensity function, which stems from the approximation of exponential-kernel shot noise processes . Additionally, our model almost performs as well as LFADS in reconstructing the Lorenz dynamics. It is to be expected that LFADS performs better than our method, since there is an additional source of observation noise in our synthetic dataset generating fluorescence transients from spikes. Notably, our model does not perform as well as the deconvolution method OASIS in reconstructing spike trains, however this does not impact the ability of our model to reconstruct the latent dynamics. In fact, constraining the reconstructed spikes by the latent dynamics may mitigate any errors in spike train reconstruction that occur by deconvolution, since the deconvolution algorithm may erroneously drop spikes during high rates, whereas our model should be less likely to do so. It will be necessary to assess this possibility further. It should be noted that the deconvolution algorithm performs much better at reconstructing spike trains in our synthetic dataset than in real datasets where ground-truth spiking is known . To our knowledge, there are no known dual recordings of population 2-photon calcium imaging with ground-truth electrophysiology in a subpopulation of neurons in-vivo during behaviour driven by hypothesized low-dimensional dynamics that we would be able to validate this with. Nevertheless, since the relationship between calcium dynamics and somatic spiking is highly non-linear, especially in dendrites, it remains to be seen how useful it is to faithfully reproduce unseen spike trains in calcium fluorescence activity. The model was then tested on synthetic data generated from a 50 cell recurrent neural network with chaotic dynamics, and an external perturbation at a random time, as described in , with parameters adjusted to make the data more representative of firing rates and time scales observed in calcium imaging experiments. Spike trains were transformed into fluorescence signals using the same procedure as with the Lorenz system dataset. (Fig 3A), spike trains (Fig 3B), and intensity functions (Fig 3C). Visually, the fluorescence traces and spike trains have been reconstructed reasonably well, whereas reconstructions of the the intensity functions are highly noisy. The latent dynamics represented by the factors (Fig 3D) and external inputs (Fig 3E) also show a lot of noise. This appears to be due to difficulty in inferring external inputs, in which it is not clear whether the timing of external perturbation has been accurately inferred, despite a slight transient at roughly the time of the perturbation in the example. Table 2 compares the R 2 goodness-of-fit on reconstructing held-out validation data, which demonstrates that our model performs very well in reconstructing the fluorescence signal and does reasonably well at reconstructing spikes. However, in this more complex benchmark where the deeper dynamic system is perturbed by external input, the pre-processing of fluorescence transients with the OASIS deconvolution algorithm reconstructs firing rates far better than our method. We present a hierarchical recurrent variational autoencoder model capable of reconstructing latent dynamics, latent spike trains, and calcium fluorescence traces in a benchmark synthetic dataset. Of the four methods tested, our model is the only one capable of reconstructing all three. Furthermore, our model performed best in reconstructing latent dynamics in our synthetic dataset We will need to assess our model on further synthetic benchmark data to assess the validity of our approach. Since our model is trained end-to-end, it should be possible to extend to reconstructing raw 2-photon imaging videos, which could enable us to train models to uncover latent dynamics from arbitrarily shaped neuronal structures. This would of great use to neuroscientists who are largely restricted to techniques that extract fluorescence traces from regions of interest with somatic shapes, whereas the morphological diversity of dendrites is much greater. An additional advantage of using our hierarchical model is that we can obtain measures of the uncertainty in both the latent dynamics, and the latent spike trains. The correlation in uncertainty between layers of this hierarchy may be what allows superior inference of the latent dynamics, despite less accurate reconstructions of the spike trains than OASIS, which provides no measure of uncertainty. We hope to improve our model to better capture the relationships between layers of this hierarchy in future. We describe a use-case in neuroscience (2-photon calcium imaging data) for which this model may be very useful. However, we are keen to investigate the general case of hierarchical dynamical systems and their utility in uncovering structure in datasets outside this domain. A APPENDIX

Represent the TLDR for finding the corresponding scientific article.

We extend a successful recurrent variational autoencoder for dynamic systems to model an instance of dynamic systems hierarchy in neuroscience using the ladder method.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.


Question: Represent the scientific article to find a short TLDR

Question: With the proliferation of specialized neural network processors that operate on low-precision integers, the performance of Deep Neural Network inference becomes increasingly dependent on the of quantization. Despite plenty of prior work on the quantization of weights or activations for neural networks, there is still a wide gap between the software quantizers and the low-precision accelerator implementation, which degrades either the efficiency of networks or that of the hardware for the lack of software and hardware coordination at design-phase. In this paper, we propose a learned linear symmetric quantizer for integer neural network processors, which not only quantizes neural parameters and activations to low-bit integer but also accelerates hardware inference by using batch normalization fusion and low-precision accumulators (e.g., 16-bit) and multipliers (e.g., 4-bit). We use a unified way to quantize weights and activations, and the outperform many previous approaches for various networks such as AlexNet, ResNet, and lightweight models like MobileNet while keeping friendly to the accelerator architecture. Additional, we also apply the method to object detection models and witness high performance and accuracy in YOLO-v2. Finally, we deploy the quantized models on our specialized integer-arithmetic-only DNN accelerator to show the effectiveness of the proposed quantizer. We show that even with linear symmetric quantization, the can be better than asymmetric or non-linear methods in 4-bit networks. In evaluation, the proposed quantizer induces less than 0.4\% accuracy drop in ResNet18, ResNet34, and AlexNet when quantizing the whole network as required by the integer processors.

Represent the short summary for finding the corresponding scientific article (long).

We introduce an efficient quantization process that allows for performance acceleration on specialized integer-only neural network accelerator.

represent the tldr for finding the corresponding scientific article!

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.