Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Data augmentation techniques, e.g., flipping or cropping, which systematically enlarge the training dataset by explicitly generating more training samples, are effective in improving the generalization performance of deep neural networks. In the supervised setting, a common practice for data augmentation is to assign the same label to all augmented samples of the same source. However, if the augmentation in large distributional discrepancy among them (e.g., rotations), forcing their label invariance may be too difficult to solve and often hurts the performance. To tackle this challenge, we suggest a simple yet effective idea of learning the joint distribution of the original and self-supervised labels of augmented samples. The joint learning framework is easier to train, and enables an aggregated inference combining the predictions from different augmented samples for improving the performance. Further, to speed up the aggregation process, we also propose a knowledge transfer technique, self-distillation, which transfers the knowledge of augmentation into the model itself. We demonstrate the effectiveness of our data augmentation framework on various fully-supervised settings including the few-shot and imbalanced classification scenarios.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a simple self-supervised data augmentation technique which improves performance of fully-supervised scenarios including few-shot learning and imbalanced classification.

We propose a novel adversarial training with domain adaptation method that significantly improves the generalization ability on adversarial examples from different attacks.


Question: Represent the article for finding a single-sentence TLDR

Question: Long short-term memory (LSTM) networks allow to exhibit temporal dynamic behavior with feedback connections and seem a natural choice for learning sequences of 3D meshes. We introduce an approach for dynamic mesh representations as used for numerical simulations of car crashes. To bypass the complication of using 3D meshes, we transform the surface mesh sequences into spectral descriptors that efficiently encode the shape. A two branch LSTM based network architecture is chosen to learn the representations and dynamics of the crash during the simulation. The architecture is based on unsupervised video prediction by an LSTM without any convolutional layer. It uses an encoder LSTM to map an input sequence into a fixed length vector representation. On this representation one decoder LSTM performs the reconstruction of the input sequence, while the other decoder LSTM predicts the future behavior by receiving initial steps of the sequence as seed. The spatio-temporal error behavior of the model is analysed to study how well the model can extrapolate the learned spectral descriptors into the future, that is, how well it has learned to represent the underlying dynamical structural mechanics. Considering that only a few training examples are available, which is the typical case for numerical simulations, the network performs very well. Data driven virtual product design is nowadays an essential tool in the automotive industry saving time and resources during the development process. For a new car model, numerical crash simulations are performed where design parameters are changed to study their effects on physical and functional properties of the car such as firewall intrusion, weight, or cost . Since one simulation run takes a couple of hours on a compute cluster, running a large number of simulation is not feasible. Therefore, a system that is able to use a limited dataset and predict new simulations would make the development process faster and more efficient. The rise of deep neural networks (DNNs) in recent years encourages further research and industrial usages. Besides manifold research for autonomous driving, it is natural for the automotive industry to seek and evaluate the possible applications of DNNs also in the product design stages. As an example, we investigate car crash tests, in which for example the plate thickness of certain parts strongly influences the bending behavior of structural beams and as a also the intrusion of the firewall into the passenger compartment. Here, numerical crash simulations for different variations of such thicknesses are used as a dataset for learning. The aim is to design a system based on a DNN architecture that learns the crash behavior and would be able to imitate the crash dynamics. Car crash simulations are based on a mathematical model of the plastic deformations and other physical and mechanical effects. They are defined on a computing mesh of currently up to three million points and up to a hundred time steps are stored. Each data instance is a simulation run-of pre-selected parts and/or time steps-that is very high dimensional. Working with this data directly exasperates any machine learning (ML) method, but a transformation of this data presented in allows to obtain a new representation that uses only a small number of coefficients to represent the high resolution numerical solutions. The transformed representation is employed here to compress the mesh geometries to feature sets suitable for neural networks, while avoiding to directly handle geometries in the machine learning method. This way, a network designed for video prediction and embedding based on a long short-term memory (LSTM) based architecture can be adapted for mesh data. Since LSTM is a recurrent neural network that allows to exhibit temporal dynamic behavior with feedback connections, it is a natural choice for learning the 3D sequences. The aim is that the network learns the observed crash behavior including translation, rotation, or deformation of the parts in the model. Since the contribution of this paper is using DNNs for analyzing car crash data, the related works are categorized into a group of publications in which DNNs are extended for 3D graphics and one that concerns the use of ML techniques for analyzing car crash simulations. For the latter, one typically uses different embedding techniques to obtain a low dimensional representation for the intrinsic underlying data space and to cluster simulations with similar characteristics together (; ; ; ;). The majority of publications about 3D DNN tried to extend CNN for 3D space and focus on description learning and shape correspondence, also known as geometric deep learning,;;;;; ) and some developed CNN filters for unorganized point clouds (a; b). The very active research is so far very compute resource consuming and there is no extension of ConvLSTM for 3D space to our knowledge, but for prediction one would need an LSTM (or GAN) approach. However, a couple of very recent works introduce new feature sets and architectures for mesh embedding using autoencoders and LSTM (b; ; a). The feature representation is using local shape deformations obtained by solving an optimization problem at each node and a global optimization for compensating for rotations. They have shown that after training the network, a sequences of 3D shapes as an animation can be generated by doing operations in the latent space. The bidirectional LSTM architecture is shown to outperform autoeconders (a). An LSTM based learning network has also been proposed in , where the obtained feature representation is then taken as the temporal data to be feed into a CNN that takes the features and represents them in a lower dimensional latent space. This information is subsequently feed into the LSTM module. Data collection is a bottleneck for deep learning applications. If the training data is not diverse enough, the network would neither be able to properly learn the intrinsic data space nor be able to return reasonable output for before unseen data. We focus on surface mesh data from numerical simulations of car crashes, where in the industrial setting the car model is divided into several physcical car parts. As a simple car model we use a Chevrolet C2500 pick-up truck, a model with around 60,000 nodes from the National Crash Analysis Center 1. The data stems from numerical crash simulation 2 of a frontal crash for random variations of the plate thickness of nine structural components, a setup similar to. The thickness variations in different deformation behavior. Figure 1 shows a snapshot of the crash simulation for the truck model. The geometries of the car model and parts are available in a regular mesh format and the correspondence of vertices between different simulations and over the time of the simulation are known by their node id. Therefore, instead of working with the meshes one can simply work with vertices and treat them like organized point clouds, while being able to recover the mesh and connectivity at any time. Now, the features for training the network are obtained from these point clouds and the network outputs a feature vector that is later post-processed to a point cloud or mesh. Instead of working directly with 3D surface meshes, a set of features is extracted for training the network. Such a feature set should be able to represent the dynamics of the crash efficiently. a compact representation for deforming shapes has been presented for the car crash case. The approach is based on the property that the Laplace Beltrami Operator (LBO) on a surface is invariant to isometric deformations. That is, the LBO is the same if a surface mesh is de- Figure 1: A snapshot of the crash simulation of the 3D model used for data collection with a zoom unto the studied longitudinal beams, from (left). The four selected parts (beams) after the crash for one selected simulation, later used for illustration (right). The colors of the parts are used to display error behavior per part. formed in such a way that it is neither stretched nor teared apart, which is the case for the considered simulations, which additionally all start from the same geometry. Consequently the eigenvectors, which form an orthogonal basis, do not change under an isometric transformation and can be used to represent mesh functions such as the deformations (as three functions, one each for x, y, z). The representation is obtained by projecting mesh functions onto the common orthogonal basis to compute so-called spectral coefficients. It turns out that most of the variations of the deformed shapes are concentrated in a small number of coefficients. Therefore an efficient representation is obtained using few spectral coefficients. As shown by Brezis & Gómez- using suitable assumptions, for the L 2 -approximation of functions controlled in the H 1 -Sobolevnorm the orthonormal basis stemming from the Laplace operator provides an optimal approximation in a certain sense; this can be extended to the LBO and functions in the Sobolev space H 2,2. Furthermore, the spectral representation can be understood as a mesh surface analogy to the Fourier decomposition of signals. This representation is introduced here for training LSTMs and allows to bypass the complexity of dealing with large meshes directly. To formalise, in B n×m we collect the first m unit eigenvectors of the LBO, where n is the number of points in the considered 3D shape. The spectral coefficients C 3×m at one time step are obtained by where R 3×n contains the x, y, z coordinates of the n points from the considered 3D shape. Four B matrices are required for four distinct parts in the dataset, which are four distinct geometries. Recovering the 3D shapes from their spectral coefficients is possible by since B n×m is orthonormal. R is an approximation of R and by choosing larger m, the approximation error gets smaller. In other words, the more eigenvectors are used in B n×m, the more details are saved in the spectral representation. Figure 2 visualizes the localization and histogram of average error for reconstructing four selected parts from their spectral coefficients. The averaging is done over 205 × 10 samples for each part independently. Note that the bounding box of the entire data is × [−600, 600] ×, determined over simulation time. Comparing the histogram with the tight bounding box's dimensions shows that the reconstruction error is not very high for m = 40, while the part with blue color coding seems to have more overall error. Note that the error is localized mostly toward the front of the parts, this area goes through very large deformations during the crash. Note that with m = 100 the observed maximal observed error would go down from 40 to 20. For frame prediction in video and to obtain latent representations a number of quite interesting applications in robotics and computer vision have been developed. Different DNN architectures have been investigated very extensively. was one of the first proposals of an unsupervised LSTM based representation learning method for video. The logic behind the choice is that the same operation must be applied at each step to propagate dynamics to the next step. This implies that the underlying dynamics of the dataset remains the same, i.e. the same dynamics acting on any state, at anytime, should produce the next state . Others extended this work by adding convolutional operations in LSTM, introducing new recurrent networks, or using model based architectures (; ; ; ;). Car crash simulation data can basically be considered a sequence of 3D geometries. In analogy to the video processing case, one can see each geometry in the sequence as a video frame. Inspired from the work on video prediction, we use a similar architecture for the case of car crash representation learning. We choose a two branch LSTM architecture with reconstruction and prediction decoders from. The encoder LSTM maps an input sequence of k time steps into a fixed length vector representation. This representation is decoded using a decoder LSTM to perform the reconstruction of the input sequence of size k. Receiving a few steps l, l < k, from the beginning of the sequence, the other decoder LSTM predicts future sequences of length k − l, see Figure 3, where we here use l = k/2 for simplicity. The architecture is in shown to have a better performance for predicting future frames compared to other approaches. The design of the prediction encoder-decoder LSTM is same as that of the autoencoder LSTM, except that it is like a supervised method in which the decoder LSTM predicts future behavior that comes after the input sequence while the encoders hidden state will capture information about the representation of the input sequence. In comparing to an autoencoder LSTM that receives the entire k time steps of the simulations and reconstruct them again, the prediction encoder-decoder LSTM receives just the first l steps and predicts the remaining time steps. Therefore, the input sequence acts as seed points for generating the rest of the sequence. This compositional architecture is supposed to address the shortcomings that one has to confront by the use of an autoencoder or encoder-predictor alone. Namely, on the one hand an autoencoder LSTM suffers from a bias by memorizing the inputs, which is not sufficient for predicting future frames. On the other hand, an encoder-decoder LSTM suffers from the bias to save information mostly about the last few frames since these carry more information for predicting the future frames, but then the representation at the end of the encoder will ignore large parts of the input. In the compositional architecture, the model is trained to also predict all of the input sequence, therefore it cannot just store information about the last few frames . It is worth to mention that the dataset contains the translation, rotation, and deformation of car parts during the crash. Therefore the network would learn the entire degrees of freedom of the crash dynamics and would be able to imitate the crash by the prediction decoder after receiving the initial time steps as seeds. Figure 3: The two branches LSTM autoencoder network architecture. The data sequence of f s are feed into the network in two lengths and a joint encoder is learned. The top decoder is learning the reconstruction, while the bottom decoder is performing future prediction. Although the 3D truck model consists of several parts, only the left and right structural beam, which are made of two parts each, are considered for evaluating our approach. These beams are structurally very important parts of the car and typically investigated by engineers, therefore it is justified to concentrate on these parts. Overall 205 different simulations were performed. For each simulation, ten snapshots equally distributed in time are selected and the deformation of the two beams are extracted from it. Therefore a data sample consists of the four selected parts during ten time steps. Together, the dataset after the extraction from the crash simulations contains 205 samples and each full sample is a long concatenated vector of 4 × 3 × 40 × 10 elements. The entire dataset is divided into training and test set of 105 and 100 samples, respectively. Each data point, i.e. each time step of a simulation, is normalized by the l 2 norm of the features at t = 0 beforehand. Note that one could consider a different scenario in which each part, out of the four, over all ten time steps is a data sample, that is treat the for beams with individual machine learning models. Since the dynamics of the two beams are coupled during the crash, considering them together can reduce the error for learning the crash dynamics by the network and we thereforce consider this setup. We employ the two branch LSTM from Figure 3, where the implementation is done with Keras. In our experiments, the encoder component of the network has 1000, the decoder has 1500, and the prediction part has 2000 LSTM units without any convolutional layer, as defining a convolutional layer for 3D graphics is not trivial and existing approaches are resource intensive. Further, we use ReLU as the activation function. Using ADAM with default parameters the entire network is trained together, which includes the encoder, reconstruction, and prediction parts over 100 iterations, until the mean squared error (over all parts) reaches the order of 10 −7 and becomes stable. The training phase takes about 30 minutes on an Intel i7-7700 CPU@3.60GHz × 8. The achieved minimum during training is a trade-off between reconstruction and prediction loss functions. The prediction part of the network receives the first five time steps as inputs and generates the next five steps of the crash (which are both a vector of 4 × 3 × 40 × 5 elements). For each grid point j we compute in the following the least squares error at time t and averaged over the simulations, as well as accumulated over time: where s is the number of simulations and S i j (t),Ŝ i j (t) are the three-dimensional (one for each direction) original mesh function and its reconstruction (or prediction) for simulation i at time t. Figure 4 shows the histograms of the reconstruction and prediction error for the training and test datasets of all four parts, only here for the spectral coefficients. The reconstruction error is larger than the prediction error since the reconstruction part recovers the entire k = 10 time steps while the prediction part just generates the last l = 5 time steps. and have reported a bifurcation for this car crash dataset. That is, two different bending behavior for the beams arise in the simulation data, which is due to changes in the plate thicknesses. One can cluster the 205 simulations into two groups of 71 and 134 for the two bending behaviors. It gives an unique opportunity to investigate the performance of the proposed system by analyzing and visualizing the error for each branch of bifurcation individually. Moreover, it is possible to see if and how well the network can recognize this bifurcation and how the spectral coefficients can preserve this. The 105 simulations used for training include samples from both branches of the bifurcation, as do the remaining 100 for testing. The distances between ground truth and the outputs of two decoders, after decompression and re-normalization from the spectral coefficient back to 3D shapes, are visualized on a template to present the spatial localization of the average error of each bifurcation branch and its histogram is also shown for a better comparison (all for the testing dataset). Figure 5 gives the accumulated error over time. Overall, the reconstruction and prediction errors are small in relation to the bounding box of the data, the network based on the spectral coefficients is able to learn the complex structural mechanics. It can be seen in first two histograms of Figure 5 that the errors accumulation for the reconstruction in the two branches is similar, which can also be seen in the first and second row in Figure 5. The part color coded red shows somewhat higher error values, while the blue part has more mid-range errors. The third and fourth row in Figure 5 for the prediction have very different error localization. Considering the localization and histograms of the average error over time for the reconstruction branch of the network, one can observe that the error behavior stays roughly the same over time, i.e. the localization of error and histograms show that the error stay the same. Regarding the localization and histogram of the average error over time for the prediction branch of the network, now the error increases in time and the behavior changes at the eighth time step. Here, one can observe that the localization of the error changes and the histograms show that the error increases more strongly for the first branch, see Figure 7 in the Appendix. The encoder LSTM weights can be used for visualizing the intrinsic underlying space of the dataset. Therefore, in order to see how well the encoder learned the representation and dynamics of the car crash simulations, the weights are visualized using different markers for the two branches of bifurcation. Due to the encoder having 1000 layers, to which each sample is mapped, one needs to use some visualization techniques for embedding from higher dimensions to a lower dimensional space. We use t-SNE and show the in Figure 6. As can be seen, the bifurcation is shown as two well-separated clusters. There are a few points from each reconstruction branch 1 reconstruction branch 2 prediction branch 1 prediction branch 2 bifurcation branch that are misaligned with the rest of their branch members. This might be because of using the spectral coefficients as an approximation which leads to losing some details about the bifurcation, therefore increasing m might rectify the issue. Or a 2D embedding is not sufficient for properly visualization the encoder LSTM weights. Nevertheless, this 2D visualization proves that the network was able to learn the complex dynamics of the crash since the bifurcation is well represented by the encoder weights. Video frames prediction has been in the center of attention of researchers for a while, but there has been only very few extensions of these works to the 3D case so far. The problem is addressed here by introducing spectral coefficients to encode functions on the geometry together with a two branch LSTM based architecture without any convolutional layer, which has already proven to be feasible for video embedding and future frames prediction. The employed LBO basis and the ing spectral coefficients provide a trade-off between accuracy and required computational resources. We encode the 3D shapes by a set of features using the eigenvectors of the LBO. For empirical evaluation, a dataset is employed from a set of numerical simulations of a car during crash under different design conditions, i.e. plate thickness variations. The appearance of a bifurcation during the crash in the dataset, motivates an error analysis done for both groups to see how good the network performs in the presence of a bifurcation. In both branches, the network is able to perform very good predictions, while we observe different error localisations for reconstruction versus prediction. Moreover, the 2D visualization of the reconstruction branch shows the bifurcation as two clusters. In any case, from a relatively small number of data, the proposed network using spectral coefficients is able to learn complex dynamical structural mechanical behaviors. Future work could go toward scaling the pipeline for learning the crash dynamics of the entire car and larger mesh sizes, which increases the needed computational effort. On the other hand, one might be able to use smaller number of eigenvectors by not simply selecting the first few ones, but those with a large variance in the spectral coefficients of the data set. Furthermore, in practical settings, re-meshing of the parts can take place, here using spectral coefficients can ease this step since one can encode shapes with different vertices number to fixed size feature vectors, as long as the geometry is (approximately) isometric. Still, there is the overall question, if and how a trained network can be evaluated for changed geometries (relevant question for any 3D DNN approach introduced so far) or different crash setups. Moreover, adding design parameters could also improve the accuracy but requires modifications of the networks architecture. For practical applications, as each crash simulation requires hours of heavy computation running computational solvers on a large cluster, a system that is able to learn the representation of experiments with very few training data and generate the predicted simulation for new design parameters would save much resources. Moreover, the ultimate goal of research along this direction would be a data driven system that receives very little information about the simulation (like design parameters) and output the crash sequences with minimum error. Another application of the current system could be feasibility detectors while running the simulation on the compute cluster. Using the network, one could check if the simulation goes well or if for some reasons it should be terminated. From the current stage of the system, one would be able to generate the parts of the future simulation simply by extrapolating the learned spectral coefficients from a few initial time steps, which are already computed on the cluster, as inputs. If the distance between network predicts and simulation gets very large over the iterations, the simulation can be terminated since it failed the feasibility check. Further, related works such as introduce a specific feature set and LSTM autoencoders, where also graph convolution operation is required. This approach could be applied for car crash data under the assumption that the local optimization can still be applied for large deformations as the ones occurring in our applications. Further, the ing features are long vectors, which in 8 hours for learning on a CPU/GPU system for a data set similar in size to ours, where we need 30 minutes. Nevertheless, a comparison of these two approach will be worthwhile future work. A APPENDIX time step 6 time step 7 time step 8 time step 9 time step 10 time step 6 time step 7 time step 8 time step 9 time step 10

A two branch LSTM based network architecture learns the representation and dynamics of 3D meshes of numerical crash simulations.

Represent the short summary for finding the corresponding scientific article (long)

A method for learning a transformation between one pair of source/target datasets and applying it a separate source dataset for which there is no target dataset


Question: Represent the scientific article to find a short TLDR:

Question: The purpose of an encoding model is to predict brain activity given a stimulus. In this contribution, we attempt at estimating a whole brain encoding model of auditory perception in a naturalistic stimulation setting. We analyze data from an open dataset, in which 16 subjects watched a short movie while their brain activity was being measured using functional MRI. We extracted feature vectors aligned with the timing of the audio from the movie, at different layers of a Deep Neural Network pretrained on the classification of auditory scenes. fMRI data was parcellated using hierarchical clustering on 500 parcels, and encoding models were estimated using a fully connected neural network with one hidden layer, trained to predict the signals for each parcel from the DNN features. Individual encoding models were successfully trained and predicted brain activity on unseen data, in parcels located in the superior temporal lobe, as well as dorsolateral prefrontal regions, which are usually considered as areas involved in auditory and language processing. Taken together, this contribution extends previous attempts on estimating encoding models, by showing the ability to model brain activity using a generic DNN (ie not specifically trained for this purpose) to extract auditory features, suggesting a degree of similarity between internal DNN representations and brain activity in naturalistic settings. One important motivation for incorporating machine learning in neuroscientific discovery is the establishment of predictive models, as opposed to models based on statistical inference. While the latter are unable to generalize to a new dataset, the former aim at sucessful generalization. In particular, encoding models aim at predicting brain activity given a model of the stimulus presented to the subject. A successful model should enable generalization to unseen data, enabling a better understanding of the underlying brain functions. Furthermore, an accurate encoding model could potentially be used to enhance machine learning, by providing an auxiliary source of training data, as recent evidence suggest that actual brain activity can guide machine learning. In this study, we tested whether a pretrained network could be used to estimate encoding models, in the case of naturalistic auditory perception. We downloaded the ds001110 dataset version 3 on openNeuro, in which 36 subjects watched a 20 minute long movie in an fMRI scanner. MRI data was collected on a 3 T full-body scanner (Siemens Skyra) with a 20-channel head coil. Functional images were acquired using a T 2 * -weighted echo planar imaging pulse sequence (TR 1500 ms, in-plane resolution 3 by 3 mm). Anatomical images were acquired using a T1-weighted magnetization-prepared rapid-acquisition gradient echo (MPRAGE) pulse sequence (0.89 mm3 resolution). More details can be found in the original paper. In this study, we report only data from 16 subjects who watched the same episde from the TV show "Sherlock". At the time of submitting this paper, OpenNeuro included only raw data. Therefore, we used a preprocessed version 1. An overview of our method is presented in figure 1, and consists in three steps 2. First, we extracted the audio track from the movie, and extracted feature vectors from all seven convolutional layers of SoundNet. The 20 minute long audio file was fed as input, and we performed interpolations according to the width of the feature maps in each layer, in order to realign the obtained feature vectors with the temporal resolution (1.5 second) of the fMRI signal. This procedure yielded a total of 946 feature vectors, for each SoundNet layer. Next, in order to reduce the dimensionality of the fMRI data, we applied hierarchical clustering using Ward criterion to parcellate each individual brain into 500 regions of interests (ROI). We subsequently realigned the fMRI data on the beginning of the movie, yielding 946 vectors of dimension 500 for each subject. Finally, encoding models were estimated separately for each subject and layer of SoundNet. We trained fully connected neural networks with one hidden layer to predict brain activity in the 500 ROI simultaneously. We report when varying the number of neurons in the hidden layer. We used ReLu activation for the hidden layer, and linear activation for the output layer. We used a learning rate of 0.001 and a L 2 penalty of 0.0001. Cross validation on the data was performed using four folds without shuffling the data (in order to ensure that the train and test data were as distant as possible temporally), and for each training fold, 10% of the data was kept for validation. We used the Adam optimiser with batches of size 50. Mean Square Error (MSE) was used as a loss function, and we applied an early stopping criterion that stops when validation MSE is not improving for 10 consecutive epochs. The final metric we use for evaluating the is the R 2 score on the test set, indicating the quality of predicting fMRI data. Additionally, we also perfomed a control analysis in which we estimated 100 null encoding models by extrating feature vectors from an untrained SoundNet, using the exact same procedure as described above. This analysis enables us to estimate the chance level of our dataset, as well as the gain obtained when using the pretrained network. 3 Results and discussion 3.1 Which layers enable the training of an encoding model? To begin with, as expected, the null models could not yield any significant training for any SoundNet layer, as indicated by R 2 scores of less than 1e −6 ± 1e −5. Next, the first four layers of SoundNet could not be used to sucessfully train encoding models, as all R 2 were less than 0.03. In the following analysis, we focus on from layers conv5, conv6 and conv7. Table 1 depicts the influence of the number of neurons in the hidden layer on the maximum over all ROI of R 2 scores, averaged across subjects and CV folds. The best were obtained using 1000 neurons in the hidden layer, which enables succesfully training an encoding model on conv5, conv6 and conv7. Furthermore, we noticed that maximum R 2 scores across folds were much higher number of neurons 50 100 500 1000 1500 Table 1: Hyper parameter exploration: average (standard error of the mean) across all subjects of maximum R 2 score, as a function of transfered layer, and number of neurons in hidden layer. than the average over folds, and we found that the second fold consistently yielded low R 2 scores (R 2 < 0.05), for all subjects, for conv5 and conv6. Regarding conv7, the first two folds yielded an average of R 2 = 0.15 across all subjects. While this issue would demand closer inspection of the data, we suspect a systematic bias in the feature vectors for conv5 and conv6 in the first half of the video. Nonetheless, we obtained good generalization for the other three folds, especially for conv7. As a consequence, in the next section we will select the fold that yielded the maximum R 2 in order to interpret the spatial maps. For all sixteen subjects, brain activity in ROIs including the superior and middle temporal gyri could be predicted with R 2 > 0.25. The corresponding SoundNet layer for which the R 2 was maximal was conv7 for 14 subjects, conv6 for one subject and conv5 for one subject. This suggest that the information in the last layers of SoundNet is linked to the fMRI activity in regions previously associated with general purpose auditory processing. Furthermore, for eleven out of sixteen subjects, brain activity in ROIs located in the left dorsolateral prefrontal cortex was predicted with R 2 > 0.15 (8 out of those eleven had a R 2 > 0.25). The corresponding layer was conv7 for 9 subjects and conv5 for 2 subjects. The left dorsolateral prefrontal cortex has been previously associated to verbal encoding in numerous studies, suggesting that the information in conv5 and conv7 might be linked to verbal content in the original stimuli, to a certain extent. Note that we also found less than five subjects for which brain activity in medial regions of the Default Mode Network, such as the medial prefrontal cortex, or the anterior cingulate cortex, could be predicted, but this didn't seem like a consistent pattern across the majority of subjects. We depict We were able to train encoding models on individual subjects to predict brain activity using the deepest layers of SoundNet, using less than 20 minutes of fMRI data. The obtained models best predicted the activity in brain areas that are part of a language-related network. However, the current study has the following limitations. First, we extracted features from the auditory part of the stimuli, while the modeled brain activity involves many other brain functions, namely visual perception, as well as higher level cognitive functions such as memory and emotional responses. This probably explains why we obtain R 2 = 0.5 in the best case. Providing a richer stimuli representation using more general purpose feature extractors would probably enable a more complete model of brain activity. Second, we estimated brain parcellations on single subject data using only 20 minutes of MRI, which might not be enough to obtain a reliable set of ROIs. Further studies should use either more repetitions on each subject, or attempt at learning parcellations across subjects, after having spatially normalized each individual to a template. Third, we didn't find a clear relationship between spatial extent of our encoding models as a function of the SoundNet layer. This could be due to the fact that SoundNet was trained independently of the brain data, and was never optimized for encoding models. One possible avenue would be to perform fine tuning, or retrain from scratch, in order to optimize the estimation of encoding models. Finally, in our approach we ignored the temporal dynamics of both the feature vectors and the fMRI data, as well as the dependencies between ROIs implied by brain connectivity. In future studies, we will consider the use of recurrent neural networks, as well as graph representation learning, in order to tackle those issues.

represent the short summary for finding the corresponding scientific article (long).
------
Fewshot example: "Simple augmentation method overcomes robustness/accuracy trade-off observed in literature and opens questions about the effect of training distribution on out-of-distribution generalization." == "Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions. While architectural advances have led to improved accuracy, building robust models remains challenging, involving major changes in training procedure and datasets. Prior work has argued that there is an inherent trade-off between robustness and accuracy, as exemplified by standard data augmentation techniques such as Cutout, which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy. We introduce Patch Gaussian, a simple augmentation scheme that adds noise to randomly selected patches in an input image. Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNet Common Corruptions benchmarks while also maintaining accuracy on clean data. We find that this augmentation leads to reduced sensitivity to high frequency noise (similar to Gaussian) while retaining the ability to take advantage of relevant high frequency information in the image (similar to Cutout). We show it can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment. Finally, we find that the idea of restricting perturbations to patches can also be useful in the context of adversarial learning, yielding models without the loss in accuracy that is found with unconstrained adversarial training. Patch Gaussian augmentation overcomes the accuracy/robustness tradeoff observed in other augmentation strategies. Larger σ of Patch Gaussian (→) improves mean corruption error (mCE) and maintains clean accuracy, whereas larger σ of Gaussian (→) and patch size of Cutout (→) hurt accuracy or robustness. More robust and accurate models are down and to the right. Modern deep neural networks can achieve impressive performance at classifying images in curated datasets (; ;). Yet, they lack robustness to various forms of distribution shift that typically occur in real-world settings. For example, neural networks are sensitive to small translations and changes in scale , blurring and additive noise , small objects placed in images , and even different images from a distribution similar to the training set (; . For models to be useful in the real world, they need to be both accurate on a high-quality held-out set of images, which we refer to as "clean accuracy," and robust on corrupted images, which we refer to as "robustness." Most of the literature in machine learning has focused on architectural changes (; ; ; ; ; ;) to improve clean accuracy but interest has recently shifted toward robustness as well. Research in neural network robustness has tried to quantify the problem by establishing benchmarks that directly measure it and comparing the performance of humans and neural networks (b;). Others have tried to understand robustness by highlighting systemic failure modes of current methods. For instance, networks exhibit excessive invariance to visual features , texture bias (a), sensitivity to worst-case (adversarial) perturbations , and a propensity to rely on non-robust, but highly predictive features for classification . Of particular relevance, has established connections between popular notions of adversarial robustness and some measures of distribution shift considered here. Another line of work has attempted to increase model robustness performance, either by projecting out superficial statistics , via architectural improvements , pretraining schemes , or with the use of data augmentations. Data augmentation increases the size and diversity of the training set, and provides a simple way to learn invariances that are challenging to encode architecturally . Recent work in this area includes learning better transformations (; ;), inferring combinations of them , unsupervised methods , theory of data augmentation , and applications for one-shot learning . Despite these advances, individual data augmentation methods that improve robustness do so at the expense of reduced clean accuracy. Further, achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets: the current state of the art in robustness benchmarks involves creating a custom dataset with styled-transferred images before training (a), and still incurs a significant drop in clean accuracy. The ubiquity of reported robustness/accuracy trade-offs in the literature have even led to the hypothesis that these trade-offs may be inevitable . Because of this, many recent works focus on improving either one or the other (; a). In this work we propose a simple data augmentation method that overcomes this trade-off, achieving improved robustness while maintaining clean accuracy. Our contributions are as follows: • We characterize a trade-off between robustness and accuracy in standard data augmentations Cutout and Gaussian (Section 2.1). • We describe a simple data augmentation method (which we term Patch Gaussian) that allows us to interpolate between the two augmentations above (Section 3.1). Despite its simplicity, Patch Gaussian achieves a new state of the art in the Common Corruptions robustness benchmark , while maintaining clean accuracy, indicating current methods have not reached this fundamental trade-off (Section 4.1). • We demonstrate that Patch Gaussian can be combined with other regularization strategies (Section 4.2) and data augmentation policies (Section 4.3). • We perform a frequency-based analysis of models trained with Patch Gaussian and find that they can better leverage high-frequency information in lower layers, while not being too sensitive to them at later ones (Section 5.1). • We show a similar method can be used in adversarial training, suggesting under-explored questions about training distributions' effect on out-of-distribution robustness (Section 5.2). We start by considering two data augmentations: Cutout and Gaussian . The former sets a random patch of the input image to a constant (mean pixel in the dataset) in order to improve clean accuracy. The latter adds independent Gaussian noise to each pixel of the input image, which directly increases robustness to Gaussian noise. We compare the effectiveness of Gaussian and Cutout data augmentation for accuracy and robustness by measuring the performance of models trained with each on clean as well as corrupted data. Here, robustness is defined as average accuracy of the model, when tested on data corrupted by various σ (0.1, 0.2, 0.3, 0.5, 0.8, 1.0) of Gaussian noise, relative to the clean accuracy: Relative Gaussian Robustness = E σ (Accuracy on Data Corrupted by σ) − Clean Accuracy Fig. 2 highlights an apparent trade-off in using these methods. In accordance to previous work , Cutout improves accuracy on clean test data. Despite this, we find it does not lead to increased robustness. Conversely, training with higher σ of Gaussian can lead to increased robustness to Gaussian noise, but also leads to decreased accuracy on clean data. Therefore, any robustness gains are offset by poor overall performance: a model with a perfect Relative Robustness of 0, but whose clean accuracy dropped to 50% will be wrong half the time, even on clean data. The y-axis is the change in accuracy when tested on data corrupted with Gaussian noise at various σ (average corrupted accuracy minus clean accuracy). The diamond indicates augmentation hyper-parameters selected by the method in Section 3.2. At first glance, these seem to reinforce the findings of previous work , indicating that robustness comes at the cost of generalization, which would offset any benefits of improved robustness. In the following sections, we will explore whether there exists augmentation strategies that do not exhibit this limitation. Each of the two methods seen so far achieves one half of our stated goal: either improving robustness or slightly improving/maintaining clean test accuracy, but never both. To explore whether this observed trade-off is fundamental, we introduce Patch Gaussian, a technique that combines the noise robustness of Gaussian with the slightly improved clean accuracy of Cutout. Our method is intentionally simple but, as we'll see, it's powerful enough to overcome the limitations described and beats complex training schemes designed to provide robustness. Patch Gaussian works by adding a W × W Figure 3: Patch Gaussian is the addition of Gaussian noise to pixels in a square patch. It allows us to interpolate between Gaussian and Cutout, approaching Gaussian with increasing patch size and Cutout with increasing σ. patch of Gaussian noise to the image (Figure 3). As with Cutout, the center of the patch is sampled to be within the image. By varying the size of this patch and the maximum standard deviation of noise sampled σ max, we can interpolate between Gaussian (which applies additive Gaussian noise to the whole image) and an approximation of Cutout (which removes all information inside the patch). See Fig. 9 for more examples. Our goal is to learn models that achieve both good clean accuracy and improved robustness to corruptions. Prior work has optimized for one or the other but, as noted before, to meaningfully improve robustness to other distributions, a method can't incur a significant drop in clean accuracy. Therefore, when selecting hyper-parameters, we focus on identifying the models that are most robust while still achieving a minimum accuracy (Z) on the clean test data. Values of Z are selected to incur negligible decrease in clean accuracy. As such, they vary per dataset and model, and can be found in the Appendix (Table 5). If no model has clean accuracy ≥ Z, we report the model with highest clean accuracy, unless otherwise specified. We find that patch sizes around 25 on CIFAR (≤250 on ImageNet, i.e.: uniformly sampled with maximum value 250) with σ ≤ 1.0 generally perform the best. A complete list of selected hyperparameters for all augmentations can be found in Table 5. We are interested in out-of-distribution robustness, and report performance of selected models on Common Corruption . However, when selecting hyper-parameters, we use Relative Gaussian Robustness as a stand-in for "robustness. " indicates that this metric is correlated with performance on Common Corruptions, so selected models should be generally robust beyond Gaussian corruptions. By picking models based on robustness to Gaussian noise, we ensure that our selection process does not overfit to the Common Corruptions benchmark. Models trained with Patch Gaussian overcome the observed trade-off and gain robustness to Gaussian noise while maintaining clean accuracy (Fig. 1). Because Gaussian robustness is only used for hyper-parameter selection, we omit these , but refer the curious reader to Appendix Fig. 7. Instead, we report how this Gaussian robustness translates into better Common Corruption robustness, which is in line with reports of the correlation between the two . In doing so, we establish a new state of the art in the Common Corruptions benchmark (Section 4.1), despite the simplicity of our method when compared with the previous best (a). We then show that Patch Gaussian can be used in complement to other common regularization strategies (Section 4.2) and data augmentation policies (Section 4.3). In this section, we look at how our augmentations impact robustness to corruptions beyond Gaussian noise. Rather than focusing on adversarial examples that are worst-case bounded perturbations, we focus on a more general set of corruptions that models are likely to encounter in real-world settings: the Common Corruptions benchmark . This benchmark, also referred to as CIFAR-C and ImageNet-C, is composed of images transformed with 15 corruptions, at 5 severities each. Each is designed to model transformations commonly found in real-world settings, such as brightness, different weather conditions, and different kinds of noise. Table 1 shows that Patch Gaussian achieves state of the art on both of these benchmarks in terms of mean Corruption Error (mCE). A "Corruption Error" is a model's average error over 5 severities of a given corruption, normalized by the same average of a baseline model. However, ImageNet-C was released in compressed JPEG format , which alters the corruptions applied to the raw pixels. Therefore, we report on the benchmark as-released ("Original mCE") 1 as well as a version of 12 corruptions without the extra compression ("mCE"). Additionally, because Patch Gaussian is a noise-based augmentation, we wanted to verify whether its gains on this benchmark were solely due to improved performance on noise-based corruptions (Gaussian Noise, Shot Noise, and Impulse Noise). To do this, we also measure the models' average performance on all other corruptions, reported as "Original mCE (-noise)", and "mCE (-noise)". The models used to normalize Corruption Errors are the "Baselines" trained with only flip and crop data augmentation. The one exception is Original mCE ImageNet, where we use the AlexNet baseline to be directly comparable with previous work (; a). On CIFAR, we compare with an adversarially-trained model . On ImageNet, we compare with a model trained with Random Erasing , as well as a shape-biased model "SIN+IN ftIN" (a). Finally, previous work has found that augmentation diversity is a key component of robustness gains. To confirm that Patch Gaussian's gains aren't simply a of using multiple augmentations, we also report on training with Cutout and Gaussian applied in sequence ("Cutout & Gaussian" and "Gaussian & Cutout"), as well as to 50% of batches ("Gaussian or Cutout"). We observe that Patch Gaussian outperforms all models, even on corruptions like fog where Gaussian hurts performance . Scores for each corruption can be found in the Table 1: Patch Gaussian achieves state of the art in the CIFAR-C (left) and ImageNet-C (right) robustness benchmarks while maintaining clean test accuracy. " Original mCE" refers to the jpegcompressed benchmark, as used in Geirhos et al. (2018a);. "mCE" is a version of it without the extra jpeg compression. Note that Patch Gaussian improves robustness even in corruptions that aren't noise-based. * Cutout 16 is presented for direct comparison with;. For Resnet-200, we also present Gaussian at a higher σ to highlight the accuracy-robustness trade-off. Augmentation hyper-parameters were selected based on the method in Section 3.2 and can be found in Appendix. See text for details. These are surprising: achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets. Training shape-biased models (a) involves creating a custom dataset of style-transferred images, which is a computationallyexpensive operation. Even with these, the most robust model reported SIN+IN displays a significant drop in clean accuracy. Because of this, our main comparison is with SIN+IN ftIN, which is fine-tuned on ImageNet. A comparison with SIN+IN can be found in Appendix Table 8. In sum, despite its simplicity, Patch Gaussian achieves a substantial decrease in mCE relative to other models, indicating that current methods have not reached the theoretical trade-off , and that complex training schemes (a) are not needed for robustness. Since Patch Gaussian has a regularization effect on the models trained above, we compare it with other regularization methods: larger weight decay, label smoothing, and dropblock (Table 2). We find that while label smoothing improves clean accuracy, it weakens the robustness in all corruption metrics we have considered. This agrees with the theoretical prediction from , which argued that increasing the confidence of models would improve robustness, whereas label smoothing reduces the confidence of predictions. We find that increasing the weight decay from the default value used in all models does not improve clean accuracy or robustness. Here, we focus on analyzing the interaction of different regularization methods with Patch Gaussian. Previous work indicates that improvements on the clean accuracy appear after training with Dropblock for 270 epochs , but we did not find that training for 270 epochs changed our analysis. Thus, we present models trained at 90 epochs for direct comparison with other . Due to the shorter training time, Dropblock does not improve clean accuracy, yet it does make the model more robust (relative to baseline) according to all corruption metrics we consider. We find that using label smoothing in addition to Patch Gaussian has a mixed effect, it improves clean accuracy while slightly improving robustness metrics except for the Original mCE. Combining Dropblock with Patch Gaussian reduces the clean accuracy relative to the Patch Gaussian-only model, as Dropblock seems to be a strong regularizer when used for 90 epochs. However, using Dropblock and Patch Gaussian together leads to the best robustness performance. These indicate that Patch Gaussian can be used in conjunction with existing regularization strategies. Knowing that Patch Gaussian can be combined with other regularizers, it's natural to ask whether it can also be combined with other data augmentation policies. Previous work has found that varied augmentation policies have a large positive impact on model robustness . In this section, we verify that Patch Gaussian can be added to these policies for further gains. Because AutoAugment leads to state of the art accuracies, we are interested in seeing how far it can be combined with Patch Gaussian to improve . Therefore, and unlike previous experiments, models are trained for 180 epochs to yield best possible. In an attempt to understand Patch Gaussian's performance, we perform a frequency-based analysis of models trained with various augmentations using the method introduced in. First, we perturb each image in the dataset with noise sampled at each orientation and frequency in Fourier space. Then, we measure changes in the network activations and test error when evaluated with these Fourier-noise-corrupted images: we measure the change in 2 norm of the tensor directly after the first convolution, as well as the absolute test error. This procedure yields a heatmap, which indicates model sensitivity to different frequency and orientation perturbations in the Fourier domain. Each image in Fig 4 shows first layer (or test error) sensitivity as a function of frequency and orientation of the sampled noise, with the middle of the image containing the lowest frequencies, and the edges of the image containing highest frequencies. For CIFAR-10 models, we present this analysis for the entire Fourier domain, with noise sampled with norm 4. For ImageNet, we focus our analysis on lower frequencies that are more visually salient add noise with norm 15.7. Note that for Cutout and Gaussian, we chose larger patch sizes and σs than those selected with the method in Section 3.2 in order to highlight the effect of these augmentations on sensitivity. Heatmaps of other models can be found in the Appendix (Figure 11). We confirm findings by that Gaussian encourages the model to learn a low-pass filter of the inputs. Models trained with this augmentation, then, have low test error sensitivity at high frequencies, which could help robustness. However, valuable high-frequency information is being thrown out at low layers, which could explain the lower test accuracy. We further find that Cutout encourages the use of high-frequency information, which could help explain its improved generalization performance. Yet, it does not encourage lower test error sensitivity, which explains why it doesn't improve robustness either. Patch Gaussian, on the other hand, seems to allow high-frequency information through at lower layers, but still encourages relatively lower test error sensitivity at high frequencies. Indeed, when we measure accuracy on images filtered with a high-pass filter, we see that Patch Gaussian models can maintain accuracy in a similar way to the baseline and to Cutout, where Gaussian fails to. See Figure 4 for full . Cutout encourages the use of high frequencies in earlier layers, but its test error remains too sensitive to them. Gaussian learns low-pass filtering of features, which increases robustness at later layers, but makes lower layers too invariant to highfrequency information (thus hurting accuracy). Patch Gaussian allows high frequencies to be used in lower layers, and its test error remains relatively robust to them. This can also be seen by the presence of high-frequency kernels in the first layer filters of the models (or lack thereof, in the case of Gaussian). (right) Indeed, Patch Gaussian models match the performance of Cutout and Baseline when presented with only the high frequency information of images, whereas Gaussian fails to effectively utilize this information (see Appendix Fig. 12 for experiment details). This pattern of reduced sensitivity of predictions to high frequencies in the input occurs across all augmentation magnitudes, but here we use larger patch sizes and σ of noise to highlight the differences in models indicated by *. See text for details. Understanding the impact of data distributions and noise on representations has been well-studied in neuroscience (; ;). The data augmentations that we propose here alter the distribution of inputs that the network sees, and thus are expected to alter the kinds of representations that are learned. Prior work on efficient coding and autoencoders has shown how filter properties change with noise in the unsupervised setting, ing in lower-frequency filters with Gaussian, as we observe in Fig. 4. Consistent with prior work on natural image statistics , we find that networks are least sensitive to low frequency noise where spectral density is largest. Performance drops at higher frequencies when the amount of noise we add grows relative to typical spectral density observed at these frequencies. In future work, we hope to better understand the relationship between naturally occurring properties of images and sensitivity, and investigate whether training with more naturalistic noise can yield similar gains in corruption robustness. Our indicate that patching a transformation can prevent overfitting to that particular transformation and maintain clean accuracy. To further confirm this, we train a model with adversarial training applied only to a patch of the training input. Adversarial training is a method of achieving robustness to worst-case perturbations. Models trained in this setting notoriously exhibit decreased clean accuracy, so it is a good candidate to verify whether our robustness gains come from patching. We train our models with PGD, in a setting equivalent to. For Patch PGD, the adversarial perturbation is calculated on the whole image for all steps, and patched after the fact. We also tried calculating PGD on a patch only and found similar . We select hyper-parameters based on PGD performance on validation set, while maintaining accuracy above 90%. However, in this section we are not interested in improving adversarial robustness performance, but on seeing its effect on robustness to Common Corruptions, to evaluate out-of-distribution (o.o.d.) robustness. We leave an analysis of the effect of patching on adversarial robustness to future work. Indeed, Table 4 shows that training with Patch PGD obtains similar PGD accuracy to training with PGD, but maintains most of the clean accuracy of the baseline model. Surprisingly, Patch PGD also improves robustness to unseen Common Corruptions, when compared to the baseline without adversarial training, indicating that patching is a generally powerful tool. This also suggests there are unexplored questions regarding the training distribution and how that translates into i.i.d and o.o.d generalization. We hope to explore these in future work. In this work, we introduced a simple data augmentation operation, Patch Gaussian, which improves robustness to common corruptions without incurring a drop in clean accuracy. For models that are large relative to the dataset size (like ResNet-200 on ImageNet and all models on CIFAR-10), Patch Gaussian improves clean accuracy and robustness concurrently. We showed that Patch Gaussian achieves this by interpolating between two standard data augmentation operations Cutout and Gaussian. Finally, we analyzed the sensitivity to noise in different frequencies of models trained with Cutout and Gaussian, and showed that Patch Gaussian combines their strengths without inheriting their weaknesses. Our method is much simpler than previous state of the art, and can be used in conjunction with other regularization and data augmentation strategies, indicating it is generally useful. We end by showing that applying perturbations in patches can be a powerful method to vary training distributions in the adversarial setting. Our indicate current methods have not reached a fundamental robustness/accuracy trade-off, and that future work is needed to understand the effect of training distributions in o.o.d. robustness. Fig. 5 shows the accuracy/robustness trade-off of models trained with various hyper-parameters of Cutout and Gaussian. Fig. 6 shows clean accuracy change of models trained with various hyper-parameters of Patch Gaussian. Fig. 7 shows how Patch Gaussian can overcome the observed trade-off and gain Gaussian robustness in various models and datasets. We run our experiments on CIFAR-10 and ImageNet datasets. On CIFAR-10, we use the Wide-ResNet-28-10 model , as well as the Shake-shake-112 model , trained for 200 epochs and 600 epochs respectively. The Wide-ResNet model uses a initial learning rate of 0.1 with a cosine decay schedule. Weight decay is set to be 5 × 10 −4 and batch size is 128. We train all models, including the baseline, with standard data augmentation of horizontal flips and pad-and-crop. Our code uses the same hyper parameters as On ImageNet, we use the ResNet-50 and Resnet-200 models , trained for 90 epochs. We use a weight decay rate of 1 × 10 −4, global batch size of 512 and learning rate of 0.2. The learning rate is decayed by 10 at epochs 30, 60, and 80. We use standard data augmentation of horizontal flips and crops. All CIFAR-10 and ImageNet experiments use the listed hyper-parameters above, unless specified otherwise. To apply Gaussian, we uniformly sample a standard deviation σ from 0 up to some maximum value σ max, and add i.i.d. noise sampled from N (0, σ 2) to each pixel. To apply Cutout, we use a fixed patch size W, and randomly set a square region with size W × W to the constant mean of each RGB channel in the dataset. As in , the patch location is randomly sampled and can lie outside of the 32 × 32 CIFAR-10 (or 224 × 224 ImageNet) image but its center is constrained to lie within it. Patch sizes and σ max are selected based on the method in Section 3.2. Table 5: Augmentation hyper-parameters selected with the method in Section 3.2 for each model/dataset. *Indicates manually-chosen stronger hyper-parameters, used to highlight the effect of the augmentation on the models. "≤" indicates that the value is uniformly sampled up to the given maximum value. Since Patch Gaussian can be combined with both regularization strategies as well as data augmentation policies, we want to see if it is generally useful beyond classification tasks. We train a RetinaNet detector with ResNet-50 backbone on the COCO dataset . Images for both baseline and Patch Gaussian models are horizontally flipped half of the time, after being resized to 640 × 640. We train both models for 150 epochs using a learning rate of 0.08 and a weight decay of 1 × 10 −4. The focal loss parameters are set to be α = 0.25 and γ = 1.5. Despite being designed for classification, Patch Gaussian improves detection performance according to all metrics when tested on the clean COCO validation set (Table 9). On the primary COCO metric mean average precision (mAP), the model trained with Patch Gaussian achieves a 1% higher accuracy over the baseline, whereas the model trained with Gaussian suffers a 2.9% loss. Next, we evaluate these models on the validation set corrupted by i.i.d. Gaussian noise, with σ = 0.25. We find that model trained with Gaussian and Patch Gaussian achieve the highest mAP of 26.1% on the corrupted data, whereas the baseline achieves 11.6%. It is interesting to note that Patch Gaussian model achieves a better on the harder metrics of small object detection and stricter intersection over union (IoU) thresholds, whereas the Gaussian model achieves a better on the easier tasks of large object detection and less strict IOU threshold metric. Overall, as was observed for the classification tasks, training object detection models with Patch Gaussian leads to significantly more robust models without sacrificing clean accuracy. FOURIER ANALYSIS Fig. 10 shows a fourier analysis of selected models reported. Fig. 11 shows complete filters for ResNet-50 models. Fig. 12 shows high-pass filters used in high-pass experiment in Fig. 4. Figure 4 for details. We again note the presence of filters of high fourier frequency in models trained with Cutout* and Patch Gaussian. We also note that Gaussian* exhibits high variance filters. We posit these have not been trained and have little importance, given the low sensitivity of this model to high frequencies. Future work will investigate the importance of filters on sensitivity. reports that PGD training helps with corruption robustness. However, they fail to report mCE values for their models. We find that, indeed, PGD helps with some corruptions, and when all corruption severities' errors are averaged, it mostly maintains performance (23.8% error, compared to baseline error of 23.51%). However, as table 4 shows, when we properly calculate mCE by normalizing with a baseline model, PGD displays much worse robustness, while Patch PGD improves performance. Figure 13 shows the frequency-based analysis for models with different hyperparameters of Patch Gaussian. First, for hyper-parameters W =16, σ=1.0 (center), the reader will note that these are very similar to the frequency sensitivity reported in Figure 4. The main difference being that the smaller patch size (16 vs 25 in Figure 4) makes the model slightly more sensitive to high frequencies. This makes sense since smaller patch size moves the model further away from a Gaussian-trained one. When we make the scale smaller (W =16, σ=0.3, left), less information is corrupted in the patch, which moves the model farther from the one trained with Cutout (and therefore closer to a Gaussiantrained one). This can be seen in the increased invariance to high frequencies at the first layer, which is reflected in invariance at test error as well. If we, instead, make the scale larger(W =16, σ=2.0, right), we move the model closer to the one trained with Cutout. Notice the higher intensity red in the first layer plot, indicating higher sensitivity to high-frequency features. We also see this sensitivity reflected in the test error, which matches the behavior for Cutout-trained models. Figure 13: Frequency-based analysis for models with different hyper-parameters of Patch Gaussian."

Feature vectors from SoundNet can predict brain activity of subjects watching a movie in auditory and language related brain regions.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.