Question:
represent the article for finding a single-sentence tldr

------

Examples:

Given Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent generative models of video have had some success, current progress is hampered by the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples. To this extent we propose Fréchet Video Distance (FVD), a new metric for generative models of video based on FID. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos. Recent advances in deep generative models have lead to remarkable success in synthesizing highquality images . A natural next challenge is to consider video generation. This is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. Generative models of video will enable many applications, including missing-frame prediction , improved instance segmentation , or complex (relational) reasoning tasks by conducting inference .While great progress has been made in recent years, video generation models are still in their infancy, and generally unable to synthesize more than a few seconds of video . Learning a good dynamics model remains a major challenge in generating real world videos. However, in order to qualitatively measure progress in synthesizing videos, we require metrics that consider visual quality, temporal coherence, and diversity of generated samples. We contribute Fréchet Video Distance (FVD), a new metric for generative models of video. FVD builds on the principles underlying Fréchet Inception Distance (FID;), which has been successfully applied to images. We introduce a feature representation that captures the temporal coherence of the content of a video, in addition to the quality of each frame. Unlike popular * Both authors contributed equally to this work while interning at Google Brain. Figure 1: Generated videos by various models ranked according to FVD (lower is better). metrics such as Peak Signal to Noise Ratio (PSNR) or the Structural Similarity (SSIM;) index, FVD considers a distribution over videos, thereby avoiding the drawbacks of framelevel metrics . We contribute extensive experiments to evaluate FVD, including a large-scale human study which confirms that FVD coincides well with qualitative human judgment of generated videos. A generative model of videos must capture the underlying data distribution with which the observed data was generated. The distance between the real world data distribution P R and the distribution defined by the generative model P G is a natural evaluation metric. An analytic expression of either distribution is usually unavailable, which prohibits straightforward application of many common distance functions. For example, the popular Fréchet Distance (or 2-Wasserstein distance) between P R and P G is defined by: DISPLAYFORM0 2 where the minimization is over all random variables X and Y with distributions P R and P G respectively. This expression is difficult to solve for the general case, but has a closed form solution when when P R and P G are multivariate Gaussians : DISPLAYFORM1 where µ R and µ G are the means and Σ R and Σ G are the co-variance matrices of P R and P G. A multivariate Gaussian is seldom an accurate representation of the underlying data distribution, but when using a suitable feature space, this is a reasonable approximation. For distributions over real world images, used a learned feature embedding to calculate the distance between P R and P G as follows: First samples from P R and P G are fed through an Inception network network that was trained on ImageNet and their feature representations (activations) in one of the hidden layers are recorded. Then the Fréchet Inception Distance (FID;) is computed via Eq. 1 using the means and covariances obtained from the recorded responses of the real, and generated samples. The feature representation learned by the pre-trained neural network greatly affects the quality of the metric. When training on ImageNet, the learned features expose information useful in reasoning about objects in images, whereas other information content may be suppressed. Likewise, different layers of the network encode features at different abstraction levels. In order to obtain a suitable feature representation for videos we require a pre-trained network that considers the temporal coherence of the visual content across a sequence of frames, in addition to its visual presentation at any given point in time. In this work we investigate several variations of a pre-trained Inflated 3D Convnet (I3D;), and name the ing metric the Fréchet Video Distance (FVD) 1 2.The I3D network generalizes the Inception architecture to sequential data, and is trained to perform action-recognition on the Kinetics data set consisting of human-centered YouTube videos. Action recognition requires visual context and temporal evolution to be considered simultaneously, and I3D has been shown to excel at this task. We explore two different feature representations (logits, avg. pool) learned by I3D networks pre-trained on Kinetics-400, and Kinetics-600.A potential downside in using Eq. 1 is the potentially large error in estimating Gaussian distributions over the learned feature space. Bińkowski et al. proposed to use the Maximum Mean Discrepancy (MMD;) as an alternative in the case of images, and we will explore this variation in the context of videos as well. MMD is a kernel-based approach, which provides a means to calculate the distance between two empirical distributions without assuming a particular form. Bińkowski et al. proposed to use a polynomial kernel k(a, b):= a T b + 1 3, which we will apply to the learned features of the I3D network to obtain the Kernel Video Distance (KVD).1 A similar adaptation of FID was used by to evaluate their vid2vid model. Here we introduce FVD as a general metric for videos and an focus on an extensive empirical study.2 Code to compute FVD is available at https://git.io/fpuEH. In the following we present of a noisy study, and human study of FVD. Additional experiments that analyze its sensitivity to sample size, and resolution are available in Appendix B & D. We test how sensitive FVD is to basic distortions by adding noise to the real videos. We consider static noise added to individual frames, and temporal noise, which distorts the entire sequence of frames. We applied these distortions (details in Appendix A) at up to six different intensities, and computed the FVD and KVD between videos from the BAIR , Kinetics-400 and HMDB51 data sets and their noisy counterparts. As potential embeddings, we considered the top-most pooling layer, and the logits layer of the I3D model pre-trained on the Kinetics-400 data set, as well as the same layers in a variant of the I3D model pre-trained on the extended Kinetics-600 data set. As a baseline, we compared to a naive extension of FID for videos in which the Inception network (pre-trained on ImageNet) is evaluated for each frame individually, and the ing embeddings (or their pair-wise differences) are averaged to obtain a single embedding for each video. This "FID" score is then computed according to Eq. 1.All variants were able to detect the various injected distortions to some degree, with the pre-trained Inception network generally being inferior at detecting temporal distortions as was expected. FIG0 shows that the logits layer of the I3D model pre-trained on Kinetics-400 has the best average rank correlation with the sequence of noise intensities. An overview of its scores on the noise experiments can be seen in FIG2. Table 1: Agreement of metrics with human judgment when considering models with a fixed value for a given metric (eq.), or with spread values over a wide range (spr.). One important criterion for the performance of generative models is the visual fidelity of the samples as judged by human observers , as a metric for generative models must ultimately correlate well with human judgment. Thus we trained several conditional video generation models, and asked human raters to compare the quality of the generated videos in different scenarios. Results The of the human evaluation studies can be seen in Table 1, and additional for KVD and Avg. FID in Appendix C. We find that FVD is the superior choice compared to all other metrics tested. The obtained for eq. FVD and spr. FVD are of key importance as they determine how users will experience FVD in practice. From the spr. FVD column we can conclude that no other metric can improve upon the ranking induced by FVD, and the eq. FVD column tells us that no other metric can reliably distinguish between good models that are equal in terms of FVD.On the other hand, FVD is able to distinguish models when other metrics can not (eq. SSIM, eq. PSNR), agreeing well with human judgment (74.9 %, and 81.0 % agreement respectively). Likewise FVD consistently improves on the ranking induced by other metrics (spr. SSIM, spr. PSNR), even though these scenarios are clearly favorable for the metric under consideration. We introduced the Fréchet Video Distance (FVD), a new evaluation metric for generative models of video, and an important step towards better evaluation of models for video generation. Our experiments confirm that FVD is accurate in evaluating videos that were modified to include static noise, and temporal noise. More importantly, a large scale human study among generated videos from several recent generative models reveals that FVD consistently outperforms SSIM and PSNR in agreeing with human judgment. To test whether FVD can detect static noise we added one of the following distortions to each frame in a sequence of video frames: a black rectangle drawn at a random location in the frame, Gaussian blur, which applies a Gaussian smoothing kernel to the frame, Gaussian noise, which interpolates between the observed frame and standard Gaussian noise, and Salt & Pepper noise, which sets each pixel in the frame to either black or white with a fixed probability. Temporal noise was injected by locally swapping a number of randomly chosen frames with its neighbor in the sequence globally swapping a number of randomly chosen pairs of frames selected across the whole sequence, interleaving the sequence of frames corresponding to multiple different videos to obtain new videos, and by switching from one video to another video after a number of frames to obtain new videos. We applied these distortions at up to six different intensities that are unique to each type, e.g. related to the size of the black rectangle, the number of swaps to perform, or the number of videos to interleave. We conduct the noise study on HMDB (Correlation of Noise Intensity and Metric FVD kinetics400 logits FVD kinetics600 logits FVD kinetics400 pool FVD kinetics600 pool KVD kinetics400 logits KVD kinetics600 logits KVD kinetics400 pool KVD kinetics600 pool FID average over frames FID average over frame-diffs We consider the accuracy with which FVD is able to calculate the true underlying distance between a distribution of generated videos and a target distribution. To calculate the FVD according to Eq. 1 we need to estimate µ R, µ G and Σ R, Σ G from the available samples. The larger the sample size, the better these estimates will be, and the better FVD will reflect the true underlying distance between the distributions. For an accurate generative model these distributions will typically be fairly close, and the noise from the estimation process will primarily affect our . This effect has been wellstudied for FID (; Bińkowski et al., 2018), and is depicted for FVD in FIG3. It can be seen that even when the underlying distributions are identical, FVD will typically be larger than zero because our estimates of the parameters µ R, µ G, Σ R and Σ G are noisy. It can also be seen that for a fixed number of samples the standard errors (measured over 50 tries) are small, and an accurate comparison can be made. Hence, it is critical that in comparing FVD values across models, one uses the same sample size 3.2 5 2 6 2 7 2 8 2 9 2 10 2 11 2 12Number of ) on the BAIR data set. Using a wide range of possible hyperparameter settings, and by including model parameters at various stages of training we obtain over 3 000 different models. Generated videos are obtained by combining 2 frames of context with the proceeding 14 output frames. Following prior work we obtain the PSNR and SSIM scores by generating 100 videos for each input context (conditioning frames) and returning the best frame-averaged value among these videos. We consider 256 video sequences (unseen by the model) to estimate the target distribution when computing FVD.We conduct several human studies based on different subsets of the trained models. In particular, we select models according to two different scenarios:One Metric Equal We consider models that are indistinguishable according to a single metric, and evaluate to what degree human raters and other competing metrics are able to distinguish these models in terms of the quality of the generated videos. We choose 10 models having roughly equal values for a given metric that are close to the best quartile of the overall distribution of that metric, i.e., the models were chosen such that they are worse than 25 % of the remaining models and better than 75 % of the remaining models as determined by the metric under consideration. We were able to choose models whose values where identical up to the first 4-5 significant digits for each metric. One Metric Spread In a second setting we consider to what degree models having very different scores for a given metric, coincide with the subjective quality of their generated videos as judged by humans. We choose 10 models which were equidistant between the 10 % and 90 % percentile of the overall distribution of that metric. In this case, there should be clear differences in terms of the quality of the generated videos among the models under consideration (provided that the metric is accurate), suggesting high agreement with human judgment for the metric under consideration in comparison to competing metrics. For the human evaluation, we used 3 generated videos from each selected model. Human raters would be shown a video from two models, and then asked to identify which of the two looked better, or alternatively report that their quality was indistinguishable. Each pair of compared videos was shown to up to 3 independent raters, where the third rater was only asked if the first two raters disagreed. The raters were given no prior indication about which video was thought to be better. We calculated the correspondence between these human ratings and the ratings determined by the various metrics under consideration. Table 3: Agreement of metrics with human judgment when considering models with a fixed value for a given metric (eq.), or with spread values over a wide range (spr.). FVD is superior at judging generated videos based on subjective quality. Table 3 contains the of the human evaluation for the KVD and Avg. FID metrics in addition to the for all other metrics from the main paper. Avg. FID is computed by averaging the Inception embedding for each frame, before computing the Fréchet distance.• Avg. FID We find that Avg. FID performs markedly worse compared to FVD in most scenarios, except on spr. SSIM, where it achieves slightly better performance. It suggests that it is preferential to judge the wide range of videos (sampled from each decile as determined by SSIM) based primarily on frame-level quality. On the other hand, when considering videos of similar quality in eq. SSIM, we find that judging based on temporal coherence (in addition to frame-level quality) is beneficial and Avg. FID performs worse.• KVD We find that KVD is highly correlated with FVD (spearman: 0.9), although in most scenarios it performs slightly worse than FVD in terms of agreement with human judgment. In general we may conclude from Table 3 that FVD is the superior choice compared to all other metrics tested. The obtained for eq. FVD and spr. FVD are of key importance as they determine how users will experience FVD in practice. From the spr. FVD column we can conclude that no other metric can improve upon the ranking induced by FVD, and the eq. FVD column tells us that no other metric can reliably distinguish between good models that are equal in terms of FVD. Table 3 also reports the agreement among raters. These are computed as the fraction of the comparisons in which the first two raters agreed for a given video pair, averaged across all comparisons to obtain the final percentage. It can be seen that in most cases the raters are confident in comparing generated videos. While the in Appendix B demonstrates that FVD are highly reproducible for a fixed sample size, it does not consider to what degree small differences in FVD can be considered meaningful. To answer this question, human raters were asked to compare videos generated by a randomly chosen model having an FVD of 200 / 400 (base200 / base400) and generated videos by models that were 10, 20, 50, 100, 200, 300, 400, and 500 FVD points worse. In each case we selected 5 models from the models available at these FVD scores and generated 3 videos for each model, ing in a total of 1 800 comparisons. For a given video comparison, raters were asked to decide which of the two videos looked better, or if they were of similar quality. For each of these pairs, we asked up to 3 human raters for their opinion. In Figure 5 it can be seen that when the difference in FVD is smaller than 50, the agreement with human raters is close to random (but never worse), and increases rapidly once two models are more than 50 FVD points apart. Hence, it can be concluded that differences of 50 FVD or more typically correspond to differences in the quality of the generated videos that can be perceived by humans. Figure 5: Fraction of human raters that agree with FVD on which of two models is better, as a function of the difference in FVD between the models. Error bars are standard errors, and raters deciding that video pairs are of similar quality are counted as not agreeing with FVD. it matches with We propose FVD: a new metric for generative models of video based on FID. A large-scale human study confirms that FVD correlates well with qualitative human judgment of generated videos. but not with This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.

Question: Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper-parameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyper-parameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that picking the right value for momentum is critical for fine-tuning performance and connect it with previous theoretical findings. Optimal hyper-parameters for fine-tuning in particular the effective learning rate are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyper-parameters for training from scratch. Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine- tuning and encourages deep learning practitioners to rethink the hyper-parameters for fine-tuning. Many real-world applications often have limited number of training instances, which makes directly training deep neural networks hard and prone to overfitting. Transfer learning with the knowledge of models learned on a similar task can help to avoid overfitting. Fine-tuning is a simple and effective approach of transfer learning and has become popular for solving new tasks in which pre-trained models are fine-tuned with the target dataset. Specifically, fine-tuning on pre-trained ImageNet classification models (; b) has achieved impressive for tasks such as object detection and segmentation and is becoming the de-facto standard of solving computer vision problems. It is believed that the weights learned on the source dataset with a large number of instances provide better initialization for the target task than random initialization. Even when there is enough training data, fine-tuning is still preferred as it often reduces training time significantly . The common practice of fine-tuning is to adopt the default hyperparameters for training large models while using smaller initial learning rate and shorter learning rate schedule. It is believed that adhering to the original hyperparameters for fine-tuning with small learning rate prevents destroying the originally learned knowledge or features. For instance, many studies conduct fine-tuning of ResNets (b) with these default hyperparameters: learning rate 0.01, momentum 0.9 and weight decay 0.0001. However, the default setting is not necessarily optimal for fine-tuning on other tasks. While few studies have performed extensive hyperparameter search for learning rate and weight decay , the momentum coefficient is rarely changed. Though the effectiveness of the hyperparameters has been studied extensively for training a model from scratch, how to set the hyperparameters for fine-tuning is not yet fully understood. In addition to using ad-hoc hyperparameters, commonly held beliefs for fine-tuning also include: • Fine-tuning pre-trained networks outperforms training from scratch; recent work has already revisited this. • Fine-tuning from similar domains and tasks works better (; ; ;). • Explicit regularization with initial models matters for transfer learning performance (; 2019). Are these practices or beliefs always valid? From an optimization perspective, the difference between fine-tuning and training from scratch is all about the initialization. However, the loss landscape of the pre-trained model and the fine-tuned solution could be much different, so as their optimization strategies and hyperparameters. Would the hyperparameters for training from scratch still be useful for fine-tuning? In addition, most of the hyperparameters (e.g., batch size, momentum, weight decay) are frozen; will the differ when some of them are changed? With these questions in mind, we re-examined the common practices for fine-tuning. We conducted extensive hyperparameter search for fine-tuning on various transfer learning benchmarks with different source models. The goal of our work is not to obtain state-of-the-art performance on each fine-tuning task, but to understand the effectiveness of each hyperparameter for fine-tuning, avoiding unnecessary computations. We explain why certain hyperparameters work so well on certain datasets while fail on others, which can guide future hyperparameter search for fine-tuning. Our main findings are as follows: • Optimal hyperparameters for fine-tuning are not only dataset dependent, but also depend on the similarity between the source and target domains, which is different from training from scratch. Therefore, the common practice of using optimization schedules derived from ImageNet training cannot guarantee good performance. It explains why some tasks are not achieving satisfactory after fine-tuning because of inappropriate hyperparameter selection. Specifically, as opposed to the common practice of rarely tuning the momentum value beyond 0.9, we verified that zero momentum could work better for fine-tuning on tasks that are similar with the source domain, while nonzero momentum works better for target domains that are different from the source domain. • Hyperparameters are coupled together and it is the effective learning rate-which encapsulates the learning rate, momentum and batch size-that matters for fine-tuning performance. While effective learning rate has been studied for training from scratch, to the best of our knowledge, no previous work investigates effective learning rate for fine-tuning and is less used in practice. Our observation of momentum can be explained as small momentum actually decreases the effective learning rate, which is more suitable for fine-tuning on similar tasks. We show that the optimal effective learning rate actually depends on the similarity between the source and target domains. • We find regularization methods that were designed to keep models close to the initial model does not apply for "dissimilar" datasets, especially for nets with Batch Normalization. Simple weight decay can in as good performance as the reference based regularization methods for fine-tuning with better search space. In transfer learning for image classification, the last layer of a pre-trained network is usually replaced with a randomly initialized fully connected layer with the same size as the number of classes in the target task . It has been shown that fine-tuning the whole network usually in better performance than using the network as a static feature extractor (; ; ;). select images that have similar local features from source domain to jointly fine-tune pre-trained networks. estimate domain similarity with ImageNet and demonstrate that transfer learning benefits from pre-training on a similar source domain. Besides image classification, many object detection frameworks also rely on fine-tuning to improve over training from scratch . Many researchers re-examined whether fine-tuning is a necessity for obtaining good performance. find that when domains are mismatched, the effectiveness of transfer learning is negative, even when domains are intuitively similar. examine the fine-tuning performance of various ImageNet models and find a strong correlation between ImageNet top-1 accuracy and the transfer accuracy. They also find that pre-training on ImageNet provides minimal benefits for some fine-grained object classification dataset. questioned whether ImageNet pre-training is necessary for training object detectors. They find the solution of training from scratch is no worse than the fine-tuning counterpart as long as the target dataset is large enough. find that transfer learning has negligible performance boost on medical imaging applications, but speed up the convergence significantly. There is much literature on the hyperparameter selection for training neural networks from scratch, mostly on batch size, learning rate and weight decay (; ;). There are few works on the selection of momentum . proposed an automatic tuner for momentum and learning rate in SGD and empirically show that it converges faster than Adam . There are also studies on the correlations of the hyperparameters, such as linear scaling rule between batch size and learning (; ;). However, most of these advances on hyperparameter tuning are designed for training from scratch, but not examined on fine-tuning tasks for computer vision problems. Most work on fine-tuning just choose fixed hyperparameters for all fine-tuning experiments or use dataset dependent learning rates in their experiments . Due to the huge computational cost for hyperparameter search, only a few works performed large-scale grid search of learning rate and weight decay for obtaining the best performance. In this section, we first introduce the notations and experimental settings, and then present our observations on momentum, effective learning rate and regularization. The fine-tuning process is not different from learning from scratch except for the weights initialization. The goal of the process is still to minimize the loss function L =, where is the loss function, N is the number of samples, x i is the input data, y i is its label, f is the neural network and θ is the model parameters. Momentum is widely used for accelerating and smoothing the convergence of SGD by accumulating a velocity vector in the direction of persistent loss reduction . The commonly used Nesterov momentum SGD iteratively updates the model in the following form: where θ t indicates the model parameter at iteration t. The hyperparameters include the learning rate η t, batch size n, momentum coefficient m ∈, and the weight decay λ. We evaluate fine-tuning on seven widely used image classification datasets, which covers tasks for fine-grained object recognition, scene recognition and general object recognition. Detailed statistics of each dataset can be seen in Table 1. We use ImageNet , Place365 and iNaturalist as source domains for pre-trained models. We resize the input images such that the aspect ratio is preserved and the shorter side is 256 pixels. The images are normalized with mean and std values calculated over ImageNet. For data augmentation, we adopt the common practices used for training ImageNet models : random mirror, random scaled cropping with scale and aspect variations, and color jittering. The augmented images are resized to 224×224. Note that state-of-the-art could achieve even better performance by using higher resolution images or better data augmentation . We mainly use ResNet-101-V2 (a) as our base network, which is pre-trained on ImageNet . Similar observations are also observed on DenseNets and MobileNet (see Appendix B). The hyperparameters to be tuned (and ranges) are: learning rate (0.1, 0.05, 0.01, 0.005, 0.001, 0.0001), momentum (0.9, 0.99, 0.95, 0.9, 0.8, 0.0) and weight decay (0.0, 0.0001, 0.0005, 0.001). We set the default hyperparameter to be batch size 256 1, learning rate 0.01, momentum 0.9 and weight decay 0.0001. To avoid insufficient training and observe the complete convergence behavior, we use 300 epochs for fine-tuning and 600 epochs for scratch-training, which is long enough for the training curves to converge. The learning rate is decayed by a factor of 0.1 at epoch 150 and 250. We report the Top-1 validation error at the end of fine-tuning. The total computation time for the experiments is more than 10K GPU hours. Momentum 0.9 is the most widely adopted value for training from scratch (; ; b), and is also widely adopted in fine-tuning . To the best of our knowledge, it is rarely changed, regardless of the network architectures or target tasks. To check the influence of momentum on fine-tuning, we first search the best momentum values for fine-tuning on the Birds dataset with different weight decay and learning rate. Figure 1(a) shows the performance of fine-tuning with and without weight decays. Surprisingly, momentum zero actually outperforms the nonzero momentum. We also noticed that the optimal learning rate increases when the momentum is disabled (Figure 1(b) and Appendix A). To verify this observation, we further compare momentum 0.9 and 0.0 on other datasets. Table 2 shows the performance of 8 hyperparameter settings on seven datasets. We find a clear pattern that disabling momentum works better for Dogs, Caltech, Indoor datasets, while momentum 0.9 works better for Cars, Aircrafts and Flowers. Interestingly, datasets such as Dogs, Caltech, Indoor and Birds are known to have high overlap with ImageNet dataset 2, while Cars/Aircrafts are identified to be difficult to benefit from fine-tuning from pre-trained ImageNet models . According to , in which the Earth Mover's Distance (EMD) is used to calculate the distance between a dataset with ImageNet, the similarity to Birds and Dogs are 0.562 and 0.620, while the similarity to Cars, Aircrafts and Flowers are 0.560 and 0.555, 0.525 3. The relative order of similarity to ImageNet is Dogs, Birds, Cars, Aircrafts and Flowers which aligns well with the transition of optimal momentum value from 0.0 to 0.9. To verify this dependency on domain similarity, we fine-tune from pre-trained models of different source domains. It is reported that Place365 and iNaturalist are better source domains than ImageNet for fine-tuning on Indoor and Birds dataset . We can expect that fine-tuning from iNaturalist works well for Birds with m = 0 and similarly, Places365 for Indoor. Indeed, as shown in Table 3, disabling momentum improves the performance when the source and target domains are similar, such as Places for Indoor and iNaturalist for Birds. Large momentum works better for fine-tuning on different domains but not for tasks that are close to source domains Our explanation for the above observations is that because the Dogs dataset is very close to ImageNet, the pre-trained ImageNet model is expected to be close to the fine-tuned solution on the Dogs dataset. In this case, momentum may not help much as the gradient direction around the minimum could be much random and accumulating the momentum direction could be meaningless. Whereas, for faraway target domains (e.g., Cars and Aircrafts) where the pre-trained ImageNet model could be much different with the fine-tuned solution, the fine-tuning process is more similar with training from scratch, where large momentum stabilizes the decent directions towards the minimum. An illustration of the difference can be found in Figure 2. Connections to early observations on decreasing momentum Early work actually pointed out that reducing momentum during the final stage of training allows finer convergence while aggressive momentum would prevent this. They recommended reducing momentum from 0.99 to 0.9 in the last 1000 parameter updates but not disabling it completely. Recent work showed that a large momentum helps escape saddle points but can hurt the final convergence within the neighborhood of the optima, implying that momentum should be reduced at the end of training. find that a larger momentum introduces higher variance of noise and encourages more exploration at the beginning of optimization, and encourages more aggressive exploitation at the end of training. They suggest that at the final stage of the step size annealing, momentum SGD should use a much smaller step size than that of vanilla SGD. When applied to fine-tuning, we can interpret that if the pre-trained model lies in the neighborhood of the optimal solution on the target dataset, the momentum should be small. Our work identifies the empirical evidence of disabling momentum helps final convergence, and fine-tuning on close domains seems to be a perfect case. for SGD with momentum is follows: which was shown to be more closely related with training dynamics and final performance rather than η . The effective learning rate with m = 0.9 is 10× higher than the one with m = 0.0 if other hyperparameters are fixed, which is probably why we see an increase in optimal learning rate when momentum is disabled in Figure 1 (b) and Appendix A. Because learning rate and momentum are coupled, looking at the performance with only one hyperparameter varied can give a misleading understanding of the effect of hyperparameters. Therefore, we report the best with and without momentum. which does not affect the maximum accuracy obtainable with and without momentum, as long as the hyperparameters explored are sufficiently close to the optimal parameters. We review previous experiments that demonstrated the importance of momentum tuning when the effective learning rate η = η/(1−m) is held fixed instead of the learning rate η. Figure 3 shows that when η is constant, momentum 0.0 and 0.9 are actually equivalent. In addition, the best performance obtained by momentum 0.9 and momentum 0 is equivalent when other hyperparameters are allowed to change. However, different effective learning rates in different performance, which indicates that it is effective learning rate that matters for the best performance. It explains why the common practice of changing only learning rate generally works, though changing momentum may in the same effect. They both change the effective learning rate. Optimal effective learning rate and weight decay depend on the similarity between source domain and target domain. Now that we have shown ELR is critical for the performance of fine-tuning, we are interested in the factors that determine the optimal ELR affected. found that there is an optimum fluctuation scale which maximizes the test set accuracy (at constant learning rate). However, the relationship between ELR and domain distance is unknown, which is important for fine-tuning. Effective learning rate encapsulates the effect of learning rate and momentum for fine-tuning. We varied other hyperparameters and report the best performance for each η. As shown in Figure 4, a smaller η works better if source and target domains are similar, such as Dogs for ImageNet and Birds for iNaturalist. On the other hand, the ELR for training from scratch is large and relative stable. We made similar observations on DenseNets and MobileNet (see Appendix B). The relationship between weight decay and effective learning rate are also well-studied (; van Laarhoven;). It was shown that the optimal weight decay value λ is inversely related with learning rate η. The'effective' weight decay is λ = λ/η. We show in Figure 5 that the optimal effective weight decay is larger when the source domain is similar with the target domain. Domain Similarity and Hyperparameter Selection Now we have made qualitative observations about the relationship between domain similarity and optimal ELR, which can help for reducing the hyperparameter search ranges. Note the original similarity score is based on models pre-trained on large scale dataset , which are not public available. We revisited the domain similarity score calculation in Appendix C and propose to use the source model as feature extractor We find there is a good correlation between our own domain similarity and the scale of optimal ELR. Based on the correlation between the similarity score and the optimal ELR, we propose a simple strategy for ELR selection in Appendix C. L 2 regularization or weight decay is widely used for constraining the model capacity . Recent work (; 2019) pointed that standard L 2 regularization, which drives the parameters towards the origin, is not adequate in transfer learning. To retain the knowledge learned by the pre-trained model, reference based regularization was used to regularize the distance between fine-tuned weights and the pre-trained weights, so that the finetuned model is not too different from the initial model. propose L 2 -SP norm, i.e.,, where θ refers to the part of network that shared with the source network, and θ refers to the novel part, e.g., the last layer with different number of neurons. While the motivation is intuitive, there are several issues for adopting reference based regularization for fine-tuning: Many applications actually adopt fine-tuning on target domains that are quite different from source domain, such as fine-tuning ImageNet models for medical imaging . The fine-tuned model does not necessarily have to be close with the initial model. The scale invariance introduced by Batch Normalization (BN) layers enable models with different parameter scales to function the same, i.e., f (θ) = f (αθ). Therefore, when L 2 regularization drives θ 2 2 towards zeros, it could still have the same functionality as the initial model. On the contrary, a model could still be different even when the L 2 -SP norm is small. It has been shown that the effect of weight decay on models with BN layers is equivalent to increasing the effective learning rate by shrinking the weights scales (van Laarhoven;). Regularizing weights with L 2 -SP norm would constrain the scale of weights to be close to the original one, therefore not increasing the effective learning rate, during fine-tuning. As a small effective learning rate is beneficial for fine-tuning from similar domains, which may explain why L 2 -SP provides better performance. If this is true, then by decreasing the effective learning rate, L 2 regularization would functions the same. To examine these conjectures, we revisited the work of with additional experiments. To show the effectiveness of L 2 -SP norm, conducted experiments on datasets such as Dogs, Caltech and Indoor, which are all datasets close to the source domain (ImageNet or Place-365) according to previous sections. We extend their experiments on other datasets that are relatively "far" away from ImageNet, such as Birds, Cars, Aircrafts and Flowers. We use the source code of to fine-tune on these datasets with both L 2 and L 2 -SP regularization. For fair comparison, we performed the same hyperparameter search for both methods (see experimental settings in Appendix E). As expected, Table 4 shows that L 2 regularization is very close to if not better than L 2 -SP on Birds, Cars, Aircrafts and Flowers, which indicates that reference based regularization methods may not be able to generalize for fine-tuning on dissimilar domains. The two extreme ways for selecting hyperparameters-performing exhaustive hyperparameter search or taking ad-hoc hyperparameters from scratch training-could be either too computationally expensive or yield inferior performance. Different with training from scratch, the default hyperparameter setting may work well for random initialization, the choice of hyperparameters for fine-tuning is not only dataset dependent but is also influenced by the similarity between the target domain and the source domains. The rarely tuned momentum value could impede the performance when the target domain and source domain are close. These observations connect with previous theoretical works on decreasing momentum at the end of training and effective learning rate. We further identify the optimal effective learning rate depends on the similarity of source domain and target domain. With this understanding, one can significant reduce the hyperparameter search space. We hope these findings could be one step towards better hyperparameter selection strategies for fine-tuning. To check the influence of momentum on fine-tuning, we first search the best momentum values for fine-tuning on the Birds dataset with fixed learning rate but different batch size and weight decay. Figure 6 provides the convergence curves for the shown in Figure 1 (a), which shows the learning curves of fine-tuning with 6 different batch sizes and weight decay combinations. Zero momentum outperforms the nonzero momentum in 5 of the 6 configurations. Optimal learning rate increases after disabling momentum. Figure 7 compares the performance of turning on/off momentum for each datasets with different learning. For datasets that are "similar" to ImageNet (Figure 7 (a-h)) and fixed learning rate (e.g., 0.01), the Top-1 validation error decreases significantly after disabling momentum. On the other hand, for datasets that are "dissimilar" to ImageNet (Figure 7(g-n) ) and fixed learning rate, disabling momentum hurts the top-1 accuracy. We can also observe that the optimal learning rate generally increase 10x after changing from 0.9 to 0.0, which is coherrent with the rule of effective learning rate. We also verified our observations on DenseNet-121 and MobileNet-1.0 with similar settings. We made similar observations among the three architectures: the optimal effective learning rate is related with the similarity to source domain. As seen in Figure 8 (b) and (c), the optimal effective learning rates for Dogs/Caltech/Indoor datasets are much smaller than these for Aircrafts/Flowers/Cars when fine-tuned from ImageNet, similar with ResNet-101. This verified our findings on ResNet-101 are pretty consistent on a variety of architectures. In Appendix C, we will show the the correlation between EMD based domain similarity and optimal ELR is quite consistent across different architectures. The domain similarity calculation based on Earth Mover Distance (EMD) is introduced in the section 4.1 of 4. Here we briefly introduce the steps. In , the authors first train ResNet-101 on the large scale JFT dataset and use it as a feature extractor. They extracted features from the penultimate layer of the model for each image of the training set of the source domain and target domain. For ResNet-101, the length of the feature vector is 2048. The features of images belonging to the same category are averaged and g(s i) denotes the average feature vector of ith label in source domain S, similarly, g(t j) denotes the average feature vector of jth label in target domain T. The distance between the averaged features of two labels is d i,j = g(s i) − g(t j). Each label is associated with a weight w ∈ corresponding to the percentage of images with this label in the dataset. So the source domain S with m labels and the target domain T with n labels can be represented as The EMD between the two domains is defined as where the optimal flow f i,j corresponds to the least amount of total work by solving the EMD optimization problem. The domain similarity is defined as where γ is 0.01. The domain similarities based on this pre-trained model for the seven dataset is listed in Table 5. , which use JFT pretrained ResNet-101 as the feature extractor. Note that the ResNet-101 pre-trained on JFT is not released and therefore we cannot calculate its similarities for datasets such as Caltech and Indoor. On the right three columns, we use three public available ImageNet pre-trained models as the feature extractor. All features are extracted from the training set of each dataset. The 1st, 2nd, 3rd and 4th scores are color coded, and the smallest three scores are marked in brown. The corresponding optimal ELR η is listed, which is the same as shown in Figure Due to the unavailability of the large-scale JFT dataset (300x larger than ImageNet) and its pre-trained ResNet-101 model, we cannot use it for extracting features for new datasets, such as Caltech256 and MIT67-Indoor. Instead of using the powerful feature representation, we use the pre-trained source model directly as the feature extractor for both source domain and target domains. We believe the similarity based on features extracted from source models captures the transfer learning process better for different source models. In Table 5, we compared the domain similarities calculated by three different ImageNet pre-trained models. We find some consistent patterns across different architectures: • The 1st and 2nd highest similarity scores are Caltech and Dogs across architectures. • The 3rd and 4th highest similarity scores refers to Birds and Indoor. • The most dissimilar datasets are Cars, Aircrafts and Flowers. Note the relative orders for the dissimilar datasets (Cars, Aircrafts and Flowers) are not consistent across difference architectures. As we can see from Table 5, the domain similarity score has some correlation with the scale of optimal ELR. Generally, the more similarity between two domains, the smaller the optimal ELR. Though the optimal ELR is not strictly corresponding to the domain similarity score, the scores provide reasonable predictions about the scale of optimal ELR, such as and therefore can reduce the search ranges of ELR. One can calculate the domain similarities and perform exhaustive hyperparameter searches for the first few target datasets, including similar and dissimilar datasets, such as Dogs and Cars and we refer these datasets as reference datasets. Then for the given new dataset to fine-tune, one can calculate its domain similarity and compare with the scores of reference datasets, and choose the range of ELRs with the closest domain similarity. D referred to the fact that the momentum parameter of BN is essential for fine-tuning. They found it useful to decrease the batch normalization momentum parameter from its ImageNet value to max(110/s, 0.9) where s is the number of steps per epoch. This will change the default BN momentum value (0.9) when s is larger than 100, but only applies when the dataset size is larger than 25.6K with batch size 256. The maximum data size used in our experiments is Caltech-256, which is 15K, so this strategy is not applicable to our experiments. We further explore the effect of BN momentum by perform similar study as to ELR. We want to identify whether there is an optimal BN momentum for each dataset. For each dataset, we finetune the pre-trained model using previously obtained best hyperparameters and only tune the BN momentum. In addition to the default 0.9, we searched 0.0, 0.95 and 0.99. If BN mommentum is critical, we can expect significant performance differences. The is shown in Figure 9 We see m bn = 0.99 slightly improves the performance for some datasets, however, we did not see the significant performance difference among values greater than 0.9. We use the code 5 provided by the authors. The base network is ImageNet pretrained ResNet-101-V1. The model is fine-tuned with batch size 64 in 9000 iterations, and the learning rate is decayed at iteration 6000. Following the original setting, we use momentum 0.9. We performed grid search on learning rate and weight decay, with the range of η: {0.02, 0.01, 0.005, 0.001, 0.0001} and λ 1: {0.1, 0.01, 0.001, 0.0001}, and report the best average error for both methods. For L 2 -SP norm, we follow the authors' setting to use constant λ 2 = 0.01. Different with the original setting for L 2 regularization, we set λ 2 = λ 1 to simulate the normal L 2 -norm. Data augmentation is an important way of increasing data quantity and diversity to make models more robust. It is even critical for transfer learning with few instances. The effect of data augmentation can be viewed as a regularization method and the choice of data augmentation method is also a hyperparameter. Most current widely used data augmentation methods have verified their effectiveness on training ImageNet models, such as random mirror flipping, random rescaled cropping 6, color jittering and etc; and they are also widely used for fine-tuning. Do these methods transfer for fine-tuning on other datasets? Here we compare three settings for data augmentation: 1) random resized cropping: our default data augmentation; 2) random crop: the same as standard data augmentation except that we use random cropping with fixed size; 3) random flip: simply random horizontal flipping. The effect of data augmentation is dataset dependent and has big impact on the convergence time The training and validation errors of fine-tuning with different data augmentation strategies are illustrated in Figure 10. We find that advanced cropping works significantly better on datasets like Cars, Aircrafts and Flowers but performs worse on Dogs. The choice of data augmentation methods has dramatic influence to the convergence behaviour. Simpler data augmentation usually converge very quickly (e.g., in 20 epochs), while the training error for random resized cropping converges much slower. We see that default hyperparemter and data augmentation method lead to overfitting on Dogs dataset. This can be solved by disabling momentum as we can see in Table 2, and in better performance than random cropping. We can expect that random resized cropping adds extra variance to the gradient direction and the effect of disabling momentum is more obvious for this case. Disabling momentum improves performance on datasets that are close to source domains Here we compare data augmentation methods with different momentum settings. As can be seen in Table 6, random resized cropping consistently outperforms random cropping on datasets like Cars, Aircrafts and Flowers. Using momentum improves the performance significantly for both methods. We see that advanced data augmentation method with default hyperparameters (m = 0.9 and η = 0.01) leads to overfitting on Dogs and Caltech dataset (Figure 11 (a) and (c)). Random resized cropping with zero momentum solves this problem and in better performance than random cropping. When momentum is disabled for random cropping, the performance is still better for Dogs, but decreases for other datasets. This can be expected as random cropping produces images with less variation and noise than random resized cropping, the gradients variation is less random and momentum can still point to the right direction. This can be further verified as we increase the learning rate for random cropping, which adds variation to the gradients, and disabling momentum shows better performance that nonzero momentum on datasets that are close. Transfer learning from similar source domains helps but does not guarantee good performance We consider two ImageNet subsets: 449 Natural objects and 551 Man-made objects, following the splits of (supplementary materials). From the bottom of Table 7, we can see that fine-tuning from ImageNet-Natural pre-trained models performs better on Birds and Dogs dataset, whereas Caltech-256 and Indoor benefit more from ImageNet-Manmade pretrained models. The performance gap between ImageNet-Manmad and ImageNet-Natural on Cars and Flowers are not as significant as for Birds and Dogs. It is surprising to see that fine-tuning from ImageNetManmade subset yields worse performance than ImageNet-Natural on the Cars and Indoor dataset. The fine-tuning on both subsets do not exceed the pre-trained models with full ImageNet. Scratch training can outperform fine-tuning with better hyperparameters We further reexamine the default hyperparameters for scratch training. For most tasks, training from scratch with default hyperparameters is much worse than fine-tuning from ImageNet. However, after slight hyperparameter tuning on learning rates, momentum and weight decay, the performance of training from scratch gets close to the default fine-tuning (e.g., Cars and Aircrafts). Scratch training HPO on Cars and Aircrafts even surpasses the default fine-tuning . Previous studies; also identified that datasets like Cars, Aircrafts do not benefit too much from fine-tuning. FT ImageNet Default is the fine-tuning with the default hyperparameters. Scratch Train use similar hyperparameters as default fine-tuning, which are η = 0.1, n = 256, λ = 0.0001 and m = 0.9 with doubled length of training schedules. HPO refers to the best with hyperparameter grid search. Note our Indoor dataset is fine-tuned from ImageNet. Results of ResNet-101 DELTA refers to , and Inception-v3 refers to (Pre-trained models on ImageNet-Natural and ImageNet-Manmade We train ResNet-101 from scratch on each subset using standard hyperparameters, i.e., initial learning rate 0.1, batch size 256, momentum 0.9. We train 180 epochs, learning rate is decayed at epoch 60 and 120 by a factor of 10. Table 8 illustrates the Top-1 errors of training ResNet-101 on each source datasets. Scratch Training HPO Figure 12 shows the training/validation errors of training from scratch on each dataset with different learning rate and weight decay. We use initial learning rate 0.1, batch size 256. For most dataset, we train 600 epochs, and decay the learning rate at epoch 400 and 550 by a factor of 10. The parameters to search is η ∈ [0.1, 0.2, 0.5] and λ ∈ [0.0001, 0.0005] with fixed momentum 0.9 and batch size 256. We observe weight decay 0.0005 consistently performs better than 0.0001.

Represent the TLDR for finding the corresponding scientific article

This paper re-examines several common practices of setting hyper-parameters for fine-tuning.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP. The experiment show that our proposed method outperforms these baselines with higher accuracy for new tasks. In many real-world applications, deep learning practitioners often have limited number of training instances. Direct training a deep neural network with a small training data set usually in the so-called over-fitting problem and the quality of the obtained model is low. A simple yet effective approach to obtain high-quality deep learning models is to perform weight fine-tuning. In such practices, a deep neural network is first trained using a large (and possibly irrelevant) source dataset (e.g. ImageNet). The weights of such a network are then fine-tuned using the data from the target application domain. Fine-tuning is a specific approach to perform transfer learning in deep learning. The weights pretrained by the source dataset with a sufficiently large number of instances usually provide a better initialization for the target task than random initializations. In a typical fine-tuning approach, weights in lower convolution layers are fixed and weights in upper layers are re-trained using data from the target domain. In this approach parameters of the target model may be driven far away from initial values, which also causes over-fitting in transfer learning scenarios. Approaches called regularization using the starting point as the reference (SPAR), were recently proposed to solve the over-fitting problem. For example, Li et al. BID10 proposed L 2 -SP that incorporates the Euclid distance between the target weights and the starting point (i.e., weights of source network) as part of the loss. Minimizing this loss function, L 2 -SP aims to minimize the empirical loss of deep learning while reducing the distance of weights between source and target networks. They achieved significant improvement compared with standard practice of using the weight decay (L 2 normalization).However such regularization method may not deliver optimal solution for transfer learning. On one side, if the regularization is not strong, even with fine-turning, the weights may still be driven far away from the initial position, leading to the lose of useful knowledge, i.e. catastrophic memory loss. On the other side, if the regularization is too strong, newly obtained model is constrained to a local neighborhood of the original model, which may be suboptimal to the target data set. Although aforementioned methods demonstrated the power of regularization in deep transfer learning, we argue that we need to perform research on at least the following two aspects in order to further improve current regularization methods. Behavior vs. Mechanisms. The practice of weight regularization for CNN is motivated by a simple intuition -the network (layers) with similar weights should produce similar outputs. However, due to the complex structures of deep neural network with strong redundancies, regulating the model parameters directly seems an over-killing of the problem. We argue that we should regularize the "Behavior", or in our case, the outer layer outputs (e.g. the feature maps) produced by each layer, rather than model parameters. With constrained feature maps, the generalization capacity could be improved through aligning the behaviors of the outer layers of the target network to the source one, which has been pre-trained using an extremely large dataset. In Convolutional Neural Networks, which we focus on exclusively in this paper, an outer layer is a convolution layer and the output of an outer layer is its feature map. Syntax vs Semantics. While regularizing the feature maps might improve the transfer of generalization capacity, it is still difficult to design such regularizers. It is challenging to measure the similarity/distance between the feature maps without understanding its semantics or representations. For example for image classification, some of the convolution kernels may be corresponding to features that are shared between the two learning tasks and hence should be preserved in transfer learning while others are specific to the source task and hence could be eliminated in transfer learning. In this paper, we propose a novel regularization approach DELTA to address the two issues. Specifically, DELTA selects the discriminative features from outer layer outputs through re-weighting the feature maps with a novel supervised attention mechanism. Through paying attention to discriminative parts of feature maps, DELTA characterizes the distance between source/target networks using their outer layer outputs, and incorporates such distance as the regularization term of the loss function. With the back-propagation, such regularization finally affects the optimization for weights of deep neural network and awards the target network generalization capacity inherited from the source network. In summary, our key insight is what we call "unactivated channel re-usage". Specifically our approach identifies those transferable channels and preserves such filters through regularization and identify those untransferable channels and reuse them, using an attention mechanism with feature map regularization. We have conducted extensive experiments using a wide range of source/target datasets and compared DELTA to the existing deep transfer learning algorithms that are in pursuit of weight similarity. The experiment show that DELTA significantly outperformed the state-of-the-art regularization algorithms including L 2 and L 2 -SP with higher accuracy on a wide group of image classification data sets. The rest of the paper is organized as follows: in Section 2 related works are summarized, in Section 3 our feature map based regularization method is introduced, in Section 4 experimental are presented and discussed, and finally in Section 5 the paper is concluded. In this section, we first review the related works to this paper, where we discuss the contributions made by this work beyond previous studies. Then, we present the s of our work. Transfer learning is a type of machine learning paradigm aiming at transferring the knowledge obtained in a source task to a target task (; BID14 . Our work primarily focuses on inductive transfer learning for deep neural networks, where the label space of the target task dif-fers from that of the source task. For example, Donahue et al BID3 proposed to train a classifier based on feature extracted from a pre-trained CNN, where a large mount of parameters, such as filters, of the source network are reused directly in the target one. This method may overload the target network with tons of irrelevant features (without discrimination power) involved, while the key features of the target task might be ignored. To understand whether a feature can be transferred to the target network, Yosinki et al. BID19 quantified the transferability of features from each layer considering the performance gain. Moreover, to understand the factors that may affect deep transfer learning performance, Huh et al. BID8 empirically analyzed the features obtained by the ImageNet pre-trained source network to a wide range of computer vision tasks. Recently, more studies to improve the inductive transfer learning from a diverse set of angles have been proposed, such as filter subset selection BID4 BID2, sparse transfer BID12, filter distribution constraining BID0, and parameter transfer BID21.For deep transfer learning problems, the most relevant work to our study is BID10, where authors investigated regularization schemes to accelerate deep transfer learning while preventing fine-tuning from over-fitting. Their work showed that a simple L 2 -norm regularization on top of the "Starting Point as a Reference" optimization can significantly outperform a wide range of regularization-based deep transfer learning mechanisms, such as the standard L 2 -norm regularization. Compared to above work, the key contributions made in this paper include 1) rather than regularizing the distance between the parameters of source network and target network, DELTA constrains the L 2 -norm of the difference between their behaviors (i.e., the feature maps of outer layer outputs in the source/target networks); and 2) the regularization term used in DELTA incorporates a supervised attention mechanism, which re-weights regularizers according to their performance gain/loss. In terms of methodologies, our work is also related to the knowledge distillation for model compression BID6 BID15. Generally, knowledge distillation focuses on teacher-student network training, where the teacher and student networks are usually based on the same task BID6. These work frequently intends to transfer the knowledge in the teacher network to the student one through aligning their outputs of some layers BID15. The most close works to this paper are BID20 BID18, where knowledge distillation technique has been studied to improve transfer learning. Compared to above work, our work, including other transfer learning studies, intends to transfer knowledge between different source/target tasks (i.e., source and target tasks), though the source/target networks can be viewed as teachers and students respectively. We follow the conceptual ideas of knowledge distillation to regularize the outer layer outputs of the network (i.e., feature maps), yet further extend such regularization to a supervised transfer learning mechanism by incorporating the labels of target task (which is different from the source task/network). Moreover, a supervised attention mechanism has been adopted to regularize the feature maps according to the importance of filters. Other works relevant to our methodology include: continual learning BID9 BID11, attention mechanism for CNN models BID13 BID16 BID17 BID20, among others. Deep convolutional networks usually consist of a great number of parameters that need fit to the dataset. For example, ResNet-110 has more than one million free parameters. The size of free parameters causes the risk of over-fitting. Regularization is the technique to reduce this risk by constraining the parameters within a limited space. The general regularization problem is usually formulated as follow. Let's denote the dataset for the desired task as {(x 1, y 1), (x 2, y 2), (x 3, y 3)..., (x n, y n)}, where totally n tuples are offered and each tuple (x i, y i) refers to the input image and its label in the dataset. We further denote ω ∈ R d be the d-dimensional parameter vector containing all d parameters of the target model. The optimization object with regularization is to obtain DISPLAYFORM0 where the first term n i=1 L(z(x i, ω), y i ) refers to the empirical loss of data fitting while the second term is a general form of regularization. The tuning parameter λ > 0 balances the trade-off between the empirical loss and the regularization loss. Without any explicit information (such as other datasets) given, one can easily use the L 0 /L 1 /L 2 -norm of the parameter vector ω as the regularization to fix the consistency issue of the network. Given a pre-trained network with parameter ω * based on an extremely large dataset as the source, one can estimate the parameter of target network through the transfer learning paradigms. Using the ω * as the initialization to solve the problem in Eq 1 can accelerate the training of target network through knowledge transfer BID7 BID1. However, the accuracy of the target network would be bottlenecked in such settings. To further improve the transfer learning, novel regularized transfer learning paradigms that constrain the divergence between target and source networks has been proposed, such that DISPLAYFORM0 where the regularization term Ω(ω, ω *) characterizes the differences between the parameters of target and source network. As ω * is frequently used as the initialization of ω during the optimization procedure, this method sometimes refers to Starting Point As the Reference (SPAR) method. To regularize weights straightforwardly, one can easily use the geometric distance between ω and ω * as the regularization terms. For example, L 2 -SP algorithm constrains the Euclid distance of the weights of convolution filters between the source/target networks BID10.In this way, we summarize the existing deep transfer learning approaches as the solution of the regularized learning problem listed in Eq 2, where the regularizer aims at constraining the divergence of parameters of the two networks while ignoring the behavior of the networks with the training data set {(x 1, y 1), (x 2, y 2),..., (x n, y n)}. More specific, the regularization terms used by the existing deep transfer learning approaches neither consider how the network with certain parameters would behave with the new data (images) or leverages the supervision information from the labeled data (images) to improve the transfer performance. In this section, we first formulate the problem, then present the overall design of proposed solution and introduce several key algorithms. In our research, instead of bounding the difference of weights, we intend to regulate the network behaviors and force some layers of the target network to behave similarly to the source ones. Specifically, we define the "behaviors" of a layer as its output, which are with semantics-rich and discriminative information. DELTA intends to incorporate a new regularizer Ω (ω, ω *, x). Given a pre-trained parameter ω * and any input image x, the regularizer Ω (ω, ω *, x) measures the distance between the behaviors of target network with parameter ω and the source one based on ω *. With such regularizer, the transfer learning problem can be reduced to learning problem as follows: DISPLAYFORM0 where n i=1 Ω(ω, ω *, x i, y i, z) characterizes the aggregated difference between the source and target network over the whole training dataset using the model z. Note that, with the input tuples Figure 1: Behavior-based Regularization using Feature Maps with Attentions (x i, y i) and for 1 ≤ i ≤ n, the proposed regularizer Ω(ω, ω *, x i, y i, z) is capable of regularizing the behavioral differences of network model z based on each labeled sample (x i, y i) in the dataset, using the parameters ω and ω * respectively. Further, inspired by SPAR method, DELTA accelerates the optimization procedure of the regularizer through incorporating a parameter-based proximal term, such that DISPLAYFORM1 where α, β are two non-negative tuning parameters to balance two terms. On top of the behavioral regularizer Ω (ω, ω *, x, y, z), DELTA includes a term Ω (ω\ω *) regularizing a subset of parameters that are privately owned by the target network w only but not exist in the source network w *. Specifically, Ω (ω\ω *) constrains the L 2 -norm of the private parameters in ω, so as to improve the consistency of inner layer parameters estimation. Note that, when using w * as the initialization of ω for optimization, DELTA indeed adopts starting point as reference (SPAR) strategy BID10 to accelerate the optimization and gains better generalizability. To regularize the behavior of the networks, DELTA considers the distance between the outer layer outputs of the two networks. Figure 1 illustrates the concepts of proposed method. Specifically, the outer layer of the network consists of a large set of convolutional filters. Given an input x i (for ∀1 ≤ i ≤ n in training set), each filter generates a feature map. Thus, DELTA characterizes the outer layer output of the network model z based on input x i and parameter ω using a set of feature maps, such as FM j (z, ω, x i) and 1 ≤ j ≤ N for the N filters in networks. In this way, the behavioral regularizer is defined as: DISPLAYFORM0 where W j (z, ω *, x i, y i) refers to the weight assigned to the j th filter and the i th image (for ∀1 ≤ i ≤ n and ∀1 ≤ j ≤ N) and the behavioral difference between the two feature maps, i.e., FM j (z, ω, x i) and FM j (z, ω *, x i), is measured using their Euclid distance (denoted as · 2).In following sections, we are going to present the design and implementation of feature map extraction FM j (z, ω, x) for 1 ≤ j ≤ N, as well as the the attention model that assigns the weight W j (z, ω *, x i, y i) to each labeled image and filter. Given each filter of the network with parameter ω and the input x i drawn from the target dataset, DELTA first uses such filter to get the corresponding output based on x, then adopts Rectified Linear Units (ReLU) to rectify the output as a matrix. Further, DELTA formats the output matrices into vectors through concatenation. In this way, DELTA obtains FM j (z, ω, x i) for 1 ≤ j ≤ N and 1 ≤ i ≤ n that have been used in Eq 5. In DELTA, the proposed regularizer measures the distance between the feature maps generated by the two networks, then aggregates the distances using non-negative weights. Our aim is to pay more attention to those features with greater capacity of discrimination through supervised learning. To obtain such weights for feature maps, we propose a supervised attention method derived from the backward variable selection, where the weights of features are characterized by the potential performance loss when removing these features from the network. For clear description, following common conventions, we first define a convolution filter as follow. The parameter of a conv2d layer is a four-dimensional tensor with the shape of (c i+1, c i, k h, k w), where c i and c i+1 represent for the number of channels of the i th and (i + 1) th layer respectively. c i+1 filters are contained in such a convolutional layer, each of which with the kernel size of c i * k h * k w, taking the feature maps with the size of c i * h i * w i of the i-th layer as input, and outputing the feature map with the size of h i+1 * w i+1.In particular, we evaluate the weight of a filter as the performance reduction when the filter is disabled in the network. Intuitively, removing a filter with greater capacity of discrimination usually causes higher performance loss. In this way, such channels should be constrained more strictly since a useful representation for the target task is already learned by the source task. Given the pre-trained parameter ω * and an input image x i, DELTA sets the weight of the j th channel, using the gap between the empirical losses of the networks on the labeled sample (x i, y i) with and without the j th channel, as follow, DISPLAYFORM0 where ω * \j refers to the modification of original parameter ω * with all elements of the j th filter set to zero (i.e., removing the j th filter from the network). We use softmax to normalize the to ensure all weights are non-negative. The aforementioned supervised attention mechanism yields a filter a higher weight for a specific image if and only if the corresponding feature map in the pretrained source network is with higher discrimination power -i.e., paying more attention to such filter on such image might bring higher performance gain. Note that, to calculate L(z(x i, ω * \j), y i ) and L(z(x i, ω *), y i ) for supervised attention mechanism, we introduce a baseline algorithm L 2 -F E that fixes the feature extractor (with all parameters copied from source networks) and only trains the discriminators using the target task. The L 2 -F E model can be viewed as an adaption of the source network (weights) to the target tasks, without further modifications to the outer layer parameters. In our work, we use L 2 -F E to evaluate L(z(x i, ω * \j), y i ) and L(z(x i, ω *), y i ) using the target datasets. We have conducted a comprehensive experimental study of the proposed DELTA method. Below we first briefly review the used datasets, followed by a description of experimental procedure and finally our observations. We evaluate the performance of three benchmarks with different tasks: Caltech 256 for general object recognition, Stanford Dogs 120 for fine-grained object recognition, and MIT Indoors 67 for scene classification. For the first two benchmarks, we used ImageNet as the source domain and Places 365 for the last one. Caltech 256. Caltech 256 is a dataset with 256 object categories containing a total of 30607 images. Different numbers of training examples are used by researchers to validate the generalization of proposed algorithms. In this paper, we create two configurations for Caltech 256, which have 30 and 60 random sampled training examples respectively for each category, following the procedure used in BID10.Stanford Dogs 120. The Stanford Dogs dataset contains images of 120 breeds of dogs from around the world. There are exactly 100 examples per category in the training set. It is used for the task of fine-grained image categorization. We do not use the bounding box annotations. MIT Indoors 67. MIT Indoors 67 is a scene classification task containing 67 indoor scene categories, each of which consists of 80 images for training and 20 for testing. Indoor scene recognition is challenging because both spatial properties and object characters are expected to be extracted. -200-2011 -200-. CUB-200-2011 contains 11,788 images of 200 bird species. Each species is associated with a wikipedia article and organized by scientific classification. Each image is annotated with bounding box, part location, and attribute labels. We use only classification labels during training. While part location annotations are used in a quantitative evaluation of show cases, to explain the transferring effect of our algorithm. Food-101. Food-101 a large scale data set of 101 food categories, with 101,000 images, for the task of fine-grained image categorization. 750 training images and 250 test images are provided for each class. This dataset is challenging because the training images contain some amount of noise. We implemented our method with ResNet-101 and Inception-V3 as the base networks. For experiment set up we followed almost the same procedure in BID10 due to the close relationship between our work and theirs. After training with the source dataset and before fine-tuning the network with the target dataset, we replace the last layer of the base network with random initialization in suit for the target dataset. For ResNet-101, the input images are resized to 256*256 and normalized to zero mean for each channel, following with data augmentation operations of random mirror and random crop to 224*224. For Inception-V3, images are resized to 320*320 and finally cropped to 229*229. We use a batch size of 64. SGD with the momentum of 0.9 is used for optimizing all models. The learning rate for the base model starts with 0.01 for ResNet-101 and 0.001 for Inception-V3, and is divided by 10 after 6000 iterations. The training is finished at 9000 iterations. We use five-fold cross validation for searching the best configurations of the hyperparameter α for each experiment. The hyperparameter β is fixed to 0.01. As was mentioned, our experiments compared DELTA to several key baseline algorithms including L 2, L 2 -SP BID10, and L 2 -F E (see also in Section 3.4), all under the same settings. Each experiment is repeated five times. The average top-1 classification accuracy and standard division is reported. DISPLAYFORM0 DISPLAYFORM1 In FIG0 plotted a sample learning curve of training with different regularization techniques. Comparing these regularization techniques, we observe that our proposed DELTA shows faster convergence than the simple L 2 -SP regularization with both step decay (StepLR) and exponential decay (ExponentialLR) learning rate scheduler. In addition, we find that the learning curve of DELTA is smoother than L 2 -SP and it is not sensitive to the learning rate decay happened at the 6000th iteration when using StepLR.In TAB0 we show the of our proposed method DELTA with and without attention, compared to the baseline of L 2 -SP reported in BID10 and also the naive L 2 -F E and L 2 methods. We find that on some datasets, fine-tuning using L 2 normalization does not perform significantly better than directly using the pre-trained model as a feature extractor(L 2 -F E), while L 2 -SP outperforms the naive methods without SPAR. We observe that greater benefits are gained using our proposed attention mechanism. Data augmentation is a widely used technique to improve image classification. Following BID10, we used a simple data augmentation method and a post-processing technique. First, we keep the original aspect ratio of input images by resizing them with the shorter edge being 256, instead of ignoring the aspect ratio and directly resizing them to 256*256. Second, we apply 10-crop testing to further improve the performance. In TAB1, we documented the experimental using these technique with different regularization methods. We observe a clear pattern that with additional data augmentation, all the three evaluated methods L 2, L 2 -SP, DELTA have improved classification accuracy while our method still delivers the best one. To better understand the performance gain of DELTA we performed an experiment where we analyzed how parameters of the convolution filters change after fine-tuning. Towards that purpose we randomly sampled images from the testing set of Stanford Dogs 120. For ResNet-101, which we use exclusively in this paper, we grouped filters into stages as described in (he et al., 2016). These stages are conv2 x, conv3 x, conv4 x, conv5 x. Each stage contains a few stacked blocks and a block is a basic inception unit having 3 conv2d layers. One conv2d layer consists of a number of output filters. We flatten each filter into a one dimension parameter vector for convenience. The Euclidian distance between the parameter vectors before and after fine-tuning is calculated. All distances are sorted as shown in FIG3.We observed a sharp difference between the two distance distributions. Our hypothesis of possible cause of the difference is that simply using L 2 -SP regularization all convolution filters are forced to be similar to the original ones. Using attention, we allow "unactivated" convolution filters to be reused for better image classification. About 90% parameter vectors of DELTA have larger distance than L 2 -SP. We also observe that a small number of filters is driven very far away from the initial value (as shown at the left end of the curves in FIG3 . We call such an effect as "unactivated channel re-usage".To further understand the effect of attention and the implication of "unactivated channel re-usage", we "attributed" the attention to the original image to identify the set of pixels having high contributions in the activated feature maps. We select some convolution filters on which the source model (the initialization before fine-tuning) has low activation. For the convenience of analyzing the effect of regularization methods, each element a i of the original activation map is normalized with DISPLAYFORM0 where the min and max terms in the formula represent for the minimum and maximum value of the whole activation map respectively. Activation maps of these convolution filter for various regularization method are presented on each row. As shown in FIG4, our first observation is that without attention, the activation maps from DELTA in different images are more or less the same activation maps from other regularization methods. This partially explains the fact that we do not observe significant improvement of DELTA without attention. Using attention, however, changes the activation map significantly. Regularization of DELTA with attention show obviously improved concentration. With attention (the right-most column in FIG4), we observed a large set of pixels that have high activation at important regions around the head of the animals. We believe this phenomenon provides additional evidence to support our intuition of "unactivated channel re-usage" as discussed in previous paragraphs. In addition, we included new statistical of activations on part locations of CUB-200-2011 supporting the above qualitative cases. The CUB-200-2011 datasets defined 15 discriminative parts of birds, e.g. the forehead, tail, beak and so on. Each part is annotated with a pixel location representing for its center position if it is visible. So for each image, we got several key points which are very important to discriminate its category. Using all testing examples of CUB-200-2011, we calculated normalized activations on these key points of these different regularization methods. As shown in TAB2, DELTA got the highest average activations on those key points, demonstrating that DELTA focused on more discriminate features for bird recognition. In this paper, we studied a regularization technique that transfers the behaviors and semantics of the source network to the target one through constraining the difference between the feature maps generated by the convolution layers of source/target networks with attentions. Specifically, we designed a regularized learning algorithm DELTA that models the difference of feature maps with attentions between networks, where the attention models are obtained through supervised learning. Moreover, we further accelerate the optimization for regularization using start point as reference (SPAR). Our extensive experiments evaluated DELTA using several real-world datasets based on commonly used convolutional neural networks. The experiment show that DELTA is able to significantly outperform the state-of-the-art transfer learning methods.

Represent the short summary for finding the corresponding scientific article (long).

improving deep transfer learning with regularization using attention based feature maps

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.