Question: We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that takes into account typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models. This article is in the context of sequential or incremental learning in Deep Neural Networks (DNNs). Essentially, this means that a DNN is not trained once, on a single task D, but successively on two or more sub-tasks D 1,..., D n, one after another. Learning tasks of this type, which we term Sequential Learning Tasks (SLTs) (see FIG0), are potentially very common in real-world applications. They occur wherever DNNs need to update their capabilities on-site and over time: gesture recognition, network traffic analysis, or face and object recognition in mobile robots. In such scenarios, neural networks have long been known to suffer from a problem termed "catastrophic forgetting"(CF) (e.g., BID7) which denotes the abrupt and near-complete loss of knowledge from previous subtasks D 1,..., D k−1 after only a few training iterations on the current sub-task D k (see FIG0 compared to FIG0). We focus on SLTs from the visual domain with two sub-tasks each, as DNNs show pronounced CF behavior even when only two sub-tasks are involved. The sequential learning tasks used in this study only have two sub-tasks: D1 and D2. During training (white ) and re-training (gray ), test accuracy is measured on D1 (blue,), D2 (green,) and D1 ∪ D2 (red,). The blue curve allows to determine the presence of CF by simple visual inspection: if there is significant degradation w.r.t. the red curve, then CF has occurred. DISPLAYFORM0 The field of incremental learning is large, e.g., BID20 and BID8. Recent systematic comparisons between different DNN approaches to avoid CF are performed in, e.g., BID23 or. Principal recent approaches to avoid CF include ensemble methods BID22 BID6, dual-memory systems BID24 BID11 BID21 BID9 and regularization approaches. Whereas BID10 suggest Dropout for alleviating CF, the EWC method BID14 proposes to add a term to the energy function that protects weights that are important for the previous sub-task (s). Importance is determined by approximating the Fisher information matrix of the DNN. A related approach is pursued by the Incremental Moment Matching technique (IMM) (see), where weights from DNNs trained on a current and a past sub-tasks are "merged" using the Fisher information matrix. Other regularization-oriented approaches are proposed in BID2; BID25 and BID13 which focus on enforcing sparsity of neural activities by lateral interactions within a layer. Number of tested datasets In general, most methods referenced here are evaluated only on a few datasets, usually on MNIST BID16 and various derivations thereof (permutation, rotation, class separation). Some studies make limited use of CIFAR10, SVHN, the Amazon sentiment analysis problem, and non-visual problems such as data from Q-learning of Atari games. A largescale evaluation on a huge number of qualitatively different datasets is still missing 1. Model selection and prescience Model selection (i.e., selecting DNN topology and hyperparameters) is addressed in some approaches BID10 but on the basis of a "prescient" evaluation where the best model is selected after all tasks have been processed, an approach which is replicated in BID14. This amounts to a knowledge of future sub-tasks which is problematic in applications. Most approaches ignore model selection BID25 BID2 BID13, and thus implicitly violate causality. Storage of data from previous sub-tasks From a technical point of view, DNNs can be retrained without storing training data from previous sub-tasks, which is done in BID10 and BID25. For regularization approaches, however, there are regularization parameters that control the retention of previous knowledge, and thus must be chosen with care. In BID14, this is λ, whereas two such quantities occur in: the "balancing" parameter α and the regularization parameter λ for L2-transfer. The only study where regularization parameters are obtained through cross-validation (which is avoided in other studies) is BID2 (for λ SN I and λ Ω) but this requires to store all previous training data. This review shows that enormous progress has been made, but that there are shortcomings tied to applied scenarios which need to be addressed. We will formalize this in Sec. 1.2 and propose an evaluation strategy that takes these formal constraints into account when testing CF in DNNs. When training a DNN model on SLTs, first of all the model must be able to be retrained at any time by new classes (class-incremental learning). Secondly, it must exhibit retention, or at least graceful decay, of performance on previously trained classes. Some forgetting is probably unavoidable, but it should be gradual and not immediate, i.e., catastrophic. However, if a DNN is operating in, e.g., embedded devices or autonomous robots, additional conditions may be applicable: Low memory footprint Data from past sub-tasks cannot be stored and used for re-training, or else to determine when to stop re-training. Causality Data from future sub-tasks, which are often known in academic studies but not in applications, must not be utilized in any way, especially not for DNN model selection. This point might seem trivial, but a number of studies such as BID14; BID10 and BID25 perform model selection in hindsight, after having processed all sub-tasks. Constant update complexity Re-training complexity (time and memory) must not depend on the number of previous sub-tasks, thus more or less excluding replay-based schemes such as BID24. Clearly, even if update complexity is constant w.r.t. the number of previous sub-tasks, it should not be too high in absolute terms either. The original contributions of our work can be summarized as follows:• We propose a training and evaluation paradigm for incremental learning in DNNs that enforces typical application constraints, see Sec. 1.2. The importance of such an applicationoriented paradigm is underlined by the fact that taking application constraints into account leads to radically different about CF than those obtained by other recent studies on CF (see Sec. 1.1).• We investigate the incremental learning capacity of various DNN approaches (Dropout, LWTA, EWC and IMM) using the largest number of qualitatively different classification datasets so far described. We find that all investigated models are afflicted by catastrophic forgetting, or else in violation of application constraints and discuss potential workarounds.• We establish that the "permuted" type of SLTs (e.g., "permuted MNIST") should be used with caution when testing for CF.• We do not propose a method for avoiding CF in this article. This is because avoiding CF requires a consensus on how to actually measure this effect: our novel contribution is a proposal how to do just that. We collect a large number of visual classification datasets, from each of which we construct SLTs according to a common scheme, and compare several recent DNN models using these SLTs. The experimental protocol is such that application constraints, see Sec. 1.2, are enforced. For all tested DNN models (see below), we use a TensorFlow (v1.7) implementation under Python (v3.4 and later). The source code for all processed models, the experiment-generator and evaluation routine can be found on our public available repository 2.FC A normal, fully-connected (FC) feed-forward DNN with a variable number and size of hidden layers, each followed by ReLU, and a softmax readout layer minimizing cross-entropy. CONV A convolutional neural network (CNN) based on the work of BID4. It is optimized to perform well on image classification problems like MNIST. We use a fixed topology: two conv-layers with 32 and 64 filters of size 5 × 5 plus ReLU and 2 × 2 max-pooling, followed by a fc-layer with 1024 neurons and softmax readout layer minimizing a cross-entropy energy function. EWC The Elastic Weight Consolidation (EWC) model presented by BID14. LWTA A fully-connected DNN with a variable number and size of hidden layers, each followed by a Local Winner Takes All (LWTA) transfer function as proposed in BID25. IMM The Incremental Moment Matching model as presented by. We examine the weight-transfer techniques in our experiments, using the provided implementation. D-FC and D-CONV Motivated by BID10 we combine the FC and CONV models with Dropout as an approach to solve the CF problem. Only FC and CONV are eligible for this, as EWC and IMM include dropout by default, and LWTA is incompatible with Dropout. We perform model selection in all our experiments by a combinatorial hyper-parameter optimization, whose limits are imposed by the computational resources available for this study. In particular, we vary the number of hidden layers L ∈ {2, 3} and their size S ∈ {200, 400, 800} (CNNs excluded), the learning rate 1 ∈ {0.01, 0.001} for sub-task D 1, and the re-training learning rate 2 ∈ {0.001, 0.0001, 0.00001} for sub-task D 2. The batch size (batch size) is fixed to 100 for all experiments, and is used for both training and testing. As in other studies, we do not use a fixed number of training iterations, but specify the number of training epochs (i.e., passes through the whole dataset) as E = 10 for each processed dataset (see Sec. 2.2), which allows an approximate comparison of different datasets. The number of training/testing batches per epoch, B, can be calculated from the batch size and the currently used dataset size. The set of all hyper-parameters for a certain model, denoted P, is formed as a Cartesian product from the allowed values of the hyperparameters L, S, 1, 2 and complemented by hyper-parameters that remain fixed (E, batch size) or are particular to a certain model. For all models that use dropout, the dropout rate for the input layer is fixed to 0.2, and to 0.5 for all hidden layers. For CNNs, the dropout rate is set to 0.5 for both input and hidden layers. All other hyper-parameters for CNNs are fixed, e.g., number and size of layers, the max-pooling and filter sizes and the strides (2 × 2) for each channel. These decisions were made based on the work of BID10. The LWTA block size is fixed to 2, based on the work of BID25. The model parameter λ for EWC is set to λ1/ 2 (set but not described in the source code of BID14). For all models except IMM, the momentum parameter for the optimizer is set to µ = 0.99 BID26. For the IMM models, the SGD optimizer is used, and the regularizer value for the L2-regularization is set to 0.01 for L2-transfer and to 0.0 for weight transfer. We select the following datasets (see Tab. 1). In order to construct SLTs uniformly across datasets, we choose the 10 best-represented classes (or random classes if balanced) if more are present. MNIST FORMULA0 BID16 ) is the common benchmark for computer vision systems and classification problems. It consist of gray scale images of handwritten digits.EMNIST BID5 is an extended version of MNIST with additional classes of handwritten letters. There are different variations of this dataset: we extract the ten best-represented classes from the By Class variation containing 62 classes. Fruits 360 BID18 ) is a dataset comprising fruit color images from different rotation angles spread over 75 classes, from which we extract the ten best-represented ones. Devanagari BID1 contains gray scale images of Devanagari handwritten letters. From the 46 character classes (1.700 images per class) we extract 10 random classes. FashionMNIST BID27 consists of images of clothes in 10 classes and is structured like the MNIST dataset. We use this dataset for our investigations because it is a "more challenging classification task than the simple MNIST digits data BID27 ". SVHN BID19 ) is a 10-class dataset based on photos of house numbers. We use the cropped digit format, where the number is centered in the color image. CIFAR10 BID15 ) contains color images of real-world objects e.g, dogs, airplanes etc. NotMNIST (Bulatov Yaroslav) contains grayscale images of the 10 letter classes from "A" to "J", taken from different publicly available fonts. DISPLAYFORM0 is a modified version of the "Arabic Digits dataBase", containing grayscale images of handwritten digits written by 700 different persons. Tab. 2). For the latter case, we include SLTs where the second sub-task adds only 1 class (D9-1 type SLTs) or 5 classes (D5-5 type SLTs), since CF may be tied to how much newness is introduced. We include permutations (DP10-10) since we suspected that this type of SLT is somehow much easier than others, and therefore not a good incremental learning benchmark. As there are far more ways to create D5-5 type SLTs than D9-1 type SLTs, we create more of the former (8-vs-3) in order to avoid misleading due to a particular choice of subdivision, whereas we create only a single permutation-type SLT. Table 2: Overview of all SLTs. The assignment of classes to sub-tasks D1 and D2 are disjunct, except for DP10-10 where two different seeded random image permutations are applied. SLT → D5-5a D5-5b D5-5c D5-5d D5-5e D5-5f D5-5g D5-5h D9-1a D9-1b D9-1c DP10-10 D 1 0-4 0 2 4 6 8 3 4 6 8 9 0 2 5 6 7 0 1 3 4 5 0 3 4 8 9 0 5 6 7 8 0 2 3 6 8 0-8 1-9 0 2-9 0-9 D 2 5-9 1 3 5 7 9 0 1 2 5 7 1 3 4 8 9 2 6 7 8 9 1 2 5 6 7 1 2 3 4 9 1 4 5 7 9 9 0 1 0-9 This study presents just one, albeit very large, experiment, whose experimental protocol implements the constraints from Sec. 1.2.Every DNN model from Sec. 2 is applied to each SLT as defined in Sec. 2.3 while taking into account model selection, see Sec. 2.1. A precise definition of our application-oriented experimental protocol is given in Alg. 1. For a given model m and an SLT (D 1 and D 2), the first step is to determine the best hyper-parameter vector p * for sub-task D 1 only (see lines 1-4), which determines the model m p * used for re-training. In a second step, m p * (from line 5) is used for re-training on D 2, with a different learning rate 2 which is varied separately. We introduce two criteria for determining the (2 -dependent) quality of a re-training phase (lines 6-10): "best", defined by the highest test accuracy on D 1 ∪ D 2, and "last", defined by the test accuracy on D 1 ∪ D 2 at the end of re-training. Although the "best" criterion violates the application constraints of Sec. 1.2 (requires D 1), we include it for comparison purposes. Finally, the is computed as the highest 2 -dependent quality (line 11). Independently of the second step, another training of m p * is conducted using D 1 ∪ D 2, ing in what we term baseline accuracy. The of the experiment described in Sec. 3 are summarized in Tab. 3, and in Tab. 4 for IMM. They lead us to the following principal :Permutation-based SLTs should be used with caution We find that DP10-10, the SLT based on permutation, does not show CF for any model and dataset, which is exemplary visualized for the FC model in FIG2 which fails completely for all other SLTs. While we show this only for SLTs with two sub-tasks, and it is unknown how experiments with more sub-tasks would turn out, we nevertheless suggest caution when intepreting on permutation-based SLTs. All examined models exhibit CF While this is not surprising for FC and CONV, D-FC as proposed in BID10 performs poorly (see FIG3), as does LWTA BID25. For EWC and IMM, the story is slightly more complex and will be discussed below. EWC is mildly effective against CF for simple SLTs. Our experiments shows that EWC is effective against CF for D9-1 type SLTs, at least when the "best" evaluation criterion is used, which makes use of D 1. This, in turn, violates the application requirements of Sec. 1.2. For the "last" criterion not making use of D 1, EWC performance, though still significant, is much less impressive. We can see the origins of this difference illustrated in FIG4. Illustrating the difference between the "best" and "last" criterion for EWC. Shown is the accuracy over time for the best model on SLT D9-1c using EMNIST (a), D9-1a using EMNIST (b) and D9-1b using Devanagari (c). The blue curve measures the accuracy on D1, green only on D2 and red the D1 ∪ D2 during the training (white) and the re-training phase (gray). Additionally, the baseline (dashed line) is indicated. In all three experiments, the "best" strategy in approximately 90% accuracy, occurring at the beginning of re-training when D2 has not been learned yet. Here, the magnitude of the best/last difference is a good indicator of CF which clearly happens in (c), partly in (b) and slightly or not at all in (a).EWC is ineffective against CF for more complex problems. Tab. 3 shows that EWC cannot prevent CF for D5-5 type SLTs, see FIG6. Apparently, the EWC mechanism cannot protect all the weights relevant for D 1 here, which is likely to be connected to the fact that the number of samples in both sub-tasks is similar. This is not the case for D9-1 type tasks where EWC does better and where D 2 has about 10% of the samples in D 1. Best EWC experiments for SLT D5-5d constructed from all datasets, to be read as FIG2. We observe that CF happens for all datasets. See also Appendix B for 2D plots. IMM is effective for all SLTs but unfeasible in practice. As we can see from Tab. 4, wtIMM clearly outperforms all other models compared in Tab. 3. Especially for the D5-5 type SLTs, a modest incremental learning quality is attained, which is however quite far away from the baseline accuracy, even for MNIST-derived SLTs. This is in contrast to the reported in for MNIST: we attribute this discrepancy to the application-oriented model selection procedure using only D 1 that we perform. In contrast, in, a model with 800/800/800 neurons, for which good on MNIST are well-established, is chosen beforehand, thus arguably making implicit use of D 2. A significant problem of IMM is the determination of the balancing parameter α, exemplarily illustrated in FIG7. Our show that the optimal value cannot simply be guessed from the relative sizes of D 1 and D 2, as it is done in, but must be determined by cross-validation, thereby requiring knowledge of D 1 (violates constraints). Apart from these conceptual issues, we find that the repeated calculation of the Fisher matrices is quite time and memory-consuming (>4h and >8GB), to the point that the treatment of SLTs from certain datasets becomes impossible even on high-end machine/GPU combinations when using complex models. This is why we can evaluate IMM only for a few datasets. It is possible that this is an artifact of the TensorFlow implementation, but in the present state IMM nevertheless violates not one but two application constraints from Sec. 1.2. FIG8 and FIG9 give a visual impression of training an IMM model on D9-1 and D5-5 type SLTs, again illustrating basic feasibility, but also the variability of the "tuning curves" we use to determine the optimal balancing parameter α. Best wtIMM experiments for SLT D5-5b constructed from datasets we were able to test. The blue surfaces (epochs 0-10) represent the test accuracy during training on D1, the green surfaces the test accuracy on D2 during training on D2 (epochs 10-20). The white bars in the middle represent baseline accuracy, whereas the right part shows accuracies on D1 ∪ D2 for different α values, computed for mean-IMM (orange surfaces) and mode-IMM (red surfaces). See also Appendix B for 2D plots. The primary from the in Sec. 4 is that CF still represents a major problem when training DNNs. This is particularly true if DNN training happens under application constraints as outlined in Sec. 1.2. Some of these constraints may be relaxed depending on the concrete application: if some prior knowledge about future sub-task exists, it can be used to simplify model selection and improve . If sufficient resources are available, a subset of previously seen data may be kept in memory and thus allow a "best" type evaluation/stopping criterion for re-training, see Alg. 1.Our evaluation approach is similar to, and we adopt some measures for CF proposed there. A difference is the setting of up to 10 sub-tasks, whereas we consider only two of them since we focus less on the degree but mainly on presence or absence of CF. Although comparable both in the number of tested models and benchmarks, BID23 uses a different evaluation methodology imposing softer constraints than ours, which is strongly focused on application scenarios. This is, to our mind, the reason why those differ significantly from ours and underscores the need for a consensus of how to measure CF.In general application scenarios without prior knowledge or extra resources, however, an essential we draw from Sec. 4 is that model selection must form an integral part of training a DNN on SLTs. Thus, a wrong choice of hyper-parameters based on D 1 can be disastrous for the remaining sub-tasks, which is why application scenarios require DNN variants that do not have extreme dependencies on hyper-parameters such as layer number and layer sizes. Lastly, our findings indicate workarounds that would make EWC or IMM practicable in at least some application scenarios. If model selection is addressed, a small subset of D 1 may be kept in memory for both methods: to determine optimal values of α for IMM and to determine when to stop re-training for EWC. FIG7 shows that small changes to α do not dramatically impact final accuracy for IMM, and FIG4 indicates that accuracy loss as a function of re-training time is gradual in most cases for EWC. The inaccuracies introduced by using only a subset of D 1 would therefore not be very large for both algorithms. To conclude, this study shows that the consideration of applied scenarios significantly changes the procedures to determine CF behavior, as well as the as to its presence in latestgeneration DNN models. We propose and implement such a procedure, and as a consequence claim that CF is still very much of a problem for DNNs. More research, either on generic solutions, or on workarounds for specific situations, needs to be conducted before the CF problem can be said to be solved. A minor but important is that obtained on permutation-type SLTs should be treated with caution in future studies on CF. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. BID11. This is achieved by dividing the "best" measure from Tab. 3 by the baseline performance. Each table entry contains two numbers: the baseline performance and Ω all, and cell coloring (indicating presence or absence of CF) is performed based on Ω all. The overall picture is similar to the one from Tab. 3, as indicated by the cell coloring. A notable exception is the performance of the CONV and D-CONV models on the SVHN dataset, where Ω all shows an increase, but we do not consider this significant since the already the baseline performance is at chance level here. That is, this problem is too hard for the simple architectures we use, in which case a small fluctuation due to initial conditions will exceed baseline performance. We therefore conclude that Ω all is an important measure whenever baseline performance is better than random, in which case is it not meaningful. On the other hand, our measure works well for random baselines but is less insightful for the opposite case (as the presence of CF is not immediately observable from the raw performances. A combination of both measures might be interesting to cover both cases. Here, we present the best of all algorithms on the MNIST, EMNIST and Devanagari datasets (according to the "best" criterion) for the D9-1b SLT, and the best EWC on the D5-5d SLT (qualitatively identical to the other D5-5 type SLTs). Such 2D representations of some experimental , to be just as FIG4, may give more clear insights into the details of each experiment. Here we can observe CF behavior for all algorithms except EWC and IMM for D9-1b. We can infer that there was no discernible dependency between the occurrence of CF and particular hyperparameter settings (number and size of layers, in particular) since these are already the best experiments for each algorithm and dataset: if these show CF, this means that non of the settings we sampled were able to prevent CF. EWC shows clear CF for the Devanagari dataset, but might conceivably do better on EMNIST given a little more time for learning D 2 (this will be investigated). For D5-5d, clear CF occurs even for EWC. IMM does not exhibit CF for D9-1b (at enormous computations cost, though), and we observe that the value for the balancing parameter cannot simply be set to 0.9 respectively 0.1, as it has its argmax elsewhere. Mean-IMM;test:All Mode-IMM;test:All

We check DNN models for catastrophic forgetting using a new evaluation scheme that reflects typical application conditions, with surprising results.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization . 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These raise new questions for FL, MAML, and broader ML research. In recent years, the growth of machine learning applications was driven by aggregation of large amounts of data in a datacenter, where a model can be trained using large scale distributed system . Both the research community and general public are becoming increasingly aware that there is a variety of scenarios where this kind of data collection comes with significant risks, mainly related to notions of privacy and trust. In the presence of user generated data, such as activity on mobile phones, Federated Learning (FL) proposes an alternative approach for training a high quality global model without ever sending raw data to the cloud. The FL system proposed by Google selects a sample of available devices and sends them a model to be trained. The devices compute an update to the model based on an optimization procedure with locally available data, and the central system aggregates the updates from different devices. Such iteration is repeated many times until the model has converged. The users' training data does not leave their devices. The basic FL algorithm, Federated Averaging (FedAvg), has been used in production applications, for instance for next word prediction in mobile keyboard , which shows that Federated Learning can outperform the best model trained in a datacenter. Successful algorithmic extensions to the central idea include training a differential private model , compression (Konečný et al., 2016b; a), secure aggregation , and a smaller number of always-participating nodes . FL applications generally face non-i.i.d and unbalanced data available to devices, which makes it challenging to ensure good performance across different devices with a FL-trained global model. Theoretical guarantees are only available under restrictive assumptions and for convex objectives, cf. Li et al. (2019b). In this work, we are interested in personalization methods that adapt the model for data available on each device, individually. We refer to a trained global model as the initial model, and the locally adapted model as the personalized model. Existing FL personalization work directly takes a converged initial model and conducts personalization evaluation via gradient descent. However, in this approach, the training and personalization procedures are completely disconnected, which in potentially suboptimal personalized models. Meta Learning optimizes the performance after adaptation given few-shot adaptation examples on heterogeneous tasks, and has increasing applications in the context of Supervised Learning and Reinforcement Learning. Model Agnostic Meta Learning (MAML) introduced by is a solely gradient-based Meta Learning algorithm, which runs in two connected stages; metatraining and meta-testing. Meta-training learns a sensitive initial model which can conduct fast adaptation on a range of tasks, and meta-testing adapts the initial model for a particular task. Both tasks for MAML, and clients for FL, are heterogeneous. For each task in MAML and client in FL, existing algorithms use a variant of gradient descent locally, and send an overall update to a coordinator to update the global model. If we present the FL training process as meta-training in the MAML language, and the FL personalization via gradient descent as meta-testing, we show in Section 2 that FedAvg and Reptile , two popular FL and MAML algorithms, are very similar to each other; see also. In order to make FL personalization useful in practice, we propose that the following objectives must all be addressed, simultaneously. Improved Personalized Model -for a large majority of the clients. Solid Initial Model -some clients have limited or even no data for personalization. Fast Convergence -reach a high quality model in small number of training rounds. Typically, the MAML algorithms only focus on objective; that was the original motivation in. Existing FL works usually focus on objectives and, and take the personalized performance as secondary. This is largely due to the fact that it was not obvious that getting a solid initial model is feasible or practical if devices are available occasionally and with limited resources. In this work, we study these three objectives jointly, and our main contributions are: • We point out the connection between two widely used FL and MAML algorithms, and interpret existing FL algorithm in the light of existing MAML algorithms. • We propose a novel modification of FedAvg, with two stages of training and fine-tuning, for optimizing the three above objectives. • We empirically demonstrate that FedAvg is already a meta learning algorithm, optimizing for personalized performance, as opposed to quality of the global model. Furthermore, we show that the fine tuning stage enables better and more stable personalized performance. • We observe that different global models with the same accuracy, can exhibit very different capacity for personalization. • We highlight that these challenge the existing objectives in the FL literature, and motivate new problems for the broader Machine Learning research community. In this section, we highlight the similarities between the FL and MAML algorithms and interpret FedAvg as a linear combination of a naive baseline and a collection of existing MAML methods. Algorithm 1 presents a conceptual algorithm with nested structure (left column), of which the MAML meta-training algorithm, Reptile (middle column), and FL-training algorithm, FedAvg (right column), are particular instances. We assume that L is a loss function common to all of the following arguments. In each iteration, a MAML algorithm trains across a random batch of tasks {T i}. For each task T i, it conducts an inner-loop update, and aggregates gradients from each sampled task with an outer-loop update. In each training round, FL uses a random selection of clients {T i}. For each client T i and its weight w i, it runs an optimization procedure for a number of epochs over the local data, and sends the update to the server, which aggregates the updates to form a new global model. If we simplify the setting and assume all clients have the same amount of data, causing the weights w i to be identical, Reptile and FedAvg in fact become the same algorithms. Several other MAML algorithms , or other non-MAML/FL methods , can also be viewed as instances of the conceptual method in left column of Algorithm 1. function ClientU pdate(θ, Ti, β) Split local dataset into batches B θi = θ for each local epoch i from 1 to E do for batch b ∈ B do θi = θi − β∇ θ L(θi, b) end for end for Return gi = θi − θ end function Require: Clients per training round M. function ServerU pdate(θ,{gi, wi}, α) In the following, we rearrange the summands comprising the update formula of FedAvg/Reptile algorithm to reveal the connection with other existing methods -a linear combination of the Federated SGD (FedSGD) and First Order MAML (FOMAML) algorithms with different number of steps. For clarity, we assume identical weights w i in FedAvg. Consider T participating clients and let θ be parameters of the relevant model. For each client i, define its local loss function as L i (θ), and let g i j be the gradient computed in j th iteration during a local gradient-based optimization process. FedSGD was proposed as a naive baseline against which to compare FL algorithms. For each client, it simply takes a single gradient step based on the local data, which is sent back to the server. It is a sensible baseline because it is a variant of what a traditional optimization method would do if we were to collect all the data in a central location, albeit inefficient in the FL setting. That means that FedSGD optimizes the performance of the initial model, as is the usual objective in datacenter training. The local update produced by FedSGD, g F edSGD, can be written as Next, we derive the update of FOMAML in similar terms. Assuming client learning rate β, the personalized model of client i, obtained after K-step gradient update is θ. Differentiating the client update formula, we get Directly optimizing the current model for the personalized performance after locally adapting K gradient steps, in the general MAML update proposed by. MAML requires to compute 2nd-order gradients, which can be computationally expensive and creates potentially infeasible memory requirements. To avoid computing the 2nd-order term, FO-MAML simply ignores it, ing in a first-order approximation of the objective . F OM AM L(K) then uses the (K + 1) th gradient as the local update, after K gradient steps. Now we have derived the building blocks of the FedAvg. As presented in Algorithm 1, the update of FedAvg, g F edAvg, is the average of client updates, which are the sums of local gradient updates. Rearranging the terms presents its interpretation as a linear combination of the above ideas. Note that interpolating to special case, g F edSGD can be seen as g F OM AM L -optimizing the performance after 0 local updates, i.e., the current model. This sheds light onto the existing Federated Averaging algorithm, as the linear combination of algorithms optimizing personalized performance after a range of local updates. Note, however, this does not mean that FedAvg optimizes for the linear combination of the objectives of the respective algorithms. Nevertheless, we show in the following section that using K = 1 in a model hard to personalize, and increasing K significantly improves the personalization performance, up until a certain point where the performance of initial model becomes unstable. In this section, we present Personalized FedAvg algorithm, which is the of experimental adaptation of the core FedAvg algorithm to improve the three objectives proposed in the introduction. We denote F edAvg(E) the Federated Averaging method from Algorithm 1, right, run for E local epochs, weighting the updates proportionally to the amount of data available locally. We denote Reptile(K) the method from Algorithm 1, middle, run in the FL setting for K local steps, irrespective of the amount of data available locally. Based on a variety of experiments we explored, we propose Personalized FedAvg in Algorithm 2. Algorithm 2 Personalized FedAvg 1: Run F edAvg(E) with momentum SGD as server optimizer and a relatively larger E. 2: Switch to Reptile(K) with Adam as server optimizer to fine-tune the initial model. 3: Conduct personalization with the same client optimizer used during training. In general, FedAvg training with several local epochs ensures reasonably fast convergence in terms of number of communication rounds. Due to complexity of production system, this measure was studied as the proxy for convergence speed of FL algorithms. We find that this method with momentum SGD as the server optimizer already optimizes for the personalized model -objective form the introduction -while the initial model -objective -is relatively unstable. Based on prior work, the recommendation to address this problem would be to decrease E or the local learning rate, stabilizing initial model at the cost of slowing down convergence ) -objective. We propose a fine-tuning stage using Reptile(K) with small K and Adam as the server optimizer, to improve the initial model, while preserving and stabilizing the personalized model. We observed that Adam yields better than other optimizers, see Table 3 in Appendix A.1, and makes the best personalized performance achievable with broader set of hyperparameters, see Figure 2. The subsequent deployment and personalization is conducted using the same client optimizer as used for training, as we observe that this choice yields the best for FedAvg/Reptile-trained models. Experimental setup. We use the EMNIST-62 dataset as our primary benchmark (b). It is the original source of the MNIST dataset, which comes with author id, and noticable variations in style, pen width, size, etc., making it a suitable source for simulated FL experiments. The dataset contains 3400 users, each with a train/test data split, with a total of 671, 585 train and 77, 483 test images. We choose the first 2, 500 users as the initial training clients, leaving the remaining 900 clients for evaluation of personalization; these clients are not touched during training. The evaluation metrics are the initial and personalized accuracy, uniformly averaged among all of the FL-personalization clients. This is preferred to a weighted average, as in a production system we care about the future performance on each device, regardless of the amount of data available for personalization. Unless specified otherwise, we use the baseline convolutional model available in TensorFlow Federated 1, using SGD with learning rate 0.02 and batch size of 20 as the client optimizer, and SGD with momentum of 0.9 and learning rate 1.0 as the server optimizer. Each experiment was repeated 9 times with random initialization, and the mean and standard deviation of initial and personalized accuracies are reported. We also use the shakespeare dataset for next-character prediction, split in a similar manner with first 500 clients used for training and remaining 215 for evaluation of personalization. In Figure 1, left, we present the convergence of both initial and personalized model during training using the Federated Averaging algorithm. The correspond to training with E being 2 and 10, with visualization of the empirical mean and variance observed in the 9 replicas of the experiment. Detailed values about performance after 500 rounds of training, and the number of rounds to reach 80% accuracy, are provided in Table 1. These provide a number of valuable insights. First, the personalized accuracy converges significantly higher than the initial accuracy. This clearly validates the EMNIST-62 as an interesting simulated dataset to use to study Federated Learning, with significantly non-i.i.d. data available to each client. Second, it provides empirical support to the claim made in Section 2, that Federated Averaging is already a Meta Learning algorithm. The personalized accuracy not only converges faster and higher, but the are also of much smaller variance than those of the initial accuracy. Third, the personalized accuracy after training with E = 10 is significantly higher than the personalized accuracy after training with E = 2. This is despite the fact that the gap in the initial accuracy between these two variants is somewhat smaller. Moreover, the variance of personalized accuracy is nearly 3-times smaller for training with E = 10, compared to E = 2, despite the variance of the initial accuracy being smaller for the E = 2 case. This supports the insight from Equation 5, that Federated Averaging with more gradient steps locally should emphasize the personalized accuracy more, potentially at the cost of initial accuracy. Finally, this challenges the objectives in the existing literature focusing on the setting of Federated Learning. FedAvg was presented as optimizing the performance of a shared, global model, while in fact it might achieve this only as a necessary partial step towards optimizing the personalized performance. We argue that in the presence of non-i.i.d. data available to different clients, the objective of Federated Learning should also be the personalized performance. Consequently, the recommendations that in order to stabilize the convergence, one might decrease the number of local epochs or the local learning rate ), are in some scenarios misguided. In this experiment, even though the initial accuracy is very noisy and roughly constant at a suboptimal level, the personalized accuracy keeps increasing. To evaluate the convergence speed more closely, Table 1 measures the accuracies after 500 rounds of training and the average number of communication rounds where the initial and personalized accuracy first time reaches 80%. While Figure 1 shows using 5 clients per round, the table below also shows the same experiment with 20 clients per round, which in general provides even better and more stable personalized accuracy. The common pattern is that increasing E initially helps, until a certain threshold. From this experiment, E in the range of 5 − 10 seems to be the best. We conduct a similar experiment with the Shakespeare data. The is in Figure 1, improvement. We conjecture that this is due to the nature of the objective -even though the data is non-i.i.d., the next-character prediction is mostly focused on a local structure of the language in general, and is similar across all users. 2 We thus do not study this problem further in this paper. It is likely that for a next-word prediction task, personalization would make a more significant difference. In this section, we study the ability of models to personalize more closely. In particular, we look at the personalization performance as a function of the number of local epochs spent personalizing, and the effect of both local personalization optimizer and the optimizer used to train the initial model. In Figure 2, left, we study the performance of three different models, personalized using different local optimizers. The models we test here are: one randomly chosen from the models trained with E = 10 in the previous section, with initial accuracy of 74.41%. The other two models are the of fine-tuning that specific model with Reptile and Reptile and Adam as the server optimizer for further 200 communication rounds, again using 5 clients per round. For all three models, we show the of personalization using Adam with default parameters and using SGD with learning rate of 0.02 and batch size of 100. In all three cases, Adam produces reasonable personalized , but inferior to the of SGD. We tried SGD with a range of other learning rates, and in all cases we observed this value to work the best. Note that this is the same local optimizer that was used during training and fine tuning, which is similar to the MAML works, where the same algorithm is user for meta-training and meta-testing. The effect of fine-tuning is very encouraging. Using Reptile, we get a model with better initial accuracy, which also gets to a slightly improved personalized accuracy. Most importantly, it gets to roughly the same performance for a wide range of local personalization epochs. Such property is of immense value for pracical deployment, as only limited tools for model quality validation can be available on the devices where personalization happens. Using Reptile significantly improves the initial accuracy, but the personalized accuracy actually drops! Even though the Equation 5 suggests that this algorithm should not take into account the personalized performance, it is not clear that such should be intuitively expected -it does not mean it actively supresses personalization, either, and it is only fine tuning of a model already trained for the personalized performance. We note that this can be seen as an analogous observation to those of , where Reptile yields a model with clearly inferior personalized performance. Motivated by this somewhat surprising , we ask the following question: If the training data were available in a central location, and an initial model was trained using standard optimization algorithms, how would the personalization change? We train such "centralized initial model" using Adam with default settings, and evaluate the personalization performance of a model snapshot after 10 and 50 training epochs. These two models have similar initial accuracies as the two models fine tuned with Reptile and Reptile, respectively. The are in Figure 2, right. The main message is that it is significantly harder to personalize these centralized initial models. Using SGD with learning rate of 0.02 is not good -a significantly smaller learning rate is needed to prevent the models from diverging, and then the personalization improvement is relatively small. In this case, using Adam does provide a better , but still below the fine tuning performance in Figure 2, left. It is worth noting that the personalized performance of this converged model is similar to that of the model we get after fine tuning with Reptile, although using different personalization optimizer. At the moment, we are unable to suggest a sound explanation for this similarity. At the start of the Section 3, we recommended using Adam as the server optimizer for fine tuning, and here we only presented such . We did try different optimizers, and found them to yield worse with higher variance, especially in terms of the initial accuracy. See Appendix A.1 for more details, where we see that all of the optimizers can deliver higher initial accuracy at the cost of slightly lower personalized accuracy. To strengthen the above observations, we look at the distribution of the initial and personalized accuracies of fine tuned models over multiple experiments. In Figure 3, we look at the same three models as in Figure 2. It is clear that the initial model has a large variance in the initial accuracy, the are in the range of 12%, but the personalized accuracies are only within the range of 1%. We chose one model, as indicated by the arrows 3, to be fine tuned with Reptile and Reptile. In both cases, fine tuning in a more consistent in both the initial and personalized accuracy. Moreover, the best personalized accuracy with Reptile is worse than the worst personalized accuracy with Reptile. In Appendix A.2, we look at a similar visualization on a per-client basis for a given model. Studying this distribution is of great importance, as in practical deployment, even a small degradation in a user's experience might incur disproportionate cost, relative to the benefit of a comparable improvement in the model quality. We do not study this question deeper in this work, though. Finally, we look at the performance of fine tuned models discussed above on both train and test clients. Table 2 shows that if we only looked at the initial accuracy, basic ML principles would suggest the Reptile models are over-fitting, due to larger gap between the train and test clients. However, the personalized accuracy tells a different story -the gap is roughly the same for both model types, and for both train and test clients, Reptile provides significantly better personalized accuracy, suggesting we need a novel way to predict the generalization of personalized models. It this work, we argue that in the context of Federated Learning, the accuracy of the global model after personalization should be of much greater interest than it has been. Investigation of the topic reveals close similarities between the fields of Federated Learning and Model Agnostic Meta Learning, and raises new questions for these areas, as well as for the broader Machine Learning community. Challenges for Federated Learning. Framing papers in the area of Federated Learning Konečný et al., 2016a; a), formulate the objective as training of a shared global model, based on a decentralized data storage where each node / client has access to a non-i.i.d sample from the overall distribution. The objective is identical to one the broader ML community would optimize for, had all the data been available in a centralized location. We argue that in this setting, the primary objective should be the adaptation to the statistical heterogeneity present at different data nodes, and demonstrate that the popular FL algorithm, Federated Averaging, does in fact optimize the personalized performance, and while doing so, also improves the performance of the global model. Experiments we perform demonstrate that the algorithm used to train the model has major influence on its capacity to personalize. Moreover, solely optimizing the accuracy of the global model tends to have negative impact on its capacity to personalize, which further questions the correctness of the commonly presented objectives of Federated Learning. Challenges for Model Agnostic Meta Learning. The objectives in the Model Agnostic Meta Learning literature are usually only the model performance after adaptation to given task . In this work, we present the setting of Federated Learning as a good source of practical applications for MAML algorithms. However, to have impact in FL, these methods need to also consider the performance of the initial model, 4 as in practice there will be many clients without data available for personalization. In addition, the connectivity constraints in a production deployment emphasize the importance of fast convergence in terms of number of communication rounds. We suggest these objectives become the subject of MAML works, in addition to the performance after adaptation, and to consider the datasets with a natural user/client structure being established for Federated Learning (b) as the source of experiments for supervised learning. Challenges for broader Machine Learning. The empirical evaluation in this work raises a number of questions of relevance to Machine Learning research in general. In particular, Figure 2 clearly shows that models with similar initial accuracy can have very different capacity to personalize to a task of the same type as it was trained on. This observation raises obvious questions for which we currently cannot provide an answer. How does the training algorithm impact personalization ability of the trained model? Is there something we can measure that will predict the adaptability of the model? Is it something we can directly optimize for, potentially leading to novel optimization methods? These questions can relate to a gap highlighted in Table 2. While the common measures could suggest the global model is overfitting the training data, this is not true of the personalized model. Transfer Learning is another technique for which our could inspire a novel solution. It is very common for machine learning practitioners to take a trained model from the research community, replace the final layer with a different output class of interest, and retrain for the new task . We conjecture that the algorithms proposed in the FL and MAML communities, could yield base models for which this kind of domain adaptation would yield better . Finally, we believe that a systematic analysis of optimization algorithms of the inner-outer structure presented in Algorithm 1 could provide novel insights into the connections between optimization and generalization. Apart from the FL and MAML algorithms, recently proposed a method that can be interpreted as outer optimizer in the general algorithm, which improves the stability of a variety of existing optimization methods used as the inner optimizer. A APPENDIX This Appendix contains further details referenced from the main body of the paper. Table 3 summarizes the attempts at fine tuning the model user in main body with different server optimizers. We see that comparing the same client optimizers, Adam consistently provides better and more stable in terms of initial accuracy. A.2 PER-CLIENT PERSONALIZATION Figure 4 visualizes the distribution of initial and personalized accuracies on a per-client basis. Each dot represents a random sample of the test clients used for personalization experiments. Studying this distribution is of great importance, as in practical deployment, degrading a user's experience might incur disproportionate cost, compared to the benefit of comparable improvement. Designing methods that robustly identify the clients below the diagonal line and at least revert to the initial model is worth of future investigation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Federated Averaging already is a Meta Learning algorithm, while datacenter-trained methods are significantly harder to personalize.

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Memorization of data in deep neural networks has become a subject of significant research interest. In this paper, we link memorization of images in deep convolutional autoencoders to downsampling through strided convolution. To analyze this mechanism in a simpler setting, we train linear convolutional autoencoders and show that linear combinations of training data are stored as eigenvectors in the linear operator corresponding to the network when downsampling is used. On the other hand, networks without downsampling do not memorize training data. We provide further evidence that the same effect happens in nonlinear networks. Moreover, downsampling in nonlinear networks causes the model to not only memorize just linear combinations of images, but individual training images. Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important phenomenon of memorization in over-parameterized deep networks. As deep convolutional neural networks (CNNs) become ubiquitous in computer vision due to their applicability and strong performance on a range of tasks BID6, recent work has begun analyzing the memorization properties of such networks in classification. For example, BID19 show that popular CNNs can achieve almost zero training error on randomly labeled datasets, indicating that CNNs have the capacity to "memorize" large training data sets. BID0 and BID15 build on the experiments from BID19 to better understand and evaluate the extent to which CNNs memorize training data. BID0 show that CNNs, when trained on large datasets, are able to learn patterns from realistic data before memorizing training images. BID15 present experiments on "membership inference" (i.e. determining whether an image was used during training) and conclude that modern architectures are capable of "remember[ing] a large number of images and distinguish [ing] them from unseen images".Although the above methods analyze memorization in the classification setting, they do not provide a mechanism through which memorization of training data occurs. We here present downsampling as one mechanism by which deep CNNs memorize specific training images. We will focus our study on the memorization properties of linear and nonlinear fully convolutional autoencoders. The architectures we use (such as U-Net, BID14) are commonly employed in imageto-image tasks, see e.g. BID17. However, we will use these architectures only in the autoencoding framework. We primarily focus on autoencoders BID1 for the following reasons: components of convolutional autoencoders are building blocks of many CNNs; and layerwise pre-training using autoencoders is a technique to initialize individual layers of CNNs to improve training BID3, BID4 ). It is important to note that there are many potential solutions to the autoencoding problem when using over-parameterized autoencoders. In particular, in the linear case, these models may range from learning the (full rank) identity function (which has 0 error in the autoencoding task) to low rank solutions where each training example corresponds to an eigenvector with eigenvalue 1. Thus, understanding how autoencoders learn is of interest in order to gain insights into how deep CNNs memorize training data. Figures 1a and 1b provide two examples of memorization: A typical U-Net architecture (the same as e.g. used in BID17 for large hole impainting) when trained on a single image "memorizes" the training image in the sense that for any input, the output always contains the training image (even if the input is random noise or an arbitrary white square). This paper provides a mechanism for this phenomenon. The outline is as follows: After introducing some notation in Section 2, we will show in Section 3 that memorization is tightly coupled with downsampling and also occurs in the simpler setting of linear autoencoding CNNs. In the linear setting, the neural network corresponds to matrix multiplication. In Section 4, we show how to extract this matrix representation and we provide our main conjecture, namely that linear combinations of the training images are stored as eigenvectors of this matrix, whose rank is given by the dimension of the span of the training set. We also provide strong evidence for this conjecture on 2 × 2 images. In Section 5, we analyze the eigenvalue decay and show in various examples that using downsampling linear CNNs, linear combinations of the training examples are stored as eigenvectors with eigenvalues close to 1. Finally, we return to the nonlinear setting in Section 6, providing evidence that memorization is an even stronger phenomenon in nonlinear networks, since the actual training images (in contrast to linear combinations of training images) are memorized. We end with a short discussion in Section 7. In this section, we introduce the mathematical framework for our work and highlight two different functions learned by autoencoding CNNs, namely the identity function and the point map. We denote a training set of n square images by X = {x 1, x 2, ..., x n}, where each x i ∈ R c×s×s with c being the number of color channels (or filter channels) and s denoting the width and height of an image. We will focus on convolutional autoencoders, i.e., CNNs trained to map between the same image. In particular, we will consider linear CNNs, by which we mean convolutional autoencoders with layers being either nearest neighbor upsampling or convolutional with kernel size 3, zero padding, no activation functions, and no biases. To simplify the computations in Section 4, we assume throughout that the input image as well as the stride size are a power of 2.We denote the function learned by a CNN on the training set X by C X: R c×s×s → R c×s×s. The training procedure minimizes a given loss function between the input image and its reconstruction by C X. We use the mean squared error loss throughout; thus the loss function is given by DISPLAYFORM0 2, where subtraction and exponentiation are taken elementwise. For linear CNNs, we denote the matrix corresponding to C X by A X. Denoting the vectorized (or "flattened") version of an image y ∈ R c×s×s by y f ∈ R cs 2, then A X satisfies A X y f = (C X (y)) f.In this work, we identify and analyze an architectural mechanism that is able to fundamentally alter the function C X learned by an autoencoding CNN on a given training set. The following two functions will play an important role in the subsequent analysis: the identity function given by C X (y) = y for any y ∈ R c×s×s and the point map given by C X (y) = C X (x 0) for any y ∈ R c×s×s, where x 0 is a particular element in span C X (X) where span C X (X) = span {C X (x i), 1 ≤ i ≤ n}. An extreme form of memorization occurs when a CNN learns the point map on a training set of size one, i.e., it maps any image to the same fixed image. In this section, we will illustrate how downsampling acts as a mechanism for learning the point map in deep CNNs. In particular, we will show that even if a downsampling network has the capacity to learn the identity function, it prefers to learn the point map. We consider the linear CNNs defined by Network ND (non-downsampling) and Network D (downsampling) in FIG2. Both networks employ 128 filters in each convolutional layer in all but the last layer, which contains 3 filters. The two networks only differ by the property that Network D uses filters of stride 8 in the first layer to downsample the image and then later uses nearest neighbor upsampling with scale factor 8 to rescale the image, while Network ND does not perform any downsampling. In concordance with the severe downsampling from 224 × 224 to 7 × 7 images performed by ResNet and VGG BID9; BID16 ), Network D downsamples from 32 × 32 to 4 × 4 images. However, we will show in Section 5 that the same memorization phenomenon is also observed in networks that only downsample using multiple convolutional layers of stride 2 as is done in U-Nets BID14 ). In the examples, the networks were initialized using the default in the PyTorch deep learning library BID13, namely each layer's weights were drawn i.i.d. from a uniform distribution U(−a, a) with a = 1/ √ P where P is the number of parameters in that layer. Both networks were trained for 2000 iteration on a single 3 × 32 × 32 image from CIFAR10 BID12 ) of a frog. Network ND reconstructed the digit 3 with a training loss of 10 −4 and Network ND with loss 10 −2. We then applied the trained networks to test images consisting of both realistic images and random images with each pixel in the image being drawn from the standard normal distribution. As shown in FIG2 and 2b, although trained on the same input for the same number of iterations Network D (with downsamling) learned the point map while Network ND (no downsampling) learned the identity function up to sign. The possible sign flip is a consequence of using no biases. In fact, this contrasting behavior is very robust throughout the training process. In FIG4, we see that regardless of when we stop training, even after only 10 iterations, Network D learns a point map, while Network ND learns a function visually similar to the identity map. One may conjecture that the memorization phenomenon that occurs using Network D could be due to the small number of filters in the downsampling layer, so that not all the features from the input could be preserved and the network would not be capable of learning a full rank identity map. This is not the case. To demonstrate that this is in general not the reason why downsampling networks learn a point map, we next present a downsampling network which, as we show, has the capacity to learn the identity function, but still prefers to learn the point map instead as the of optimization. Example. Consider the linear CNN defined by the small downsampling Network DS presented in FIG22, which is trained on a single 1×2×2 image for 2000 iterations. Figure 4b shows that there exists an initialization such that the network learns the identity function. A full description of how we computed this manual initialization is provided in Appendix A. In contrast, as shown in Figure 4c initializing Network DS with the default PyTorch uniform distribution in the point map. This provides the first example showing that even though the network has the capacity to learn the identity function, it prefers to learn a point map using the default PyTorch initialization. We observed the same effects using the Xavier uniform initialization, which is based on the assumption that activations are linear BID5 ). However, the are not observed for linear networks when using Kaiming initialization, which is expected since this initialization is meant for nonlinear networks with ReLU/PReLU activations BID8 ). In Appendix E, we show that nonlinear networks memorize training images under any of these initializations. We now turn to extracting and analyzing the linear operator that describes the function learned by the linear CNN. While the full algorithms for extracting the matrix A X from the network are described in Appendix B, we here provide intuition for this procedure. Since the matrix A X can be decomposed as a product of matrices, one for each layer, it suffices to provide intuition for converting a general convolutional layer and a general upsampling layer into a matrix. We then discuss our main conjecture and proposition linking the eigenvectors of A X to the mechanism of memorization. To simplify notation, we assume that the inputs and outputs are already zero padded images (with 1 layer of zero padding on each side), i.e., the original images lie in R c×(s−2)×(s−2) such that the images after vectorization and zero padding lie in R stride size t, kernel volume f × 3 × 3, and 1 layer of zero padding, which operates on an incoming zero padded vectorized image of f s 2 voxels, where f is the depth and s the width and height of the incoming image. Assuming that the stride and image sizes are powers of 2, it follows that (s −2) is a multiple of t. Hence the matrix corresponding to the operator of this particular layer is of size f (DISPLAYFORM0 t + 2 corresponds to the width and height of the output after striding and zero padding. The matrix itself is circulant-like, where the zero padding must be carefully accounted for by additional shifts and rows of zeros. A particular example is shown in Appendix B.Next, we provide intuition on how to extract the matrix corresponding to the nearest neighbor upsampling operation. The full algorithm together with an example is given in Appendix B. Given a vectorized zero-padded image of f s 2 voxels as input, then nearest neighbor upsampling with scale factor k corresponds to a matrix of size f (k(s − 2) + 2) 2 × f s 2, where (k(s − 2) + 2) comes from having to scale only the non-zero-padded elements of the representation and adding 2 for the new zero padding. This matrix is composed of blocks of circulant-like matrices consisting of one-hot vectors, where blocks of s identical rows are shifted by 1 element. Finally, the full operator A X for a linear network C X is obtained by multiplying the matrices corresponding to each layer. Being able to extract the linear operator A X from a linear CNN is critical for analyzing the mechanism of memorization, since it allows an examination of its eigenvalues and eigenvectors. Note that A X operates on the space of c × s × s images. Hence every eigenvector (when stripped of zero-padding and reshaped to c × s × s) represents an image in this space. However, since A X is in general not symmetric, eigenvectors can be complex and hence the real and imaginary components of the eigenvectors represent separate images in the original space. We can easily obtain a bound on the rank of A X: Since A X is given by a product of matrices, one for each layer in the linear CNN, the rank of A X is at most equal to the rank of the minimal rank matrix in the factorization of A X. We denote this number by r min. We now present our main conjecture. Conjecture. Let X = {x 1, . . ., x n}, x i ∈ R c×s×s, be a training set of images of size c × s × s.(a) If A X is the linear operator of a downsampling network trained to zero loss, then DISPLAYFORM0 where all eigenvalues are equal to 1 and the corresponding eigenvectors are linear combinations of the input training images.(b) If A X is the linear operator of a non-downsampling network trained to zero loss, then DISPLAYFORM1. In other words, we conjecture that linear downsampling networks learn a low rank solution when the number of linearly independent training examples is not sufficiently large (i.e. larger than cs 2), even when they have the capacity to learn the identity function. In particular, we conjecture that the rank is given by the dimension of the span of the training images. Most importantly, we conjecture that the mechanism of memorization in downsampling networks is by storing linear combinations of the training images as eigenvectors of the linear operator. On the other hand, we conjecture that linear non-downsampling networks learn a much higher rank solution, thereby often rendering the solution visually indistinguishable from the identity function. Our conjecture also implies that when training a linear downsampling CNN on images of size 3 · 224 · 224, which corresponds to the input image size for VGG and ResNet BID9, BID16 ), the number of linearly independent training examples needs to be at least 3 · 224 · 224 = 153, 228 before the network can learn the identity function. However, assuming that realistic images lie on a low-dimensional manifold, this conjecture indicates that a linear downsampling network will learn a basis for the low-dimensional manifold instead of the identity when trained on sufficiently many realistic images. Before providing empirical evidence for our conjecture in Section 5, we end this section by summarizing our theoretical evidence in the following proposition showing that the network in FIG22 can learn solutions of all ranks 1 through 4 for 1 × 2 × 2 images depending on the dimension of the span of the training set. Proposition. The linear Network DS presented in FIG22 can learn a linear operator of rank min (dim(span (X), 4)) with 1 ≤ dim(span (X)) ≤ 4.The proof is given in Appendix C. Note that this proposition does not imply that A X must obtain rank r = min (dim (span (X), 4)) for all training sets X on 1 × 2 × 2 images, but rather that the conjecture holds for any example of training sets that we tried, in particular also training sets covering each possible nonzero value of min (dim (span (X), 4)). We now provide empirical evidence for our main conjecture by analyzing the operator A X for networks trained on CIFAR10 color images (which have width and height 32). First, we show that when training on one or two images respectively, a downsampling network learns an operator that has very low effective rank with the top eigenvector(s) corresponding to the training image(s). On the other hand, we show that a non-downsampling network trained on the same data learns an operator with much higher rank, closer to the full dimensionality of the data. Finally, we will train downsampling networks on tens to thousands of images from a single class of CIFAR10 and present the rank of the learned solutions in Appendix D.All of our downsampling networks downsample using 3 convolutional layers with stride 2 and then upsample using a 1 nearest neighbor upsampling layer with scale factor 8. The general architecture is shown in FIG6. Importantly, we note that if we use such a network on color CIFAR10 images, we need at least b = 3·8·8 = 192 filters in order to be able to learn the identity function since otherwise r min < 3 · 32 · 32. Our non-downsampling networks have the same convolutional layer scheme as the downsampling networks. We denote a downsampling (respectively non-downsampling) network with a filters on upsampled representations and b filters on downsampled representations by D a,b (respectively N D a,b).The networks are initialized using the default initialization in PyTorch BID13 ), trained using the Adam optimizer BID11 ) with learning rate 10 −4, and trained until the loss decreased by less than 10 −4 or for 20, 000 epochs. Note that we disregard overfitting in training, since we expect an overfitted autoencoder to learn the identity function. As pointed out in FIG4, the memorization effect is robust throughout the training process. However, in this case, we wish to train until the training error is sufficiently low (i.e. around 10 −3) so that we can understand the rank of the operator in relation to our conjecture. FIG8 shows the for various downsampling and non-downsampling networks when trained on 1 ≤ k ≤ 2 images. As shown in FIG8, in both cases Network D 64,192 stores the k images (up to a shift in the color channels) as eigenvectors with eigenvalues close to 1 and produces roughly a rank k solution. In particular, when trained on a single picture of a plane from CIFAR10 this network stores the training image as the real component of the top eigenvector; it is an inverted color image of the plane and the rank of the learned solution is approximately 1 FIG8. When trained on a picture of the plane and a car, the top two eigenvectors are linear combinations of the plane and the car with the plane having inverted colors FIG8 ). In fact, the plane is more visible in the complex component of the second eigenvector while the car is more visible in the real components. The learned solution is of rank approximately 2 with the top two eigenvalues being complex conjugates (hence having the same magnitude). On the other hand, FIG8 show that training the non-downsampling network N D 4,4 on a single image or on two images in a higher-rank solution, with rank and spectrum being independent of k. When training on k > 2 images from different classes of CIFAR10, our conjecture stipulates that the rank of the learned solution indicates the dimension of the span of the training set. Since visu- alization of the eigenvectors when trained on k > 2 images is not as insightful (it becomes difficult to visually identify the training images in the linear combination), the corresponding visualizations and eigenvalue plots are moved to Appendix D. In this section, we analyze the phenomenon of memorization for non-linear networks. In the following experiments, we use the linear CNNs D a,b and N D a,b as the backbone architecture but with LeakyReLU activations BID18 after every convolutional layer. We denote the - For FIG12 we trained the non-linear downsampling network N LD 128,128 on a set of 10 images, one from each class of CIFAR10. Interestingly, when fed in a new image from CIFAR10, Gaussian noise or artificial white squares, the output was always one of the training images. This suggests that the effect of memorization is even stronger for non-linear downsampling networks, since they output specific training examples as compared to linear combinations of training examples, which we had observed for linear downsampling networks. In other words, the individual training examples act as strongly attracting fixed points for non-linear downsampling networks. We end this section with a remark on the impact of different initializations on training. While the experiments shown in FIG10 were initialized in the default uniform distribution from PyTorch, Appendix E shows with different initializations. In particular, we show that memorization still occurs when training N LD a,b with either Xavier or Kaiming uniform/normal initializations. This paper identified downsampling as a mechanism through which linear CNNs memorize training images. We demonstrated that downsampling convolutional autoencoders memorize training images in both the linear and nonlinear setting. In particular, we showed that it is not just the dimensionality reduction of downsampling that causes these models to learn point maps by demonstrating that a downsampling CNN architecture with the capacity to learn the identity function still prefers the point map. In the linear case, this preference for low-rank over the equally valid high-rank solutions is highly suggestive of similar phenomena observed in problems such as matrix completion (e.g., Gunasekar et al.).In the non-linear case, memorization in downsampling networks is manifested even more strikingly with nearly arbitrary input images being mapped to output images that are visually identifiable as one of the training images. While the exact mechanism still needs to be explored, this is reminiscent of FastICA in Independent Component Analysis BID10 or more general non-linear eigen-problems BID2, where every "eigenvector" for certain iterative maps has its own basin of attraction. On the other hand, non-downsampling auto-encoders do not memorize the training data and consistently learn a "high rank" map, similar to the identity map, at least visually. We conjecture that our findings will help to shed light on the strong generalization properties of downsampling networks for image classification and recognition tasks. Indeed, if downsampling networks memorize images or linear combinations of images, when trained on large datasets, they may be capable of learning representations within the space of all realisitic images instead of learning the standard full rank basis. We conclude with a mention of further areas of exploration spurred on by our work. We still need to understand why downsampling forces the network to learn low rank solutions even when the network has the capacity to learn the identity. This requires developing a better grasp of optimization and initialization, starting with linear autoencoders and proceeding to the non-linear settings. Finally, we need to explore connections between our conjecture and the manifold hypothesis to better understand the space of realistic images. Here we provide the initialization of the identity function for the network from FIG22. The procedure is as follows (the positions are 0 indexed):1. There are 4 filters in the first layer each of size and with stride 2. Filter 1 should have indices set to 1. Filter 2 should have indices set to 1. Filter 3 should have indices set to 1. Filter 4 should have indices set to 1.2. Now nearest neighbor upsampling will be applied to each of the outputs. 6. The last filter is of size. This filter should have indices (i, 1, 1) set to 1 for 0 ≤ i ≤ 3. In this section, we present algorithms for converting convolutional layers and nearest neighbor upsampling layers into matrices. We first present how to construct a block of this matrix for a single filter in Algorithm 1. To construct a matrix for multiple filters, one need only apply the provided algorithm to construct separate matrix blocks for each filter and then concatenate them. We now provide an example of how to convert the first layer from the network in FIG22 into a single matrix for 1 × 2 × 2 images. First suppose we have a zero padded 1 × 2 × 2 matrix as input, which is shown vectorized to the right: DISPLAYFORM0 Now we apply the first convolutional layer to this 16 × 1 vector. We have 4 convolutional filters with parameter C (j) i denote the ith parameter (read in row-major order) from filter j for 1 ≤ i ≤ 9 and 1 ≤ j ≤ 4. The ing convolutional matrix is of size 4 · 3 · 3 × 1 · 4 · 4, but here we show just the first 9 columns (i.e. the pattern for the ith filter): DISPLAYFORM1 DISPLAYFORM2 We similarly create matrices for the other convolutional layers. Next we present how to construct the rows of such a matrix in Algorithm 2. for f ilterIndex ← 0 to f − 1 do 6:for kernelIndex ← 0 to 8 do 7:rowIndex ← kernelIndex mod 3 + paddedSize C ← zeros matrix of size ((resized + 2) 2, f · paddedSize 2 )12:index ← resized + 2 + 1 for shif t ← 0 to resized − 1 do nextBlock ← zeros matrix of size (resized, f · paddedSize 2)15: DISPLAYFORM0 for rowShif t ← 1 to resized − 1 do return C 24: end functionWe now provide the upsampling matrix for an upsampling layer with scale factor 2 operating on a vectorized zero padded 1 × 1 × 1 image: DISPLAYFORM1 DISPLAYFORM2 We now give an example for the network from FIG4 when it is applied to vectorized zero padded 1 × 2 × 2 input images. Suppose that the ith convolutional layer can be written as a matrix A i and the upsampling layer is written as a matrix U. Then we have the following matrix factorization for our network:. In each of the above examples, we have thus demonstrated that when training on k linearly independent examples, A X obtains rank k. DISPLAYFORM3 Now we show that when X's span has dimension k, then A X has rank k. To do this, we will consider a selection of k basis elements from the above 4 matrices, and then show that introducing linear combinations of these basis elements into X does not affect the rank of A X.Namely, we consider selecting the training set: DISPLAYFORM4 which has dimension 3. After training, the ing eigenvalues are 1, 1, 1, −0.005. The corresponding top 3 eigenvectors are:. Hence we again see that the rank of A X is given by the dimension of the span of the training set. In this section, we examine the operator of D 64,128 when trained using a larger dataset. Suppose we first train network D 64,128 for 20, 000 iterations using 10 examples, 1 from each class of CIFAR10 with training examples shown in FIG21. Network D 64,128 achieves an MSE of 0.0046 after 20, 000 iterations. In FIG21, there are 10 eigenvalues with magnitude close to 1. These top ten eigenvalues are the following: FIG21 presents the real and imaginary components of 64 eigenvectors corresponding to the top 64 eigenvalues (the grids should be read in row-major order). However, as the eigenvectors are now linear combinations of the training input, it is much harder to decipher exactly how the training examples are presented in the first 10 eigenvectors. However, we can clearly see that some of the training examples are present in these eigenvectors. For example, the real components of eigenvectors 1, 2, and 3 contain the bird training example, but with inverted colors. Similarly, the real components of eigenvectors 4 and 5 contain the the boat example again with inverted colors; the real components of eigenvectors 6 and 7 contain the frog example; the real components of eigenvectors 8 and 9 contain the car example. Next we consider the rank of the learned operators when training on thousands of examples from a single class of CIFAR10. In particular, we trained using 2000 dogs, 5000 dogs, or 5000 planes from CIFAR10's training set. If our conjecture holds, then the ing solution should be of rank equal to the dimension of the span of each of these training sets. Note that the rank of the learned solution should be less than 3072 even though we are training on greater than 3072 images because it is unlikely that there are 3072 linearly independent images in each of the sets. The training losses on each of these datasets is shown in Table 1 and the corresponding eigenvalue and eigenvector plots are shown in FIG1.From the plot of eigenvalues, it appears that the rank of the ing matrix for D 64,192 on 2000 dogs, 5000 dogs, and 5000 planes is around 500, 900, 750 respectively. However, it could be the case that the remaining eigenvalues are providing small nonzero contributions to the learned solution. To rectify this issue, we took the SVD of the ing operator and zeroed out the lowest singular values and compared the MSE of the reconstructions using the ing operator against those of the original. In particular, we compared the MSE of the reconstructions (after min-max scaling) to the training set used. The are summarized in the Table 1.From this table it is evident that only the top 500, 900, and 750 components were necessary to achieve low training error. Assuming our conjecture holds, this would indicate that these values are approximations for the dimension of the span of the corresponding training sets. We note that this is an approximation because the true dimension would be given when the training error approaches 0. The training errors here are of the order of 10 −3, thereby justifying such approximations. In FIG1, we present the effect of different initializations on learning the point map in a nonlinear downsampling network (N LD 128, 128 LD 128, 128) learns the point map regardless of the initialization used. However, we note that for Kaiming intializations, the ing reconstruction after feeding in new inputs is a blurred version of the training image.

We identify downsampling as a mechansim for memorization in convolutional autoencoders.

represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.