Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task. In this work, we investigate the efficiency of current lifelong approaches, in terms of sample complexity, computational and memory cost. Towards this end, we first introduce a new and a more realistic evaluation protocol, whereby learners observe each example only once and hyper-parameter selection is done on a small and disjoint set of tasks, which is not used for the actual learning experience and evaluation. Second, we introduce a new metric measuring how quickly a learner acquires a new skill. Third, we propose an improved version of GEM , dubbed Averaged GEM (A-GEM), which enjoys the same or even better performance as GEM, while being almost as computationally and memory efficient as EWC and other regularization-based methods. Finally, we show that all algorithms including A-GEM can learn even more quickly if they are provided with task descriptors specifying the classification tasks under consideration. Our experiments on several standard lifelong learning benchmarks demonstrate that A-GEM has the best trade-off between accuracy and efficiency Intelligent systems, whether they are natural or artificial, must be able to quickly adapt to changes in the environment and to quickly learn new skills by leveraging past experiences. While current learning algorithms can achieve excellent performance on a variety of tasks, they strongly rely on copious amounts of supervision in the form of labeled data. The lifelong learning (LLL) setting attempts at addressing this shortcoming, bringing machine learning closer to a more realistic human learning by acquiring new skills quickly with a small amount of training data, given the experience accumulated in the past. In this setting, the learner is presented with a stream of tasks whose relatedness is not known a priori. The learner has then the potential to learn more quickly a new task, if it can remember how to combine and re-use knowledge acquired while learning related tasks of the past. Of course, for this learning setting to be useful, the model needs to be constrained in terms of amount of compute and memory required. Usually this means that the learner should not be allowed to merely store all examples seen in the past (in which case this reduces the lifelong learning problem to a multitask problem) nor should the learner be engaged in computations that would not be feasible in real-time, as the goal is to quickly learn from a stream of data. Unfortunately, the established training and evaluation protocol as well as current algorithms for lifelong learning do not satisfy all the above desiderata, namely learning from a stream of data using limited number of samples, limited memory and limited compute. In the most popular training paradigm, the learner does several passes over the data BID1 BID22, while ideally the model should need only a handful of samples and these should be provided one-by-one in a single pass BID15. Moreover, when the learner has several hyper-parameters to tune, the current practice is to go over the sequence of tasks several times, each time with a different hyper-parameter value, again ignoring the requirement of learning from a stream of data and, strictly speaking, violating the assumption of the LLL scenario. While some algorithms may work well in a single-pass setting, they unfortunately require a lot of computation BID15 or their memory scales with the number of tasks, which greatly impedes their actual deployment in practical applications. In this work, we propose an evaluation methodology and an algorithm that better match our desiderata, namely learning efficiently -in terms of training samples, time and memory -from a stream of tasks. First, we propose a new learning paradigm, whereby the learner performs cross validation on a set of tasks which is disjoint from the set of tasks actually used for evaluation (Sec. 2). In this setting, the learner will have to learn and will be tested on an entirely new sequence of tasks and it will perform just a single pass over this data stream. Second, we build upon GEM BID15, an algorithm which leverages a small episodic memory to perform well in a single pass setting, and propose a small change to the loss function which makes GEM orders of magnitude faster at training time while maintaining similar performance; we dub this variant of GEM, A-GEM (Sec. 4). Third, we explore the use of compositional task descriptors in order to improve the fewshot learning performance within LLL showing that with this additional information the learner can pick up new skills more quickly (Sec. 5). Fourth, we introduce a new metric to measure the speed of learning, which is useful to quantify the ability of a learning algorithm to learn a new task (Sec. 3). And finally, using our new learning paradigm and metric, we demonstrate A-GEM on a variety of benchmarks and against several representative baselines (Sec. 6). Our experiments show that A-GEM has a better trade-off between average accuracy and computational/memory cost. Moreover, all algorithms improve their ability to quickly learn a new task when provided with compositional task descriptors, and they do so better and better as they progress through the learning experience. Currently, most works on lifelong learning BID23 BID16 adopt a learning protocol which is directly borrowed from supervised learning. There are T tasks, and each task consists of a training, validation and test sets. During training the learner does as many passes over the data of each task as desired. Moreover, hyperparameters are tuned on the validation sets by sweeping over the whole sequence of tasks as many times as required by the cross-validation grid search. Finally, metrics of interest are reported on the test set of each task using the model selected by the previous cross-validation procedure. Since the current protocol violates our stricter definition of LLL for which the learner can only make a single pass over the data, as we want to emphasize the importance of learning quickly from data, we now introduce a new learning protocol. We consider two streams of tasks, described by the following ordered sequences of datasets DISPLAYFORM0 } is the dataset of the k-th task, T CV < T (in all our experiments T CV = 3 while T = 20), and we assume that all datasets are drawn from the same distribution over tasks. To avoid cluttering of the notation, we let the context specify whether D k refers to the training or test set of the k-th dataset. CV is the stream of datasets which will be used during cross-validation; D CV allows the learner to replay all samples multiple times for the purposes of model hyper-parameter selection. Instead, D EV is the actual dataset used for final training and evaluation on the test set; the learner will observe training examples from D EV once and only once, and all metrics will be reported on the test sets of D EV. Since the regularization-based approaches for lifelong learning BID29 are rather sensitive to the choice of the regularization hyper-parameter, we introduced the set D CV, as it seems reasonable in practical applications to have similar tasks that can be used for tuning the system. However, the actual training and testing are then performed on D EV using a single pass over the data. See Algorithm 1 for a summary of the training and evaluation protocol. Each example in any of these dataset consists of a triplet defined by an input (x k ∈ X), task descriptor (t k ∈ T, see Sec. 5 for examples) and a target vector (y k ∈ y k), where y k is the set of labels specific to task k and y k ⊂ Y. While observing the data, the goal is to learn a predictor f θ: X × T → Y, parameterized by θ ∈ R P (a neural network in our case), that can map any test pair (x, t) to a target y. Below we describe the metrics used to evaluate the LLL methods studied in this work. In addition to Average Accuracy (A) and Forgetting Measure (F) BID4, we define a new measure, the Learning Curve Area (LCA), that captures how quickly a model learns. The training dataset of each task, D k, consists of a total B k mini-batches. After each presentation of a mini-batch of task k, we evaluate the performance of the learner on all the tasks using the corresponding test sets. Let a k,i,j ∈ be the accuracy evaluated on the test set of task j, after the model has been trained with the i-th mini-batch of task k. Assuming the first learning task in the continuum is indexed by 1 (it will be T CV + 1 for D EV) and the last one by T (it will be T CV for D CV), we define the following metrics:Average Accuracy (A ∈) Average accuracy after the model has been trained continually with all the mini-batches up till task k is defined as: DISPLAYFORM0 In particular, A T is the average accuracy on all the tasks after the last task has been learned; this is the most commonly used metric used in LLL.Forgetting Measure (F ∈ [−1, 1]) BID4 Average forgetting after the model has been trained continually with all the mini-batches up till task k is defined as: DISPLAYFORM1 where f k j is the forgetting on task'j' after the model is trained with all the mini-batches up till task k and computed as: DISPLAYFORM2 Measuring forgetting after all tasks have been learned is important for a two-fold reason. It quantifies the accuracy drop on past tasks, and it gives an indirect notion of how quickly a model may learn a new task, since a forgetful model will have little knowledge left to transfer, particularly so if the new task relates more closely to one of the very first tasks encountered during the learning experience. Learning Curve Area (LCA ∈) Let us first define an average b-shot performance (where b is the mini-batch number) after the model has been trained for all the T tasks as: DISPLAYFORM3 LCA at β is the area of the convergence curve Z b as a function of b ∈ [0, β]: DISPLAYFORM4 LCA has an intuitive interpretation. LCA 0 is the average 0-shot performance, the same as forward transfer in BID15. LCA β is the area under the Z b curve, which is high if the 0-shot performance is good and if the learner learns quickly. In particular, there could be two models with the same Z β or A T, but very different LCA β because one learns much faster than the other while they both eventually obtain the same final accuracy. This metric aims at discriminating between these two cases, and it makes sense for relatively small values of β since we are interested in models that learn from few examples. So far we discussed a better training and evaluation protocol for LLL and a new metric to measure the speed of learning. Next, we review GEM BID15, which is an algorithm that has been shown to work well in the single epoch setting. Unfortunately, GEM is very intensive in terms of computational and memory cost, which motivates our efficient variant, dubbed A-GEM. In Sec. 5, we will describe how compositional task descriptors can be leveraged to further speed up learning in the few shot regime. GEM avoids catastrophic forgetting by storing an episodic memory M k for each task k. While minimizing the loss on the current task t, GEM treats the losses on the episodic memories of tasks DISPLAYFORM0, as inequality constraints, avoiding their increase but allowing their decrease. This effectively permits GEM to do positive backward transfer which other LLL methods do not support. Formally, at task t, GEM solves for the following objective: DISPLAYFORM1 Where f t−1 θ is the network trained till task t − 1. To inspect the increase in loss, GEM computes the angle between the loss gradient vectors of previous tasks g k, and the proposed gradient update on the current task g. Whenever the angle is greater than 90°with any of the g k's, it projects the proposed gradient to the closest in L2 norm gradientg that keeps the angle within the bounds. Formally, the optimization problem GEM solves is given by: DISPLAYFORM2 Eq.7 is a quadratic program (QP) in P -variables (the number of parameters in the network), which for neural networks could be in millions. In order to solve this efficiently, GEM works in the dual space which in a much smaller QP with only t − 1 variables: DISPLAYFORM3 where DISPLAYFORM4 ×P is computed at each gradient step of training. Once the solution v * to Eq. 8 is found, the projected gradient update can be computed asg = G v * + g. While GEM has proven very effective in a single epoch setting BID15, the performance gains come at a big computational burden at training time. At each training step, GEM computes the matrix G using all samples from the episodic memory, and it also needs to solve the QP of Eq. 8. Unfortunately, this inner loop optimization becomes prohibitive when the size of M and the number of tasks is large, see Tab. 7 in Appendix for an empirical analysis. To alleviate the computational burden of GEM, next we propose a much more efficient version of GEM, called Averaged GEM (A-GEM).Whereas GEM ensures that at every training step the loss of each individual previous tasks, approximated by the samples in episodic memory, does not increase, A-GEM tries to ensure that at every training step the average episodic memory loss over the previous tasks does not increase. Formally, while learning task t, the objective of A-GEM is: DISPLAYFORM5 The corresponding optimization problem reduces to: DISPLAYFORM6 where g ref is a gradient computed using a batch randomly sampled from the episodic memory, DISPLAYFORM7, of all the past tasks. In other words, A-GEM replaces the t − 1 constraints of GEM with a single constraint, where g ref is the average of the gradients from the previous tasks computed from a random subset of the episodic memory. The constrained optimization problem of Eq. 10 can now be solved very quickly; when the gradient g violates the constraint, it is projected via: DISPLAYFORM8 The formal proof of the update rule of A-GEM (Eq. 11) is given in Appendix C. This makes A-GEM not only memory efficient, as it does not need to store the matrix G, but also orders of magnitude faster than GEM because 1) it is not required to compute the matrix G but just the gradient of a random subset of memory examples, 2) it does not need to solve any QP but just an inner product, and 3) it will incur in less violations particularly when the number of tasks is large (see Tab. 7 and Fig. 6 in Appendix for empirical evidence). All together these factors make A-GEM faster while not hampering its good performance in the single pass setting. Intuitively, the difference between GEM and A-GEM loss functions is that GEM has better guarantess in terms of worst-case forgetting of each individual task since (at least on the memory examples) it prohibits an increase of any task-specific loss, while A-GEM has better guaratees in terms of average accuracy since GEM may prevent a gradient step because of a task constraint violation although the overall average loss may actually decrease, see Appendix Sec. D.1 and D.2 for further analysis and empirical evidence. The pseudo-code of A-GEM is given in Appendix Alg. 2. In this section, we discuss how we can improve forward transfer for all the LLL methods including A-GEM. In order to speed up learning of a new task, we consider the use of compositional task descriptors where components are shared across tasks and thus allow transfer. Examples of compositional task descriptors are, for instance, a natural language description of the task under consideration or a matrix specifying the attribute values of the objects to be recognized in the task. In our experiments, we use the latter since it is provided with popular benchmark datasets BID26 BID12 ). For instance, if the model has already learned and remembers about two independent properties (e.g., color of feathers and shape of beak), it can quickly recognize a new class provided a descriptor specifying the values of its attributes (yellow feathers and red beak), although this is an entirely unseen combination. Borrowing ideas from literature in few-shot learning BID13 BID30 BID5 BID27, we learn a joint embedding space between image features and the attribute embeddings. Formally, let x k ∈ X be the input (e.g., an image), t k be the task descriptor in the form of a matrix of size C k × A, where C k is the number of classes in the kth task and A is the total number of attributes for each class in the dataset. The joint embedding model consists of a feature extraction module, φ θ: DISPLAYFORM0, and a task embedding module, ψ ω: DISPLAYFORM1 In this work, φ θ is implemented as a standard multi-layer feed-forward network (see Sec. 6 for the exact parameterization), whereas ψ ω is implemented as a parameter matrix of dimensions A × D. This matrix can be interpreted as an attribute look-up table as each attribute is associated with a D dimensional vector, from which a class embedding vector is constructed via a linear combination of the attributes present in the class; the task descriptor embedding is then the concatenation of the embedding vectors of the classes present in the task (see FIG9 for the pictorial description of the joint embedding model). During training, the parameters θ and ω are learned by minimizing the cross-entropy loss: DISPLAYFORM2 where (DISPLAYFORM3 where [a] i denotes the i-th element of the vector a. Note that the architecture and loss functions are general, and apply not only to A-GEM but also to any other LLL model (e.g., regularization based approaches). See Sec. 6 for the actual choice of parameterization of these functions. We consider four dataset streams, see Tab.1 in Appendix Sec. A for a summary of the statistics. Permuted MNIST ) is a variant of MNIST dataset of handwritten digits where each task has a certain random permutation of the input pixels which is applied to all the images of that task. Split CIFAR BID29 consists of splitting the original CIFAR-100 dataset BID11 ) into 20 disjoint subsets, where each subset is constructed by randomly sampling 5 classes without replacement from a total of 100 classes. Similarly to Split CIFAR, Split CUB is an incremental version of the fine-grained image classification dataset CUB BID26 of 200 bird categories split into 20 disjoint subsets of classes. Split AWA, on the other hand, is the incremental version of the AWA dataset of 50 animal categories, where each task is constructed by sampling 5 classes with replacement from the total 50 classes, constructing 20 tasks. In this setting, classes may overlap among multiple tasks, but within each task they compete against different set of classes. Note that to make sure each training example is only seen once, the training data of a each class is split into disjoint sets depending on the frequency of its occurrence in different tasks. For Split AWA, the classifier weights of each class are randomly initialized within each head without any transfer from the previous occurrence of the class in past tasks. Finally, while on Permuted MNIST and Split CIFAR we provide integer task descriptors, on Split CUB and Split AWA we stack together the attributes of the classes (specifying for instance the type of beak, the color of feathers, etc.) belonging to the current task to form a descriptor. In terms of architectures, we use a fully-connected network with two hidden layers of 256 ReLU units each for Permuted MNIST, a reduced ResNet18 for Split CIFAR like in BID15, and a standard ResNet18 BID8 for Split CUB and Split AWA. For a given dataset stream, all models use the same architecture, and all models are optimized via stochastic gradient descent with mini-batch size equal to 10. We refer to the joint-embedding model version of these models by appending the suffix'-JE' to the method name. As described in Sec. 2 and outlined in Alg. 1, in order to cross validate we use the first 3 tasks, and then report metrics on the remaining 17 tasks after doing a single training pass over each task in sequence. Lastly, we compared A-GEM against several baselines and state-of-the-art LLL approaches which we describe next. VAN is a single supervised learning model, trained continually without any regularization, with the parameters of a new task initialized from the parameters of the previous task. ICARL BID17 ) is a class-incremental learner that uses nearest-exemplar-based classifier and avoids catastrophic forgetting by regularizing over the feature representation of previous tasks using a knowledge distillation loss. EWC, PI BID29, RWALK BID4 and MAS BID1 A T, F T and LCA 10 values and confidence intervals are computed over 10 runs. Other baselines are given in Tab. 5, 6 and 7 in the Appendix, which are used to generate the plots.whereby a new "column" with lateral connections to previous hidden layers is added once a new task arrives. GEM described in Sec. 4 is another natural baseline of comparison since A-GEM builds upon it. The amount of episodic memory per task used in ICARL, GEM and A-GEM is set to 250, 65, 50, and 100, and the batch size for the computation of g ref (when the episodic memory is sufficiently filled) in A-GEM is set to 256, 1300, 128 and 128 for MNIST, CIFAR, CUB and AWA, respectively. While populating episodic memory, the samples are chosen uniformly at random for each task. Whereas the network weights are randomly initialized for MNIST, CIFAR and AWA, on the other hand, for CUB, due to the small dataset size, a pre-trained ImageNet model is used. Finally, we consider a multi-task baseline, MULTI-TASK, trained on a single pass over shuffled data from all tasks, and thus violating the LLL assumption. It can be seen as an upper bound performance for average accuracy.6.1 Fig. 1 and 2 show the overall on all the datasets we considered (for brevity we show only representative methods, see detailed in Appendix Tab. 4, 5, 6 and 7). First, we observe that A-GEM achieves the best average accuracy on all datasets, except Permuted MNIST, where PROG-NN works better. The reason is because on this dataset each task has a large number of training examples, which enables PROG-NN to learn its task specific parameters and to leverage its lateral connections. However, notice how PROG-NN has the worst memory cost by the end of training -as its number of parameters grows super-linearly with the number of tasks. In particular, in large scale setups (Split CUB and AWA), PROG-NN runs out of memory during training due to its large size. Also, PROG-NN does not learn well on datasets where tasks have fewer training examples. Second, A-GEM and GEM perform comparably in terms of average accuracy, but A-GEM has much lower time (about 100 times faster) and memory cost (about 10 times lower), comparable to regularizationbased approaches like EWC. Third, EWC and similar methods perform only slightly better than VAN on this single pass LLL setting. The analysis in Appendix Sec. F demonstrates that EWC requires several epochs and over-parameterized architectures in order to work well. Fourth, PROG-NN has no forgetting by construction and A-GEM and GEM have the lowest forgetting among methods that use a fixed capacity architecture. Next, all methods perform similarly in terms of LCA, with PROG-NN being the worst because of its ever growing number of parameters and A-GEM slightly better than all the other approaches. And finally, the use of task descriptors improves average accuracy across the board as shown in Fig.2, with A-GEM a bit better than all the other methods we tried. All joint-embedding models using task descriptors have better LCA performance, although this is the same across all methods including A-GEM. Overall, we conclude that A-GEM offers the best trade-off between average accuracy performance and efficiency in terms of sample, memory and computational cost. different methods are best in terms of LCA, although A-GEM is always top-performing. Fig. 4 shows in more detail the gain brought by task descriptors which greatly speed up learning in the few-shot regime. On these datasets, A-GEM performs the best or on par to the best. Finally, in Fig. 5, we report the 0-shot performance of LLL methods on Split CUB and Split AWA datasets over time, showing a clear advantage of using compositional task descriptors with joint embedding models, which is more significant for A-GEM. Interestingly, the zero-shot learning performance of joint embedding models improves over time, indicating that these models get better at forward transfer or, in other words, become more efficient over time. Continual or Lifelong Learning (LLL) BID25 have been the subject of extensive study over the past two decades. One approach to LLL uses modular compositional models BID6 BID0 BID19 BID3 BID28 BID7, which limit interference among tasks by using different subset of modules for each task. Unfortunately, these methods require searching over the space of architectures which is not sample efficient with current methods. Another approach is to regularize parameters important to solve past tasks BID29 BID4, which has been proven effective for over-parameterized models in the multiple epoch setting (see Appendix Sec. F), while we focus on learning from few examples using memory efficient models. Methods based on episodic memory BID17 BID15 ) require a little bit more memory at training time but can work much better in the single pass setting we considered BID15 ).The use of task descriptors for LLL has already been advocated by BID9 but using a sparse coding framework which is not obviously applicable to deep nets in a computationally efficient way, and also by although they did not explore the use of compositional descriptors. More generally, tasks descriptors have been used in Reinforcement Learning with similar motivations by several others BID24 BID21 BID2, and it is also a key ingredient in all the zero/few-shot learning algorithms BID13 BID27 BID5 BID26 BID12 ). We studied the problem of efficient Lifelong Learning (LLL) in the case where the learner can only do a single pass over the input data stream. We found that our approach, A-GEM, has the best tradeoff between average accuracy by the end of the learning experience and computational/memory cost. Compared to the original GEM algorithm, A-GEM is about 100 times faster and has 10 times less memory requirements; compared to regularization based approaches, it achieves significantly higher average accuracy. We also demonstrated that by using compositional task descriptors all methods can improve their few-shot performance, with A-GEM often being the best. Our detailed experiments reported in Appendix E also show that there is still a substantial performance gap between LLL methods, including A-GEM, trained in a sequential learning setting and the same network trained in a non-sequential multi-task setting, despite seeing the same data samples. Moreover, while task descriptors do help in the few-shot learning regime, the LCA performance gap between different methods is very small; suggesting a poor ability of current methods to transfer knowledge even when forgetting has been eliminated. Addressing these two fundamental issues will be the focus of our future research. In Sec. A we report the summary of datasets used for the experiments. Sec. B details our A-GEM algorithm and Sec. C provides the proof of update rule of A-GEM discussed in Sec. 4 of the main paper. In Sec. D, we analyze the differences between A-GEM and GEM, and describe another variation of GEM, dubbed Stochastic GEM (S-GEM). The detailed of the experiments which were used to generate Fig 1 and 2 in the main paper are given in Sec. E. In Sec. F, we provide empirical evidence to the conjecture that regularization-based approaches like EWC require over-parameterized architectures and multiple passes over data in order to perform well as discussed in the Sec. 6.1 of the main paper. In Sec. G, we provide the grid used for the cross-validation of different hyperparameters and report the optimal values for different models. Finally, in Sec. H, we pictorially describe the joint embedding model discussed in Sec. 5.A DATASET STATISTICS DISPLAYFORM0 end if 14: DISPLAYFORM1 At,: DISPLAYFORM2 end for 19:return f θ, A 20: end procedure DISPLAYFORM3 end for 10: return a 11: end procedure DISPLAYFORM4 end for 7:return M 8: end procedure C A-GEM UPDATE RULE Here we provide the proof of the update rule of A-GEM (Eq. 11),g = g − Proof. The optimization objective of A-GEM as described in the Eq. 10 of the main paper, is: DISPLAYFORM5 Replacingg with z and rewriting Eq. 14 yields: DISPLAYFORM6 Note that we discard the term g g from the objective and change the sign of the inequality constraint. The Lagrangian of the constrained optimization problem defined above can be written as: DISPLAYFORM7 Now, we pose the dual of Eq. 16 as: DISPLAYFORM8 Lets find the value z * that minimizes the L(z, α) by setting the derivatives of L(z, α) w.r.t. to z to zero: DISPLAYFORM9 The simplified dual after putting the value of z * in Eq. 17 can be written as: DISPLAYFORM10 The solution α * = max α;α>0 θ D (α) to the dual is given by: DISPLAYFORM11 By putting α * in Eq. 18, we recover the A-GEM update rule: DISPLAYFORM12 In this section, we empirically analyze the differences between A-GEM and GEM, and report experiments with another computationally efficient but worse performing version of GEM.D.1 FREQUENCY OF CONSTRAINT VIOLATIONS FIG4 shows the frequency of constraint violations (see Eq. 8 and 10) on Permuted MNIST and Split CIFAR datasets. Note that, the number of gradient updates (training steps) per task on MNIST and CIFAR are 5500 and 250, respectively. As the number of tasks increase, GEM violates the optimization constraints at almost each training step, whereas A-GEM plateaus to a much lower value. Therefore, the computational efficiency of A-GEM not only stems from the fact that it avoids solving a QP at each training step (which is much more expensive than a simple inner product) but also from the fewer number of constraint violations. From the figure, we can also infer that as the number of tasks grows the gap between GEM and A-GEM would grow further. Thus, the computational and memory overhead of GEM over A-GEM, see also Tab. 7, gets worse as the number of tasks increases. In Tab. 2, we empirically demonstrate the different properties induced by the objective functions of GEM and A-GEM. GEM enjoys lower worst-case task forgetting while A-GEM enjoys better overall average accuracy. This is particularly true on the training examples stored in memory, as on the test set the is confounded by the generalization error. In this section we report experiments with another variant of GEM, dubbed Stochastic GEM (S-GEM). The main idea in S-GEM is to randomly sample one constraint, at each training step, from the possible t − 1 constraints of GEM. If that constraint is violated, the gradient is projected only taking into account that constraint. Formally, the optimization objective of S-GEM is given by: DISPLAYFORM0 In other words, at each training step, S-GEM avoids the increase in loss of one of the previous tasks sampled randomly. In Tab. 3 we report the comparison of GEM, S-GEM and A-GEM on Permuted MNIST and Split CIFAR.Although, S-GEM is closer in spirit to GEM, as it requires randomly sampling one of the GEM constraints to satisfy, compared to A-GEM, which defines the constraint as the average gradient of the previous tasks, it perform slightly worse than GEM, as can be seen from Tab. 3. In Tab. 4, 5, 6 and 7 we report the detailed which were used to generate Fig.1 and 2. In this section we provide empirical evidence to the conjecture that regularization-based approaches like EWC need over-parameterized architectures and multiple passes over the samples of each task in order to perform well. The intuition as to why models need to be over-parameterized is because it is easier to avoid cross-task interference when the model has additional capacity. In the single-pass setting and when each task does not have very many training samples, regularization-based appraches also suffer because regularization parameters cannot be estimated well from a model that has not fully converged. Moreover, for tasks that do not have much data, rgularization-based approaches do not enable any kind of positive backward transfer BID15 which further hurts performance as the predictor cannot leverage knowledge acquired later to improve its prediction on past tasks. Finally, regularization-based approaches perform much better in the multi-epoch setting simply because in this setting the baseline un-regularized model performs much worse, as it overfits much more to the data of the current task, every time unlearning what it learned before. We consider Permuted MNIST and Split CIFAR datasets as described in Sec. 6 of the main paper. For MNIST, the two architecture variants that we experiment with are; 1) two-layer fully-connected network with 256 units in each layer (denoted by −S suffix), and 2) two-layer fully-connected network with 2000 units in each layer (denoted by −B suffix).For CIFAR, the two architecture variants are; 1) ResNet-18 with 3 times less feature maps in all the layers (denoted by −S suffix), and 2) Standard ResNet-18 (denoted by −B token).We run the experiments on VAN and EWC with increasing the number of epochs from 1 to 10 for Permuted MNIST and from 1 to 30 for CIFAR. For instance, when epoch is set to 10, it means that the training samples of task t are presented 10 times before showing examples from task t + 1. In Fig. 7 and 8 we plot the Average Accuracy (Eq. 1) and Forgetting (Eq. 2) on Permuted MNIST and Split CIFAR, respectively. We observe that the average accuracy significantly improves with the number of epochs only when EWC is applied to the big network. In particular, in the single epoch setting, EWC peforms similarly to the baseline VAN on Split CIFAR which has fewer number of training examples per task. Below we report the hyper-parameters grid considered for different experiments. Note, as described in the Sec. 6 of the main paper, to satisfy the requirement that a learner does not see the data of a task more than once, first T CV tasks are used to cross-validate the hyper-parameters. In all the datasets, the value of T CV is set to'3'. The best setting for each experiment is reported in the parenthesis.• Pictorial description of the joint embedding model discussed in the Sec. 5 of the main paper. Modules; φ θ and ψ ω are implemented as feed-forward neural networks with P and A × D parameters, respectively. The descriptor of task k (t k) is a matrix of dimensions C k × A, shared among all the examples of the task, constructed by concatenating the A-dimensional class attribute vectors of C k classes in the task.

Represent the short summary for finding the corresponding scientific article (long)

An efficient lifelong learning algorithm that provides a better trade-off between accuracy and time/ memory complexity compared to other algorithms.

represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the scientific article to find a short tldr!

Question: Reduced precision computation is one of the key areas addressing the widening’compute gap’, driven by an exponential growth in deep learning applications. In recent years, deep neural network training has largely migrated to 16-bit precision,with significant gains in performance and energy efficiency. However, attempts to train DNNs at 8-bit precision have met with significant challenges, because of the higher precision and dynamic range requirements of back-propagation. In this paper, we propose a method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16)and a broader set of workloads (Resnet-18/34/50, GNMT, and Transformer) than previously reported. We propose an enhanced loss scaling method to augment the reduced subnormal range of 8-bit floating point, to improve error propagation. We also examine the impact of quantization noise on generalization, and propose a stochastic rounding technique to address gradient noise. As a of applying all these techniques, we report slightly higher validation accuracy compared to full precision baseline. The unprecedented success of Deep Learning models in a variety of tasks including computer vision, machine translation and speech recognition has led to the proliferation of deeper and more complex models. Algorithmic innovations such as large batch training and neural architecture search have enabled models to scale on large compute cluster to accelerate training. This enhanced performance has enabled the adoption of larger neural networks. As a consequence, the computational requirements for training Deep Learning models have been growing at an exponential rate (Amodei & Hernandez) over the past few years, outperforming Moore's Law and hardware capabilities by a wide margin. One of the promising areas of research to address this growing compute gap is to reduce the numeric precision requirements for deep learning. Reduced precision methods exploit the inherent noise resilient properties of deep neural networks to improve compute efficiency, while minimizing the loss of model accuracy. Recent studies have shown that, deep neural networks can be trained using 16-bits of precision without any noticeable impact on validation accuracy across a wide range of networks. Today, state-of-the-art training platforms support 16-bit precision in the form of high-performance systolic array or GEMM engine (General Matrix Multiply) implementations (; Köster et al., 2017a). There have been numerous attempts (; ; ; ;) to train deep neural networks at lower precision (below 16-bits) with varying degrees of success. With the abundance of 8-bit integer deep learning'ops' deployed to accelerate inference tasks, much of the research into training methods have also focused on integer based fixed-point numeric formats (; ;). Training with 8-bit integers has been significantly more challenging because the dynamic range of such formats is not sufficient to represent error gradients during back-propagation. More recently, have shown that 8-bit floating representation can be used to train convolutional neural networks, with the help of specialized chunk-based accumulation and stochastic rounding hardware. While this method has shown promising , it requires expensive stochastic rounding hardware built into the critical compute path making it unattractive for systolic array and GEMM accelerator implementations. Our paper extends the state of the art in 8-bit floating point (FP8) training with the following key contributions: • We propose a scalable training solution that eliminates the need for specialized hardware designs , thereby enabling efficient MAC designs with higher compute density. • We demonstrated state-of-the-art training using 8-bit floating point representation (for weight, activation, error and gradient tensors), across multiple data sets (Imagenet-1K, WMT16) and a broader set of workloads (Resnet, GNMT, Transformer) than previously reported . • We propose enhanced loss scaling method to compensate for the reduced subnormal range of 8-bit floating point representation for improved error propagation leading to better model accuracy. • We present a detailed study of the impact of quantization noise on model generalization and propose a stochastic rounding technique to address the gradient noise in the early epochs leading to better generalization. The study of reduced precision methods for deep learning training is an active area of research. In the pursuit of improving compute efficiency, researchers have experimented with various numeric formats and hardware implementations. demonstrated that deep neural networks can be trained with minimal loss in accuracy, using 16-bit fixed point representation. This was followed by studies employing other numeric formats such as, half-precision floating point and dynamic fixed point (Köster et al., 2017b;), demonstrating state of the art across residual, recurrent and generative networks. Today most of the neural network training in a production deployment has migrated to 16-bit hardware, ing in significant improvements in performance . There have been several attempts to further reduce the precision requirements of DNNs to boost training performance. have trained DoReFa-Net, a derivative of AlexNet using bit-convolutions with 1-bit and 2-bits to represent weights and activations respectively, while the gradients are quantized to 6-bits of precision. have trained AlexNet using 8-bit precision for activations, errors and weight gradients, while the weights are quantized to 2-bits of precision. However, both these methods have reported significant loss in validation accuracy. More recently, have successfully trained Resnet-50 ) using 8-bit floating point numeric format with the help of a specialized hardware to compute chunk-based dotproduct computation and stochastic rounding on a 16-bit accumulator. The authors of this study have focused on reducing the accumulator precision and based on studies on smaller networks (AlexNet Resnet-18), attributed training issues related to error propagation and generalization on the choice of accumulator size. However, our studies on larger networks (Resnet-34/50) using 32-bit accumulator indicate that, these issues are not related to the choice of accumulator size and should be addressed independently. We discuss these issues and our proposed solutions in greater detail in Sections3.1and 3.2. Guided by these , we decided to focus on studying the impact of using FP8 numeric format on training, while maintaining a high precision accumulator(FP32). We further believe that modern GEMM engine designs implementing progressive multiplier reduction techniques can effectively amortize the cost of a larger final accumulator, and do not benefit from building 16-bit accumulator with additional hardware overheads in the critical compute path. The choice of bit-level representation of floating point (sign, exponent, mantissa), has a significant impact on the effectiveness of the numerical format -the trade-off between the dynamic range and precision is especially tricky at low bit-width representations. While it is important to maintain higher dynamic range for effective propagation of error gradients , it leads to having values that are too few and scattered to maintain fidelity required for gradient computations. After careful consideration of these facts and several failed experiments with other formats (ex: with more exponent bits), we decided to use s=1,e=5,m=2 numeric format for representing 8-bit floating point. We also use a 32-bit floating point accumulator to accommodate the dynamic range of the output ing from the dot-product operation. Therefore, each GEMM/convolution operation takes two input tensors in 8-bit floating point format and produces a 32-bit single precision floating point output. The 32-bit output must be down-converted to a 8-bit floating point value before passing to the next operation. We believe rounding plays an extremely important role during down-conversion operation to help recover the numeric accuracy. We present the from the study of different rounding modes applied to this numeric format and their impact on training performacne in Section.3.2 Figure. 1a shows the precision settings of various compute operations used in our mixed precision training setup. The'GEMM' operator shown in Figure. 1a represents the key compute kernel used by deep neural networks during forward, backward, and gradient computation passes. Quantization nodes identified with the letter'Q' perform down-conversion and rounding on 32-bit output from GEMM operators and convert them them to 8-bit floating point format before passing on to the next layer. In our training experiments, we quantized the weight, activation, error and gradient tensors of all convolution and MatMul kernels to 8-bit floating point format in forward, backward and weight update paths. More details about workload specific quantization flow is discussed in Section 4. Figure.1b shows the data flow during optimization and weight update steps. In the optimization path the L2-regularization term is added to the cross entropy, and the ing loss is multiplied with loss scale parameter before initiating back propagation. The weight gradients are computed during back propagation and converted to 8-bit floating point format. During weight update, the weight gradients are re-scaled using the loss scale parameter, this step is performed in full precision to prevent any potential underflow. The gradients are then passed to the momentum optimizer, and the final gradients are applied to the master copy of weights. In our experiments, the master copy of weights are stored in half-precision floating point format, these values are up-converted to 32-bit during the update step and the weight update is performed at full precision. After the update, the master weights are converted to half-precision format before they are stored back into memory. Since this is a bandwidth bound operation, compute precision will not have any noticeable impact on the performance. Previous studies on half-precision floating point have shown that loss scaling technique can be used to push smaller error gradients into representable range and train neural networks successfully. The full range of numeric values represented by a floating point format include the'subnormal' values, the range of which is determined by the number of mantissa bits. Because of this property of floating point numbers, the proposed 8-bit floating point format will have significantly smaller subnormal range compared to a half-precision floating point which has the same number of exponent bits. Table. 1 shows the dynamic range comparison between fullprecision(FP32), half-precision(FP16) and the proposed 8-bit floating point (FP8) formats. Half-precision training for convolution networks has been shown to converge using a constant loss scaling value (ex: Resnet-50 uses 1000) . Networks like GNMT and Transformer experience significant variations in gradient distributions through the training cycle and perform better with dynamic loss scaling methods. We adopt the same loss scaling methods used by FP16 training for FP8. For Resnet-50, we increase the scaling factor to compensate for the smaller subnormal range of FP8. Figure.2a shows from our convergence studies on Resnet-50 using different loss scaling values. The model failed to converge with a scaling factor of 1000, and progressively performed better with increasing loss scale values, converging at 10 000. Transformer (big) model, uses the standard'DynamicLossScale' method implemented in TensorFlow and it works out-of-the-box without any additional changes. We use the OpenSeq2Seq implementation for GNMT model, and it uses'back-off' method for automatic loss scaling. Both these loss scaling methods try to maintain a higher loss scale value while checking for numeric overflows at regular intervals(=2000), that in a'NaN' during gradient computation. Our experiments showed that loss scaling experiences frequent gradient overflows for GNMT because of a few outliers, while a significant chunk of the gradients underflow. This irregular gradient distribution is likely a of not having normalization layers in GNMT. The loss scaling algorithm responds to these overflows by dropping the loss scale value significantly. We addressed this by ignoring a few overflows which are likely a of the outliers and continue to maintain a higher loss scale value. We accomplish this by setting a'minimum threshold' for the loss scale value to prevent it from going below a certain threshold value even if an overflow occurs. Figure.2b shows the loss scaling schedule that worked for GNMT. The goal of this exercises was to maintain a higher loss scale value, while ignoring a few spurious overflows. This can also achieved by enhancing the loss scaling algorithm with few additional checks to ignore overflows unless they occur in succession for a few times that is predefined threshold value (nan threshold). Overflows occurring at successive intervals is a more reliable indicator of a actual shift in the gradient distribution. We can also reduce the interval between loss scale updates (from 2000 iterations to 500 iterations), to improve chances of recovering from any inadvertent drop in loss scale value. We describe this algorithm in 1. Reduced precision methods introduce significant amount of noise that can adversely effect convergence and accuracy of deep neural networks. Rounding techniques applied to quantization methods can be effective in regulating some of this noise. For extremely low precision representations with large rounding errors such as the one proposed here(= 0.125), the choice of rounding method can have significant influence on the numeric accuracy and overall applicability of the numeric format. Previous studies have shown that stochastic rounding can be effective for training neural networks using low-precision fixed point formats. The most widely supported rounding failed to converge with loss scale=1000, performed better with 2.3% accuracy loss at loss scale=4000 and showed full convergence at loss scale=10 000, (right) Dynamic loss scaling with gradually increasing minimum threshold for the scaling factor. Ensure: minval = 2, maxval = 2 15, interval = 500, scale = 2, nan threshold = 2 1: nan count = 0, last iter = 0, last nan iter = 0 2: procedure UPDATE LOSS SCALE (loss scale, has nan, iter) return loss scale method in hardware today is RNE (round to nearest even), because it is easier to implement and requires smaller silicon area. In this section, we explore the impact of both RNE and stochastic rounding methods on model convergence and generalization. Our early experiments showed that, for smaller networks such as Resnet-18, RNE proved quite effective when trained on Imagenet-1K data set. However, when we trained ResNet-50 we observed some interesting . Figure.3 shows the convergence plots for Resnet-50 using RNE rounding method applied to quantized weights, activations and gradients. The model displayed significant over-fitting behaviour as indicated by the increased validation error, while the training error mostly follows the baseline as shown in as shown in Figure. 3b, and 3a. Multiple experiments indicated that this behaviour is caused by the noisy error gradients during early epochs which lead to unconstrained growth in model parameters. This is indicated by steep increase in L2 regularization parameter as shown in Figure. 3c. Regularization loss is computed using the formula shown in Equation.1. Increased regularization loss leads to more noisy gradients, which further exacerbates this behaviour. An interesting observation about the L2 regularization loss is that for ResNet-18, the L2-loss is low at the beginning and increases with gradually with iterations. On the other hand for ResNet-50, the L2-loss is high at the beginning due to the initialization of low fan-in 1x1 convolutions, and needs to dip a little before gradually rising again. We suspect that this property of the initialization leads to more noisy behavior of ResNet-50 in the earlier iterations as compared to ResNet-18. Therefore for the ResNet-50 model stochastic rounding is essential. Where, λ is the weight decay parameter and W is the total number of weights. In order to understand the issue of regularization independent of the choice of rounding method, we conducted additional experiments using RNE with other forms of regularization. Figure.4a compares the'Dropout' method with'no regularization' method which uses quantization noise as implicit regularizer with no explicit regularization term. In both these cases, the models performed much better than using L2 regularization with RNE, leading us to the that RNE was ineffective in regulating quantization noise in gradients causing unconstrained growth in model parameters. Unlike deterministic rounding techniques, stochastic rounding computes the probability of rounding using information from several discarded bits of the input making it less prone to introducing large rounding errors. We studied the error behaviour of Resnet-50 by applying stochastic rounding on activations and gradients to regulate quantization noise in the gradients, which in-turn can improve the effectiveness of explicit regularization methods. Our stochastic rounding method is defined as follows: x k, with probability 1 − P Where, k is the target precision, is machine epsilon, and r is random value generated by a pseudo random number generator. We built a TensorFlow based training platform that can accurately emulate the numeric properties of 8-bit floating point on existing floating point hardware. Training experiments were conducted using open source model implementations from TensorFlow 1 and OpenSeq2Seq . Our training framework internally updates the training graph by inserting quantization nodes in the forward, backward, weight update paths for all convolution and GEMM kernels as described in Section.3. Using the proposed training method, we have successfully trained two residual networks (Resnet-34/50) on Imagenet-1K data set. For 8-bit training experiments, we have quantized the Convolution, BatchNorm (stats are accumulated at full precision), ReLU and EltWise layers, while the data layer and SoftMax layers use full precision. We have maintained the first Convolution layer, the following BatchNorm layer, and final Dense layer at half-precision to minimize the accuracy loss. The first convolution layer is expected to have a minimum of eight mantissa bits to accurately represent the normalized image data passed from the data layer. We also store the master copy of weights in half-precision; this can potentially speed up weight update operation by up to 2×, because the operation is memory bound. Since the weight update computation is performed at full precision (as descried in Section.3), to realize these bandwidth benefits the up/down conversion of the master weights between full and half precision formats must be strictly performed in the cache or near memory. We have used identical batch size, and hyper parameters for both full precision and 8-bit training experiments. Both sets of experiments also employ'warm up' for the first 5 epochs, with an initial learning rate of 0.128, and train for a total of 90 epochs to achieve the final accuracy. Table. 2 summarizes the validation accuracy achieved by convolution networks on imagenet-1K dataset. In addition to convolution networks, we have also trained two state of the art machine translation workloads (GNMT and Transformer ) and demonstrated BLEU scores matching single precision baselines. We trained an 8-layer GNMT encoder/decoder LSTM model with 1024 recurrent units and 1024 attention units. We trained this network using 8-bit floating point format for all MatMul operations, including the LSTM, attention modules, embedding lookup is performed at 8-bit while the updates are computed at full precision. The activation functions such as tanh and sigmoid use half-precision data type. Our 8-bit training experiments use identical batch size and hyper parameters to that of the full precision baseline and the network is trained for 10 epochs. We used the loss scaling schedule described in Section.3.1. We also trained a 6-layer Transformer-big translation network with roughly 200M parameters. For the 8-bit experiments, we quantized all the MatMul operations in the encoder, decoder, including attention layers. Embedding lookup operation is performed at 8-bit precision while the updates are computed at full precision. We use the identical configuration for both baseline and 8-bit training experiments. Both GNMT and Transformer models were trained on large scale, WMT2016 English− →German dataset consisting of 4.5 million sentence pairs. The are summarized in Table. 4. We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16) and a broader set of workloads (Resnet-18/34/50, GNMT, Transformer) than previously reported. We propose easy to implement and scalable solution for building FP8 compute primitives, eliminating the need for stochastic rounding hardware in the critical compute path, as proposed by , thereby reducing the cost and complexity of the MAC unit. We explore issues around gradient underflow and quantization noise that arise as a of using the proposed 8-bit numeric format for large scale neural network training. We propose solutions to deal with these problems in the form of enhanced loss scaling and stochastic rounding.

We demonstrated state-of-the-art training results using 8-bit floating point representation, across Resnet, GNMT, Transformer.

represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise or tripletwise similarity constraints while others take the advantage of structured similarity information among multiple data points. In this work, we approach deep metric learning from a novel perspective. We propose instance cross entropy (ICE) which measures the difference between an estimated instance-level matching distribution and its ground-truth one. ICE has three main appealing properties. Firstly, similar to categorical cross entropy (CCE), ICE has clear probabilistic interpretation and exploits structured semantic similarity information for learning supervision. Secondly, ICE is scalable to infinite training data as it learns on mini-batches iteratively and is independent of the training set size. Thirdly, motivated by our relative weight analysis, seamless sample reweighting is incorporated. It rescales samples’ gradients to control the differentiation degree over training examples instead of truncating them by sample mining. In addition to its simplicity and intuitiveness, extensive experiments on three real-world benchmarks demonstrate the superiority of ICE. Deep metric learning (DML) aims to learn a non-linear embedding function (a.k.a. distance metric) such that the semantic similarities over samples are well captured in the feature space . Due to its fundamental function of learning discriminative representations, DML has diverse applications, such as image retrieval , clustering , verification , few-shot learning and zero-shot learning . A key to DML is to design an effective and efficient loss function for supervising the learning process, thus significant efforts have been made (; ; ; ; ;). Some loss functions learn the embedding function from pairwise or triplet-wise relationship constraints (; ;). However, they are known to not only suffer from an increasing number of non-informative samples during training, but also incur considering only several instances per loss computation. Therefore, informative sample mining strategies are proposed (; ; b). Recently, several methods consider semantic relations among multiple examples to exploit their similarity structure (; ;). Consequently, these structured losses achieve better performance than pairwise and triple-wise approaches. In this paper, we tackle the DML problem from a novel perspective. Specifically, we propose a novel loss function inspired by CCE. CCE is well-known in classification problems owing to the fact that it has an intuitive probabilistic interpretation and achieves great performance, e.g., ImageNet classification . However, since CCE learns a decision function which predicts the class label of an input, it learns class-level centres for reference (; a). Therefore, CCE is not scalable to infinite classes and cannot generalise well when it is directly applied to DML . With scalability and structured information in mind, we introduce instance cross entropy (ICE) for DML. It learns an embedding function by minimising the cross entropy between a predicted instance-level matching distribution and its corresponding ground-truth. In comparison with CCE, given a query, CCE aims to maximise its matching probability with the class-level context vector (weight vector) of its ground-truth class, whereas ICE targets at maximising its matching probability with it similar instances. As ICE does not learn class-level context vectors, it is scalable to infinite training classes, which is an intrinsic demand of DML. Similar to (; ; ; ;), ICE is a structured loss as it also considers all other instances in the mini-batch of a given query. We illustrate ICE with comparison to other structured losses in Figure 1. A common challenge of instance-based losses is that many training examples become trivial as model improves. Therefore, we integrate seamless sample reweighting into ICE, which functions similarly with various sample mining schemes (; ; ;). Existing mining methods require either separate time-consuming process, e.g., class mining , or distance thresholds for data pruning (; ;). Instead, our reweighting scheme works without explicit data truncation and mining. It is motivated by the relative weight analysis between two examples. The current common practice of DML is to learn an angular embedding space by projecting all features to a unit hypersphere surface (; ;). We identify the challenge that without sample mining, informative training examples cannot be differentiated and emphasised properly because the relative weight between two samples is strictly bounded. We address it by sample reweighting, which rescales samples' gradient to control the differentiation degree among them. Finally, for intraclass compactness and interclass separability, most methods (; ; ;) use distance thresholds to decrease intraclass variances and increase interclass distances. In contrast, we achieve the target from a perspective of instance-level matching probability. Without any distance margin constraint, ICE makes no assumptions about the boundaries between different classes. Therefore, ICE is easier to apply in applications where we have no prior knowledge about intraclass variances. Our contributions are summarised: We approach DML from a novel perspective by taking in the key idea of matching probability in CCE. We introduce ICE, which is scalable to an infinite number of training classes and exploits structured information for learning supervision. A seamless sample reweighting scheme is derived for ICE to address the challenge of learning an embedding subspace by projecting all features to a unit hypersphere surface. We show the superiority of ICE by comparing with state-of-the-art methods on three real-world datasets. Heated-up, NormFace, TADAM, DRPR, Prototypical Networks, Proxy-NCA. These methods calculate the similarities between a query and class centres (a.k.a. proxies or prototypes) instead of other instances (; a; ; ; ;). In Heated-up and NormFace, the class centres are learned parameters of a fully connected layer, which is similar to Center Loss. While in TADAM, DRPR, and Prototypical Networks, a class centre is the mean over all embeddings of a class. By comparing a sample with other examples other than class centres, more informative instances can contribute more in ICE. NCA , S-NCA . NCA learns similarity relationships between instances. Since original NCA learns the whole training data and its time complexity is quadratically proportional to the scale of training data, S-NCA is proposed recently with linear time complexity with respect to the training data size. Instead, ICE is scalable to infinite training data by iterative learning on randomly sampled small-scale instances matching tasks. S-NCA and NCA share the same learning objective. However, they treat the event of all similar instance being correctly recognised as a whole by a sum accumulator. Instead, we maximise the probability of every similar sample being correctly identified individually. Therefore, ICE's optimisation task is harder, leading to better generalisation. N-pair-mc . The aim of N-pair-mc is to identify one positive example from N − 1 negative examples of N − 1 classes (one negative example per class). In other words, only one (a) A query versus learned parametric class centroids. All T classes in the training set are considered. Prior work: CCE, Heated-up , NormFace (a). (b) A query versus non-parametric class means. Only classes in the mini-batch are considered. Representative work: TADAM , DRPR , Prototypical Networks . (c) N-pair-mc : A query versus one instance per class. A mini-batch has to be 2 examples per class rigidly. Only one instance per negative class is randomly sampled out of 2. (d) NCA and S-NCA : A query versus the rest instances. Figure 1: Our ICE and related losses. The first row shows prior work of a query versus class centres/means while the second row displays the work of a query versus instances. Note that the cross entropy computation and interpretation are different in different losses. For a mini-batch, we show two classes, i.e., circle and rectangle, with 3 examples per class except N-pair-mc which requires 2 samples per class. The icons are at the right bottom. GT means ground-truth matching distribution. When illustrating the losses of a query versus instances in (c), (d) and (e), we index those instances with numbers for clarity, except the query. positive and one negative instance per class are considered per loss computation by simulating CCE exactly. Instead, ICE exploits all negative examples to benefit from richer information. When constructing mini-batches, N-pair-mc requires expensive offline class mining and samples 2 images per class. According to N-pair-mc is superior to NCA. Hyperbolic . It aims to preserve the similarity structures among instances as well. However, it learns a hyperbolic embedding space where the distance depends only on norm of embeddings. Instead, we learn an angular space where the similarity depends only on the angle between embeddings. Besides, Hyperbolic requires a separate sampling of semantic subtrees when the dataset is large. Mining informative examples or emphasising on them are popular strategies in DML: 1) Mining non-trivial samples during training is crucial for faster convergence and better performance. Therefore, sample mining is widely studied in the literature. In pairwise or triplet-wise approaches (; ; ;, data pairs with higher losses are emphasized during gradient backpropagation. As for structured losses, Lifted Struct also focuses on harder examples. Furthermore, and propose to mine hard negative classes to construct informative input mini-batches. Proxy-NCA addresses the sampling problem by learning class proxies. 2) Assigning higher weights to informative examples is another effective scheme (a; c). Beyond, there are some other novel perspectives to address sample mining or weighting, e.g., hardness-aware examples generation and divide-and-conquer of the embedding space . Our proposed ICE has a similarity scaling factor which helps to emphasise more on informative examples. Moreover, as described in , very hard negative pairs are likely to be outliers and it is safer to mine semi-hard ones. In ICE, the similarity scaling factor is flexible in that it controls the emphasis degree on harder samples. Therefore, a proper similarity scaling factor can help mine informative examples and alleviate the disturbance of outliers simultaneously. What makes ours different is that we do not heuristically design the mining or weighting scheme. Instead, it is built-in and we simply scale it as demonstrated in Section 3.4. We remark that Prototypical Networks, Matching Networks and NCA are also scalable and do not require distance thresholds. Therefore, they are illustrated and differentiated in Figure 1. Matching Networks are designed specifically for one-shot learning. Similarly, design mAP-SSVM and mAP-DLM for few-shot learning, which directly optimises the retrieval performance mAP when multiple positives exist. FastAP is similar to and optimises the ranked-based average precision. Instead, ICE processes one positive at a time. Beyond, the setting of few-shot learning is different from deep metric learning: Each mini-batch is a complete subtask and contains a support set as training data and a query set as validation data in few-shot learning. Few-shot learning applies episodic training in practice. Remarkably, TADAM formulates instances versus class centres and also has a metric scaling parameter for adjusting the impact of different class centres. Contrastively, ICE adjusts the influence of other instances. Furthermore, ours is not exactly distance metric scaling since we simply apply naive cosine similarity as the distance metric at the testing stage. That is why we interpret it as a weighting scheme during training. is an input mini-batch, where x i ∈ R h×w×3 and y i ∈ {1, ..., C} represent i-th image and the corresponding label, respectively; {x is a set of N c images belonging to c-th class, ∀c, N c ≥ 2. The number of classes C is generally much smaller than the total number of classes T in the training set (C T). Note that T is allowed to be extremely large in DML. Given a sufficient number of different mini-batches, our goal is to learn an embedding function f that captures the semantic similarities among samples in the feature space. We represent deep embeddings of X as {{f . Given a query, 'positives' and 'negatives' refer to samples of the same class and different classes, respectively. CCE is widely used in a variety of tasks, especially classification problems. As demonstrated in , a deep classifier consists of two joint components: deep feature learning and linear classifier learning. The feature learning module is a transformation (i.e., embedding function f) composed of convolutional and non-linear activation layers. The classifier learning module has one neural layer, which learns T class-level context vectors such that any image has the highest compatibility (logit) with its ground-truth class context vector: where is the learned parameters of the classifier. During training, the goal is to maximise the joint probability of all instances being correctly classified. The identical form is to minimise the negative log-likelihood, i.e., L CCE (X; f, W). Therefore, the learning objective of CCE is: In contrast to CCE, ICE is a loss for measuring instance matching quality (lower ICE means higher quality) and does not need class-level context vectors. We remark that an anchor may have multiple positives, which are isolated in separate matching distributions. There is a matching distribution for every anchor-positive pair versus their negatives as displayed in Figure 1e. Let f c a be a random query, we compute its similarities with the remaining points using dot product. We define the probability of the given anchor x c a matching one of its positives x c i (i = a) as follows: where f We remark: Dot product measures the similarity between two vectors; Eq. represents the probability of a query matching a positive while Eq. is the probability of a query matching its ground-truth class. To maximise p(x between the predicted and ground-truth distributions, which is equivalent to minimise their cross entropy. Since the ground-truth distribution is one-hot encoded, the cross-entropy is − log p(x c i |x c a). To be more general, for the given anchor x c a, there may exist multiple matching points when N c > 2, i.e., |{x c i} i =a | = N c − 1 > 1. In this case, we predict one matching distribution per positive point. Our goal is to maximise the joint probability of all positive instances being correctly identified, i.e.,. A case of two positives matching a given query is described in Figure 1e. In terms of mini-batch, each image in X serves as the anchor iteratively and we aim to maximise the joint probability of all queries {{p x c a} Nc a=1 } C c=1. Equivalently, we can achieve this by minimising the sum of all negative log-likelihoods. Therefore, our proposed ICE on X is as follows: Following the common practice in existing DML methods, we apply L 2 -normalisation to feature embeddings before the inner product. Therefore, the inner product denotes the cosine similarity. The similarity between two feature vectors is determined by their norms and the angle between them. Without L 2 normalisation, the feature norm can be very large, making the model training unstable and difficult. With L 2 normalisation, all features are projected to a unit hypersphere surface. Consequently, the semantic similarity score is merely determined by the direction of learned representations. Therefore, L 2 normalisation can be regarded as a regulariser during training 1. Note that the principle is quite different from recent hyperspherical learning methods (a; a; a). They enforce the learned weight parameters to a unit hypersphere surface and diversify their angles. In contrast, feature normalisation is output regularisation and invariant to the parametrisation of the underlying neural network . In summary, our learning objective is: The feature L 2 -normalisation layer is implemented according to Wang et al. (2017a). It is a differentiable layer and can be easily inserted at the output of a neural net. Intrinsic sample weighting. We find that ICE emphasises more on harder samples from the perspective of gradient magnitude. We demonstrate this by deriving the partial derivatives of L ICE (X; f) with respect to positive and negative examples. Given the query x c a, the partial derivative of its any positive instance is derived by the chain rule: Since ||f Similarly, the partial derivative of its any negative sample is:. Clearly, the harder negative samples own higher matching probabilities and weights. Relative weight analysis. In general, the relative weight is more notable as the exact weight will be rescaled during training, e.g., linear post-processing by multiplying the learning rate. Therefore, we analyse the relative weight between two positive points of the same anchor (i = k = a): Similarly, the relative weight between two negative points of the same anchor (o = c, l = c) is: Note that the positive relative weight in Eq. is only decided by f Non-linear scaling for controlling the relative weight. Inspired by , we introduce a scaling parameter to modify the absolute weight non-linearly: where s ≥ 1 is the scaling parameter. In contrast to p and w,p andŵ represent the rescaled matching probability and partial derivative weight, respectively. We remark that we scale the absolute weight Batch setting: C classes, N c images from c-th class, batch size N = C c=1 N c. Hyper-setting: The scaling parameter s and the number of iterations τ. Input: Initialised embedding function f, iteration counter iter = 0. Output: Updated f. for iter < τ do iter = iter + 1. Sample one mini-batch randomly X = {{x Step 1: Feedforward X into f to obtain feature representations {{f Step 2: Compute the similarities between an anchor and the remaining instances. Every example serves as the anchor iteratively. Step 3: Gradient back-propagation to update the parameters of f using Eq.. end for non-linearly, which is an indirect way of controlling the relative weight. We do not modify the relative weight directly and Eq. and Eq. are only for introducing our motivation. Our objective is to maximise an anchor's matching probability with its any positive instance competing against its negative set. Therefore, we normalise the rescaled weights based on each anchor:. Note that the denominators in Eq. and are the accumulated weights of positives and negatives w.r.t. x c a, respectively. Although there are much more negatives than positives, the negative set and positive set contribute equally as a whole, as indicated by 1/2. N = C c=1 N c is the total number of instances in X. We select each instance as the anchor iteratively and treat all anchors equally, as indicated by 1/N. It is worth noting that during back-propagation, the magnitudes of partial derivatives in Eq. and Eq. To make it more clear and intuitive for understanding, we now analyse a naive case of ICE, where there are two samples per class in every mini-batch, i.e., ∀c, N c = 2, |{x c i} i =a | = N c − 1 = 1. In this case, for each anchor (query), there is only one positive among the remaining data points. As a , the weighting schemes in Eq. for positives and Eq. for negatives can be simplified: Firstly, we have N anchors that are treated equally as indicated by 1/N. Secondly, for each anchor, we aim to recognise its positive example correctly. However, there is a sample imbalance problem because each anchor has only one positive and many negatives. ICE addresses it by treating the positive set (single point) and negative set (multiple points) equally, i.e., 1/2 in Eq. and Eq. 2. Finally, as there are many negative samples, we aim to focus more on informative ones, i.e., harder negative instances with higher matching probabilities with a given anchor. The non-linear transformation can help control the relative weight between two negative points. The weighting scheme shares the same principle as the popular temperature-based categorical cross entropy . The key is that we should consider not only focusing on harder examples, but also the emphasis degree. Algorithm 1 summarises the learning process with ICE. As presented there, the input data format of ICE is the same as CCE, i.e., images and their corresponding labels. In contrast to other methods which require rigid input formats , e.g., triplets and n-pair tuplets, ICE is much more flexible. We iteratively select one image as the anchor. For each anchor, we aim to maximise its matching probabilities with its positive samples against its negative examples. Therefore, the computational complexity over one mini-batch is O(N 2), being the same as recent online metric learning approaches (; b). Note that in FaceNet and N -pair-mc , expensive sample mining and class mining are applied, respectively. For data augmentation and preprocessing, we follow (; . In detail, we first resize the input images to 256 × 256 and then crop it at 227 × 227. We use random cropping and horizontal mirroring for data augmentation during training. To fairly compare with the reported in , we use a centre cropping without horizontal flipping in the test phase. For the embedding size, we set it to 512 on all datasets following (; ; a). To compare fairly with (; ;), we choose GoogLeNet V2 (with batch normalisation) as the backbone architecture initialised by the publicly available pretrained model on ImageNet . We simply change the original 1000-neuron fully connected layers followed by softmax normalisation and CCE to 512-neuron fully connected layers followed by the proposed ICE. For faster convergence, we randomly initialise the new layers and optimise them with 10 times larger learning rate than the others as in . We implement our algorithm in the Caffe framework . The source code will be available soon. Datasets. Following the evaluation protocol in (;, we test our proposed method on three popular fine-grained datasets including CARS196 , CUB-200-2011 (and SOP . A summary of the datasets is given in 2 The weight sum of negatives: Table 1. We also keep the same train/test splits. We remark that to test the generalisation and transfer capability of the learned deep metric, the training and test classes are disjoint. Evaluation protocol. We evaluate the learned representations on the image retrieval task in terms of Recall@K performance . Given a query, its K nearest neighbours are retrieved from the database. Its retrieval score is one if there is an image of the same class in the K nearest neighbours and zero otherwise. Recall@K is the average score of all queries. Training settings. All the experiments are run on a single PC equipped with Tesla V100 GPU with 32GB RAM. For optimisation, we use the stochastic gradient descent (SGD) with a weight decay of 1e −5 and a momentum of 0.8. The base learning rate is set as 1e Remarks. For a fair comparison, we remark that the methods group (; ; b; ;) using GoogLeNet V1 and another group (; ;) using ResNet-50 (are not benchmarked. Besides, ensemble models ; ;) are not considered. HTL also uses GoogLeNet V2, but it constructs a hierarchical similarity tree over the whole training set and updates the tree every epoch, thus being highly unscalable and expensive in terms of both computation and memory. That is why HTL achieves better performance on small datasets but performs worse than ours on the large dataset SOP. Finally, there are some other orthogonal deep metric learning research topics that are worth studying together in the future, e.g., a robust distance metric and metric learning with continuous labels. In GoogLeNet V2, there are three fully connected layers of different depth. We refer them based on their depth: L for the low-level layer (inception-3c/output), M for the mid-level layer (inception-4e/output) and H for the high-level layer (inception5b/output). By default, we use only'H'. We also report the of their combination (L, M, H) for reference following RLL (a). Competitors. All the compared baselines, Triplet Semihard , Lifted Struct , N -pair-mc , Struct Clust , Spectral Clust (, Proxy-NCA , RLL (a) and our ICE are trained and evaluated using the same settings: GoogLeNet V2 serves as the backbone network; All models are initialised with the same pretrained model on ImageNet; All apply the same data augmentation during training and use a centre-cropped image during testing. The of some baselines (; ;) are from , which means they are reimplemented there for a fair comparison. In addition, the of vanilla GoogLeNet V2 pretrained on ImageNet without fine-tuning and with fine-tuning via minimising CCE are reported in , which can be regarded as the most basic baselines. Among these baselines, Proxy NCA is not scalable as class-level proxies are learned during training. Struct Clust and Spectral Clust are clustering-motivated methods which explicitly aim to optimise the clustering quality. We highlight that clustering performance Normalised Mutual Information (NMI) (Schütze et al., 2008) is not a good assessment for SOP because SOP has a large number of classes but only 5.3 images per class on average. Therefore, we only report and compare Recall@K performance. Results. Table 3 compares the of our ICE and those of the state-of-the-art DML losses. ICE achieves the best Recall@1 performance on all benchmarks. We observe that only RLL achieves comparable performance in a few terms. However, RLL is more complex since it has three hyperparameters in total: one weight scaling parameter and two distance margins for positives and negatives, respectively. In addition, its perspective is different since it processes the positive set together similarly with (; c). We note that (c) is also complex in designing weighting schemes and contains four control hyper-parameters. However, our Recall@1 on SOP is 77.3%, which is only 0.9% lower than 78.2% of (c). It is also worth mentioning that among these approaches, except fine-tuned models with CCE, only our method has a clear probability interpretation and aims to maximise the joint instance-level matching probability. As observed, apart from being unscalable, CCE's performance is much worse than the state-of-the-art methods. Therefore, ICE can be regarded as a successful exploration of softmax regression for learning deep representations in DML. The t-SNE visualisation of learned embeddings are available in the supplementary material. We empirically study the impact of the weight scaling parameter s, which is the only hyperparameter of ICE. It functions similarly with the popular sample mining or example weighting (a; b; c) widely applied in the baselines in Table 3. Generally, different s corresponds to different emphasis degree on difficult examples. When s is larger, more difficult instances are assigned with relatively higher weights. In general, small datasets are more sensitive to minor changes of hyper-settings and much easier to overfit. Therefore, the experiments are conducted on the large dataset SOP. The are shown in Table 2. Note that when s is too small, e.g., s = 1, we observe that the training does not converge, which demonstrates the necessity of weighting/mining samples. The most significant observation is that focusing on difficult samples is better but the emphasis degree should be properly controlled. When s increases from 16 to 64, the performance grows gradually. However, when s = 80, we observe the performance drops a lot. That may be because extremely hard samples, e.g., outliers, are emphasised when s is too large. In this paper, we propose a novel instance-level softmax regression framework, named instance cross entropy, for deep metric learning. Firstly, the proposed ICE has clear probability interpretation and exploits structured semantic similarity information among multiple instances. Secondly, ICE is scalable to infinitely many classes, which is required by DML. Thirdly, ICE has only one weight scaling hyper-parameter, which works as mining informative examples and can be easily selected via cross-validation. Finally, distance thresholds are not applied to achieve intraclass compactness and interclass separability. This indicates that ICE makes no assumptions about intraclass variances and the boundaries between different classes. Therefore ICE owns general applicability. A.1 BATCH CONTENT We evaluate the impact of batch content which consists of C classes and k images per class, i.e., ∀c, N c = k. The batch size N = C × k is set to 180. In our experiments, we change the number of classes C from 36 to 90, and the number of images k from 2 to 5, while keeping the batch size unchanged. Table 4 shows the on SOP dataset. We observe that when there are more classes in the mini-batch, the performance is better. We conjecture that as the number of classes increases, the mini-batch training becomes more difficult and helps the model to generalise better. To explore different batch size N, we fix k = 2 and only change C. In this case, N = C × 2. Table 5 shows that as the number of classes increases, the performance grows. In detail, when the number of classes increases from 50 to 90, the performance raises from 74.4% to 77.3% accordingly. One reason may be that as the number of classes increases, it fits the global structure of the test set better, where there are a large number of classes but only a few positive examples. In addition, the increasing difficulty of mini-batch training can help the model to generalise better. The dimension of feature representations is an important factor in many DML methods. We conduct experiments on SOP to see the influence of different embedding size. The are presented in Table 6. We observe that when the embedding size is very small, e.g., 64, the performance is much worse. The performance increases gradually as the embedding size grows. The t-SNE visualisation of learned embeddings is available in Figures 2, 3, 4.

Represent the TLDR for finding the corresponding scientific article

We propose instance cross entropy (ICE) which measures the difference between an estimated instance-level matching distribution and its ground-truth one.

Represent the short summary for finding the corresponding scientific article (long)