Question: Represent the text to find a very short summary (1-2 sentences max).

Question: In order to efficiently learn with small amount of data on new tasks, meta-learning transfers knowledge learned from previous tasks to the new ones. However, a critical challenge in meta-learning is the task heterogeneity which cannot be well handled by traditional globally shared meta-learning methods. In addition, current task-specific meta-learning methods may either suffer from hand-crafted structure design or lack the capability to capture complex relations between tasks. In this paper, motivated by the way of knowledge organization in knowledge bases, we propose an automated relational meta-learning (ARML) framework that automatically extracts the cross-task relations and constructs the meta-knowledge graph. When a new task arrives, it can quickly find the most relevant structure and tailor the learned structure knowledge to the meta-learner. As a , the proposed framework not only addresses the challenge of task heterogeneity by a learned meta-knowledge graph, but also increases the model interpretability. We conduct extensive experiments on 2D toy regression and few-shot image classification and the demonstrate the superiority of ARML over state-of-the-art baselines. Learning quickly is the key characteristic of human intelligence, which remains a daunting problem in machine intelligence. The mechanism of meta-learning is widely used to generalize and transfer prior knowledge learned from previous tasks to improve the effectiveness of learning on new tasks, which has benefited various applications, such as computer vision (;, natural language processing and social good (; a). Most of existing meta-learning algorithms learn a globally shared meta-learner (e.g., parameter initialization (;, meta-optimizer , metric space (; ;) ). However, globally shared meta-learners fail to handle tasks lying in different distributions, which is known as task heterogeneity (; b). Task heterogeneity has been regarded as one of the most challenging issues in meta-learning, and thus it is desirable to design meta-learning models that effectively optimize each of the heterogeneous tasks. The key challenge to deal with task heterogeneity is how to customize globally shared meta-learner by using task-specific information? Recently, a handful of works try to solve the problem by learning a task-specific representation for tailoring the transferred knowledge to each task (; ;). However, the expressiveness of these methods is limited due to the impaired knowledge generalization between highly related tasks. Recently, learning the underlying structure across tasks provides a more effective way for balancing the customization and generalization. Representatively, Yao et al. propose a hierarchically structured meta-learning method to customize the globally shared knowledge to each cluster (b). Nonetheless, the hierarchical clustering structure completely relies on the handcrafted design which needs to be tuned carefully and may lack the capability to capture complex relationships. Hence, we are motivated to propose a framework to automatically extract underlying relational structures from historical tasks and leverage those relational structures to facilitate knowledge customization on a new task. This inspiration comes from the way of structuring knowledge in knowledge bases (i.e., knowledge graphs). In knowledge bases, the underlying relational structures across text entities are automatically constructed and applied to a new query to improve the searching efficiency. In the meta-learning problem, similarly, we aim at automatically establishing the metaknowledge graph between prior knowledge learned from previous tasks. When a new task arrives, it queries the meta-knowledge graph and quickly attends to the most relevant entities (vertices), and then takes advantage of the relational knowledge structures between them to boost the learning effectiveness with the limited training data. The proposed meta-learning framework is named as Automated Relational Meta-Learning (ARML). Specifically, the ARML automatically builds the meta-knowledge graph from meta-training tasks to memorize and organize learned knowledge from historical tasks, where each vertex represents one type of meta-knowledge (e.g., the common contour between birds and aircrafts). To learn the meta-knowledge graph at meta-training time, for each task, we construct a prototype-based relational graph for each class, where each vertex represents one prototype. The prototype-based relational graph not only captures the underlying relationship behind samples, but alleviates the potential effects of abnormal samples. The meta-knowledge graph is then learned by summarizing the information from the corresponding prototype-based relational graphs of meta-training tasks. After constructing the meta-knowledge graph, when a new task comes in, the prototype-based relational graph of the new task taps into the meta-knowledge graph for acquiring the most relevant knowledge, which further enhances the task representation and facilitates its training process. Our major contributions of the proposed ARML are three-fold: it automatically constructs the meta-knowledge graph to facilitate learning a new task; it empirically outperforms the state-ofthe-art meta-learning algorithms; the meta-knowledge graph well captures the relationship among tasks and improves the interpretability of meta-learning algorithms. Meta-learning designs models to learn new tasks or adapt to new environments quickly with a few training examples. There are mainly three research lines of meta-learning: black-box amortized methods design black-box meta-learners to infer the model parameters (; ; ;); gradient-based methods aim to learn an optimized initialization of model parameters, which can be adapted to new tasks by a few steps of gradient descent (; ; ;); non-parametric methods combine parametric meta-learners and non-parametric learners to learn an appropriate distance metric for few-shot classification (; ; ; ; ;). Our work is built upon the gradient-based meta-learning methods. In the line of gradient-based meta-learning, most algorithms learn a globally shared meta-learners from previous tasks (; ;), to improve the effectiveness of learning process on new tasks. However, these algorithms typically lack the ability to handle heterogeneous tasks (i.e., tasks sample from sufficient different distributions). To tackle this challenge, recent works tailor the globally shared initialization to different tasks by customizing initialization (; b) and using probabilistic models (; . Representatively, HSML customizes the globally shared initialization with a manually designed hierarchical clustering structure to balance the generalization and customization (b). However, the handcrafted designed hierarchical structure may not accurately reflect the real structure and the clustering structure constricts the complexity of relationship. Compared with these methods, ARML leverages the most relevant structure from the automatically constructed meta-knowledge graph. Thus, ARML not only discovers more accurate underlying structures to improve the effectiveness of meta-learning algorithms, but also the meta-knowledge graph further enhances the model interpretability. Few-shot Learning Considering a task Ti, the goal of few-shot learning is to learn a model with a dataset Di = {D, θ), and obtain the optimal parameters θi. For the regression problem, the loss function is defined based on the mean square error (i.e., (x j,y j)∈D tr i f θ (xj)−yj 2 2 ) and for the classification problem, the loss function uses the cross entropy loss (i.e., − (x j,y j)∈D tr i log p(yj|xj, f θ)). Usually, optimizing and learning parameter θ for the task Ti with a few labeled training samples is difficult. To address this limitation, meta-learning provides us a new perspective to improve the performance by leveraging knowledge from multiple tasks. Meta-learning and Model-agnostic Meta-learning In meta-learning, a sequence of tasks {T1, ..., TI} are sampled from a task-level probability distribution p(T), where each one is a few-shot learning task. To facilitate the adaption for incoming tasks, the meta-learning algorithm aims to find a well-generalized meta-learner on I training tasks at meta-learning phase. At meta-testing phase, the optimal meta-learner is applied to adapt the new tasks Tt. In this way, meta-learning algorithms are capable of adapting to new tasks efficiently even with a shortage of training data for a new task. Model-agnostic meta-learning (MAML) , one of the representative algorithms in gradient-based meta-learning, regards the meta-learner as the initialization of parameter θ, i.e., θ0, and learns a well-generalized initialization θ * 0 during the meta-training process. The optimization problem is formulated as (one gradient step as exemplary): At the meta-testing phase, to obtain the adaptive parameter θt for each new task Tt, we finetune the initialization of parameter θ * 0 by performing gradient updates a few steps, i.e., In this section, we introduce the details of the proposed ARML. To better explain how it works, we show its framework in Figure 1. The goal of ARML is to facilitate the learning process of new tasks by leveraging transferable knowledge learned from historical tasks. To achieve this goal, we introduce a meta-knowledge graph, which is automatically constructed at the meta-training time, to organize and memorize historical learned knowledge. Given a task, which is built as a prototypebased relational structure, it taps into the meta-knowledge graph to acquire relevant knowledge for enhancing its own representation. The enhanced prototype representations further aggregate and incorporate with meta-learner for fast and effective adaptions by utilizing a modulating function. In the following subsections, we elaborate three key components: prototype-based sample structuring, automated meta-knowledge graph construction and utilization, and task-specific knowledge fusion and adaptation, respectively. Given a task which involves either classifications or regressions regarding a set of samples, we first investigate the relationships among these samples. Such relationship is represented by a graph, called prototype-based relational graph in this work, where the vertices in the graph denote the prototypes of different classes while the edges and the corresponding edge weights are created based on the similarities between prototypes. Constructing the relational graph based on prototypes instead of raw samples allows us to alleviate the issue raised by abnormal samples. As the abnormal samples, which locate far away from normal samples, could pose significant concerns especially when only a limited number of samples are available for training. Specifically, for classification problem, the prototype, denoted by c, is defined as: where N tr k denotes the number of samples in class k. E is an embedding function, which projects xj into a hidden space where samples from the same class are located closer to each other while samples from different classes stay apart. For regression problem, it is not straightforward to construct Figure 1: The framework of ARML. For each task T i, ARML first builds a prototype-based relational structure R i by mapping the training samples D tr i into prototypes, with each prototype represents one class. Then, R i interacts with the meta-knowledge graph G to acquire the most relevant historical knowledge by information propagation. Finally, the task-specific modulation tailors the globally shared initialization θ 0 by aggregating of raw prototypes and enriched prototypes, which absorbs relevant historical information from the meta-knowledge graph. the prototypes explicitly based on class information. Therefore, we cluster samples by learning an assignment matrix Pi ∈ R K×N tr. Specifically, we formulate the process as: where Pi[k] represents the k-th row of Pi. Thus, training samples are clustered to K clusters, which serve as the representation of prototypes. After calculating all prototype representations {c k i |∀k ∈ [1, K]}, which serve as the vertices in the the prototype-based relational graph Ri, we further define the edges and the corresponding edge weights. The edge weight A Ri (c where Wr and br represents learnable parameters, γr is a scalar and σ is the Sigmoid function, which normalizes the weight between 0 and 1. For simplicity, we denote the prototype-based relational graph K×d represent a set of vertices, with each one corresponds to the prototype from a class, while gives the adjacency matrix, which indicates the proximity between prototypes. In this section, we first discuss how to organize and distill knowledge from historical learning process and then expound how to leverage such knowledge to benefit the training of new tasks. To organize and distill knowledge from historical learning process, we construct and maintain a meta-knowledge graph. The vertices represent different types of meta-knowledge (e.g., the common contour between aircrafts and birds) and the edges are automatically constructed to reflect the relationship between meta-knowledge. When serving a new task, we refer to the meta-knowledge, which allows us to efficiently and automatically identify relational knowledge from previous tasks. In this way, the training of a new task can benefit from related training experience and get optimized much faster than otherwise possible. In this paper, the meta-knowledge graph is automatically constructed at the meta-training phase. The details of the construction are elaborated as follows: Assuming the representation of an vertex g is given by h g ∈ R d, we define the meta-knowledge graph as G = (HG, AG), where HG = {h G×G denote the vertex feature matrix and vertex adjacency matrix, respectively. To better explain the construction of the meta-knowledge graph, we first discuss the vertex representation HG. During meta-training, tasks arrive one after another in a sequence and their corresponding vertices representations are expected to be updated dynamically in a timely manner. Therefore, the vertex representation of meta-knowledge graph are defined to get parameterized and learned at the training time. Moreover, to encourage the diversity of meta-knowledge encoded in the meta-knowledge graph, the vertex representations are randomly initialized. Analogous to the definition of weight in the prototype-based relational graph Ri in equation 4, the weight between a pair of vertices j and m is constructed as: where Wo and bo represent learnable parameters and γo is a scalar. To enhance the learning of new tasks with involvement of historical knowledge, we query the prototype-based relational graph in the meta-knowledge graph to obtain the relevant knowledge in history. The ideal query mechanism is expected to optimize both graph representations simultaneously at the meta-training time, with the training of one graph facilitating the training of the other. In light of this, we construct a super-graph Si by connecting the prototype-based relational graph Ri with the meta-knowledge graph G for each task Ti. The union of the vertices in Ri and G contributes to the vertices in the super-graph. The edges in Ri and G are also reserved in the super-graph. We connect Ri with G by creating links between the prototype-based relational graph with the meta-knowledge graph. The link between prototype c j i in prototype-based relational graph and vertex h m in metaknowledge graph is weighted by the similarity between them. More precisely, for each prototype c and {h m |∀m ∈ [1, G]} as follows: where γ s is a scaling factor. We denote the intra-adjacent matrix as AS = {AS (c K×G . Thus, for task T i, the adjacent matrix and feature matrix of super-graph After constructing the super-graph Si, we are able to propagate the most relevant knowledge from meta-knowledge graph G to the prototype-based relational graph Ri by introducing a Graph Neural Networks (GNN). In this work, following the "message-passing" framework , the GNN is formulated as: where MP(·) is the message passing function and has several possible implementations (; ; Veličković et al., 2018) is the vertex embedding after l layers of GNN and W (l) is a learnable weight matrix of layer l. The input After stacking L GNN layers, we get the information-propagated feature representation for the prototype-based relational graph Ri as the top-K rows of H After propagating information form meta-knowledge graph to prototype-based relational graph, in this section, we discuss how to learn a well-generalized meta-learner for fast and effective adaptions to new tasks with limited training data. To tackle the challenge of task heterogeneity, in this paper, we incorporate task-specific information to customize the globally shared meta-learner (e.g., initialization here) by leveraging a modulating function, which has been proven to be effective to provide customized initialization in previous studies ). The modulating function relies on well-discriminated task representations, while it is difficult to learn all representations by merely utilizing the loss signal derived from the test set D ts i. To encourage such stability, we introduce two reconstructions by utilizing two auto-encoders. There are two collections of parameters, i.e, CR i andĈR i, which contribute the most to the creation of the task-specific meta-learner. CR i express the raw prototype information without tapping into the meta-knowledge graph, whileĈR i give the prototype representations after absorbing the relevant knowledge from the meta-knowledge graph. Therefore, the two reconstructions are built on CR i andĈR i. To reconstruct CR i, an aggregator AG q (·) (e.g., recurrent network, fully connected layers) is involved to encode CR i into a dense representation, which is further fed into a decoder AG q dec (·) to achieve reconstructions. Compute the similarity between each prototype and meta-knowledge vertex in equation 6 and construct the super-graph Si 8: Apply GNN on super-graph Si and get the information-propagated representationĈR i 9: Aggregate CR i in equation 8 andĈR i in equation 9 to get the representations qi, ti and reconstruction loss Lq, Lt Compute the task-specific initialization θ0i in equation 10 and update end for 12: 13: end while Then, the corresponded task representation qi of CR i is summarized by applying a mean pooling operator over prototypes on the encoded dense representation. Formally, Similarly, we reconstructĈR i and get the corresponded task representation ti as follows: The reconstruction errors in Equations 8 and 9 pose an extra constraint to enhance the training stability, leading to improvement of task representation learning. After getting the task representation qi and ti, the modulating function is then used to tailor the task-specific information to the globally shared initialization θ0, which is formulated as: where Wg and b g is learnable parameters of a fully connected layer. Note that we adopt the Sigmoid gating as exemplary and more discussion about different modulating functions can be found in ablation studies of Section 5. For each task Ti, we perform the gradient descent process from θ0i and reach its optimal parameter θi. Combining the reconstruction loss Lt and Lq with the meta-learning loss defined in equation 1, the overall objective function of ARML is: where µ1 and µ2 are introduced to balance the importance of these three items. Φ represents all learnable parameters. The algorithm of meta-training process of ARML is shown in Alg. 2. The details of the meta-testing process of ARML are available in Appendix A. In this section, we conduct extensive experiments to demonstrate the effectiveness of the ARML on 2D regression and few-shot classification. We compare our proposed ARML with two types of baselines: Gradient-based meta-learning methods: both globally shared methods (MAML , Meta-SGD ) and task-specific methods (MT-Net , MUMO-MAML , HSML (b), BMAML ) are considered for comparison. Other meta-learning methods (non-parametric and black box amortized methods): we select globally shared methods VERSA , Prototypical Network (ProtoNet) , TapNet , we use the GRU as the encoder and decoder in this structure. We adopt one layer GCN with tanh activation as the implementation of GNN in equation 7. For the modulation network, we test sigmoid, tanh and Film modulation, and find that sigmoid modulation achieves best performance. Thus, in the future experiment, we set the sigmoid modulation as modulating function. More detailed discussion about experiment settings are presented in Appendix B. Dataset Description In 2D regression problem, we adopt the similar regression problem settings as; b; ), which includes several families of functions. In this paper, to model more complex relational structures, we design a 2D regression problem rather than traditional 1D regression. Results and Analysis In Figure 2, we summarize the interpretation of meta-knowledge graph (see top figure, and more cases are provided in Figure 8 of Appendix G.4) and the the qualitative (see bottom table) of 10-shot 2D regression. In the bottom table, we can observe that ARML achieves the best performance as compared to competitive gradient-based meta-learning methods, i.e., globally shared models and task-specific models. This finding demonstrates that the meta-knowledge graph is necessary to model and capture task-specific information. The superior performance can also be interpreted in the top figure. In the left, we show the heatmap between prototypes and meta-knowledge vertices (darker color means higher similarity). We can see that sinusoids and line activate V1 and V4, which may represent curve and line, respectively. V1 and V4 also contribute to quadratic and quadratic surface, which also show the similarity between these two families of functions. V3 is activated in P0 of all functions and the quadratic surface and ripple further activate V1 in P0, which may show the different between 2D functions and 3D functions (sinusoid, line, quadratic and cubic lie in the subspace). Specifically, in the right figure, we illustrate the meta-knowledge graph, where we set a threshold to filter the link with low similarity score and show the rest. We can see that V3 is the most popular vertice and connected with V1, V5 (represent curve) and V4 (represent line). V1 is further connected with V5, demonstrating the similarity of curve representation. In the few-shot classification problem, we first use the benchmark proposed in (b), where four fine-grained image classification datasets are included (Aircraft), and FGVCx-Fungi (Fungi)). For each few-shot classification task, it samples classes from one of four datasets. In this paper, we call this dataset as Plain-Multi and each fine-grained dataset as subdataset. Then, to demonstrate the effectiveness of our proposed model for handling more complex underlying structures, in this paper, we increase the difficulty of few-shot classification problem by introducing two image filters: blur filter and pencil filter. Similar as , for each image in PlainMulti, one artistic filters are applied to simulate a changing distribution of few-shot classification tasks. After applying the filters, the total number of subdatasets is 12 and each tasks is sampled from one of them. This data is named as Art-Multi. More detailed descriptions of the effect of different filters is discussed in Appendix C. Following the traditional meta-learning settings, all datasets are divided into meta-training, metavalidation and meta-testing classes. The traditional N-way K-shot settings are used to split training and test set for each task. We adopt the standard four-block convolutional layers as the base learner for ARML and all baselines for fair comparison. The number of vertices of meta-knowledge graph for Plain-Multi and Art-Multi datasets are set as 4 and 8, respectively. Additionally, for the miniImagenet and tieredImagenet , similar as, which tasks are constructed from a single domain and do not have heterogeneity, we compare our proposed ARML with baseline models and present the in Appendix D. Overall Performance Experimental for Plain-Multi and Art-Multi are shown in Table 1 and Table 2, respectively. For each dataset, the performance accuracy with 95% confidence interval is reported. Due to the space limitation, in Art-Multi dataset, we only show the average value of each filter here. The full are shown in Table 8 of Appendix E. In these two tables, first, we can observe that task-specific gradient-based models (MT-Net, MUMOMAML, HSML, BMAML) significantly outperforms globally shared models (MAML, Meta-SGD). Second, compared ARML with other task-specific gradient-based meta-learning methods, the better performance confirms that ARML can model and extract task-specific information more accurately by leveraging the constructed meta-knowledge graph. Especially, the performance gap between the ARML and HSML verifies the benefits of relational structure compared with hierarchical clustering structure. Third, as a gradientbased meta-learning algorithm, ARML can also outperform methods of other research lines (i.e., ProtoNet, TADAM, TapNet and VERSA). Finally, to show the effectiveness of proposed components in ARML, we conduct comprehensive ablation studies in Appendix F. The further demonstrate the effectiveness of prototype-based relational graph and meta-knowledge graph. In this section, we conduct extensive qualitative analysis for the constructed meta-knowledge graph, which is regarded as the key component in ARML. Due to the space limit, we present the on Art-Multi datasets here and the analysis of Plain-Multi with similar observations are discussed in Appendix G.1. We further analyze the effect To analyze the learned meta-knowledge graph, for each subdataset, we randomly select one task as exemplary (see Figure 9 of Appendix G.4 for more cases). For each task, in the left part of Figure 3, we show the similarity heatmap between prototypes and vertices in meta-knowledge graph, where deeper color means higher similarity. V0-V8 and P1-P5 denotes the different vertices and prototypes, respectively. The meta-knowledge graph is also illustrated in the right part. Similar as the graph in 2D regression, we set a threshold to filter links with low similarity and illustrate the rest of them. First, we can see that the V1 is mainly activated by bird and aircraft (including all filters), which may reflect the shape similarity between bird and aircraft. Second, V2, V3, V4 are firstly activated by texture and they form a loop in the meta-knowledge graph. Especially, V2 also benefits images with blur and pencil filters. Thus, V2 may represent the main texture and facilitate the training process on other subdatasets. The meta-knowledge graph also shows the importance of V2 since it is connected with almost all other vertices. Third, when we use blur filter, in most cases (bird blur, texture blur, fungi blur), V7 is activated. Thus, V7 may show the similarity of images with blur filter. In addition, the connection between V7 and V2 and V3 show that classify blur images may depend on the texture information. Fourth, V6 (activated by aircraft mostly) connects with V2 and V3, justifying the importance of texture information to classify the aircrafts. In this paper, to improve the effectiveness of meta-learning for handling heterogeneous task, we propose a new framework called ARML, which automatically extract relation across tasks and construct a meta-knowledge graph. When a new task comes in, it can quickly find the most relevant relations through the meta-knowledge graph and use this knowledge to facilitate its training process. The experiments demonstrate the effectiveness of our proposed algorithm. In the future, we plan to investigate the problem in the following directions: we are interested to investigate the more explainable semantic meaning in the meta-knowledge graph on this problem; Figure 3: Interpretation of meta-knowledge graph on Art-Multi dataset. For each subdataset, we randomly select one task from them. In the left, we show the similarity heatmap between prototypes (P0-P5) and meta-knowledge vertices (V0-V7). In the right part, we show the meta-knowledge graph. we plan to extend the ARML to the continual learning scenario where the structure of meta-knowledge graph will change over time; our proposed model focuses on tasks where the feature space, the label space are shared. We plan to explore the relational structure on tasks with different feature and label spaces. The work was supported in part by NSF awards #1652525 and #1618448. The views and contained in this paper are those of the authors and should not be interpreted as representing any funding agencies. Algorithm 2 Meta-Testing Process of ARML Require: Training data D tr t of a new task T t 1: Construct the prototype-based relational graph R t by computing prototype in equation 2 and weight in equation 4 2: Compute the similarity between each prototype and meta-knowledge vertice in equation 6 and construct the super-graph St 3: Apply GNN on super-graph St and get the updated prototype representationĈR t 4: Aggregate CR t in equation 8,ĈR t in equation 9 and get the representations qt, tt 5: Compute the task-specific initialization θ0t in equation 10 6: Update parameters In 2D regression problem, we set the inner-loop stepsize (i.e., α) and outer-loop stepsize (i.e., β) as 0.001 and 0.001, respectively. The embedding function E is set as one layer with 40 neurons. The autoencoder aggregator is constructed by the gated recurrent structures. We set the meta-batch size as 25 and the inner loop gradient steps as 5. In few-shot image classification, for both Plain-Multi and Art-Multi datasets, we set the corresponding inner stepsize (i.e., α) as 0.001 and the outer stepsize (i.e., β) as 0.01. For the embedding function E, we employ two convolutional layers with 3 × 3 filters. The channel size of these two convolutional layers are 32. After convolutional layers, we use two fully connected layers with 384 and 128 neurons for each layer. Similar as the hyperparameter settings in 2D regression, the autoencoder aggregator is constructed by the gated recurrent structures, i.e., AG t, AG t dec AG q, AG q dec are all GRUs. The meta-batch size is set as 4. For the inner loop, we use 5 gradient steps. For the gradient-based baselines (i.e., MAML, MetaSGD, MT-Net, BMAML. MUMOMAML, HSML), we use the same inner loop stepsize and outer loop stepsize rate as our ARML. As for non-parametric based meta-learning algorithms, both TADAM and Prototypical network, we use the same meta-training and meta-testing process as gradient-based models. Additionally, TADAM uses the same embedding function E as ARML for fair comparison (i.e., similar expressive ability). In this dataset, we use pencil and blur filers to change the task distribution. To investigate the effect of pencil and blur filters, we provide one example in Figure 4. We can observe that different filters in different data distributions. All used filter are provided by OpenCV 1. For miniimagenet and tieredImagenet, since it do not have the characteristic of task heterogeneity, we show the in Table 3 and Table 4, respectively. In this table, we compare our model with other gradient-based meta-learning models (the top baselines are globally shared models and the bottom baselines are task-specific models). Similar as, we also apply the standard 4-block convolutional layers for each baseline. For MT-Net on MiniImagenet, we use the reported in (b), which control the model with the same expressive power. Most task-specific models including ARML achieve the similar performance on the standard benchmark due to the homogeneity between tasks. 48.70 ± 1.84% LLAMA 49.40 ± 1.83% Reptile 49.97 ± 0.32% MetaSGD 50.47 ± 1.87% MT-Net 49.75 ± 1.83% MUMOMAML 49.86 ± 1.85% HSML (b) 50.38 ± 1.85% PLATIPUS 50.13 ± 1.86% ARML 50.42 ± 1.73% Table 4: Performance comparison on the 5-way, 1-shot tieredImagenet dataset. Algorithms 5-way 1-shot Accuracy MAML 51.37 ± 1.80% Reptile 49.41 ± 1.82% MetaSGD 51.48 ± 1.79% MT-Net 51.95 ± 1.83% MUMOMAML 52.59 ± 1.80% HSML (b) 52.67 ± 1.85% ARML 52.91 ± 1.83% We provide the full table of Art-Multi Dataset in Table 8. In this table, we can see our proposed ARML outperforms almost all baselines in every sub-datasets. In this section, we perform the ablation study of the proposed ARML to demonstrate the effectiveness of each component. The of ablation study on 5-way, 5-shot scenario for Art-Multi and PlainMulti datasets are presented in Table 5 and Table 6, respectively. Specifically, to show the effectiveness of prototype-based relational graph, in ablation I, we apply the mean pooling to aggregate each sample and then feed it to interact with meta-knowledge graph. In ablation II, we use all samples to construct the sample-level relational graph without constructing prototype. In ablation III, we remove the links between prototypes. Compared with ablation I, II and III, the better performance of ARML shows that structuring samples can better handling the underlying relations alleviating the effect of potential anomalies by structuring samples as prototypes. In ablation IV, we remove the meta-knowledge graph and use the prototype-based relational graph with aggregator AG q as the task representation. The better performance of ARML demonstrates the effectiveness of meta-knowledge graph for capturing the relational structure and facilitating the classification performance. We further remove the reconstruction loss in ablation V and replace the encoder/decoder structure as MLP in ablation VI. The demonstrate that the autoencoder structure benefits the process of task representation learning and selected encoder and decoder. In ablation VII, we share the gate value within each filter in Convolutional layers. Compared with VII, the better performance of ARML indicates the benefit of customized gate for each parameter. In ablation VIII and IX, we change the modulate function to Film and tanh, respectively. We can see that ARML is not very sensitive to the modulating activation, and sigmoid function is slightly better in most cases. Figure 5: Interpretation of meta-knowledge graph on Plain-Multi dataset. For each subdataset, one task is randomly selected from them. In the left figure, we show the similarity heatmap between prototypes (P1-P5) and meta-knowledge vertices (denoted as E1-E4), where deeper color means higher similarity. In the right part, we show the meta-knowledge graph, where a threshold is also set to filter low similarity links. We first investigate the impact of vertice numbers in meta-knowledge graph. The of Art-Multi (5-way, 5-shot) are shown in Table 7. From the , we can notice that the performance saturates as the number of vertices around 8. One potential reason is that 8 vertices are enough to capture the potential relations. If we have a larger datasets with more complex relations, more vertices may be needed. In addition, if the meta-knowledge graph do not have enough vertices, the worse performance suggests that the graph may not capture enough relations across tasks. In this part, we provide the case study to visualize the task structure of HSML and ARML. HSML is one of representative task-specific meta-learning methods, which adapts transferable knowledge by introducing a task-specific representation. It proposes a tree structure to learn the relations between tasks. However, the structure requires massive labor efforts to explore the optimal structure. By contrast, ARML automatically learn the relation across tasks by introducing the knowledge graph. In addition, ARML fully exploring there types of relations simultaneously, i.e., the prototype-prototype, prototype-knowledge and knowledge-knowledge relations. To compare these two models, we show the case studies of HSML and ARML in Figure 6 and Figure 7. For tasks sampled from bird, bird blur, aircraft and aircraft blur are selected for this comparison. Following case study settings in the original paper (b), for each task, we show the soft-assignment probability to each cluster and the learned hierarchical structure. For ARML, like 3, we show the learned meta-knowledge and the similarity heatmap between prototypes and meta-knowledge vertices. In this figures we can observe that ARML constructs relations in a more flexible way by introducing the graph structure. More specifically, while HSML activate relevant node in a fixed two-layer hierarchical way, ARML provides more possibilities to leverage previous learned tasks by leveraging prototypes and the learned meta-knowledge graph. Published as a conference paper at ICLR 2020 We provide additional case study in this section. In Figure 8, we show the cases of 2D regression and the additional cases of Art-Multi are illustrated in Figure 9. We can see the additional cases also support our observations and interpretations.

Represent the short summary for finding the corresponding scientific article (long)

Addressing task heterogeneity problem in meta-learning by introducing meta-knowledge graph

This paper proposes a deep generative classifier which is effective to detect out-of-distribution samples as well as classify in-distribution samples, by integrating the concept of Gaussian discriminant analysis into deep neural networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, a deep boosting algorithm is developed to learn more discriminative ensemble classifier by seamlessly combining a set of base deep CNNs (base experts) with diverse capabilities, e.g., these base deep CNNs are sequentially trained to recognize a set of object classes in an easy-to-hard way according to their learning complexities. Our experimental have demonstrated that our deep boosting algorithm can significantly improve the accuracy rates on large-scale visual recognition. The rapid growth of computational powers of GPUs has provided good opportunities for us to develop scalable learning algorithms to leverage massive digital images to train more discriminative classifiers for large-scale visual recognition applications, and deep learning BID19 BID20 BID3 has demonstrated its outstanding performance because highly invariant and discriminant features and multi-way softmax classifier are learned jointly in an end-to-end fashion. Before deep learning becomes so popular, boosting has achieved good success on visual recognition BID21. By embedding multiple weak learners to construct an ensemble one, boosting BID15 can significantly improve the performance by sequentially training multiple weak learners with respect to a weighted error function which assigns larger weights to the samples misclassified by the previous weak learners. Thus it is very attractive to invest whether boosting can be integrated with deep learning to achieve higher accuracy rates on large-scale visual recognition. By using neural networks to replace the traditional weak learners in the boosting frameworks, boosting of neural networks has received enough attentions BID23 BID10 BID7 BID9. All these existing deep boosting algorithms simply use the weighted error function (proposed by Adaboost ) to replace the softmax error function (used in deep learning) that treats all the errors equally. Because different object classes may have different learning complexities, it is more attractive to invest new deep boosting algorithm that can use different weights over various object classes rather than over different training samples. Motivated by this observation, a deep boosting algorithm is developed to generate more discriminative ensemble classifier by combining a set of base deep CNNs with diverse capabilities, e.g., all these base deep CNNs (base experts) are sequentially trained to recognize different subsets of object classes in an easy-to-hard way according to their learning complexities. The rest of the paper is organized as: Section 2 briefly reviews the related work; Section 3 introduce our deep boosting algorithm; Section 4 reports our experimental ; and we conclude this paper at Section 5. In this section, we briefly review the most relevant researches on deep learning and boosting. Even deep learning has demonstrated its outstanding abilities on large-scale visual recognition BID6 BID19 BID20 BID3 BID4, it still has room to improve: all the object classes are arbitrarily assumed to share similar learning complexities and a multi-way softmax is used to treat them equally. For recognizing large numbers of object classes, there may have significant differences on their learning complexi-ties, e.g., some object classes may be harder to be recognized than others. Thus learning their deep CNNs jointly may not be able to achieve the global optimum effectively because the gradients of their joint objective function are not uniform for all the object classes and such joint learning process may distract on discerning some object classes that are hard to be discriminated. For recognizing large numbers of object classes with diverse learning complexities, it is very important to organize them in an easy-to-hard way according to their learning complexities and learn their deep CNNs sequentially. By assigning different weights to the training samples adaptively, boosting BID15 BID1 BID16 has provided an easy-to-hard approach to train a set of weak learners sequentially. Thus it is very attractive to invest whether we can leverage boosting to learn a set of base deep CNNs sequentially for recognizing large numbers of object classes in an easy-to-hard way. Some deep boosting algorithms have been developed by seamlessly integrating boosting with deep neural networks to improve the performance in practice. BID17 BID18 proposed the first work to integrate Adaboost with neural networks for online character recognition application. BID23 extended the Adaboosting neural networks algorithm for credit scoring. Recently, BID11 developed a deep incremental boosting method which increases the size of neural network at each round by adding new layers at the end of the network. Moreover, BID10 integrated residual networks with incremental boosting and built an ensemble of residual networks via adding one more residual block to the previous residual network at each round of boosting. All these methods combine the merits of boosting and neural networks; they train each base network either using a different training set by resampling with a probability distribution derived from the error weight, or directly using the weighted cost function for the base network. Alternatively, BID14 proposed a margin enforcing loss for multi-class boosting and presented two ways to minimize the ing risk: the one is coordinate descent approach which updates one predictor component at a time, the other way is based on directional functional derivative and updates all components jointly. By applying the first way, i.e., coordinate descent, BID0 designed ensemble learning algorithm for binary-class classification using deep decision trees as base classifiers and gave the data-dependent learning bound of convex ensembles, and BID7 furthermore extended it to multi-class version. By applying the second way, i.e., directional derivative descent, BID9 developed an algorithm for boosting deep convolutional neural networks (CNNs) based on least squares between weights and directional derivatives, which differs from the original method based on inner product of weights and directional derivative in BID14. All above algorithms focus on seeking the optimal ensemble predictor via changing the error weights of samples; they either update one component of the predictor per boosting iteration, or update all components simultaneously. On the other hand, our deep boosting algorithm focuses on combining a set of base deep CNNs with diverse capabilities: large numbers of object classes are automatically organized in an easyto-hard way according to their learning complexities; all these base deep CNNs (base experts) are sequentially learned to recognize different subsets of object classes; and these base deep CNNs with diverse capabilities are seamlessly combined to generate more discriminative ensemble classifier. In this paper, a deep boosting algorithm is developed by seamlessly combining a set of base deep CNNs with various capabilities, e.g., all these base deep CNNs are sequentially trained to recognize different subsets of object classes in an easy-to-hard way according to their learning complexities. Our deep boosting algorithm uses the base deep CNNs as its weak learners, and many well-designed deep networks (such as AlexNet BID6, VGG BID19, ResNet BID3, and huang2016densely), can be used as its base deep CNNs. It is worth noting that all these well-designed deep networks [] optimize their structures (i.e., numbers of layers and units in each layer), their node weights and their softmax jointly in an end-to-end manner for recognizing the same set of object classes. Thus our deep boosting algorithm is firstly implemented for recognizing 1,00 object classes, however, it is straightward to extend our current implementation Normalization: DISPLAYFORM0 Training the t th base deep CNNs f t (x) via Loss t with respect to the importance distribution DISPLAYFORM1 Calculating the error per category for f t (x): ε t (l), (l = 1, ..., C); DISPLAYFORM2 Computing the weighted error for f t (x): DISPLAYFORM3 Setting DISPLAYFORM4.., C), so that hard object classes misclassified by f t (x) can receive larger weights (importances) when training the (t + 1) th base deep CNNs at the next round; 8: end for 9: Ensembling: DISPLAYFORM5 when huge deep networks (with larger capacities) are available in the future and being used as the base deep CNNs. As illustrated in Algorithm 1, our deep boosting algorithm contains the following key components: (a) Training the t th base deep CNNs (base expert) f t (x) by focusing on achieving higher accuracy rates for some particular object classes; (b) Estimating the weighted error function for the t th base deep CNNs f t (x) according to the distribution of importances D t for C object classes; (c) Updating the distribution of importances D t+1 for C object classes to train the (t + 1) th base deep CNNs by spending more efforts on distinguishing the hard object classes which are not classified very well by the previous base deep CNNs; (d) Such iterative training process stops when the maximum number of iterations is reached or a certain level of the accuracy rates is achieved. For the t th base expert f t (x), we firstly employ deep CNNs to map x into more separable feature space h t (x; θ t), followed by a fully connected discriminant layer and a C-way softmax layer. The output of the t th base expert is the predicted multi-class distribution, denoted as DISPLAYFORM0 ⊤, whose each component p t (l|x) is the probability score of x assigned to the object class l, (l = 1, ..., C): DISPLAYFORM1 where θ t and w lt, (l = 1, ..., C) are the model parameters for the t th base expert f t (x). Based on the above probability score, the category label of x can be predicted by the t th base expert as follows: DISPLAYFORM2 Suppose that training set consists of N labeled samples from C classes: DISPLAYFORM3. To train the t th base expert f t (x), the model parameters can be learned by maximizing the objective function in the form of weighted margin as follows: DISPLAYFORM4 where DISPLAYFORM5 Herein the indicator function 1(y i = l) is equal to 1 if y i = l; otherwise zero. N l denotes the number of samples belonging to the l th object class. D t (l) is the normalized importance score for class l in the t th base expert f t (x). By using the distribution of importances [D t,..., D t (C)] to approximate the learning complexities for C object classes, our deep boosting algorithm can push the current base deep CNNs to focus on distinguishing the object classes which are hard classified by the previous base deep CNNs, thus it can support an easy-to-hard solution for large-scale visual recognition.ξ lt measures the margin between the average confidence on correctly classified examples and the average confidence on misclassified examples for the l th object class. If the second item in Eq. FORMULA11 is small enough and negligible, DISPLAYFORM6, then maximizing the objective function in Eq. FORMULA10 is equivalent to maximizing the weighted likelihood. For the t th base expert f t (x), the classification error rate over the training samples in l th object class is as follows: DISPLAYFORM7 This error rate is used to update category weight and the loss function of the next weak learner, and above definition encourages predictors with large margin to improve the discrimination between correct class and incorrect classe competing with it. Error rate calculated by Eq. FORMULA15 is in soft decision with probability; alternatively, we can also simply compute the error rate in hard decision as DISPLAYFORM8 where the hyperparameter λ controls the threshold, and we constrain λ > 1 2 (i.e., 1 2λ < 1) such that the threshold makes sense. The larger the hyper-parameter λ is, the more strict the precision requirement becomes. We then compute the weighted error rate ε t over all classes for f t (x) such that hard object classes are focused on by the next base expert. DISPLAYFORM9 The distribution of importances is initialized equally for all C object classes: DISPLAYFORM10, and it is updated along the iterative learning process by emphasizing the object classes which are heavily misclassified by the previous base deep CNNs: DISPLAYFORM11 where β t should be an increasing function of ε t, and its range should be 0 < β t < 1. It should be pointed out that λϵ t (l) denotes the product of λ and ϵ t (l). Such update of distribution encourages the next base network focusing on the categories that are hard to classify. As shown in Section 4, to guarantee the upper boundary of ratio (the number of heavily misclassified categories over the number of all classes) to be minimized, we set DISPLAYFORM12 Normalization of the updated importances distribution can be easily carried out: DISPLAYFORM13 The distribution of importances is used to: (a) separate the hard object classes (heavily misclassified by the previous base deep CNNs) from the easy object classes (which have been classified correctly by the previous base deep CNNs); (b) estimate the weighted error function for the (t + 1) th base deep CNNs f t+1 (x), so that it can spend more efforts on distinguishing the hard object classes misclassified by the previous base deep CNNs. After T iterations, we can obtain T base deep CNNs (base experts) {f 1, · · ·, f t, · · ·, f T}, which are sequentially trained to recognize different subsets of C object classes in an easy-to-hard way according to their learning complexities. All these T base deep CNNs are seamlessly combined to generate more discriminative ensemble classifier g(x) for recognizing C object classes: DISPLAYFORM14 where DISPLAYFORM15 is a normalization factor. By diversifying a set of base deep CNNs on their capabilities (i.e., they are trained to recognize different subsets of C object classes in an easyto-hard way), our deep boosting algorithm can obtain more discriminative ensemble classifier g(x) to significantly improve the accuracy rates on large-scale visual recognition. To apply such ensembled classifier for recognition, for a given test sample x test, it firstly goes through all these base deep CNNs to obtain T deep representations {h 1, · · ·, h T} and then its final probability score p(l|x test) to be assigned into the lth object class is calculated as follows: DISPLAYFORM16 3.2 SELECTION OF β tIn our deep boosting algorithm, β t is selected to be an increasing function of error rate ε t, with its range. β t is employed in two folds: (i) As seen in Eq., β t helps to update the importance of different categories such that hard object classes are emphasized; (ii) As seen in Eq. FORMULA7 and Eq., reciprocals of β t are the combination coefficients for the final ensemble classifier such that those base experts with low error rate have large weight. The criterion of hard object classes for the tth expert is DISPLAYFORM17 for each t, (t = 1, ..., T); it implies that the lth object class is hard for all T experts. Let ♯{l : ϵ min (l) > 1 2λ } denote the the number of hard object classes for all T experts. Inspired by BID1, we now show that the selection of β t as in Eq. guarantees the upper boundary of ratio (the number of heavily misclassified categories over the number of all classes) to be minimized. It can be shown that for 0 < x < 1 and 0 < α < 1, we have x α ≤ 1 − (1 − x)α. According to Eq.: DISPLAYFORM18 According to Eq. and Eq., we get: DISPLAYFORM19 By substituting Eq. into Eq. FORMULA7, we get DISPLAYFORM20 DISPLAYFORM21 Combining Eq. FORMULA7 with Eq., we get DISPLAYFORM22 To minimize the rightside, we set its partial derivative with respect to β t to zero: DISPLAYFORM23 Since β t only exists in the tth factor, above equation is equivalent to DISPLAYFORM24 Solving it, we find that β t can be optimally selected as: DISPLAYFORM25 We substitute β t = λεt 1−λεt into Eq., and get the upper boundary of ratio (the number of hard object categories over the number of all classes): DISPLAYFORM0 Now we discuss the range for the hyper-parameter λ. Recall that the criterion of hard object classes for the tth expert is ϵ t (l) > From the relation between λε t and λε t (1 − λε t), as illustrated in Fig.1, we can see the effect of λ on the upper boundary of ratio (the number of hard object categories over the number of all classes) in Eq..• In the yellow shaded region, λ ∈ [1 2, 1 2εt], i.e., εt 2 < λε t < 1 2, the condition 0 < β t < 1 is satisfied, and the upper boundary of hard category percentage in Eq. increases with λ increasing, the reason for which is that when λ increases, the precision requirement increases, thus the number of hard categories increases too.• On the right side of the yellow shaded region, λ > 1 2εt, i.e., λε t > 1 2. In this case, the condition 0 < β t = λεt 1−λεt < 1 is not satisfied, thus the update of importance distribution in Eq. can not effectively emphasize the object classes which are heavily misclassified by the previous experts. In hard classification task, large error rates ε t tend to in λε t larger than or approaching 1 2, and β t larger than or approaching 1. The value of λ should be set smaller to alleviate large ε t such that λε t < 1 2 and 0 < β t < 1.• On the left side of the yellow shaded region, λ < 1 2, i.e., The procedure of learning the t th base expert repeatedly adjusts the parameters of the corresponding deep network so as to maximize the objective function O t in Eq.. To maximize O t, it is necessary to calculate its gradients with respect to all parameters, including the weights {w lt} C l=1 and the set of model parameters θ t.For clearance, we denote DISPLAYFORM0 Thus, the probability score of x assigned to the object class l, (l = 1, ..., C), in Eq. can be written as DISPLAYFORM1 Then, the objective function in Eq. can be denoted as DISPLAYFORM2 From above presentations, it can be more clearly seen that the objective is a composite function. DISPLAYFORM3 Herein, J is Jacobi matrix. Such gradients are back-propagated [] through the t th base deep CNNs to fine-tune the weights {w lt} C l=1 and the set of model parameters θ t simultaneously. Denote X as the instance space, denote Ω as the distribution over X, and denote S as a training set of N examples chosen i.i.d according to Ω. We are to investigate the gap between the generalization error on Ω and the empirical error on S.Suppose that F is the set from which the base deep experts are chosen, and let G = Note that g is a C-dim vetor, and each component of g is the category confidence, i.e., g y (x) = p(y|x), (y = 1, ..., C). Based on Eq., the category label of test sample can be predicted by arg max y g y (x) = p(y|x). The ensembled classifier g predicts wrong if g y (x) ≤ maxȳ ̸ =y gȳ(x). The generalization error rate for the final ensembled classifier can be measured by the probability DISPLAYFORM0 DISPLAYFORM1 According to probability theory, for any events B 1 and B 2, P(B 1) ≤ P(B 2) + P(B 2 |B 1), therefore DISPLAYFORM2 where ξ > 0 measures the margin between the confidences from ground-truth and incorrect categories. Using Chernoff bound BID16, the the second term in the right side of Eq. FORMULA7 is bounded as: DISPLAYFORM3 Assume that the base-classifier space F is with VC-dimension d, which can be approximately estimated by the number of neurons ν and the number of weigths ω in the base deep network, i.e., d = O(νω). Recall that S is a sample set of N examples from C categories. Then the effective number of hypotheses for F over S is at most DISPLAYFORM4 Thus, the effective number of hypotheses over S forĜ = DISPLAYFORM5 Applying Devroye Lemma as in BID16, it holds with probability at least 1 − δ Γ that DISPLAYFORM6 where DISPLAYFORM7 Likewise, in probability theory for any events B 1 and B 2, P(B 1) ≤ P(B 2) + P(B 1 |B 2), thus DISPLAYFORM8 Because DISPLAYFORM9 So, combining Eq.(19−23) together, it can be derived that DISPLAYFORM10 As can be seen from above, large margin ξ over the training set corresponds to narrow gap between the generalization error on Ω and the empirical error on S, which leads to the better upper bound of generalization error. In this section we evaluate the proposed algorithms on three real world datasets MNIST , CIFAR-100 BID5, and ImageNet . For MNIST and CIFAR-100, we train all networks from scrach in each AdaBoost iteration stage. On ImageNet, we use the pretrained model as the of iteration #1 and then train weighted models sequentially. The pretrained model is available in TorchVision 1. In each iteration, we adopt the weight initialization menthod proposed by BID2. All the networks are trained using stochastic gradient descent (SGD) with the weight decay 0f 10 −4 and the momentum of 0.9 in the experiments. MNIST dataset consists of 60,000 training and 10,000 test handwritten digit samples. BID18 showed the accuracy improvement of MLP via AdaBoost on MNIST dataset by updating sample weights according to classification errors. For fair comparison, we firstly use the similar network architecture (MLP) as the base experts in experiments. We train two sets of networks with the only difference that one updates weights w.r.t the class errors on training datasets while the other one updates weights w.r.t the sample errors on training datasets. The former is the proposed method in this paper, and the latter is the traditional AdaBoost method. In the two sets of weak learners, we share the same weak learner in iteration #1 and train other two weak learners seperately. For data pre-processing, we normalize data via subtracting means and dividing standard deviations. In the experiment on MNIST, we simply train the network with learning rate 0.01 through out the whole 120 epoches. With our proposed method, the top 1 error on test datasets decreases from 4.73% to 1.87 % after three iterations (table 1). After the interation #1, the top 1 error of our method drops more quickly than the method which update weights w.r.t sample errors. Our method, which updates weights w.r.t the class errors, leverages the idea that different class should have different learning comlexity and should not be treated equally. Through the iterations, our method trains a set of classifiers in an easy-to-hard way. Class APs vary from each weak learner in each iteration to others, increasing for marjor weighted classes while decreasing for minor weighted classes(FIG1 . Therefore, in each iteration, the weighted learner classifier behaves like a expert different from the classfier in the previous iteration. Though some APs for certain classes may decrease in some degree with each weak learner, the boosting models improve the accuracy for hard classes while preservering the accuracy for easy classes ( FIG1 . Our method cordinates the set of weak learners trained sequeentially with diversified capabilities to improve the classfication capability of boosting model. We also carry out experiments on CIFAR-100 dataset. CIFAR-100 dataset consists of 60,000 images from 100 classes. There are 500 training images and 100 testing images per class. We adopt padding, mirroring, shifting for data augumentation and normalization as in BID3 BID4 . In training stage, we hold out 5,000 training images for validation and leave 45,000 for training. Because the error per class on training datasets approaches zero and training errors could be all zeros with even simple networks, we update the category distribution w.r.t the class errors on validation datasets. We do not use any sample of validation datasets to update parameters of the networks itself. When training networks on CIFAR-100, the initial learning rate is set to 0.1 and divided by 0.1 at epoch. Similar to BID2 BID4, we train the network for 300 epoches. We show the with various models including ResNet56(λ = 0.7) and DenseNet-BC(k=12) BID4 on test set. The performances of emsembled classifier with different number of base networks are shown in the middle two rows of (table 2).As illustrated in section 3.3, λ controls the weight differences among classes. In comparison, we use λ={0.7, 0.5, 0.1}. As shown in FIG2 -left, with smaller lambda, the weitht differences become bigger. We use ResNet model in BID3 with 56 layers on CIFAR-100 datasets. Overall, the models with lambda=0.7 performs the best, ing in 24.15% test error after four iterations. Comparing with lambda=0.5 and lambda=0.7, we find that both model performs well in the initial several iterations, but the model with lambda=0.7 would converge to a better optimal(FIG2 . However, with lambda=0.1 which weights classes more discriminately, top 1 error fluctuates along the iterations ( FIG2 . We conclude that lambda should be merely used to insure that the value of β is below 0.5 and may harm the performance in the ensemble models if set to a low value. In FIG3, we show the comparison of weak leaner #1 and weak learner #2 without boosting. Though with minor exeptions, most classes with low APs improve their class APs in the proceeding weak learner. Our method is based on the motivation that different classes should have different learning comlexity. Thus, those classes with higher learning complexity should be paid more attention along the iterations. Based on the class AP of the privious iteration, we suppose those classes with lower APs should have higher learning complexity and be paid for attention in the subsequent iterations. We furthermore carry out experiments on ILSVRC2012 Classification dataset which consists of 1.2 million images for training, and 50,000 for validation. There are 1,000 classes in the dataset. For data augmentation and normalization, we adopt scaling, ramdom cropping and horizontal flipping as in BID3 BID4. Similar to the experiments on CIFAR-100, the error per class on training datasets approaches zero, we update the category distribution w.r.t the class errors on validation datasets. Since the test dataset of ImageNet are not available, we just report the on the validation sets, following BID3; BID4 for ImageNet. When we train ResNet50 networks on ImageNet, the initial learning rates are set to 0.1 and divided by 0.1 at epoch. Similar to BID2 BID4 again, we train the network for 90 epoches. The performances of emsembled classifier with different number of base networks are shown in the bottom rows of (table 2). These base ResNet networks with diverse capabilities are combined to generate more discriminative ensemble classifier. In this paper, we develop a deep boosting algorithm is to learn more discriminative ensemble classifier by combining a set of base experts with diverse capabilities. The base experts are from the family of deep CNNs and they are sequentially trained to recognize a set of object classes in an easy-to-hard way according to their learning complexities. As for the future network, we would like to investigate the performance of heterogeneous base deep networks from different families.

Represent the TLDR for finding the corresponding scientific article

A deep boosting algorithm is developed to learn more discriminative ensemble classifier by seamlessly combining a set of base deep CNNs.

Represent the short summary for finding the corresponding scientific article (long)!

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.