Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Lifelong machine learning focuses on adapting to novel tasks without forgetting the old tasks, whereas few-shot learning strives to learn a single task given a small amount of data. These two different research areas are crucial for artificial general intelligence, however, their existing studies have somehow assumed some impractical settings when training the models. For lifelong learning, the nature (or the quantity) of incoming tasks during inference time is assumed to be known at training time. As for few-shot learning, it is commonly assumed that a large number of tasks is available during training. Humans, on the other hand, can perform these learning tasks without regard to the aforementioned assumptions. Inspired by how the human brain works, we propose a novel model, called the Slow Thinking to Learn (STL), that makes sophisticated (and slightly slower) predictions by iteratively considering interactions between current and previously seen tasks at runtime. Having conducted experiments, the empirically demonstrate the effectiveness of STL for more realistic lifelong and few-shot learning settings. Deep Learning has been successful in various applications. However, it still has a lot of areas to improve on to reach human's lifelong learning ability. As one of its drawbacks, neural networks (NNs) need to be trained on large datasets before giving satisfactory performance. Additionally, they usually suffer from the problem of catastrophic forgetting -a neural network performs poorly on old tasks after learning a novel task. In contrast, humans are able to incorporate new knowledge even from few examples, and continually throughout much of their lifetime. To bridge this gap between machine and human abilities, effort has been made to study few-shot learning (; ; ; ; ; Ravi & Larochelle (2017b);;;; ), lifelong learning (; ; ; ; ; ; SerrÃ et al.;;; ), and both . The learning tasks performed by humans are, however, more complicated than the settings used by existing lifelong and few-shot learning works. Task uncertainty: currently, lifelong learning models are usually trained with hyperparameters (e.g., number of model weights) optimized for a sequence of tasks arriving at test time. The knowledge about future tasks (even their quantity) may be a too strong assumption in many real-world applications, yet without this knowledge, it is hard to decide the appropriate model architecture and capacity when training the models. Sequential few-shot tasks: existing few-shot learning models are usually (meta-)trained using a large collection of tasks. 1 Unfortunately, this collection is not available in the lifelong learning scenarios where tasks come in sequentially. Without seeing many tasks at training time, it is hard for an existing few-shot model to learn the shared knowledge behind the tasks and use the knowledge to speed up the learning of a novel task at test time. Humans, on the other hand, are capable of learning well despite having only limited information and/or even when not purposely preparing for a particular set of future tasks. Comparing how humans learn and think to how the current machine learning models are trained to learn and make predictions, we observe that the key difference lies on the part of thinking, which is the decision-making counterpart of models when making predictions. While most NN-based supervised learning models use a single forward pass to predict, humans make careful and less error-prone decisions in a more sophisticated manner. Studies in biology, psychology, and economics have shown that, while humans make fast predictions (like machines) when dealing with daily familiar tasks, they tend to rely on a slow-thinking system that deliberately and iteratively considers interactions between current and previously learned knowledge in order to make correct decisions when facing unfamiliar or uncertain tasks. We hypothesize that this slow, effortful, and less error-prone decision-making process can help bridge the gap of learning abilities between humans and machines. We propose a novel brain-inspired model, called the Slow Thinking to Learn (STL), for taskuncertain lifelong and sequential few-shot machine learning tasks. STL has two specialized but dependent modules, the cross-task Slow Predictor (SP) and per-task Fast Learners (FLs), that output lifelong and few-shot predictions, respectively. We show that, by making the prediction process of SP more sophisticated (and slightly slower) at runtime, the learning process of all modules can be made easy at training time, eliminating the need to fulfill the aforementioned impractical settings. Note that the techniques for slow predictions (; Ravi & Larochelle (2017b);; ) and fast learning (; ;) have already been proposed in the literature. Our contributions lie in that we 1) explicitly model and study the interactions between these two techniques, and 2) demonstrate, for the first time, how such interactions can greatly improve machine capability to solve the joint lifelong and few-shot learning problems encountered by humans everyday. 2 Slow Thinking to Learn (STL) Figure 1: The Slow Thinking to Learn (STL) model. To model the interactions between the shared SP f and per-task FLs {(g (t), M (t) )} t, we feed the output of FLs into the SP while simultaneously letting the FLs learn from the feedback given by SP. We focus on a practical lifelong and fewshot learning set-up:, · · · arriving in sequence and the labeled examples also coming in sequence, the goal is to design a model such that it can be properly trained by data ) collected up to any given time point s, and then make correct predictions for unlabeled data X (t) = {x (t,i) } i in any of the seen tasks, t ≤ s. Note that, at training time s, the future tasks To solve Problem 1, we propose the Slow Thinking to Learn (STL) model, whose architecture is shown in Figure 1. The STL is a cascade where the shared Slow Predictor (SP) network f parameterized by θ takes the output of multiple task-specific Fast Learners (FLs) {(g (t), M (t) )} t, t ≤ s, as input. An FL for task T (t) consists of an embedding network g (t)2 parameterized by φ (t) and augmented with an external, episodic, non-parametric memory Here, we use the Memory Module as the external memory which saves the clusters of seen examples {(x (t,i), y (t,i) )} i to achieve better storage efficiency-the h (t,j) of an entry (h (t,j), v (t,j) ) denotes the embedding of a cluster of x (t,i)'s with the same label while the v (t,j) denotes the shared label. We use the FL (g (t), M (t) ) and SP f to make few-shot and lifelong predictions for task T (t), respectively. We let the number of FLs grow with the number of seen tasks in order to ensure that the entire STL model will have enough complexity to learn from possibly endless tasks in lifelong. This does not imply that the SP will consume unbounded memory space to make predictions at runtime, as the FL for a specific task can be stored on a hard disk and loaded into the main memory only when necessary. Slow Predictions. The FL predicts the label of a test instance x using a single feedforward pass just like most existing machine learning models. As shown in Figure 2 (a), the FL for task T (t) first embed the instance to get h = g (t) (x) and then predicts the labelŷ FL of x by averaging the cluster labels where KNN(h) is the set of K nearest neighboring embeddings of h. We havê where h, h denotes the cosine similarity between h (t,j) and h. On the other hand, the SP predicts the label of x with a slower, iterative process, which is shown in Figure 2 (b). The SP first adapts (i.e., fine-tunes) its weights θ to KNN(h) and their corresponding values stored in M (t) to getθ by solving where loss(·) denotes a loss function. Then, the SP makes a prediction byŷ SP = f (h ;θ). The adapted network fθ is discarded after making the prediction. The slower decision-making process of SP may seem unnecessary and wasteful of computing resources at first glance. Next, we explain why it is actually a good bargain. Life-Long Learning with Task Uncertainty. Since the SP makes predictions after runtime adaptation, we define the training objective of θ for task T (s) such that it minimizes the losses after being adapted for each seen task The term loss(f (h;θ *), v) denotes the empirical slow-prediction loss of the adapted SP on an example (x, y) in M (t), whereθ * denotes the weights of the adapted SP for x following Eq.: requires recursively solvingθ * for each (x, y) remembered by the FLs. We use an efficient gradient-based approach proposed by ) to solve Eq.. Please refer to Section 2.1 of the Appendix for more details. Since the SP learns from the output of FLs, theθ * in Eq. approximates a hypothesis used by an FL to predict the label of x. The θ, after being trained, will be close to everyθ * and can be fine-tuned to become a hypothesis, meaning that θ encodes the invariant principles 3 underlying the hypotheses for different tasks. (a) (b) (c) Figure 3: The relative positions between the invariant representations θ and the approximate hypothesesθ (t)'s of FLs for different tasks T (t)'s on the loss surface defined by FLs after seeing the (a) first, (b) second, and (c) third task. Since θ−θ (t) ≤ R for any t in Eq., the effective capacity of SP (at runtime) is the union of the capacity of all possible points within the dashed R-circle centered at θ. Furthermore, after being sequentially trained by two tasks using Eq., the θ will easily get stuck in the middle ofθ andθ. To solve the third task, the third FL needs to change its embedding function (and therefore the loss surface) such thatθ falls into the R-circle centered at θ. Recall that in Problem 1, the nature of tasks arriving after a training process is unknown, thus, it is hard to decide the right model capacity at training time. A solution to this problem is to use an expandable network and expand the network when training it for a new task, but the number of units to add during each expansion remains unclear. Our STL walks around this problem by not letting the SP learn the tasks directly but making it learn the invariant principles behind the tasks. Assuming that the underlying principles of the learned hypotheses for different tasks are universal and relatively simple, 4 one only needs to choose a model architecture with capacity that is enough to learn the shared principles in lifelong manner. Note that limiting the capacity of SP at training time does not imply underfitting. As shown in Figure 3, the postadaptation capacity of SP at runtime can be much larger than the capacity decided during training. Sequential Few-Shot Learning. Although each FL is augmented with an external memory that has been shown to improve learning efficiency by the theory of complementary learning systems , it is not sufficient for FLs to perform few-shot predictions. Normally, these models need to be trained on many existing few-shot tasks in order to obtain good performance at test time. Without assuming s in Problem 1 to be a large number, the STL takes a different approach that fast stabilizes θ and then let the FL for a new incoming task learn a good hypothesis by extrapolating from θ. We define the training objective of g (s), which is parameterized by φ (s) and augmented with memory M (s), for the current task T (s) as follows: where ) is the empirical loss term whose specific form depends on the type of external memory used (see Section 2.2 of the Appendix for more details), and ) is a regularization term, which we call the feedback term, whose inverse value denotes the usefulness of the FL in helping SP (f parameterized by θ) adapt. Specifically, it is written as The feedback term encourages each FL to learn unique and salient features for the respective task so the SP will not be confused by two tasks having similar embeddings. As shown in Figure 3 (b), the relative position of θ gets "stuck" easily after seeing a few of previous tasks. To solve the current task, g (s) needs to change the loss surface for θ such thatθ (s) falls into the R-circle centered at θ (Figure 3(c) ). This makes θ an efficient guide (through the feedback term) to finding g (s) when there are only few examples and also few previous tasks. We use an alternate training procedure to train the SP and FLs. Please see Section 2.3 of the Appendix for more details. Note that when sequentially training STL for task T (s) in lifelong, we can safely discard the data in the previous tasks because the FLs are task-specific (see Eq.) and the SP does not require raw examples to train (see Eq.). In this section, we discuss related works that are not mentioned in Sections 1 and 2. For a complete discussion, please refer to Section 1 of the Appendix. Runtime Adaptation. Our idea of adapting SP at runtime is similar to that of MbPA , which is a method proposed for lifelong learning only. In MbPA, the embedder and output networks are trained together, in a traditional approach, as one network for the current task. Its output network adapts to examples stored in an external memory for a previous task before making lifelong predictions. Nevertheless, there is no discussion of how the runtime adaptation could improve the learning ability of a model, which is the main focus of this paper. Meta-Learning. The idea of learning the invariant representations in SP is similar to meta-learning (; Ravi & Larochelle (2017b); ), where a model (meta-)learns good initial weights that can speed up the training of the model for a new task using possibly only few shots of data. To learn the initial weights (which correspond to the invariant representations in our work), existing studies usually assume that the model can sample tasks, including training data, following the task distribution of the ground truth. However, the Problem 1 studied in this paper does not provide such a luxury. Memory-Augmented Networks. An FL is equipped with an external episodic memory module, which is shown to have fast-learning capability due to its nonparametric nature. Although we use the Memory Module in this work, our model can integrate with other types of external memory modules, such as;;. This is left as our future work. FewShot Learning without Forgetting. proposed a new few-shot learning approach that does not forget previous tasks when trained on a new one. However, it still needs to be trained on a large number of existing tasks in order to make few-shot predictions and therefore cannot be applied to Problem 1. In this section, we evaluate our model in different aspects. We implement STL and the following baselines using TensorFlow : Vanilla NN. A neural network without any technique for preventing catastrophic forgetting or preparation for few-shot tasks. EWC. A regularization technique protecting the weights that are important to previous tasks in order to mitigate catastrophic forgetting. Memory Module. An external memory module that can make predictions (using KNNs) by itself. It learns to cluster rows to improve prediction accuracy and space efficiency. MbPA+. A memory-augmented model (trained (FLs first, and then SP), and we use MAML to solve Eq. when training the SP. Separate-MbPA. This is similar to Separate-MAML, except that the SP is not trained to prepare for run-time adaptation, but it still applies run-time adaptation at test time. Next, we evaluate the abilities of the models to fight against catastrophic forgetting using the permuted MNIST and CIFAR-100 datasets. Then, we investigate the impact of task-uncertainty on model performance. Permuted MNIST. We create a sequence of 10 tasks, where each task contains MNIST images whose pixels are randomly permuted using the same seed. The seeds are different across tasks. We train models for one task at a time, and test their performance on all tasks seen so far. We first use a setting where all memory-augmented models can save raw examples in their external memory. This eliminates the need for an embedding network, and, following the settings in , we use a 2-layer MLP with 400 units for all models. We trained all models using the Adam optimizer for 10,000 iterations per task, with their best-tuned hyperparameters. Figure 4(a) shows the average performance of models for all tasks seen so far. The memory-augmented models outperform the Vanilla NN and EWC and do not suffer from forgetting. This is consistent with previous findings . However, saving raw examples for a potentially infinite number of tasks may be infeasible as it consumes a lot of space. We therefore use another setting where memory-augmented models save only the embedded examples. This time, we let both the embedder and the output network (in STL, it is SP) consist of 1-layer MLP with 400 units. Figure 4(b) shows that the memory-augmented models do not forget even when saving the embeddings. The only exception is MbPA+, because it uses the same embedder network for all tasks, the embedder network is prone to forgetting. CIFAR-100. Here, we design more difficult lifelong learning tasks using the CIFAR-100 dataset. The CIFAR-100 dataset consists of 100 classes. Each class belongs to a superclass, and each superclass has 5 classes. We create a sequence of tasks, called CIFAR-100 Normal, where the class labels in one task belong to different superclasses, but the labels across different tasks are from the same superclass. This ensures that there is transferable knowledge between tasks in the ground truth. We also create another sequence of tasks, called CIFAR-100 Hard, where the class labels in one task belong to the same superclasses, while the labels across different tasks are from different superclass. The tasks in CIFAR-100 Hard share less information, making the lifelong learning more difficult. For CIFAR-100 tasks, we let the memory-augmented models store embeddings in external memory. The embedding networks of all models consist of 4 convolutional layers followed by one fully connected layer, and all output networks (SP in STL) are a 2-layer MLP with 400 units. We search for the best hyperparameters for each model but limit the memory size to 100 embeddings, apply early stopping during training, and use Adam as the optimizer. As shown in Figures 5(a)(b), our SP clearly outperforms the baseline models for both the Normal and Hard tasks. Task Uncertainty and Hyperparameters. To understand why the SP outperforms other baselines, we study how the performance of each model changes with model capacity. Figure 4 (c) shows the performance of different models on the permuted MNIST dataset when we deliberately limit the size of external memory to 10 embeddings. Only the SP performs well in this case. We also vary the size of external memory used by our FLs and find out that the performance of SP does not drastically change like the other baselines, except when the memory size is extremely small, as shown in Figure 5 (c). The above justify that our STL can avoid the customization of memory size (a hyperparameter) to be specifically catered to expected future tasks, whose precise characteristics may not be known at training time. In addition to memory size, we also conduct experiments with models whose architectures of the output networks (SP in our STL) are changed based on LeNet . We consider two model capacities. The larger model has 4 convolutional layers with 128, 128, 256, and 256 filters followed by 3 fully-connected layers with 256, 256, and 128 units; whereas the small model has 4 convolutional layers with 16, 16, 32, and 32 filters followed by 3 fully-connected layers with 64, 32, 16 units. Figure 6 compares the performance of different parametric models for the current and previous CIFAR-100 Normal tasks. We can see that the performance of EWC on current task is heavily affected by model capacity. EWC with small model size can learn well at first, but struggles to fit the following tasks, which was not a problem when it has larger model size. MbPA has good performance on current task but forgets the previous tasks no matter how large the model is. On the other hand, STL is able to perform well on both the previous and current tasks regardless of model size. This proves the advantage of SP's runtime adaptation ability, that is, it mitigates the need for a model that is carefully sized to the incoming uncertain lifelong tasks. CIFAR-100. Existing few-shot learning models are usually trained using a large collection of tasks as training batches. However, in sequential continual learning settings, collections of these tasks are not available. Here we designed an experiment setting that simulates an incoming few-shot task during lifelong learning. We modified the CIFAR Normal and CIFAR Hard sequential tasks, where we trained the models with sequential tasks just like conventional lifelong learning set-up, except that the last task is a "few-shot" task. 5 In this experiment, we assume that the input domains are the same, which means we can use the network parameters (e.g. embedder's weights) learned from previous tasks as initial weights. We consider three baselines, namely the Memory Module, Separate-MAML, and Vanilla NN. The Memory Module is the only known model designed for both lifelong and few-shot learning. We use Separate-MAML to simulate the STL without the feedback term in Eq and the Vanilla NN to indicate "default" fine-tuning performance for each task. Figure 7 shows the performance on the few-shot task with different number of batches of available training data. Each batch contains 16 examples, and the memory size for each task is 20 in all memory-augmented models. We can see that both the FLs and SP in our model outperform other baselines. The Memory Module cannot learn well without seeing a large collection of tasks at training time, and the Separate-MAML gives unstable performance due to the lack of feedback from the SP (MAML). Interestingly, these two sophisticated models perform worse than the Vanilla NN sometimes, justifying that the interactions between the fast-leaning and slow-thinking modules are crucial to the joint lifelong and few-shot learning. Our above observations still hold on the even more challenging dataset, CIFAR Hard. Please refer to Section 3.3 of the Appendix for more details. Comparing the of FLs and SP, we can see that an FL gives better performance when the training data is small. This justifies that the invariant representations learned by the SP can indeed guide an FL to better learn from the few shots. Interestingly, the intersection of the predictive ability of an FL and the SP seem to be stable across tasks and usually falls within the range of 48 to 192 examples. In Section 3.4 of the Appendix, we visualize the embeddings stored in the FLs and the Memory Module to understand how the feedback from SP guide the representation learning of FLs. Inference Time. The SP makes "slow" predictions because of runtime adaptation. Here, we study the time required by the SP to make a single prediction. We run trained models on a machine with a commodity NVIDIA GTX-1070 GPU. The number of adaptation steps used is 3 as in previous experiments. For an FL, SP, and a non-adaptive Vanilla NN trained for the CIFAR-100 Normal tasks, we get 0.24 ms, 2.62 ms, and 0.79 ms per-example inference time on average. We believe that trading delay of a few milliseconds at runtime for a great improvement on lifelong and few-shot learning abilities is a good bargain in many applications. Space Efficiency. The STL also has an advantage in space efficiency. Please see Section 3 of the Appendix for more details. Inspired by the thinking process that humans undergo when making decisions, we propose STL, a cascade of per-task FLs and shared SP. To the best of our knowledge, this is the first work that studies the interactions between the fast-learning and slow-prediction techniques and shows how such interactions can greatly improve machine capability to solve the joint lifelong and few-shot learning problems under challenging settings. For future works, we will focus on integrating the STL with different types of external memory and studying the performance of STL in real-world deployments. Memory-Augmented Neural Network bridged the gap of leveraging an external memory for one-shot learning. MANN updates its external memory by learning a content-based memory writer. The Memory Module learns a Matching Network but includes an external memory that retains previously seen examples or their representatives (cluster of embeddings). Unlike MANN, the Memory Module has a deterministic way of updating its memory. Memory Module has the ability for providing ease and efficiency in grouping of incoming data and selecting class representations. However, Memory Module encounters limitation in learning and making precise predictions when the given memory space becomes extremely small. Our proposed STL focuses on the interaction between the per-task memory-augmented Fast Learners (FLs) and the Slow Predictor (SP) to optimize the data usage stored in the memory. This interaction allows an FL to learn better representations for a better lifelong and few-shot predictions. It is common for lifelong learning algorithms to store a form of knowledge from previously learned tasks to overcome forgetting. Some remember the task specific models , while some store raw data, the hessian of the task, or the attention mask of the network for the task (; ; SerrÃ et al. ). Some approaches such as not only attempts to consolidate the model but also expands the network size. Other works like; tried to solve the problem with fixed storage consumption. Except for , the previously mentioned works need to predefine the model capacity, and lacks the flexibility to unknown number of future tasks. Although can expand its capacity when training for a new task, the challenge of deciding how many number of units to add during each expansion still remains. Some of the recent models (; ; ; ; ; ;) in lifelong learning have taken inspiration on how the brain works . Our proposed framework is closely related to other dual-memory systems that are inspired by the complementary learning systems (CLS) theory, which defines the contribution of the hippocampus for quick learning and the neocortex for memory consolidation. A version of GeppNet that is augmented with external memory stores some of its training data for rehearsal after each new class is trained. FearNet ) is composed of three networks for quick recall, memory consolidation, and network selection. Both GeppNet and FearNet have dedicated sleep phases for memory replay, a mechanism to mitigate catastrophic forgetting. STL, however, does not require a dedicated sleep or shutdown to consolidate the memory. This choice is based on considering that there are cases wherein a dedicated sleep time is not feasible, such as when using a machine learning model to provide a frontline service that needs to be up and running all the time and cannot be interrupted by a regular sleep schedule. In this section, we discuss more technical details about the design and training of STL. There are different ways to solve Eq.. One can use either the gradient-based MAML or Reptile to get an approximated solution efficiently. The constraint θ − θ ≤ R can be implemented by either adding a Lagrange multiplier in the objective or limiting the number of gradient steps in MAML/Reptile. In this paper, we use MAML due to its simplicity, ease of implementation, and efficiency, and we enforce the constraint θ − θ ≤ R by limiting the number of adaptation steps of SP at runtime. An FL in STL is compatible with different types of external memory modules, such as;;. We choose the Memory Module in this paper due to its clustering capabilities, which increase space efficiency. For completeness, we briefly discuss how an FL based on the Memory Module are optimized. ) be the sorted K nearest neighbors (from the closest to the farthest) of the embedding of x, and where ·, · denotes the cosine similarity between two vectors, and p and b are the smallest indices such that v (p) = y and v (b) = y, respectively; h (p) and h (b) are the closest positive and negative neighbors. As this loss is minimized, g (s) maximizes the similarity of embedding of training data points to their positive neighbors, while minimizing the similarity to the negative neighbors by a margin of ε. also has deterministic update and replacement rules for records in M (s). In effect, an h represents the embedding of a cluster of data points, and its value v denotes the shared label of points in that group. We sequentially train the STL for tasks T, T, · · · coming in lifelong. For the current task T (s), we train the STL using an alternate training approach. First, the weights φ (s) of g (s) for T (t) is updated by taking some descent steps following Eq. in the main paper. Next, the θ that parametrizes f is updated following Eq. in the main paper. One alternate training iteration involves training the FL for the current task for a steps, and then the SP for b steps. We set the alternate ratio a: b to 1:1 by default. The pseudo-code of STL's training procedure is shown in Algorithms 1, 2, and 3. One important hyperparameter to decide before training the STL is R, which affects the number of adaptation steps used by SP. A larger R allows the adapted weightsθ's to move farther from θ, which may lead the SP to better lifelong predictions but will in higher computation cost at runtime. A smaller R helps stabilize θ after the model is trained on previous tasks and enables θ to guide the FLs for new incoming tasks sooner. We experimented on different values of R by adjusting the number of adaptation steps of SP, and found out that it does not need to be large to achieve good performance. Normally, it suffices to have less than 5 adaptation steps. The SP in STL can work with FTs having very small external memory. Figure 8(a) shows the trade-off between the average all-task performance at the 10-th sequential task on the permuted MNIST dataset and the size (in number of embedded examples) of external memory. While the performance of most memory-augmented models drops when the memory size is 10, the SP can still perform well. This justifies that the invariant principles learned by the SP can indeed guide FLs to find better representations that effectively "bring back" the knowledge of SP for a particular task. Figure 8(b) shows the memory space required by different models in order to achieve at least 0.9 average all-task accuracy. The STL consumes less than 1% space as compared to MbPA and Memory Module. The SP also has high adaptation efficiency. Figure 8 (c) shows the performance gain of different adaptive models after runtime adaptation. When the memory size is 1000, all models can adapt for the current task to give improved performance. However, the adaptation efficiency of the baseline models drops when the memory size is 10. The SP, on the other hand, achieves good performance in this case even after being adapted for just one step thanks to 1) the SP is trained to be ready for adaptation and 2) the invariant principles learned by the SP are useful by themselves and require only few examples (embeddings) to transform to a good hypothesis. 3.3 Sequential Few-shot Learning on CIFAR Hard Figure 9 shows the sequential few-shot learning of different models on the CIFAR Hard dataset. In this dataset, the labels of different tasks come from different superclasses in the original CIFAR 100 dataset. So, it is very challenging for a model to learn from only a few examples in a new task without being able to see a lot of previous tasks. As we can see, our STL model still outperforms other baselines. In particular, Figure 9 (a) shows that the STL is able to perform few-shot learning after seeing just one previous task. Again, the above demonstrates that the interactions between the FLs and SP are a key to improving machine ability for the joint lifelong and few-shot learning. In order to understand how the feedback from SP guide the representation learning of FLs, we visualize the embeddings stored in the FLs and the Memory Module. We sample 300 testing examples per task, get their embeddings from the two models, and then project them onto a 2D space using the t-SNE algorithm . The are shown in Figure 10. We can see that the embeddings produced by different FLs for different tasks are more distant from each other than those output by the Memory Module. Recall that the feedback term in Eq. encourages each FL to learn features that help the SP adapt. Therefore, each FL learns more salient features for the respective task so the SP will not be confused by two tasks having similar embeddings. This, in turn, quickly stabilizes SP and makes it an efficient guide (through the feedback term) to learning the FL for a new task when there are only few examples and also few previous tasks, as Figure 3 shows.

This paper studies the interactions between the fast-learning and slow-prediction models and demonstrate how such interactions can improve machine capability to solve the joint lifelong and few-shot learning problems.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the scientific article to find a short TLDR

Question: Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like and , when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence. Meta-learning describes a broad family of techniques in machine learning that deals with the problem of learning to learn. An emerging branch of meta-learning involves the use of hypernetworks, which are meta neural networks that generate the weights of a main neural network to solve a given task in an end-to-end differentiable manner. Hypernetworks were originally introduced by as a way to induce weight-sharing and achieve model compression by training the same meta network to learn the weights belonging to different layers in the main network. Since then, hypernetworks have found numerous applications including but not limited to: weight pruning , neural architecture search (;, Bayesian neural networks (; ; ; ;), multi-task learning (; ; ; Serrà et al., 2019;), continual learning (von), generative models , ensemble learning , hyperparameter optimization , and adversarial defense . Despite the intensified study of applications of hypernetworks, the problem of optimizing them to this day remains significantly understudied. Given the lack of principled approaches to training hypernetworks, prior work in the area is mostly limited to ad-hoc approaches based on trial and error (c.f. Section 3). For example, it is common to initialize the weights of a hypernetwork by sampling a "small" random number. Nonetheless, these ad-hoc methods do lead to successful hypernetwork training primarily due to the use of the Adam optimizer , which has the desirable property of being invariant to the scale of the gradients. However, even Adam will not work if the loss diverges (i.e. integer overflow) at initialization, which will happen in sufficiently big models. The normalization of badly scaled gradients also in noisy training dynamics where the loss function suffers from bigger fluctuations during training compared to vanilla stochastic gradient descent (SGD). showed that while adaptive optimizers like Adam may exhibit lower training error, they fail to generalize as well to the test set as non-adaptive gradient methods. Moreover, Adam incurs a computational overhead and requires 3X the amount of memory for the gradients compared to vanilla SGD. Small random number sampling is reminiscent of early neural network research before the advent of classical weight initialization methods like Xavier init and Kaiming init . Since then, a big lesson learned by the neural network optimization community is that architecture specific initialization schemes are important to the ro-bust training of deep networks, as shown recently in the case of residual networks . In fact, weight initialization for hypernetworks was recognized as an outstanding open problem by prior work that had questioned the suitability of classical initialization methods for hypernetworks. Our We show that when classical methods are used to initialize the weights of hypernetworks, they fail to produce mainnet weights in the correct scale, leading to exploding activations and losses. This is because classical network weights transform one layer's activations into another, while hypernet weights have the added function of transforming the hypernet's activations into the mainnet's weights. Our solution is to develop principled techniques for weight initialization in hypernetworks based on variance analysis. The hypernet case poses unique challenges. For example, in contrast to variance analysis for classical networks, the case for hypernetworks can be asymmetrical between the forward and backward pass. The asymmetry arises when the gradient flow from the mainnet into the hypernet is affected by the biases, whereas in general, this does not occur for gradient flow in the mainnet. This underscores again why architecture specific initialization schemes are essential. We show both theoretically and experimentally that our methods produce hypernet weights in the correct scale. Proper initialization mitigates exploding activations and gradients or the need to depend on Adam. Our experiments reveal that it leads to more stable mainnet weights, lower training loss, and faster convergence. Section 2 briefly covers the relevant technical preliminaries and Section 3 reviews problems with the ad-hoc methods currently deployed by hypernetwork practitioners. We derive novel weight initialization formulae for hypernetworks in Section 4, empirically evaluate our proposed methods in Section 5, and finally conclude in Section 6. Definition. A hypernetwork is a meta neural network H with its own parameters φ that generates the weights of a main network θ from some embedding e in a differentiable manner: θ = H φ (e). Unlike a classical network, in a hypernetwork, the weights of the main network are not model parameters. Thus the gradients ∆θ have to be further backpropagated to the weights of the hypernetwork ∆φ, and then trained via gradient descent φ t+1 = φ t − λ∆φ t. This fundamental difference suggests that conventional knowledge about neural networks may not apply directly to hypernetworks and novel ways of thinking about weight initialization, optimization dynamics and architecture design for hypernetworks are sorely needed. We propose the use of Ricci calculus, as opposed to the more commonly used matrix calculus, as a suitable mathematical language for thinking about hypernetworks. Ricci calculus is useful because it allows us to reason about the derivatives of higher-order tensors with notational ease. For readers not familiar with the index-based notation of Ricci calculus, please refer to for a good introduction to the topic written from a machine learning perspective. For a general nth-order tensor T i1,...,in, we use d i k to refer to the dimension of the index set that i k is drawn from. We include explicit summations where the relevant expressions might be ambiguous, and use Einstein summation convention otherwise. We use square brackets to denote different layers for added clarity, so for example W [t] denotes the t-th weight layer. derived weight initialization formulae for a feedforward neural network by conducting a variance analysis over activations and gradients. For a linear layer suppose we make the following Xavier Assumptions at initialization: Thus, the forward pass and backward pass in symmetrical formulae. proposed an initialization based on their harmonic mean: Var(W i j) = 2 dj +di. In general, a feedforward network is non-linear, so these assumptions are strictly invalid. But symmetric activation functions with unit derivative at 0 in a roughly linear regime at initialization. extended's analysis by looking at the case of ReLU activation functions, i.e. This in an extra factor of 2 in the variance formula. W i j have to be symmetric around 0 to enforce Xavier Assumption 3 as the activations and gradients propagate through the layers. argued that both the forward or backward version of the formula can be adopted, since the activations or gradients will only be scaled by a depth-independent factor. For convolutional layers, we have to further divide the variance by the size of the receptive field.' Xavier init' and'Kaiming init' are terms that are sometimes used interchangeably. Where there might be confusion, we will refer to the forward version as fan-in init, the backward version as fan-out init, and the harmonic mean version as harmonic init. In the seminal paper, the authors identified two distinct classes of hypernetworks: dynamic (for recurrent networks) and static (for convolutional networks). They proposed Orthogonal init for the dynamic class, but omitted discussion of initialization for the static class. The static class has since proven to be the dominant variant, covering all kinds of non-recurrent networks (not just convolutional), and thus will be the central object of our investigation. Through an extensive literature and code review, we found that hypernet practitioners mostly depend on the Adam optimizer, which is invariant to and normalizes the scale of gradients, for training and resort to one of four weight initialization methods: M3 Kaiming init, but with the output layer scaled by 1 10 (as found in). M4 Kaiming init, but with the hypernet embedding set to be a suitably scaled constant (as found in). M1 uses classical neural network initialization methods to initialize hypernetworks. This fails to produce weights for the main network in the correct scale. Consider the following illustrative example of a one-layer linear hypernet generating a linear mainnet with T + 1 layers, given embeddings sampled from a standard normal distribution and weights sampled entry-wise from a zero-mean distribution. We leave the biases out for now. is likely to vanish, since the size of the embedding vector is typically small relatively to the width of the mainnet weight layer being generated. Where the fan-in is of a different scale than the fan-out, the harmonic mean has a scale close to that of the smaller number. Therefore, the fan-in, fan-out, and harmonic variants of Xavier and Kaiming init will all in activations and gradients that scale exponentially with the depth of the mainnet. M2 and M3 introduce additional hyperparameters into the model, and the ad-hoc manner in which they work is reminiscent of pre deep learning neural network research, before the introduction of classical initialization methods like Xavier and Kaiming init. This ad-hoc manner is not only inelegant and consumes more compute, but will likely fail for deeper and more complex hypernetworks. kt to a suitable constant (d in this case), such that both itkt can seem to be initialized with the same variance as Kaiming init. This ensures that the variance of the activations in the mainnet are preserved through the layers, but the restrictions on the embeddings might not be desirable in many applications. Luckily, the fix appears simple -set Var(. This in the variance of the generated weights in the mainnet Var( resembling conventional neural networks initialized with fan-in init. This suggests a general hypernet weight initialization strategy: initialize the weights of the hypernet such that they approximate weights coming from classical neural network initialization. We elaborate on and generalize this intuition in Section 4. Most hypernetwork architectures use a linear output layer so that gradients can pass from the mainnet into the hypernet directly without any non-linearities. We make use of this fact in developing methods called hyperfan-in init and hyperfan-out init for hypernetwork weight initialization based on the principle of variance analysis. Proposition. Suppose a hypernetwork comprises a linear output layer. Then, the variance between the input and output activations of a linear layer in the mainnet y i = W i j x j + b i can be preserved using fan-in init in the hypernetwork with appropriately scaled output layers. Case 1. The hypernet generates the weights but not the biases of the mainnet. The bias in the mainnet is initialized to zero. We can write the weight generation in the form where h computes all but the last layer of the hypernet and (H, β) form the output layer. We make the following Hyperfan Assumptions at initialization: Xavier assumptions hold for all the layers in the hypernet. Use fan-in init to initialize the weights for h. Then, Var(h(e) k ) = Var(e l). If we initialize H with the formula Var(and β with zeros, we arrive at Var(W i j) = 1 dj, which is the formula for fan-in init in the mainnet. The Hyperfan assumptions imply the Xavier assumptions hold in the mainnet, thus preserving the input and output activations. Case 2. The hypernet generates both the weights and biases of the mainnet. We can write the weight and bias generation in the form respectively, where h and g compute all but the last layer of the hypernet, and (H, β) and (G, γ) form the output layers. We modify Hyperfan Assumption 2 so it includes G i l, g(e) l, and γ i, and further assume Var(x j) = 1, which holds at initialization with the common practice of data standardization. Use fan-in init to initialize the weights for h and g. Then,, and β, γ with zeros, then the input and output activations in the mainnet can be preserved. If we initialize G i j to zeros, then its contribution to the variance will increase during training, causing exploding activations in the mainnet. Hence, we prefer to introduce a factor of 1/2 to divide the variance between the weight and bias generation, where the variance of each component is allowed to either decrease or increase during training. This becomes a problem if the variance of the activations in the mainnet deviates too far away from 1, but we found that it works well in practice. A similar derivation can be done for the backward pass using analogous assumptions on gradients flowing in the mainnet: and through mainnet biases: If we initialize the output layer H with the analogous hyperfan-out formula Var(and the rest of the hypernet with fan-in init, then we can preserve input and output gradients on the mainnet: Var. However, note that the gradients will shrink when flowing from the mainnet to the hypernet: Var(Var( and scaled by a depth-independent factor due to the use of fan-in rather than fan-out init. In the classical case, the forward version (fan-in init) and the backward version (fan-out init) are symmetrical. This remains true for hypernets if they only generated the weights of the mainnet. However, if they were to also generate the biases, then the symmetry no longer holds, since the biases do not affect the gradient flow in the mainnet but they do so for the hypernet (c.f. Equation 4). Nevertheless, we can initialize G so that it helps hyperfan-out init preserve activation variance on the forward pass as much as possible (keeping the assumption that Var(x j) = 1 as before): We summarize the variance formulae for hyperfan-in and hyperfan-out init in Table 1. It is not uncommon to re-use the same hypernet to generate different parts of the mainnet, as was originally done in. We discuss this case in more detail in Appendix Section A. We initialize h and g with fan-in init, and For convolutional layers, we have to further divide Var(H i jk) by the size of the receptive field. Uniform init: X ∼ U(− 3Var(X), 3Var(X)). Normal init: X ∼ N (0, Var(X)). Variance Formula Initialization Variance Formula We evaluated our proposed methods on four sets of experiments involving different use cases of hypernetworks: feedforward networks, continual learning, convolutional networks, and Bayesian neural networks. In all cases, we optimize with vanilla SGD and sample from the uniform distribution according to the variance formula given by the init method. More experimental details can be found in Appendix Section B. As an illustrative first experiment, we train a feedforward network with five hidden layers (500 hidden units), a hyperbolic tangent activation function, and a softmax output layer, on MNIST across four different settings: a classical network with Xavier init, a hypernet with Xavier init that generates the weights of the mainnet, a hypernet with hyperfan-in init that generates the weights of the mainnet, and a hypernet with hyperfan-out init that generates the weights of the mainnet. The use of hyperfan init methods on a hypernetwork reproduces mainnet weights similar to those that have been trained from Xavier init on a classical network, while the use of Xavier init on a hypernetwork causes exploding activations right at the beginning of training (see Figure 1). Observe in Figure 2 that when the hypernetwork is initialized in the proper scale, the magnitude of generated weights stabilizes quickly. This in turn leads to a more stable training regime, as seen in Figure 3. More visualizations of the activations and gradients of both the mainnet and hypernet can be viewed in Appendix Section B.1. Qualitatively similar observations were made when we replaced the activation function with ReLU and Xavier with Kaiming init, with Kaiming init leading to even bigger activations at initialization. Suppose now the hypernet generates both the weights and biases of the mainnet instead of just the weights. We found that this architectural change leads the hyperfan init methods to take more time (but still less than Xavier init), to generate stable mainnet weights (c.f. Figure 25 in the Appendix). Continual learning solves the problem of learning tasks in sequence without forgetting prior tasks. von used a hypernetwork to learn embeddings for each task as a way to efficiently regularize the training process to prevent catastrophic forgetting. We compare different initialization schemes on their hypernetwork implementation, which generates the weights and biases of a ReLU mainnet with two hidden layers to solve a sequence of three regression tasks. In Figure 4, we plot the training loss averaged over 15 different runs, with the shaded area showing the standard error. We observe that the hyperfan methods produce smaller training losses at initialization and during training, eventually converging to a smaller loss for each task. applied a hypernetwork on a convolutional network for image classification on CIFAR-10. We note that our initialization methods do not handle residual connections, which were in their chosen mainnet architecture and are important topics for future study. Instead, we implemented their hypernetwork architecture on a mainnet with the All Convolutional Net architecture that is composed of convolutional layers and ReLU activation functions. After searching through a dense grid of learning rates, we failed to enable the fan-in version of Kaiming init to train even with very small learning rates. The fan-out version managed to begin delayed training, starting from around epoch 270 (see Figure 5). By contrast, both hyperfan-in and hyperfan-out init led to successful training immediately. This shows a good init can make it possible to successfully train models that would have otherwise been unamenable to training on a bad init. In the work of , it was noticed that even with the use of batch normalization in the mainnet, classical initialization approaches still led to diverging losses (due to exploding activations, c.f. Section 3). We observe similar in our experiment (see Figure 6) -the fan-in version of Kaiming init, which is the default initialization in popular deep learning libraries like PyTorch and Chainer, ed in substantially higher initial losses and led to slower training than the hyperfan methods. We found that the observation still stands even when the last layer of the mainnet is not generated by the hypernet. This shows that while batch normalization helps, it is not the solution for a bad init that causes exploding activations. Our approach solves this problem in a principled way, and is preferable to trial-and-error based heuristics that had to resort to in order to train their model. Surprisingly, the fan-out version of Kaiming init led to similar as the hyperfan methods, suggesting that batch normalization might be sufficient to correct the bad initializations that in vanishing activations. That being said, hypernet practitioners should not expect batch normalization to be the panacea for problems caused by bad initialization, especially in memory-constrained scenarios. In a Bayesian neural network application (especially in hypernet architectures without relaxed weight-sharing), the blowup in the number of parameters limits the use of big batch sizes, which is essential to the performance of batch normalization . For example, in this experiment, our hypernet model requires 32 times as many parameters as a classical MobileNet. To the best of our knowledge, the interaction between batch normalization and initialization is not well-understood, even in the classical case, and thus, our findings prompt an interesting direction for future research. In all our experiments, hyperfan-in and hyperfan-out both lead to successful hypernetwork training with SGD. We did not find a good reason to prefer one over the other (similar to 's observation in the classical case for fan-in and fan-out init). For a long time, the promise of deep nets to learn rich representations of the world was left unfulfilled due to the inability to train these models. The discovery of greedy layer-wise pre-training and later, Xavier and Kaiming init, as weight initialization strategies to enable such training was a pivotal achievement that kickstarted the deep learning revolution. This underscores the importance of model initialization as a fundamental step in learning complex representations. In this work, we developed the first principled weight initialization methods for hypernetworks, a rapidly growing branch of meta-learning. We hope our work will spur momentum towards the development of principled techniques for building and training hypernetworks, and eventually lead to significant progress in learning meta representations. Other non-hypernetwork methods of neural network generation can also be improved by considering whether their generated weights in exploding activations and how to avoid that if so. A.1 FOR MAINNET WEIGHTS OF THE SAME SIZE For model compression or weight-sharing purposes, different parts of the mainnet might be generated by the same hypernet function. This will cause some assumptions of independence in our analysis to be invalid. Consider the example of the same hypernet being used to generate multiple different mainnet weight layers of the same size, i.e. The relaxation of some of these independence assumptions does not always prove to be a big problem in practice, because the correlations introduced by repeated use of H can be minimized with the use of flat distributions like the uniform distribution. It can even be helpful, since the re-use of the same hypernet for different layers causes the gradient flowing through the hypernet output layer to be the sum of the gradients from the weights of these layers: itk, thus combating the shrinking effect. Similar reasoning applies if the same hypernet was used to generate differently sized subsets of weights in the mainnet. However, we encourage avoiding this kind of hypernet architecture design if not otherwise essential, since it will complicate the initialization formulae listed in Table 1.'s hypernetwork architecture. Their two-layer hypernet generated weight chunks of size (K, n, n) for a main convolutional network where K = 16 was found to be the highest common factor among the size of mainnet layers, and n 2 = 9 was the size of the receptive field. We simplify the presentation by writing i for i t, j for j t, k for k t,m, and l for l t,m. Because the output layer (H, β) in the hypernet was re-used to generate mainnet weight matrices of different sizes (i.e. in general, i t = i t+1, j t = j t+1), G effectively becomes the output layer that we want to be considering for hyperfan-in and hyperfan-out initialization. Hence, to achieve fan-in in the mainnet Var, and hyperfan-in init for ). Analogously, to achieve fan-out in the mainnet Var(W [t] ), and hyperfan-out init for ). The networks were trained on MNIST for 30 epochs with batch size 10 using a learning rate of 0.0005 for the hypernets and 0.01 for the classical network. The hypernets had one linear layer with embeddings of size 50 and different hidden layers in the mainnet were all generated by the same hypernet output layer with a different embedding, which was randomly sampled from U(− √ 3, √ 3) and fixed. We use the mean cross entropy loss for training, but the summed cross entropy loss for testing. We show activation and gradient plots for two cases: (i) the hypernet generates only the weights of the mainnet, and (ii) the hypernet generates both the weights and biases of the mainnet. (i) covers Figures 3, 1, 7, 8, 9, 10, 11, 12, 2, 13, 14, 15, and 16. (ii) covers Figures 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, and 29. The activations and gradients in our plots were calculated by averaging across a fixed held-out set of 300 examples drawn randomly from the test set. In Figures 1, 8, 9, 11, 12, 13, 14, 16, 18, 20, 21, 23, 24, 26, 27, and 29, the y axis shows the number of activations/gradients, while the x axis shows the value of the activations/gradients. The value of activations/gradients from the hypernet output layer correspond to the value of mainnet weights. In Figures 2, 7, 10, 15, 19, 22, 25, and 28, the y axis shows the mean value of the activations/gradients, while each increment on the x axis corresponds to a measurement that was taken every 1000 training batches, with the bars denoting one standard deviation away from the mean. proposed to use the harmonic mean of the two different initialization formulae derived from the forward and backward pass. commented that either version suffices for convergence, and that it does not really matter given that the difference between the two will be a depth-independent factor. We experimented with the harmonic, geometric, and arithmetic means of the two different formulae in both the classical and the hypernet case. There was no indication of any significant benefit from taking any of the three different means in both cases. Thus, we confirm and concur with's original observation that either the fan-in or the fan-out version suffices. The mainnet is a feedforward network with two hidden layers (10 hidden units) and the ReLU activation function. The weights and biases of the mainnet are generated from a hypernet with two hidden layers (10 hidden units) and trainable embeddings of size 2 sampled from U(− √ 3, √ 3). We keep the same continual learning hyperparameter β output value of 0.005 and pick the best learning rate for each initialization method from {10 −2, 10 −3, 10 −4, 10 −5}. Notably, Kaiming (fan-in) could only be trained from learning rate 10 −5, with losses diverging soon after initialization using the other learning rates. Each task was trained for 6000 training iterations using batch size 32, with Figure 4 plotted from losses measured at every 100 iterations. The networks were trained on CIFAR-10 for 500 epochs starting with an initial learning rate of 0.0005 using batch size 100, and decaying with γ = 0.1 at epochs 350 and 450. The hypernet is composed of two layers (50 hidden units) with separate embeddings and separate input layers but shared output layers. The weight generation happens in blocks of where K = 96 is the highest common factor between the different sizes of the convolutional layers in the mainnet and n = 3 is the size of the convolutional filters (see Appendix Section A.2 for a more detailed explanation on the hypernet architecture). The embeddings are size 50 and fixed after random sampling from U(− √ 3, √ 3). We use the mean cross entropy loss for training, but the summed cross entropy loss for testing. showed that a Bayesian neural network can be developed by using a hypernetwork to express a prior distribution without substantial changes to the vanilla hypernetwork setting. Their methods simply require putting L 2 -regularization on the model parameters and sampling from stochastic embeddings. We trained a linear hypernet to generate the weights of a MobileNet mainnet architecture (excluding the batch normalization layers), using the block-wise sampling strategy described in , with a factor of 0.0005 for the L 2 -regularization. We initialize fixed embeddings of size 32 sampled from U(− √ 3, √ 3), and sample additive stochastic noise coming from U(−0.1, 0.1) at the beginning of every mini-batch training. The training was done on ImageNet with batch size 256 and learning rate 0.1 for 25 epochs, or equivalently, 125125 iterations. The testing was done with 10 Monte Carlo samples. We omit the test loss plots due to the computational expense of doing 10 forward passes after every mini-batch instead of every epoch.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

The first principled weight initialization method for hypernetworks

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.

Represent the TLDR for finding the corresponding scientific article

A novel Bayesian deep learning framework that captures and relates hierarchical semantic and visual concepts, performing well on a variety of image and text modeling and generation tasks.

We proposed a progressive learning method to improve learning and disentangling latent representations at different levels of abstraction.