Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Learning-to-learn or meta-learning leverages data-driven inductive bias to increase the efficiency of learning on a novel task. This approach encounters difficulty when transfer is not mutually beneficial, for instance, when tasks are sufficiently dissimilar or change over time. Here, we use the connection between gradient-based meta-learning and hierarchical Bayes to propose a mixture of hierarchical Bayesian models over the parameters of an arbitrary function approximator such as a neural network. Generalizing the model-agnostic meta-learning (MAML) algorithm, we present a stochastic expectation maximization procedure to jointly estimate parameter initializations for gradient descent as well as a latent assignment of tasks to initializations. This approach better captures the diversity of training tasks as opposed to consolidating inductive biases into a single set of hyperparameters. Our experiments demonstrate better generalization on the standard miniImageNet benchmark for 1-shot classification. We further derive a novel and scalable non-parametric variant of our method that captures the evolution of a task distribution over time as demonstrated on a set of few-shot regression tasks. Meta-learning algorithms aim to increase the efficiency of learning by treating task-specific learning episodes as examples from which to generalize BID39. The central assumption of a meta-learning algorithm is that some tasks are inherently related and so inductive transfer can improve generalization and sample efficiency BID4 BID2. Recent metalearning algorithms have encoded this assumption by learning global hyperparameters that provide a task-general inductive bias. In learning a single set of hyperparameters that parameterize, for example, a metric space BID46 or an optimizer for gradient descent BID31 BID8, these meta-learning algorithms make the assumption that tasks are equally related and therefore mutual transfer is appropriate. This assumption has been cemented in recent few-shot learning benchmarks, which consist of a set of tasks generated in a systematic manner (e.g., BID8 BID46 .However, the real world often presents scenarios in which an agent must decide what degree of transfer is appropriate. In the case of positive transfer, a subset of tasks may be more strongly related to each other and so non-uniform transfer poses a strategic advantage. Negative transfer in the presence of dissimilar or outlier tasks worsens generalization performance BID34 . Moreover, when the underlying task distribution is non-stationary, inductive transfer to initial tasks should exhibit graceful degradation to address the catastrophic forgetting problem BID16 . However, the consolidation of all inductive biases into a single set of hyperparameters cannot flexibly account for variability in the task distribution. In contrast, in order to deal with this degree of task heterogeneity, extensive task-switching literature reveals that people detect and readily adapt even in the face of significantly novel contexts (see BID5, for a review).In this work, we learn a mixture of hierarchical models that allows the meta-learner to adaptively select over a set of learned parameter initializations for gradient-based fast adaptation BID8 to a new task. The method is equivalent to clustering task-specific parameters in the hierarchical model induced by recasting gradient-based meta-learning as hierarchical Bayes BID13 and generalizes the model-agnostic meta-learning (MAML) algorithm introduced in BID8.By treating the assignment of task-specific parameters to clusters as latent variables in a probabilistic model, we can directly detect similarities between tasks on the basis of the task-specific likelihood, which may be parameterized by a black-box model such as a neural network. Our approach therefore alleviates the need for explicit geometric or probabilistic modelling assumptions about the weights of a parametric model and provides a scalable method to regulate information transfer between episodes. We extend our latent variable model to the non-parametric setting and leverage stochastic point estimation for scalable inference in a Dirichlet process mixture model (DPMM) BID30. To the best of our knowledge, no previous work has considered a scalable stochastic point estimation in a non-parametric mixture model. Furthermore, we are not aware of prior work applying nonparametric mixture modelling techniques to high-dimensional parameter spaces such as those of deep neural networks. The non-parametric extension allows the complexity of a meta-learner to evolve by introducing or removing clusters in alignment with the changing composition of the dataset and preserves performance on previously encountered tasks better than a parametric counterpart. The goal of a meta-learner is to extract task-general knowledge from the experience of solving a number of related tasks. By leveraging this acquired prior knowledge, the meta-learner can quickly adapt to novel tasks even in the face of limited data or limited computation time BID40. Recent approaches to meta-learning consolidate information from a set of training tasks into the parameters of a mapping to be applied at test time to a novel task. This mapping has taken the form of, for instance, a learned metric space (e.g., BID46 BID42, a trained recurrent neural network (e.g., BID38, or an gradient-based optimization algorithm with learned parameters (e.g., BID31 .Model-agnostic meta-learning (MAML) BID8 ) is a gradient-based meta-learning approach that estimates global parameters to be shared among task-specific models as an initialization for a few steps of gradient descent. MAML also admits a natural interpretation as parameter estimation in a hierarchical model, where the learned initialization acts as data-driven regularization for task-specific parameters BID13. In particular, BID13 cast MAML as posterior inference for task-specific parameters φ j given a batch of task data x j 1:N and a prior over φ j that is induced by early stopping of an iterative descent procedure. A few steps of gradient descent on the negative log-likelihood − log p(x j 1:N | φ j ), starting from φ j = θ can be then understood as mode estimation of the posterior p(φ DISPLAYFORM0 DISPLAYFORM1 Subroutine 3: The E-STEP and M-STEP for a finite mixture of hierarchical Bayesian models. If the task distribution is heterogeneous, assuming a single parameter initialization θ is not suitable because it is unlikely that the point estimate computed by a few steps of gradient descent will sufficiently adapt the task-specific parameters φ to a diversity of tasks. Moreover, explicitly estimating relatedness between tasks has the potential to aid the efficacy of a meta-learning algorithm by modulating both positive and negative transfer BID44 BID51 BID35 BID52 BID48 . Nonetheless, defining an appropriate notion of task relatedness is a difficult problem in the high-dimensional parameter or activation space of models such as neural networks. Using the probabilistic interpretation of Section 2, we may deal with the variability in the tasks T j by assuming that each set of task-specific parameters φ j is drawn from a mixture of base distributions each of which is parameterized by a hyperparameter θ . Accordingly, we capture task relatedness by estimating the likelihood of assigning each task to a mixture component based only on the task loss itself after a single step of fast adaptation BID8. The is a scalable end-to-end meta-learning algorithm that jointly learns task-specific cluster assignments and network parameters. This algorithm, further detailed in the following section, is capable of modulating the transfer of information across tasks to better generalize to heterogeneous or evolving task distributions. Let z j be the Categorical latent variable indicating the cluster assignment of each task-specific parameter φ j. A direct maximization of the mixture model likelihood is a combinatorial optimization problem that can grow intractable. This intractability is equally problematic for the posterior distribution over the cluster assignment variables z j and the task-specific parameters φ j, which are both treated as latent variables in the probabilistic formulation of meta-learning. A standard approach for estimation in latent variable models such as probabilistic mixtures is to represent the distribution using samples with a sampler. The most widely used is the Gibbs Model 1-shot matching network FCE BID46 a 43.56 ± 0.84 meta-learner LSTM BID31 43.44 ± 0.77 SNAIL BID20 b 45.1 ± --prototypical networks BID42 c 46.61 ± 0.78 MAML BID8 48.70 ± 1.84 LLAMA BID13 49.40 ± 1.83 KNN + GNN embedding BID10 49.44 ± 0.28 mAP-DLM BID45 49.82 ± 0.78 fwCNN (Hebb) BID22 50.21 ± 0.37 GNN BID10 50.33 ± 0.36 Our method (clustering all layers of a neural network)50.80 ± 1.70 sampler BID24 BID11, which draws from the conditional distribution of each latent variable given the others until convergence to the posterior distribution over all the latents. However, in the setting of latent variables defined over high-dimensional parameter spaces such as those of neural network models, a sampling approach such as Gibbs sampling is prohibitively expensive BID25 BID21.Instead of maintaining samples to represent the distribution over the latent variables, a scalable approximation involves representing the conditional distribution for each latent variable with either a maximum a posteriori (MAP) value or an expectation. In our meta-learning setting of a mixture of hierarchical Bayesian models, this suggests an augmented expectation maximization (EM) procedure BID7 alternating between an E-STEP that computes an expectation of the task-to-cluster assignments, which itself involves the computation of a MAP estimate for the task-specific parameters, and an M-STEP that computes a local maximum of the hyperparameters θ (1:L).To ensure scalability, we use the minibatch variant of stochastic optimization BID33 to compute both the E-STEP and M-STEPs; such approaches to EM are motivated by a view of the algorithm as optimizing a single free energy at both the E-STEP and the M-STEP BID26. In particular, for each task j and cluster, we follow the gradients to minimize the negative log-likelihood on the training data points using the cluster parameters θ as initialization. This allows us to obtain a modal point estimate of the task parametersφ DISPLAYFORM0 The E-STEP in Subroutine 3 leverages the connection between gradient-based meta-learning and hierarchical Bayes (HB) BID13 to employ the task-specific parameters to compute the posterior probability of cluster assignment. Accordingly, based on the likelihood of the same training data points under the model parameterized byφ j, we compute the cluster assignment probabilities as DISPLAYFORM1 The cluster means θ are then updated by gradient descent on the validation loss in the M-STEP, given in Subroutine 3; this M-STEP is similar to the MAML algorithm in BID8. Note that, unlike other recent approaches to probabilistic clustering (e.g., BID1 we adhere to the episodic meta-learning setup for both training and testing since only the task support set x j 1:N is used to compute both the point estimateφ j and the cluster responsibilities γ j. See Algorithm 2 for the full algorithm, whose high-level structure is shared with the non-parametric variant of our method. Clustering task-specific parameters provides a way for a meta-learner to deal with task heterogeneity since each cluster can be associated with a subset of the tasks that would benefit most from inductive DISPLAYFORM0 Subroutine 4: The E-STEP and M-STEP for an infinite mixture of hierarchical Bayesian models.transfer. We apply Algorithm 2 with L = 5 components to the 1-shot 5-way classification few-shot classification benchmark miniImageNet BID46 using the same data split, architecture, and hyperparameter values as in BID8. We additionally use τ = 1 for the softmax temperature and the same initialization as BID8 for our global prior G 0 (which reduces to a fixed initialization in the parametric case).While we do not expect the standard miniImageNet dataset to present much heterogeneity given the uniform sampling assumptions behind its design, we demonstrate in TAB1 that our parametric meta-learner can improve the generalization of gradient-based meta-learning on this task. This suggests some level of cluster differentiation even on this non-heterogeneous benchmark. The mixture of meta-learners developed in Section 3 addresses a drawback of meta-learning approaches such as MAML that consolidate task-general information into a single set of hyperparameters, but adds another dimension to model validation in the form of identifying the correct number of mixture components. While this may be resolved by cross-validation if the dataset is static and therefore the number of components may be fixed, adhering to a fixed number of components throughout training is not appropriate in the nonstationary regime, where the underlying task distribution changes and therefore different types of tasks are presented sequentially. In this regime, it is important to add mixture components sequentially to enable specialization of the component meta-learners to the different types of tasks that constitute the dataset. To address this, we derive a scalable stochastic estimation procedure to compute the expectation of task-tocluster assignments (E-Step) for a growing number of task clusters using an infinite or non-parametric mixture model BID30 called the Dirichlet process mixture model (DPMM). This approach obviates the need for an a priori fixed number of components and enables the model to unboundedly adapt its complexity according to the observed data. The formulation of the DPMM that is most appropriate for incremental learning is the sequential draws formulation that corresponds to an instantiation of the Chinese restaurant process (CRP) BID30. A CRP prior over z allows some probability to be assigned to a new mixture component while the task identities are inferred in a sequential manner, and has therefore been key to recent online and stochastic learning of the DPMM BID19. A draw from a CRP proceeds as follows: For a sequence of tasks (1, . . ., J), the first task is assigned to the first cluster and the jth subsequent task is then assigned to the th cluster drawn with probability DISPLAYFORM0 where L indicates the number of non-empty clusters, n indicates the number of tasks already occupying a cluster, and ζ is a fixed positive concentration parameter. The prior probability associated with a new mixture component is therefore p(z j = L + 1 | z 1:j−1, ζ), and the joint log-likelihood of the infinite mixture model can be written as DISPLAYFORM1 where γ j is the responsibility of cluster for task-specific parameters j. We refer the reader to BID30 for more details on the likelihood function associated with the DPMM.In a similar spirit to Section 3, we develop a stochastic EM procedure for this estimation problem. However, while the computation of the mode estimate of the task-specific parameters φ is mostly unchanged from finite variant, the estimation of the cluster assignment variables z in the E-STEP requires revisiting the Gibbs conditional distributions due to the potential addition of a new cluster at each step. For a DPMM, the conditional distributions for z are DISPLAYFORM2 with G 0 as the base measure over the components of the CRP, also known as the global prior. G is the prior over each cluster's parameters, initialized with a draw from a Gaussian centered at G 0 with a fixed variance. Using a mode estimateφ j for task-specific φ j, the distribution over task-to-cluster assignments z j can be expressed as DISPLAYFORM3 We thus obtain a set of local objective functions amenable to gradient-based optimization in a similar fashion to the parametric M-STEP of Subroutine 3. We can also omit the prior term log p(φ | θ ) as it arises as an implicit prior ing from truncated gradient descent, as explained in Section 3 of BID13.One marked difference between the objective of the M-STEP in Subroutine 3 and that in Subroutine 4 is the penalty term of log n or log ζ which incentivizes larger clusters in order to deter overspawning. Accordingly, this approximate inference routine still preserves the preferential attachment ("rich get richer") dynamics of Bayesian nonparametrics BID32. Another is that is not immediate from the Gibbs conditionals is the use of a threshold on the cluster responsibilities to account for mini-batch noise when spawning a cluster based on a single batch. This threshold is necessary for our stochastic mode estimation of Algorithm 4 as it maintains that a new cluster's responsibility needs to exceed a certain value before being permanently added to the set of components (see the E-STEP in Subroutine 4 for more details). Intuitively, if a cluster has close to an equal share of responsibilities to existing clusters after accounting for the CRP penalty (log n vs. log ζ), it is spawned. A sequential approximation for nonparametric mixtures with a similar threshold was proposed in BID19, where variational Bayes was used instead of point estimation in a DPMM.Overall, unlike traditional nonparametric algorithms, our model does not refine the cluster assignments of previously observed points by way of multiple expensive passes over the whole data set. Instead, we incrementally infer model parameters and add components during episodic training based on noisy estimates but unbiased of the log-likelihood gradients. To the best of our knowledge, no previous work has considered a scalable stochastic point estimation of the parameters in a non-parametric mixture model. Furthermore, we are not aware of prior work applying non-parametric mixture models to high-dimensional function approximators such as modern neural networks with end-to-end learning. A non-parametric mixture model should be able to detect and adapt to a changing distribution of tasks, without any external information to signal the start of a new task type (i.e.,, in a "taskagnostic" manner), by adjusting its capacity. On the other hand, current meta-learning methods coerce Figure 5: Mixture component validation log-likelihoods (MSE losses) on an evolving dataset that generates even polynomial regression tasks for 700 iterations, then odd polynomial regression tasks until iteration 1400 at which point it generates sinusoidal regression tasks. We plot the validation negative log likelihood of the data for each task. Note, for example, the change in loss to the second (red) cluster at 700 iterations when the odd polynomial tasks are introduced.the inductive biases into a single set of hyperparameters and therefore inevitably saturate model parameters in the evolving dataset regime. Here, we present our main experimental using synthetic few-shot regression tasks to demonstrate that our non-parametric meta-learning algorithm is able to acquire new inductive biases as new tasks are introduced without necessarily over-writing existing ones by adding new components. Experimental Setup To demonstrate the unconstrained and adaptive capacity of our non-parametric meta-learning model, we consider alternating sinusoidal, even-degree polynomial, and odd-degree polynomial regression tasks with input x sampled uniformly from [−5, 5] during the meta-training procedure. For the sinusoidal regression, we consider a sine wave with phase sampled uniformly from [0, π] and with amplitudes sampled from a 1 ∼ N in a similar fashion to the synthetic regression tasks in BID8 As for the polynomial regression, y = Figure 5, we presented even-degree polynomial regression tasks for 700 iterations, followed by odd-degree polynomial regression tasks until 1400 iterations, before switching to sinusoidal regression tasks. We use the mean-squared error loss function for each task as the inner loop and meta-level objectives. Hyperparameter choices Our architecture is a feedforward neural network with 2 hidden layers with ReLU nonlinearities, each of size 40. We use a meta-batch size of 25 tasks (both for the inner updates and the meta-gradient updates) as in the setup for 10-shot sinusoidal regression in BID8. Our non-parametric algorithm starts with a single cluster (L 0 = 1 in Algorithm 4). In these experiments, we set the threshold = 0.95T /(L + 1), with L the number of non-empty clusters and T the size of the meta-batch. We also compute the cluster sizes using a moving window of size 20 (which is a dataset-dependent hyperparameter) to accommodate the order of magnitude in comparison to the small training losses used for cluster responsibility evaluation. This is necessary since we do not keep track of the exact prior cluster assignments for our randomly selected task mini-batches nor do we re-assess our assignments at each iteration. Otherwise, non-empty clusters can accumulate an extremely large number of assignments, making cluster spawning impossible after only a few meta-learning episodes. An additional practical reason for the stochastic setting is that it would be extremely expensive to store the assignments for an entire dataset in memory. Finally, preserving task assignments is potentially training iteration index Figure 6: (Top) An evolving dataset of quadratic, sinusoidal, and logistic regression tasks. The dataset generates quadratic regression tasks for 700 iterations, then switches to generating sinusoidal regression tasks until iteration 5700 at which point it generates logistic regression tasks. (Below) Note the change in responsibilities to the second (red) cluster at 700 iterations when the sinusoidal tasks are introduced. At 5700 iterations, the logistic regression tasks (third row) also cause a third (brown) cluster to be introduced. harmful due to stale parameter values since the task assignments in our framework are meant to be easily reconstructed on-the-fly using the E-STEP with updated parameters θ. Results. In Figure 5, we report the mean-squared error (MSE) validation loss for each task as the 3 different tasks are introduced sequentially and disjoint (only one task for each training phase) to the non-parametric meta-learner. Overall, our algorithm consistently outperforms MAML in terms of validation loss on the three tasks across the three training phases. More interestingly, our algorithm preserves its performance on old tasks when switching to a new training phase, whereas MAML suffers from a clear degradation. While it seems our algorithm does not perfectly preserve old inductive biases, we can conclude from a parallel experiment in Figure 6 that it can increase its capacity, when needed, to adjust for new training phases. This allows for better preservation of previously learnt knowledge which is key for continual learning. In this section, we turn our focus to task differentiation. We thus investigate the cluster responsibilities on validation data from each of the 3 tasks: quadratic regression, sinusoidal regression, and logistic regression on data from the same input range as specified in Section 5.1.In Figure 5, we notice a clear differentiation between the tasks as indicated by the cluster responsibilities. The responsibilities under the first cluster (in blue) decreases to almost zero for the 3rd task (sinusoid) while staying evenly split for the related odd and even polynomial regression tasks. Furthermore, a second cluster (in red) is spawned to account for the difference between odd and even degree polynomials. However, we also notice that the second cluster responsibilities are not zero for the first task, indicating similarities between even and odd polynomial regressions. The same behavior can be seen for the third cluster on the third task. Note that the sinusoidal regression task is the more difficult task which explains the different order of magnitude of the losses and the motivation for a longer period of training. Also note that regression losses are unbounded and thus pose difficulties to any optimization-based continual learner; accordingly, most continual learning datasets such as Moving MNIST consist of classification tasks that make use of a bounded cross-entropy error. Multi-task learning. Rosenstein et al. BID34 demonstrated that negative transfer can worsen generalization performance. This has motivated much work on HB in transfer learning and domain adaptation (e.g., BID17 BID49 BID9 BID6 BID47 . Closest to our proposed approach is early work on hierarchical Bayesian multi-task learning with neural networks BID15 BID0 BID43 . These approaches are different from ours in that they place a prior, which could be nonparametric as in and BID43, only on the output layer. Furthemore, none of these approaches were applied to the episodic training setting of meta-learning. BID15 and BID43 also propose training a mixture model over the output layer weights using MAP inference. However, this approach does not scale well to all the layers in a network and performing full passes on the dataset for inference of the full set of weights can become computationally intractable. Continual learning. Techniques developed specifically to address the catastrophic forgetting problem in continual learning, such as elastic weight consolidation (EWC) BID16, synaptic intelligence (SI) BID50, and variational continual learning (VCL) BID27 require access to an explicit delineation between tasks that acts as a catalyst to grow model size, which we refer to as "task-aware." In contrast, our nonparametric algorithm tackles the "task-agnostic" setting of continual learning, where the meta-learner does not receive information about task changes but instead learns to recognize a shift in the task distribution and adapt accordingly. Clustering. Incremental or stochastic clustering has been considered in the EM setting in BID26, and similarly for minibatch K-means BID41. Online learning of nonparametric mixture models is also a way of perform clustering in the mini-batch setting using sequential variational inference BID19.A key distinction between our work and these approaches is that we leverage the connection between empirical Bayes in this model and gradient-based metalearning BID13 to use the MAML BID8 objective as a log posterior surrogate. This allows our algorithm to scale and easily integrate with minibatch stochastic gradient-based meta-learning instead of alternating multiple backpropagation steps with multiple inference passes over the full dataset BID43 BID1. Our approach is distinct from recent work on gradient-based clustering BID14 since we adhere to the more challenging setting of episodic meta-learning for both training and testing. This can be a challenging setting for a clustering algorithm, as the assignments need to be computed using K = 1 examples per class as is the case in 1-shot learning. Meta-learning is a source of learned inductive bias. Occasionally, the inductive bias is harmful because the experience gained from solving one task does not transfer well to another. On the other hand, if tasks are closely related, they can benefit from a greater amount of inductive transfer. Here, we present an approach that allows a gradient-based meta-learner to explicitly modulate the amount of transfer between tasks, as well as to adapt its parameter dimensionality when the underlying task distribution evolves. We formulate this as probabilistic inference in a mixture model that defines a clustering of task-specific parameters. To ensure scalability, we make use of the recent connection between gradient-based meta-learning and hierarchical Bayes BID13 to perform approximate maximum a posteriori (MAP) inference in both a finite and an infinite mixture model. This approach admits non-conjugate likelihoods parameterised with a black-box function approximator such as a deep neural network, and therefore learns to identify underlying genres of tasks using the standard gradient descent learning rule. We demonstrate that this approach allows the model complexity to grow along with the evolving complexity of the observed tasks in both a few-shot regression and a few-shot classification problem. BID31 43.44 ± 0.77 60.60 ± 0.71 SNAIL BID20 b 45.1 ± --55.2 ± --prototypical networks BID42 c 46.61 ± 0.78 65.77 ± 0.70 MAML BID8 48.70 ± 1.84 63.11 ± 0.92 LLAMA BID13 49.40 BID45 49.82 ± 0.78 63.70 ± 0.67 KNN + GNN embedding BID10 49.44 ± 0.28 64.02 ± 0.51 GNN BID10 50.33 ± 0.36 66.41 ± 0.63 fwCNN (Hebb) BID22 50.21 ± 0.37 64.75 ± 0.49 fwResNet (Hebb) BID22 56.84 ± 0.52 71.00 ± 0.34 SNAIL BID20 55. BID23 57.10 ± 0.70 70.04 ± 0.63 MAML BID8 Figure 7: An evolving dataset of miniImageNet few-shot classification tasks where for the first 20k iterations we train on the standard dataset, then switch to a "pencil" effect set of tasks for 10k iterations before finally switching to a "blurred" effect set of tasks until 40k. Responsibilities γ for each cluster are plotted over time. Note the change in responsibilities as the dataset changes at iterations 20k and 30k. We apply Algorithm 4 to an evolving variant of the miniImageNet few-shot classification dataset while using the same standard architecture of BID46 BID8. In this variant, different artistic filters are applied to the images in the few-shot classification tasks over the metatraining procedure to simulate a changing distribution of classification tasks. More specifically, we first train on the standard mini-imagenet tasks for 20000 iterations then introduce "pencil" effect tasks for 10000 iterations, before finally switching to a "radial blur" effect for another 10000 iterations. Cluster responsibilities during training can be found in Figure 7. To compare more closely to the single-cluster baseline, we restrict the number of clusters to 1 for the first phase of training (the standard mini-imagenet tasks). However, the moment we start introducing new datasets, this restriction is lifted. This allows a better evaluation of the transfer from the first cluster to the new tasks as more datasets are introduced in an online setting. We notice some differentiation between the tasks that is not as pronounced as what we observed on the toy data. This is potentially due to the fact that all the tasks are derived from the same core set of images. Accordingly, inductive biases learned on the unfiltered dataset for the first 20000 iterations can transfer to the filtered datasets more easily than the different regression tasks we experimented with.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We use the connection between gradient-based meta-learning and hierarchical Bayes to learn a mixture of meta-learners that is appropriate for a heterogeneous and evolving task distribution.

Represent the TLDR for finding the corresponding scientific article:

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the article for finding a single-sentence TLDR

Question: We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. Unlike previously proposed routing algorithms, the parent's ability to reconstruct the child is not explicitly taken into account to update the routing probabilities. This simplifies the routing procedure and improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces sequential iterative routing with concurrent iterative routing. Besides outperforming existing capsule networks, our model performs at-par with a powerful CNN (ResNet-18), using less than 25% of the parameters. On a different task of recognizing digits from overlayed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks. Capsule Networks (CapsNets) represent visual features using groups of neurons. Each group (called a "capsule") encodes a feature and represents one visual entity. Grouping all the information about one entity into one computational unit makes it easy to incorporate priors such as "a part can belong to only one whole" by routing the entire part capsule to its parent whole capsule. Routing is mutually exclusive among parents, which ensures that one part cannot belong to multiple parents. Therefore, capsule routing has the potential to produce an interpretable hierarchical parsing of a visual scene. Such a structure is hard to impose in a typical convolutional neural network (CNN). This hierarchical relationship modeling has spurred a lot of interest in designing capsules and their routing algorithms (; ; ; ; ; ; . In order to do routing, each lower-level capsule votes for the state of each higher-level capsule. The higher-level (parent) capsule aggregates the votes, updates its state, and uses the updated state to explain each lower-level capsule. The ones that are well-explained end up routing more towards that parent. This process is repeated, with the vote aggregation step taking into account the extent to which a part is routed to that parent. Therefore, the states of the hidden units and the routing probabilities are inferred in an iterative way, analogous to the M-step and E-step, respectively, of an Expectation-Maximization (EM) algorithm. Dynamic Routing and EMrouting can both be seen as variants of this scheme that share the basic iterative structure but differ in terms of details, such as their capsule design, how the votes are aggregated, and whether a non-linearity is used. We introduce a novel routing algorithm, which we called Inverted Dot-Product Attention Routing. In our method, the routing procedure resembles an inverted attention mechanism, where dot products are used to measure agreement. Specifically, the higher-level (parent) units compete for the attention of the lower-level (child) units, instead of the other way around, which is commonly used in attention models. Hence, the routing probability directly depends on the agreement between the parent's pose (from the previous iteration step) and the child's vote for the parent's pose (in the current iteration step). We also propose two modifications for our routing procedure - using Layer Normalization as normalization, and doing inference of the latent capsule states and routing probabilities jointly across multiple capsule layers (instead of doing it layer-wise). These modifications help scale up the model to more challenging datasets. Our model achieves comparable performance as the state-of-the-art convolutional neural networks (CNNs), but with much fewer parameters, on CIFAR-10 (95.14% test accuracy) and CIFAR-100 (78.02% test accuracy). We also introduce a challenging task to recognize single and multiple overlapping objects simultaneously. To be more precise, we construct the DiverseMultiMNIST dataset that contains both single-digit and overlapping-digits images. With the same number of layers and the same number of neurons per layer, the proposed CapsNet has better convergence than a baseline CNN. Overall, we argue that with the proposed routing mechanism, it is no longer impractical to apply CapsNets on real-world tasks. We will release the source code to reproduce the experiments. An example of our proposed architecture is shown in Figure 1. The backbone is a standard feedforward convolutional neural network. The features extracted from this network are fed through another convolutional layer. At each spatial location, groups of 16 channels are made to create capsules (we assume a 16-dimensional pose in a capsule). LayerNorm is then applied across the 16 channels to obtain the primary capsules. This is followed by two convolutional capsule layers, and then by two fully-connected capsule layers. In the last capsule layer, each capsule corresponds to a class. These capsules are then used to compute logits that feed into a softmax to computed the classification probabilities. Inference in this network requires a feed-forward pass up to the primary capsules. After this, our proposed routing mechanism (discussed in the next section) takes over. In prior work, each capsule has a pose and some way of representing an activation probability. In Dynamic Routing CapsNets , the pose is represented by a vector and the activation probability is implicitly represented by the norm of the pose. In EM Routing CapsNets , the pose is represented by a matrix and the activation probability is determined by the EM algorithm. In our work, we consider a matrix-structured pose in a capsule. We denote the capsules in layer L as P L and the i-th capsule in layer L as p form and will be reshaped to R d L when representing it as a matrix, where d L is the number of hidden units grouped together to make capsules in layer L. The activation probability is not explicitly represented. By doing this, we are essentially asking the network to represent the absence of a capsule by some special value of its pose. The proposed routing process consists of two steps. The first step computes the agreement between lower-level capsules and higher-level capsules. The second step updates the pose of the higher-level capsules. Procedure 1 Inverted Dot-product Attention Routing algorithm returns updated poses of the capsules in layer L + 1 given poses in layer L and L + 1 and weights between layer L and L + 1. for all capsule i in layer L and capsule j in layer for all capsule i in layer L and capsule j in layer for all capsule j in layer (L + 1): return P Step 1: Computing Agreement: where the matrix W The pose p L+1 j is obtained from the previous iteration of this procedure, and will be set to 0 initially. Step 2 where r L ij is an inverted attention score representing how higher-level capsules compete for attention of lower-level capsules. Using the routing probabilities, we update the pose p L+1 j for capsule j in layer L + 1 from all capsules in layer L: We adopt Layer Normalization as the normalization, which we empirically find it to be able to improve the convergence for routing. The routing algorithm is summarized in Procedure 1 and Figure 2. To explain how inference and learning are performed, we use Figure 1 as an example. Note that the choice of the backbone, the number of capsules layers, the number of capsules per layer, the design of the classifier may vary for different sets of experiments. We leave the discussions of configurations in Sections 5 and 6, and in the Appendix. For ease of exposition, we decompose a CapsNet into pre-capsule, capsule and post-capsule layers. The goal is to obtain a backbone feature F from the input image I. The backbone model can be either a single convolutional layer or ResNet computational blocks . for L in layers 2 to N: P L ← 0s non-primary capsules /* Capsules Layers (1st Iteration): sequential routing */ for L in layers 1 to (N − 1) do 6: non-primary capsules /* Capsules Layers (2nd to tth Iteration): concurrent routing */ 7: for L in layers 2 to N: P L ←P returnŷ Capsule Layers: The primary capsules P 1 are computed by applying a convolution layer and Layer Normalization to the backbone feature F. The non-primary capsules layers P 2:N are initialized to be zeros 1. For the first iteration, we perform one step of routing sequentially in each capsule layer. In other words, the primary capsules are used to update their parent convolutional capsules, which are then used to update the next higher-level capsule layer, and so on. After doing this first pass, the rest of the routing iterations are performed concurrently. Specifically, all capsule layers look at their preceding lower-level capsule layer and perform one step of routing simultaneously. This procedure is an example of a parallel-in-time inference method. We call it "concurrent routing" as it concurrently performs routing between capsules layers per iteration, leading to better parallelism. Figure 3 illustrates this procedure from routing iteration 2 to t. It is worth noting that, our proposed variant of CapsNet is a weight-tied concurrent routing architecture with Layer Normalization, which empirically showed could converge to fixed points. Previous CapsNets used sequential layer-wise iterative routing between the capsules layers. For example, the model first performs routing between layer L − 1 and layer L for a few iterations. Next, the model performs routing between layer L and L + 1 for a few iterations. When unrolled, this sequential iterative routing defines a very deep computational graph with a single path going from the inputs to the outputs. This deep graph could lead to a vanishing gradients problem and limit the depth of a CapsNet that can be trained well, especially if any squashing non-linearities are present. With concurrent routing, the training can be made more stable, since each iteration has a more cumulative effect. The goal is to obtain the predicted class logitsŷ from the last capsule layer (the class capsules) P N. In our CapsNet, we use a linear classifier for class i in class capsules: This classifier is shared across all the class capsules. We update the parameters θ, W 1:N −1 by stochastic gradient descent. For multiclass classification, we use multiclass cross-entropy loss. For multilabel classification, we use binary cross-entropy loss. We also tried Margin loss and Spread loss which are introduced by prior work . However, these losses do not give us better performance against cross-entropy and binary cross-entropy losses. The concurrent routing is a parallel-in-time routing procedure for all capsules layers. CIFAR-10 and CIFAR-100 datasets consist of small 32 × 32 real-world color images with 50, 000 for training and 10, 000 for evaluation. CIFAR-10 has 10 classes, and CIFAR-100 has 100 classes. We choose these natural image datasets to demonstrate our method since they correspond to a more complex data distribution than digit images. Comparisons with other CapsNets and CNNs: In Table 1, we report the test accuracy obtained by our model, along with other CapsNets and CNNs. Two prior CapsNets are chosen: Dynamic Routing CapsNets and EM Routing CapsNets . For each CapsNet, we apply two backbone feature models: simple convolution followed by ReLU nonlinear activation and a ResNet backbone. For CNNs, we consider a baseline CNN with 3 convolutional layers followed by 1 fully-connected classifier layer. First, we compare previous routing approaches against ours. In a general trend, the proposed CapsNets perform better than the Dynamic Routing CapsNets, and the Dynamic Routing CapsNets perform better than EM Routing CapsNets. The performance differs more on CIFAR-100 than on CIFAR-10. For example, with simple convolutional backbone, EM Routing CapsNet can only achieve 37.73% test accuracy while ours can achieve 57.32%. Additionally, for all CapsNets, we see improved performance when replacing a single convolutional backbone with ResNet backbone. This is not surprising since ResNet structure has better generalizability than a single convolutional layer. Second, we discuss the performance difference between CNNs and CapsNets. We see that, with a simple backbone (a single convolutional layer), it is hard for CapsNets to reach the same performance as CNNs. For instance, our routing approach can only achieve 57.32% test accuracy on CIFAR-100 while the baseline CNN achieves 62.30%. However, with a SOTA backbone structure (ResNet backbone), the proposed routing approach can reach competitive performance (95.14% on CIFAR-10) as compared to the SOTA CNN model (ResNet-18 with 95.11% on CIFAR-10). Convergence Analysis: In Figure 4, top row, we analyze the convergence for CapsNets with respect to the number of routing iterations. The optimization hyperparameters are chosen optimally for each routing mechanism. For Dynamic Routing CapsNets , we observe a mild performance drop when the number of iterations increases. For EM Routing CapsNets , the best-performed number of iterations is 2. Increasing or decreasing this number severely hurts the performance. For our proposed routing mechanism, we find a positive correlation between performance and number of routing iterations. The performance variance is also the smallest among the three routing mechanisms. This suggests our approach has better optimization and stable inference. However, selecting a larger iteration number may not be ideal since memory usage and inference time will also increase (shown in the bottom right in Figure 4). Note that, we observe sharp performance jitters during training when the model has not converged (especially when the number of iterations is high). This phenomenon is due to applying LayerNorm on a low-dimensional vector. The jittering is reduced when we increase the pose dimension in capsules. Ablation Study: Furthermore, we inspect our routing approach with the following ablations: 1) Inverted Dot-Product Attention-A: without Layer Normalization; 2) Inverted Dot-Product Attention-B: replacing concurrent to sequential iterative routing; and 3) Inverted Dot-Product Attention-C: Figure 4 bottom row. When removing Layer Normalization, performance dramatically drops from our routing mechanism. Notably, the prediction becomes uniform when the iteration number increases to 5. This implies that the normalization step is crucial to the stability of our method. When replacing concurrent with sequential iterative routing, the positive correlation between performance and iteration number no longer exists. This fact happens in the Dynamic Routing CapsNet as well, which also uses sequential iterative routing. When adding activations to our capsule design, we obtain a performance deterioration. Typically, squashing activations such as sigmoids make it harder for gradients to flow, which might explain this. Discovering the best strategy to incorporate activations in capsule networks is an interesting direction for future work. The goal in this section is to compare CapsNets and CNNs when they have the same number of layers and the same number of neurons per layer. Specifically, we would like to examine the difference of the representation power between the routing mechanism (in CapsNets) and the pooling operation (in CNNs). A challenging setting is considered in which objects may be overlapping with each other, and there may be a diverse number of objects in the image. To this end, we construct the DiverseMultiMNIST dataset which is extended from MNIST , and it contains both single-digit and two overlapping digit images. The task will be multilabel classification, where the prediction is said to be correct if and only if the recognized digits match all the digits in the image. We plot the convergence curve when the model is trained on 21M images from DiverseMultiMNIST. Please see Appendix B.2 for more details on the dataset and Appendix B.1 for detailed model configurations. The are reported in Figure 5. First, we compare our routing method against the Dynamic routing one. We observe an improved performance from the CapsNet * to the CapsNet (83.39% to 85.74% with vector-structured poses). The suggests a better viewpoint generalization for our routing mechanism. Second, we compare baseline CNN against our CapsNet. From the table, we see that CapsNet has better test accuracy compared to CNN. For example, the CapsNet with vector-structured poses reaches 85.74% test accuracy, and the baseline CNN reaches 79.81% test accuracy. In our CNN implementation, we use average pooling from the last convolutional layer to its next fully-connected layer. We can see that having a routing mechanism works better than pooling. However, one may argue that the pooling operations requires no extra parameter but routing mechanism does, and hence it may not be fair to compare their performance. To address this issue, in the baseline CNN, we replace the pooling operation with a fully-connected operation. To be more precise, instead of using average pooling, we learn the entire transformation matrix from the last convolutional layer to its next fully-connected layer. This procedure can be regarded as considering pooling with learnable parameters. After doing this, the number of parameters in CNN increases to 42.49M, and the corresponding test accuracy is 84.84%, which is still lower than 85.74% from the CapsNet. We conclude that, when recognizing overlapping and diverse number of objects, the routing mechanism has better representation power against the pooling operation. Last, we compare CapsNet with different pose structures. The CapsNet with vector-structured poses works better than the CapsNet with matrix-structured poses (80.59% vs 85.74%). However, the former requires more parameters, more memory usage, and more inference time. If we increase the number of parameters in the matrix-pose CapsNet to 42M, its test accuracy rises to 91.17%. Nevertheless, the model now requires more memory usage and inference time as compared to using vector-structured poses. We conclude that more performance can be extracted from vector-structured poses but at the cost of high memory usage and inference time. The idea of grouping a set of neurons into a capsule was first proposed in Transforming AutoEncoders . The capsule represented the multi-scale recognized fragments of the input images. Given the transformation matrix, Transforming Auto-Encoders learned to discover capsules' instantiation parameters from an affine-transformed image pair. extended this idea to learn part-whole relationships in images systematically. cast the routing mechanism as fitting a mixture of Gaussians. The model demonstrated an impressive ability for recognizing objects from novel viewpoints. Recently, Stacked Capsule AutoEncoders proposed to segment and compose the image fragments without any supervision. The work achieved SOTA on unsupervised classification. However, despite showing promising applications by leveraging inherent structures in images, the current literature on capsule networks has only been applied on datasets of limited complexity. Our proposed new routing mechanism instead attempts to apply capsule networks to more complex data. Our model also relates to Transformers and Set Transformers , where dot-product attention is also used. In the language of capsules, a Set Transformer can be seen as a model in which a higher-level unit can choose to pay attention to K lower-level units (using K attention heads). Our model inverts the attention direction (lower-level units "attend" to parents), enforces exclusivity among routing to parents and does not impose any limits on how many lower-level units can be routed to any parent. Therefore, it combines the ease and parallelism of dot-product routing derived from a Transformer, with the interpretability of building a hierarchical parsing of a scene derived from capsule networks. There are other works presenting different routing mechanisms for capsules. formulated the Dynamic routing as an optimization problem consisting of a clustering loss and a KL regularization term. generalized the routing method within the framework of weighted kernel density estimation. approximated the routing process with two branches and minimized the distributions between capsules layers by an optimal transport divergence constraint. replaced standard convolutional structures before capsules layers by densely connected convolutions. It is worth noting that this work was the first to combine SOTA CNN backbones with capsules layers. proposed DeepCaps by stacking 10+ capsules layers. It achieved 92.74% test accuracy on CIFAR-10, which was the previous best for capsule networks. Instead of looking for agreement between capsules layers, proposed to learn deterministic attention scores only from lower-level capsules. Nevertheless, without agreement, their best-performed model achieved only 88.61% test accuracy on CIFAR-10. In contrast to these prior work, we present a combination of inverted dotproduct attention routing, layer normalization, and concurrent routing. To the best of our knowledge, we are the first to show that capsule networks can achieve comparable performance against SOTA CNNs. In particular, we achieve 95.14% test accuracy for CIFAR-10 and 78.02% for CIFAR-100. In this work, we propose a novel Inverted Dot-Product Attention Routing algorithm for Capsule networks. Our method directly determines the routing probability by the agreements between parent and child capsules. Routing algorithms from prior work require child capsules to be explained by parent capsules. By removing this constraint, we are able to achieve competitive performance against SOTA CNN architectures on CIFAR-10 and CIFAR-100 with the use of a low number of parameters. We believe that it is no longer impractical to apply capsule networks to datasets with complex data distribution. Two future directions can be extended from this paper: • In the experiments, we show how capsules layers can be combined with SOTA CNN backbones. The optimal combinations between SOTA CNN structures and capsules layers may be the key to scale up to a much larger dataset such as ImageNet. • The proposed concurrent routing is as a parallel-in-time and weight-tied inference process. The strong connection with Deep Equilibrium Models can potentially lead us to infinite-iteration routing. Suofei Zhang, Quan Zhou, and Xiaofu Wu. Fast dynamic routing based on weighted kernel density estimation. In International Symposium on Artificial Intelligence and Robotics, pp. 301-309.. A MODEL CONFIGURATIONS FOR CIFAR-10/CIFAR-100 The configuration choices of Dynamic Routing CapsNets and EM Routing CapsNets are followed by prior work . We empirically find their configurations perform the best for their routing mechanisms (instead of applying our network configurations to their routing mechanisms). The optimizers are chosen to reach the best performance for all models. We list the model specifications in Table 2, 3, 4, 5, 6, 7, 8, and 9. We only show the specifications for CapsNets with a simple convolutional backbone. When considering a ResNet backbone, two modifications are performed. First, we replace the simple feature backbone with ResNet feature backbone. Then, the input dimension of the weights after the backbone is set as 128. A ResNet backbone contains a 3 × 3 convolutional layer (output 64-dim.), three 64-dim. residual building block with stride 1, and four 128-dim. residual building block with stride 2. The ResNet backbone returns a 16 × 16 × 128 tensor. For the optimizers, we use stochastic gradient descent with learning rate 0.1 for our proposed method, baseline CNN, and ResNet-18 . We use Adam with learning rate 0.001 for Dynamic Routing CapsNets and Adam with learning rate 0.01 for EM Routing CapsNets. We decrease the learning rate by 10 times when the model trained on 150 epochs and 250 epochs, and there are 350 epochs in total. We consider the same data augmentation for all networks. During training, we first pad four zerovalue pixels to each image and randomly crop the image to the size 32 × 32. Then, we horizontally flip the image with probability 0.5. During evaluation, we do not perform data augmentation. All the model is trained on a 8-GPU machine with batch size 128. To fairly compare CNNs and CapsNets, we fix the number of layers and the number of neurons per layer in the models. These models consider the design: 36x36 image → 18x18x1024 neurons → 8x8x1024 neurons → 6x6x1024 neurons → 640 neurons → 10 class logits. The configurations are presented in Table 10, 11, and 12. We also fix the optimizers across all the models. We use stochastic gradient descent with learning rate 0.1 and decay the learning rate by 10 times when the models trained on 150 steps and 250 steps. One step corresponds to 60, 000 training samples, and we train the models with a total of 350 steps. Diverse MultiMNIST contains both single-digit and overlapping-digit images. We generate images on the fly and plot the test accuracy for training models over 21M (21M = 350(steps) × 60, 000(images)) generated images. We also generate the test images, and for each evaluation step, there are 10, 000 test images. Note that we make sure the training and the test images are from the disjoint set. In the following, we shall present how we generate the images. We set the probability of generating a single-digit image as 1 6 and the probability of generating an overlapping-digit image as 5 6. The single-digit image in DiverseMultiMNIST training/ test set is generated by shifting digits in MNIST training/ test set. Each digit is shifted up to 4 pixels in each direction and in 36 × 36 image. , we generate overlapping-digit images in DiverseMultiMNIST training/ test set by overlaying two digits from the same training/ test set of MNIST. Two digits are selected from different classes. Before overlaying the digits, we shift the digits in the same way which we shift for the digit in a single-digit image. After overlapping, the generated image has size 36 × 36. We consider no data augmentation for both training and evaluation. All the model is trained on a 8-GPU machine with batch size 128. Output Size input dim=3, output dim=256, 9x9 conv, stride=1, padding=0 24x24x256 ReLU input dim=256, output dim=256, 9x9 conv, stride=2, padding=0 8x8x256 Capsules reshape 8x8x32x8 Squash Linear Dynamic Routing to 100 16-dim. capsules 100x16 Squash

represent the tldr for finding the corresponding scientific article

We present a new routing method for Capsule networks, and it performs at-par with ResNet-18 on CIFAR-10/ CIFAR-100.

Represent the TLDR for finding the corresponding scientific article

We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.


Question: Represent the scientific article to find a short TLDR

Question: We introduce Doc2Dial, an end-to-end framework for generating conversational data grounded in business documents via crowdsourcing. Such data can be used to train automated dialogue agents performing customer care tasks for the enterprises or organizations. In particular, the framework takes the documents as input and generates the tasks for obtaining the annotations for simulating dialog flows. The dialog flows are used to guide the collection of utterances produced by crowd workers. The outcomes include dialogue data grounded in the given documents, as well as various types of annotations that help ensure the quality of the data and the flexibility to (re)composite dialogues. There has been growing interest in using automated dialogue agents to assist customers through online chat. However, despite recent effort in training automated agents with human-human dialogues, it often faces the bottleneck that a large number of chat logs or simulated dialogues with various scenarios are required. Meanwhile, enterprises and organizations often own a large number of business documents that could address customers' requests, such as technical documentation, policy guidance and Q&A webpages. However, customers would still prefer having interactive conversations with agents instead of searching and reading through lengthy documents. Taken together, a promising solution is to build machine assisted agents that could perform task-oriented dialogues that are based on the content of the business documents. In task-oriented dialogues for customer care, a recurrent theme is a diagnostic process -identifying the contextual conditions that apply to the customer to retrieve the most relevant solutions. Meanwhile, business documents often contain similar information, with prior conditions, in for example if-clauses or subtitles, followed by corresponding solutions. Therefore, these documents can be used to guide diagnostic dialogues-we call them document-grounded dialogues. For example, the sample business document in Figure 1 contains information for an agent to perform the dialogue on the right, where P-n (S-n) denotes text span n labeled a precondition (solution) and "O-D" denotes "out of domain". The preconditions are expressed in various ways such as subtitles or if-clauses, followed by corresponding solution if that precondition applies. In this work, we hypothesize that an essential capability for a dialogue agent to perform goal-oriented information retrieval tasks should be to recognize the preconditions and their associated solutions covered by the given documents and then use them to carry out the diagnostic interactions. Towards this goal, we introduce DOC2DIAL, an end-to-end framework for generating conversational data grounded in business documents via crowdsourcing. It aims to minimize the effort for handcrafting dialog flows that is specific to the document but still introduce dynamic dialog scenes. It also provides quality control over the data collection process. We guide our investigation with the following principles: 1) We aim to identify the document content that provides solution(s) to a user's request as well as describes the prerequisites required. 2) The generated dialog flows should be specific to the given document without relying on heavily supervised or handcrafted work. 3) The generated data tasks should be easy to scale -feasible to crowdsourcing platforms and could be updated with respect to changes in the document. Thus, we propose a pipeline of three interconnected tasks for dialogue composition based on business documents: labeling text spans as preconditions or solutions in a given documents (TextAnno); identifying the relation(s) between these preconditions or solutions (RelAnno; simulating dialog flows based on the linked preconditions/solutions and applying them to guide the collection of human generated utterances(DialAnno). For the dialogue collection, we could deploy it via both synchronized and asynchronized processes. An asychronized process allows crowd workers to work on the production and evaluation of individual turns without the constraints of timing or having a dialog partner. The outcome includes the document grounded dialogues as well as various types of implicit and explicit annotations that help ensure the quality of the data. Such data sets can be used to develop various types of dialogue agent technologies in the context of customer care. Our primary contributions can be summarized as follows: we introduce DOC2DIAL, an end-to-end framework 1 to generate task-oriented conversational data from business documents; we propose a novel pipeline approach of three interconnected tasks that minimizes the dialog flow crafting manual effort and enables comprehensive cross-task validation to ensure the quality of dialogue data; the system supports both synchronized and asynchronized dialogue collection processes. We demonstrate that such setting allows flexible dialogue composition by utterances, which are guided by the given document content and its annotations. Our work is mostly motivated by the success of research in training automatic agents with documents, as evidenced by the wide usage of data sets for machine comprehension tasks, e.g.. Two recent data tasks, CoQA and QuAC, aim to support conversational QA which involves understanding contexts and interconnected questions with multi-turn conversations. These data sets were created by pairing crowd workers to chat about a passage in the form of multi-turn questions and answers. Although getting closer to enabling automatic dialogue agents with documents, these data tasks do not tackle the understanding of preconditions, which is common in real-world task-oriented dialogues. In a recent paper, ShARC data task was proposed to address under-specified questions in conversational QAs by asking follow-up questions, which can be created by crowd workers based on supporting rules presented in a document. While it shares similar goals as our work on supporting dialogues that derive answers by understanding preconditions, it is a much simpler task focusing only on asking boolean follow-up questions. In contrast, our DOC2DIAL framework enables the end-to-end pipeline, from extracting complex precondition and solutions that present in business documents, to generating complete task-oriented dialogues that fit different scenarios. Figure 2 presents the overview of our framework. Next, we first introduce the main tasks, and then describe a bit more about the core components on processing the documents for generating the tasks. Main Tasks There are three tasks in our pipeline approach: TextAnno, as in Figure 3, is for labeling individual text spans as precondition, solution or otherwise in a selected document. The ed annotations can be used to automatically determine if a document is feasible for creating dynamic dialogues, thus enters the next steps in the pipeline. RelAnno as seen in Figure 4 is to identify the relations between precondition pairs labeled in TextAnno, such as hierarchical relations i.e. is-sibling, is-parent-of, as R-1 and R-2 in Figure 1. The task can also include text pairs to be labeled as is-precondition-of or is-solution-of for cross validating the annotations from TextAnno (e.g. R-3 in Figure 1).DialAnno, as shown in Figure 5, collects the dialog utterances based on the pre-generated dialog flow. Each turn correspond in the flow corresponds to a dialog scene, which is a composite of the interlocutor role (agent or user), a dialog action, a selected text span in the document based on the annotations from TextAnno and RelAnno, and dialogue history if applies. For instance, if the selected text is a precondition, the worker is given the role of an agent, the dialog action could be request/query/open. The worker is then given the instruction that You need to know whether the highlighted precondition applies to the customer in order to narrow down the solution. More examples of dialogue scenarios are given in Table 1. Document Processing prepares the input documents for the three annotation tasks. We first obtain various syntactic-semantic analysis ranging from computational linguistic statistics to HTML-based tree structures. For instance, we apply constituency parsing for splitting long sentences to text spans for TextAnno. We also extract sub-clauses with certain discourse connectives (e.g. "if", "as long as") via for identifying linguistic indicators of preconditions. In addition, we try to capture the outline patterns embedded in HTML tree structure in the documents. Documents that are well structured and clearly written with descriptive sub-titles and the discourse connectives are considered as good candidates for generating the dialogues with dynamic flows. The system is equipped with the capabilities to automatically assign labels based on the linguistic indicators mentioned above. We also employ heuristics based on the HTML tree structures and text proximity for the relation annotations. Such fuzzy labels are mainly for allowing generating and testing the data tasks without human labels. They can also be used as pseudo-gold labels for quality control for the crowdsourced tasks. Dialog Flow Composition generates the dialog flows for DialAnno from the labels of precondition/solution text and their relations obtained via TextAnno and RelAnno. For each turn, the dialog scene is determined by three factors, i.e., selected text span content, role and dialog act, which are determined sequentially. The dynamics of the dialog flows are introduced by varying the three aforementioned factors that are constrainted by the relations collected via RelAnnp. First, we randomly select content from a candidate pool of preconditions and solutions identified in the document, which is updated after every scenario generated. The general rule for updating the candidate pool is to eliminate preconditions/solutions that are already verified or eliminated. Then role is randomly selected between AGENT and USER. For our pilot study, we mainly consider dialog acts corresponding to the preconditions/solutions as selectively shown in As a pilot study, we demonstrate how to apply DOC2DIAL to various documents and task contexts. The input were internal customer care documents on topics such as technical trouble shooting, policy guidance, etc. We experimented with 1900 documents from the candidate document pool after the automatic filtering. With these documents, we obtained 26,000 text spans labeled as either precondition or solution via TextAnno and RelAnno. Then, we generated 5 dialog flows per document and selected those with more than 5 turns. Next, we evaluated the dialog flows by asking the crowd worker if the highlighted text matched the given dialogue scenario of a turn in a dialog flow. 67% of the turns were labeled as "match". Most of the mismatches were due to disagreement on the precondition/solution labeled by crowd workers in earlier steps. Figure 5 shows the sample task on collecting the 6-th turn of a dialog flow of 7 turns by providing the dialogue scenario and the chat history. The sample dialogues show that when the crowd contributors were able to understand the selected text in the context of the document, they could properly interpret the dialog scene to produce utterance or evaluate the task.

Represent the TLDR for finding the corresponding scientific article

We introduce Doc2Dial, an end-to-end framework for generating conversational data grounded in business documents via crowdsourcing for train automated dialogue agents

Represent the short summary for finding the corresponding scientific article (long).

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.