Question: Represent the article for finding a single-sentence TLDR.

Question: Recent pretrained transformer-based language models have set state-of-the-art performances on various NLP datasets. However, despite their great progress, they suffer from various structural and syntactic biases. In this work, we investigate the lexical overlap bias, e.g., the model classifies two sentences that have a high lexical overlap as entailing regardless of their underlying meaning. To improve the robustness, we enrich input sentences of the training data with their automatically detected predicate-argument structures. This enhanced representation allows the transformer-based models to learn different attention patterns by focusing on and recognizing the major semantically and syntactically important parts of the sentences. We evaluate our solution for the tasks of natural language inference and grounded commonsense inference using the BERT, RoBERTa, and XLNET models. We evaluate the models' understanding of syntactic variations, antonym relations, and named entities in the presence of lexical overlap. Our show that the incorporation of predicate-argument structures during fine-tuning considerably improves the robustness, e.g., about 20pp on discriminating different named entities, while it incurs no additional cost at the test time and does not require changing the model or the training procedure. Transformer-based language models like BERT , XLNET , and RoBERTa achieved stateof-the-art performances on various NLP datasets including those of natural language inference (NLI) , and grounded commonsense reasoning (GCI) . 1 Natural language inference is the task of determining whether the hypothesis entails, contradicts, or is neutral to the given premise. Grounded commonsense reasoning, as it is defined by the SWAG dataset , is the task of reasoning about what is happening and predict what might come next given a premise that is a partial description about a situation. Despite their great progress on individual datasets, pretrained language models suffer from various biases, including lexical overlap (b). For instance, given the premise "Neil Armstrong was the first man who landed on the Moon", the model may recognize the sentence "Moon was the first man who landed on the Neil Armstrong" as an entailing hypothesis or a plausible ending because it has a high lexical overlap with the premise. In this paper, we enhance the text of the input sentences of the training data, which is used for fine-tuning the pretrained language model on the target task, with automatically detected predicateargument structures. Predicate-argument structures identify who did what to whom for each sentence. The motivation of using predicate-argument structures is to provide a higher-level abstraction over different surface realizations of the same underlying meaning. As a , they can help the model to focus on the more important parts of the sentence and abstract away from the less relevant details. We show that adding this information during fine-tuning considerably improves the robustness of the examined models against various adversarial settings including those that evaluate models' understanding of syntactic variations, antonym relations, and named entities in the presence of high lexical overlap. Our solution imposes no additional cost over the linguistic-agnostic counterpart at the test time since it does not require predicateargument structures for the test data. Besides, compared to existing methods for handling the lexical overlap bias; ), it does not require introducing new models or training procedures and the model's complexity remains unchanged. The contributions of this work are as follows: 1. We provide three adversarial evaluation sets for the SWAG dataset to evaluate the lexical overlap bias. These adversarial test sets evaluate the model's understanding of syntactic variation, antonym relation, and named entities. The performance of all the examined models drops substantially on these datasets. We will release the datasets to encourage the community to develop models that better capture the semantics of the task instead of relying on surface features. 2. We propose a simple solution for improving the robustness against the lexical overlap bias by adding predicate-argument structures to the fine-tuning data. Our solution in no additional cost during the test time, it does not require oracle predicate-argument structures, and it also does not require any changes in the model or the training procedure. We will release the augmented training data for MultiNLI and SWAG training data. The findings of this work include: • While lexical overlap is a known bias for NLI, we show that models that are fine-tuned on SWAG are more prone to this bias. • The RoBERTa model performs the best on all adversarial test sets and is therefore more robust against the lexical overlap bias. • Among the examined evaluation settings, discriminating different named entities in the presence of high lexical overlap is the most challenging. The best accuracy, i.e., the accuracy of the RoBERTa-large model fine-tuned with augmented training data, is 59%. • Previous work showed that pretrained transformer-based language models capture various linguistic phenomena, e.g., POS tags, syntax, named entities, and predicate-argument structures, without explicit supervision . Yet, our work shows that explicit incorporation of such information is beneficial for improving robustness. Overcoming the Lexical Overlap Bias. This bias is investigated for NLI. The existing solutions for tackling this bias include using debiasing methods; ). All these models use a separate model to recognize the training examples that contain the bias. They then use various approaches to either not learn from biased examples or down-weight their importance during training. The ing improvements from these methods on the HANS dataset, using the BERT model, is on-par as those reported in this work. Our proposed solution, on the other hand, makes the model itself, e.g., BERT, more robust against the lexical overlap bias by learning better attention patterns and does not require a separate model for recognizing or skipping biased examples during training. The use of linguistic information in recent neural models is not very common. The use of such information has been mainly investigated for tasks in which there is a clear relation between the linguistic features and the target task. For instance, various neural models use syntactic information for the task of semantic role labeling (SRL) (; ; ;), which is closely related to syntactic relations, i.e., some arcs in the syntactic dependency tree can be mirrored in semantic dependency relations. build a graph representation from the input text using their corresponding dependency relations and use graph convolutional networks (GCNs) to process the ing graph for SRL. They show that the incorporation of syntactic relations improves the in-domain but decreases the out-of-domain performance. and incorporate linguistic information, i.e., coreference relations, in their model and show improvements in in-domain evaluations. use linguistic information, i.e., dependency parse, part-of-speech tags, and predicates for SRL using a transformer-based encoder . They make use of this linguistic information by using multi-task learning, and supervising the neural attention of the transformer model to predict syntactic dependencies. They use gold syntax information during training and predicted information during the test time. Their model substantially improves both indomain and out-of-domain performance in SRL. However, these are then outperformed by a simple BERT model without using any additional linguistic information . examine the use of various linguistic features, e.g., syntactic dependency relations and gender and number information, as additional input features to a neural coreference resolver. They show that using informative linguistic features substantially improves the generalization of the examined model. All the above approaches require additional linguistic information, e.g., syntax, both during the training and the test time. , on the other hand, only make use of the additional syntactic information during training. They use multi-task learning by considering syntax parsing as an auxiliary task and minimizing the combination of the losses of the main and auxiliary tasks. They use syntactic information for the tasks of SRL and coreference resolution. They show that this information slightly improves the in-domain performance. In this work, we do not change the loss function and only augment the input sentences of the training data. The advantage of our solution is that it does not require any changes in the model or its training objective. It can be applied to all the transformer-based models without changing the training procedure. Predicate-Argument Structures. Predicate-argument structures have been used for improving the performance of downstream tasks like machine translation , reading comprehension , and dialogue systems . However, these approaches are based on pre-neural models. The proposed model by for neural machine translation is a sample neural model that incorporates predicate-argument structures. Unlike this work, incorporate these linguistic structures at the model-level. They add two layers of semantic GCNs on top of a standard encoder, e.g., convolutional neural network or bidirectional LSTM. The Premise: A man in a black polo shirt is sitting in front of an electronic drum set. Correct ending: The tutorial starts by showing each part of the drum set up close. semantic structures are used for determining nodes and edges in the GCNs. In this work, however, we incorporate these structures at the input level, and only for the training data. Therefore, we can use the state-of-the-art models without any changes. Overall, this work differs from the related work because it evaluates the use of predicateargument structures for improving the robustness of state-of-the-art models for the tasks of NLI and GCI, and it uses these structures at the input level to extend raw inputs, it only employs this information during training, and it requires no changes in the model or the training procedure. 3 Experimental Setup 3.1 Tasks Grounded Commonsense Inference. Given a premise that is a partial description about a situation, GCI is the task of reasoning about what is happening and predicting what might come next. SWAG models this task as a multiple choice answer selection, in which the premise is given and the correct and three incorrect endings are presented as candidate answers. Figure 1 shows a sample premise and its correct ending from SWAG. Natural Language Inference. Given a premise and a hypothesis, NLI is the task of determining whether the hypothesis entails, contradicts, or is neutral to the premise. For instance, the hypothesis in Figure 2 entails the given premise. For the experiments of this paper, we use MultiNLI dataset, which is the largest available dataset for NLI. Premise: As spacecraft commander for Apollo XI, the first manned lunar landing mission, Armstrong was the first man to walk on the Moon. " That's one small step for a man, one giant leap for mankind." With these historic words, man's dream of the ages was fulfilled. Hypothesis: Neil Armstrong was the first man who landed on the Moon. Premise: A woman is packing a suitcase. Hypothesis: A suitcase is packing a woman. Premise: A lot of people are sitting on terraces in a big field and people is walking in the entrance of a big stadium. Ending: A lot of people are standing on terraces in a big field and people is walking in the entrance of a big stadium. In this section, we describe the adversarial datasets that we use to evaluate the robustness of the model against the lexical overlap bias. We created three different adversarial datasets based on the SWAG development set for evaluating the lexical overlap bias. These datasets evaluate the model's understanding of syntactic variations, antonym relations, and named entities in the presence of high lexical overlap. Syntactic Variations. In this evaluation set, premises which contain subject-verb-object structures are taken from the SWAG development set. We then construct a new negative ending by swapping the subject and object of the premise and replace one of the existing negative endings with the new one. This dataset includes 20 006 samples. 2 Figure 3 contains an example of this test set. Antonym Relations. In this test set, we create a new negative ending by replacing the first verb of the premise (from the SWAG development set) with its antonym. We use WordNet for the antonym relations. This adversarial setting is also common in NLI, e.g., . Figure 4 shows a sample premise and its corresponding incorrect ending that is created based on antonym relations. This set contains 7476 samples. Named Entities. In order to evaluate the capability of the examined models in discriminating different named entities, we create a new adversarial dataset in which a new incorrect ending is Premise: The reflection he sees is Harrison Ford as someone Solo winking back at him. Ending: The reflection he sees is Eve as someone Solo winking back at him. created by replacing one of the named entities of the premise with an unrelated named entity, i.e., "Eve". Figure 5 shows an example of this adversarial set. This test set contains 190 samples. We use the Stanford named entity recognizer for determining the named entities. For the adversarial evaluation of natural language inference, we use the Heuristic Analysis for NLI Systems (HANS) dataset (b). Sentence pairs in HANS include various forms of lexical overlap, which are created based on various syntactic variations, namely lexical overlap, subsequence, and constituent. In the lexical overlap subset, all words of the hypothesis appear in the premise. The subsequence subset contains hypotheses which are a contiguous subsequence of their corresponding premise. Finally, in the constituent subset, hypotheses are a complete subtree of the premise. Constituent is a special case of the subsequence heuristic, and they are both special cases of lexical overlap. Figure 6 includes an example for each of these three subsets. We incorporate predicate-argument structures of sentences by augmenting the raw text of each input sentence with its corresponding predicatearguments. This way, no change is required in the model's architecture. For the main experiments, we use the ProbBank-style semantic role labeling model of 3, which has the state-of-the-art on the CoNLL-2009 dataset, to get predicate-argument structures. We specify the beginning of the augmentation by the [PRD] special token that indicates that the next tokens are the detected predicate. We then specify the ARG0 and ARG1 arguments, if any, with [AG0] and [AG1] special tokens, respectively. The end of the detected predicate-argument structure is also specified by the [PRE] special token. If more than one predicate is detected for a sentence, they would all be added at the end of the input sentence. Figure 7 shows an example for an augmented sentence. For our experiments, we use BERT, XLNET, and RoBERTa. BERT is jointly trained on a masked language modeling task and a next sentence prediction task. It is pre-trained on the BookCorpus and English Wikipedia. XLNET is trained with a permutationbased language modeling objective for capturing bidirectional contexts. The XLNet-base model is trained with the same data as BERT-base. The RoBERTa model has the same architecture as BERT. However, it is trained with dynamic masking and without the next sentence prediction task. It is also trained using larger batchsize, vocabulary size, and training data. We use the Huggingface Transformers library 4, and we initialize the models with bert-base-uncased, roberta-base, and xlnet-base-cased, respectively. We finetune each of the above models on MultiNLI and SWAG training data for the NLI and GCI experiments, respectively. We report all the in two different settings including original: in which the model is fine-tuned on the original training data, and augmented: in which the input sentences of the bert_base_srl.jsonnet 4 https://github.com/huggingface/ transformers training data are extended with their corresponding predicate-argument structures. Except for the fine-tuning data, the other settings are exactly the same for both original and augmented experiments. Please note that we only augment the training data of the target task that is required for fine-tuning, and not the training data of the underlying language model. In this section, we evaluate the impact of augmenting the training data with predicate-argument structures for both in-domain and adversarial evaluations on the grounded commonsense reasoning and natural language inference tasks. Table 2 shows the of the examined models, based on both original and augmented settings, on the SWAG development set (in-domain) and the adversarial evaluation sets. From the of Table 2, we observe that: 1. While the examined models achieve humanlevel performance on in-domain evaluation, their performance drops drastically on the adversarial sets, e.g., below the random baseline on the antonym and named entities evaluation sets. This shows that all these models overly rely on shallow heuristics such as word overlap for predicting whether two sentences contain successive events. 2. Discriminating different named entities is the most challenging adversarial evaluation. 3. The augmentation of training data with predicate-argument structures slightly decreases the performance on in-domain evaluations. However, it significantly improves the robustness of all models in all three adversarial evaluation sets, i.e., from 8 to 22 points and on average by 13pp. 4. While the augmentation of the training data improves the performance on the examined adversarial evaluation sets, there is still room for improvement, i.e., the highest accuracy on the named entities adversarial set is 43.99. 5. RoBERTa has the highest performance in both original and augmented experiments and is, therefore, more robust against the lexical overlap bias. Table 1: Impact of data augmentation on the HANS dataset for the "entailment" and "non-entailment" labels. All models are fine-tuned on MultiNLI training data. The highest accuracy for each subset is boldfaced. Table 2: Accuracy of the examined models using the original vs. augmented SWAG training data. Table 3 and Table 1 show the of the examined models on the MultiNLI development set and HANS, respectively. The main challenge in HANS is the detection of non-entailment labels because, as a of the dataset creation artifacts in MultiNLI, most of the samples that have high lexical overlap are labeled as entailment. Therefore, models that are trained on this dataset tend to classify most of Table 3: Accuracy on MultiNLI development sets when the models are fine-tuned on original vs. augmented training data. such samples as entailment. Based on the , fine-tuning the models on the augmented data slightly (0.2pp-0.6pp) decreases the performance on the original dataset. The exception is the of the XLNET model on the match subset of MultiNLI in which the performance increases slightly. However, it improves the performance of BERT and XLNET models on the NLI adversarial subsets. The improvements are not as large as those of GCI, e.g., 5pp in NLI vs. 22pp in GCI. We hypothesize that models that are fine-tuned on the NLI dataset recognize predicateargument structures better than those trained on the SWAG dataset. As mentioned by McCoy et al. (2019a), the on the HANS dataset can vary by a large margin using different random seeds during fine-tuning, e.g., BERT accuracy on the lexical overlap subset Figure 8: BERT attention weights on an example from the HANS dataset based on original (top weights) and augmented (bottom weights) training. Attention weights are visualized using BertViz . They highlight the attention between the hypothesis and premise words and for the predicate-argument structures of the hypothesis. can vary from 6% to 54%. For reproducibility, all the in this paper are reported using the default random seed in Huggingface Transformers. We have evaluated the impact of data augmentation on the BERT model using different random seeds. The on the SWAG adversarial test sets do not change notably using different random seeds. However, using a different random seed, the improvement of augmented compared to original experiment on the HANS lexical overlap subset and for the non-entailment label can increase to 21.3pp. 5. Figure 8 shows the difference of the BERT attention weights, using BertViz 6 , on an example from the HANS dataset. In this example, the premise and hypothesis are "The senators supported the secretary in front of the doctor." and "The doctor supported the senators.", respectively. For instance, for the predicate "supported" in the hypothesis, the BERT model that is trained on augmented data (bottom subfigure), has high attention weights on "senators", "supported", and "secretary", while for original the attention weights of this predicate are more distributed. Similarly, for the subject "doctor" in the hypothesis, augmented mainly attends to the corresponding subject in the 5 The model is always trained using the same random seed for both original and augmented experiments 6 https://github.com/jessevig/bertviz premise, i.e., "senators". 5.1 Are predicate-argument useful for large models as well? The examined language models have two variations, base and large models. For instance, the RoBERTa-base model contains 125M parameters while RoBERTa-large contains 355M parameters. In this section, we examine whether the addition of predicate-argument structures still improves the robustness of large models. For the experiments of this section, we use RoBERTa and the GCI datasets. The are shown in Table 4. As we see from the , while RoBERTa-large model has higher performance on all GCI evaluation sets compared to RoBERTa-base, the addition of predicate-argument structures still considerably improves the performance on adversarial evaluation sets, i.e., 7pp-27pp. Table 4: Accuracy of RoBERTa-large on the GCI adversarial sets. In all the experiments of this paper, we use the predicted predicate-argument structures of a stateof-the-art semantic role labeling system, i.e.,. In this section, we examine the impact of the used SRL system to see how the ing errors in SRL impact the . Therefore, in this section we use OpenIE for augmenting the training data instead of the stateof-the-art SRL model. OpenIE is a less accurate but more efficient tool for extracting relations from the sentences. Table 5 report the of the RoBERTa-base model on GCI evaluation sets when the model is fine-tuned on the augmented data with relations that are extracted using OpenIE. As we see, the use of different models to extract predicate-argument structures does not considerably impact the ing robustness. Both augmentations considerably improve the robustness against the lexical overlap, even though one is less accurate than the other. Table 5: Accuracy of the RoBERTa-base model on the GCI adversarial sets when the training data is augmented using OpenIE. Based on our experiments, the addition of the predicate-argument structures to the test data does not have a benefit for pretrained transformer-based models. The reason is that the addition of predicateargument structures to the training data helps the transformer models to learn a different attention pattern. However, this is not possible for other neural models. Table 6 shows the of the ESIM model , when it is trained and tested on the original vs. augmented SWAG training data. As we see, the performance of the ESIM model is considerably lower than the pretrained language models on the adversarial datasets for the lexical overlap, and the addition of predicate-argument structures to the ESIM training and test data in an improvement in the robustness. However, the improvements are not as large as those of In this paper, we propose a solution to improve the robustness of the state-of-the-art NLP models, i.e., BERT, XLNET, and RoBERTa, against the lexical overlap bias. We improve the model robustness by extending the input sentences with their corresponding predicate-argument structures. The addition of these structures helps the transformer model to better recognize the major semantically and syntactically important parts of the sentences and learns more informative attention patterns accordingly. Our finding, regarding the benefit of explicit incorporation of predicate-argument structures, is despite the fact that transformer-based models already captures various linguistic phenomena, including predicate-argument structures . Our proposed solution in considerable improvements in the robustness, e.g., 20pp in accuracy, incurs no additional cost during the test time, does not require ant change in the model or the training procedure, and works with noisy predicate-argument structures. We evaluate the effectiveness of our solution on the task of natural language inference and grounded commonsense reasoning. However, since our solution only includes enhancing the training examples, it is not limited to a specific task and it is applicable to other tasks and datasets that suffer from this bias, e.g., paraphrase identification , and question answering . We will release the new adversarial evaluation sets for the lexical overlap bias as well as the augmented training data for MultiNLI ans SWAG datasets upon the publication.

Represent the TLDR for finding the corresponding scientific article.

Enhancing the robustness of pretrained transformer models against the lexical overlap bias by extending the input sentences of the training data with their corresponding predicate-argument structures

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the scientific article to find a short TLDR

Question: We propose a method for quantifying uncertainty in neural network regression models when the targets are real values on a $d$-dimensional simplex, such as probabilities. We show that each target can be modeled as a sample from a Dirichlet distribution, where the parameters of the Dirichlet are provided by the output of a neural network, and that the combined model can be trained using the gradient of the data likelihood. This approach provides interpretable predictions in the form of multidimensional distributions, rather than point estimates, from which one can obtain confidence intervals or quantify risk in decision making. Furthermore, we show that the same approach can be used to model targets in the form of empirical counts as samples from the Dirichlet-multinomial compound distribution. In experiments, we verify that our approach provides these benefits without harming the performance of the point estimate predictions on two diverse applications: distilling deep convolutional networks trained on CIFAR-100, and predicting the location of particle collisions in the XENON1T Dark Matter detector. Artificial neural networks are typically trained by maximizing the conditional likelihood of output targets given input features. Each target is modeled as a sample from a distribution p(y|x) parameterized by the output activity of the neural network, where the choice of parametric distribution is implied by the choice of objective function. Thus, the support of the probability distribution should match the target space, but in practice, this is often not the case. Today, the vast majority of neural network output layers implicitly model the targets as samples from one of four distributions: a binomial, a categorical, a Gaussian, or a Laplacian distributionrespectively corresponding to the binomial cross-entropy loss, multi-class cross-entropy loss, mean squared error, and mean absolute error. These distributions are commonly used even when the target space does not match the support, because the gradient calculations for these distributions are simple (and easy to compute) when paired with the appropriate output layer activation functions. These distributions dominate to such a degree that few alternatives are even available in most common deep learning software packages such as Keras BID3 and PyTorch BID15.Alternatives do exist -using neural networks to parameterize more complex distributions is not new. The standard regression approach can be generalized to a heteroskedastic Gaussian output layer BID14 BID18, where the neural network predicts both a mean and a variance for each target. Multi-model distributions can be modeled with a mixture density BID1. And more recently, the Gamma output layer was proposed to model targets in R >0 BID13. In principle, any parametric distribution with well-defined gradients could serve as a probabilistic prediction at the output of a neural network model. The approach proposed here is simpler than the one taken by Conditional Variational Autoencoders (CVAEs) BID10 BID16. While CVAEs can, in theory, model arbitrary high-dimensional conditional distributions, computing the exact conditional likelihood of a target requires marginalizing over intermediate representations, making exact gradient calculations intractable. Thus, training a CVAE requires approximating the gradients through sampling. In this work we show that restricting the output to a particular class of distributions, namely the Dirichlet or Dirichlet-multinomial compound distributions, enables a calculation of the exact likelihood of the targets and the exact gradients. Interpreting the output of a neural network classifier as a probability distribution has obvious benefits. One can derive different point estimates, define confidence intervals, or integrate over possible outcomes -a necessity for managing risk in decision making. Potentially, it could also lead to better learning -matching the output support to the target space essentially constrains the learning problem by incorporating outside knowledge. Allowing the network to output "uninformative" distributions -e.g. a uniform distribution over the support -could make training faster by allowing the network to focus on the easiest training examples first -a self-guided form of curriculum learning. In the present work, we derive gradients for the Beta distribution, Dirichlet distribution, and Dirichlet-multinomial compound distribution. We then propose activation functions that stabilize numerical optimization with stochastic gradient descent. Finally, we demonstrate through experiments that this approach can be used to model three common types of targets: targets over the multivariate simplex, real-valued scalar targets with lower and upper bounds, and nonnegative integer-valued counts (samples from the Dirichlet-multinomial compound distribution). The experiments demonstrate that our approach provides interpretable predictions with learned uncertainty, without decreasing the performance of the point estimates. Consider a supervised learning scenario, in which the goal is to model the relationship between a set of input, target pairs (x (t), y (t) ), where DISPLAYFORM0 We can construct a neural network that takes in x (t) and outputs a length-d vector DISPLAYFORM1, that parameterizes a Dirichlet distribution over ∆ d with probability density function DISPLAYFORM2 DISPLAYFORM3 where Γ is the Gamma function, which generalizes the factorial function to real values. Thus, given a set of neural network weights and some input x (t), we have a conditional density function over the domain of the target y (t). The network can be trained to minimize the negative log-likelihood (NLL) of the training set, − t ln p α (t) (y (t) ), using gradient descent. Dropping the example index t for clarity, we write the log-likelihood for a single example as DISPLAYFORM4 The gradient w.r.t. the network output α i is then DISPLAYFORM5 where ψ is the digamma function ψ(x) = ∂ ∂x ln(Γ(x)), and the gradient for a particular target is DISPLAYFORM6 Accurate numerical approximations of the digamma function are readily available, and obtaining point estimates from the network output is as simple as computing the mean of the Dirichlet distribution, DISPLAYFORM7, or the mode α − 1 DISPLAYFORM8 The Dirichlet distribution can also be parameterized by a vector µ ∈ ∆ d, together with a scalar γ > 0, where DISPLAYFORM9 DISPLAYFORM10 In this alternative parameterization, µ predicts the expectation of the Dirichlet, and the summation to one could be enforced using a softmax activation. This is conceptually similar to the heteroskedastic Gaussian model where the neural network computes the mean and standard deviation of a Gaussian distribution. The Dirichlet output layer can also be used to model vectors of non-negative integer targets y ∈ N d 0 as samples from the Dirichlet-multinomial compound distribution. This allows us to treat each target as a collection of related trials, conditioned on the same input. The distribution is parameterized by the Dirichlet parameters α and the number of multinomial trials n. The number of trials n can be fixed for each training example, fixed for the entire data set, or treated as a random variable sampled from a conditional probability distribution parameterized by an additional neural network output. Here we assume that n = i y i is given for each training example, so that the probability of target y is given by the following: DISPLAYFORM0 where we use the definition of the multivariate beta function to integrate over the simplex ∆ d and marginalize out z. This leads to the log-likelihood DISPLAYFORM1 and the gradient of the log-likelihood w.r.t. network output α i, DISPLAYFORM2 When the targets are real-valued scalars with lower and upper bounds, we can shift and rescale the values to be in the range and model the target as a sample from a Beta distribution. The Beta distribution is the d = 2 case of the Dirichlet, and can be used to predict univariate targets y ∈. It can be parameterized by a neural network that outputs two values α, β > 0, with DISPLAYFORM0 with log-likelihood DISPLAYFORM1 and gradients DISPLAYFORM2 DISPLAYFORM3 Alternatively, we can parameterize the Beta using two scalar network outputs: µ ∈ and γ > 0. DISPLAYFORM4 DISPLAYFORM5 In this parameterization, µ = α α+β = E[z] is the expectation of the distribution, and γ = α + β controls the width of the density function. Inverse-Linear DISPLAYFORM6 Figure 1: The choice of activation function in the output layer of a neural network affects the shape of the objective function. To stabilize learning in a Dirichlet output layer, we propose that the activation function f should approach 0 asymptotically as x → −∞, and that DISPLAYFORM7 )f (x) should be bounded. Two such functions are the Inverse-Linear (left) and Exponential-Linear (right) piecewise functions. In order to stabilize learning with stochastic gradient descent, the activation function should be chosen carefully to shape the objective function. For the models proposed here, we need an activation function with a strictly positive range, and that asymptotically approaches zero (or some minimum value > 0) as x → −∞. Moreover, the digamma terms in the gradient of the Dirichlet become large for small α i (lim x→0 + ψ(x) = −∞), so to avoid large gradients (which can destabilize learning), we propose two piecewise activation functions for which ∂ ∂x ψ(f (x)) is bounded: the Inverse-Linear (IL) and Exponential-Linear (EL) functions (Figure 1). The latter is simply a strictly-positive variant of the popular Exponential Linear Unit, or ELU BID4. DISPLAYFORM0 We also propose an activation function that saturates at hyper-parameter value τ > 1, the Exponential-Tanh (ET): DISPLAYFORM1 Each of these activation functions were tested in experiments. We found that in both parameterizations of the Dirichlet output, using the IL activation ed in a more stable learning trajectory than with the EL, presumably because 1 1−x approaches zero more slowly than e x. We also observed overflow errors when some α i became very small. We found that this could be avoided through weight regularization or by adding a small stability factor to the output activation function. Even with these activation functions, the increased flexibility of the models could lead to unexpected behavior. The model is able to concentrate probability mass at a particular target value, which could allow a network to devote its limited capacity to maximizing the likelihood of a single example. While this was not observed in our experiments, additional regularization and hyper-parameter tuning might be required for some applications. Figure 2: Example evolution of the probability density function for a simple neural network with a Beta output, trained to optimize the log-likelihood of a single data point with y = 0.1. The density functions are given by p θ1,θ2 (y) = y α−1 (1 − y) β−1 /B(α, β) where α = EL(θ 1) and β = EL(θ 2). Initializing θ 1 and θ 2 to zero gives a uniform distribution (dotted line). After 10 iterations of gradient descent updates to θ 1, θ 2, the probability mass accumulates in the left corner (dashed line), with α < 1. Then α increases and the mode of the distribution becomes non-zero when α > 0. At 1000 iterations, the mode is close to the target at y = 0.1 (solid line). As a simple illustration, consider a single data point y = 0.1 modeled as a sample from a Beta distribution parameterized by α = EL(θ 1) and β = EL(θ 2), with the exponential linear activation function EL defined as in Equation 19. We update θ 1 and θ 2 using gradient descent as if they were parameters of a very simple neural network. Figure 2 shows that this model quickly learns to concentrate probability mass at the target. For real-valued targets in a bounded interval, one might expect performance to improve when these bounds are incorporated into the model. We tested this idea on a simulated data set from the XENON1T dark matter detection experiment (b), where the task is to predict the x and y locations of individual particle collisions from detector data -essentially a video of the sensor activities over time. The location of each collision is bounded by the dimensions of the detector. The training data consists of 160,000 simulated collision events (a), while another 20,000 events are used for early stopping and validation. The detector has rotational symmetry isomorphic to the cyclic group of order 6, which is accounted for by randomly rotating each example during training. For each event, the neural network input consists of real-valued recordings from 248 detector elements for 1000 time steps, and a neural network predicts both the x and y locations of the event, each normalized to be in the range.A "typical" deep neural network regression model was constructed with an MSE objective and a linear output layer, where one output unit predicted the x-location, and another predicted the ylocation. The architecture and other hyperparameters were optimized using Sherpa BID6, including the number and shape of the layers, the learning rate, momentum, and dropout regularization. The best architecture used a two-column siamese network design to process the data from the top and bottom of the detector, with three convolutional layers of shape 300-300-10 followed by two dense layers of 2000-1000 in each column. (The kernel shape and step size were set to 1, as the sensors were not arranged in a grid.) These were followed by two dense layers of 300 units that combine all the information from the two columns. Layers were initialized from a scaled normal distribution BID5, and optimized using the Adam algorithm BID9 ) (learning rate = 0.0001, β 1 = 0.999, β 2 = 0.999, decay = 0.0001) on mini-batches of size 20. A dropout rate of 50% was used in the top 3 layers BID17 BID0. The selu activation BID11 was used in each hidden layer. Figure 3: Left: Example prediction from the XENON1T network with a Beta output layer. The target is at 0.95, and the Beta distribution predicted by the network has mean 0.95 and mode 0.96. Right: Validation set performance on the XENON1T regression task using a linear output layer, a heteroskedastic Gaussian output layer, a Beta output layer, or a Beta output layer with the alternative parameterization (Beta2). Rather than compare the NLL objective, we compare the more intuitive MSE loss, using the means of the Beta distributions as point estimates. This standard neural network regression model was then compared to three neural networks with more expressive output distributions: a heteroskedastic Gaussian, where the mean was predicted by neurons with linear activation, and the standard deviation was predicted by neurons with exponential activation; a Beta output layer (Equation 11) with exponential activation; and a Beta output layer with the alternative parameterization in Equation 17 where the µ units have a sigmoid activation and the γ units have a exponential activation. The heteroskedastic Gaussian and Beta layers not only provide more informative predictions in the form of distributions that can be used in the downstream analysis, but they also in better performance than the standard approach (Figure 3). This approach is also appropriate for tasks in which the targets are probabilities. In model compression, or network distillation BID2 BID7, a large model (or ensemble of models) is trained for a supervised learning task, and then the information learned is transferred to a separate, smaller model by training the small model to predict the probabilistic output, or "soft targets," of the large model. In many cases, the smaller "student" model will train faster and generalize better than if it had been trained on the actual "hard targets" from the training data set, because there is information, or "dark knowledge", contained in the imperfect predictions of the large model. When the "teacher" model has a sigmoid or softmax output, the targets of the student model will be probability values on a multi-dimensional simplex, which can be modelled as samples from a Dirichlet distribution. We tested this approach by first training a typical convolutional neural network on the CIFAR-100 BID12 ) benchmark data set to serve as a teacher model. The classification data set consisted of 60,000 32-by-32 RGB images from 100 classes, with 50,000 training examples and 10,000 test examples. We used the 18-layer convolutional network architecture from BID4, with the selu transfer function in the hidden layers BID11, a softmax output layer, constant dropout rates of (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.0) in each stack, batch size of 100, and the Adam optimizer (η = 0.0001, β 1 = 0.99, β 2 = 0.999). Training was stopped when the validation loss did not improve over 10 epochs. Training samples were augmented using horizontal flipping and random translations of up to 10% vertically or horizontally. Next, student neural networks were trained to predict the predictions of the teacher network. The student networks had the same architecture and optimization hyperparameters as the original network, except that no dropout regularization was used, and gradients were clipped to a maximum value of 100 for stability. We compared three different types of student network output layers: a standard softmax layer with categorical cross-entropy objective, a Dirichlet output consisting of. Rather than compare the NLL objective, we compare the more intuitive MSE loss, using the mean of the Dirichlet distributions as point estimates. Each network was also trained using an MSE objective (using the mean of the Dirichlet as a point-estimate); this had little effect on the softmax output, but hurt the final performance of the Dirichlet outputs., plus a stability factor of = 10 −6, and a Dirichlet with the alternative parameterization given in Equation 7, where the mean of the Dirichlet is specified by a softmax of size 100, and the scale is specified with a single IL unit. FIG3 compares the training trajectories of the networks, and shows that after training, the mean-value point estimate from the Dirichlet output layers have very similar MSE to the predictions of the softmax layer. We also include training trajectories for another three networks trained using the MSE objective on the mean-value point prediction from each distribution, rather than the NLL. This initially made training faster for the two versions of the Dirichlet output, but the final performance was not as good; for the softmax layer, using the MSE as the objective instead of the cross-entropy made little difference. To test the Dirichlet-multinomial output, we trained an autoencoder network to learn a 2-dimensional embedding of data simulated from high-dimensional, semi-sparse multinomials. This situation is encountered in metagenomics, where the goal is to understand the structure of microbial communities from mixed sequence reads .The data set was constructed by first parameterizing 10 clusters by sampling 10 times from a Dirichlet (d = 100, α =< 0.1, 0.1, . . ., 0.1 >), and then sampling 1,000 times from each cluster, with the number of trials n sampled from the uniform distribution U, for a total of 10,000 examples, with each 100-dimensional example consisting of a vector of non-negative counts. Training was performed on 80% of these examples, while 20% was used as a validation set. We trained a neural network autoencoder model consisting of three tanh hidden layers of shape 100-2-100, with the 2-dimensional layer serving as the low-dimensional bottleneck, and a Dirichlet-multinomial output layer. The network was trained for 100 epochs, using the Adam optimizer (η = 0.0001, β 1 = 0.99, β 2 = 0.999), batch size of 100, and L2 regularization in the hidden layer (0.0001). As shown in Figure 5, the model has no problem converging to an embedding in which the 10 clusters are clearly separated in the validation set. In most artificial neural network models, supervised learning corresponds to maximizing the NLL of the training set targets conditioned on the inputs. In this interpretation, each neural network prediction is a distribution over possible target values. While the vast majority of neural network classifiers in use today rely on a small set of distributions -the binomial distribution, the categorical distribution, the Gaussian distribution, or the Laplacian distribution -there are many situations for which none of these distributions are appropriate. Here we propose the use of the Beta distribution, Figure 5: A deep Dirichlet-multinomial autoencoder was used to learn a two-dimensional embedding of simulated samples from 100-dimensional multinomials. The 10 different clusters are readily apparent in the embedding of the validation set examples. The samples shown are colored by their true cluster identity. Dirichlet distribution, and the Dirichlet-multinomial compound distribution as outputs of neural networks. We show that a neural network can parameterize these distributions and the entire model can be trained using gradient descent on the NLL of the training data targets. This provides a particularly elegant approach to modelling certain types of network targets. The Beta and Dirichlet provide a better way to model targets that lie on a simplex, such as probabilities or realvalues that lie on a bounded interval, and the Dirichlet-multinomial enables us to model vectors of counts using the elegant mathematical properties of the Dirichlet. The predicted distributions have the correct support, so we can use them in decision making and for confidence intervals. Moreover, we have demonstrated through experiments that the expectation over the Dirichlet serves as a good point estimate, with a mean squared error that is similar to optimizing the MSE directly.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Neural network regression should use Dirichlet output distribution when targets are probabilities in order to quantify uncertainty of predictions.

Represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the article for finding a single-sentence TLDR:

Question: Deep neural networks have achieved outstanding performance in many real-world applications with the expense of huge computational resources. The DenseNet, one of the recently proposed neural network architecture, has achieved the state-of-the-art performance in many visual tasks. However, it has great redundancy due to the dense connections of the internal structure, which leads to high computational costs in training such dense networks. To address this issue, we design a reinforcement learning framework to search for efficient DenseNet architectures with layer-wise pruning (LWP) for different tasks, while retaining the original advantages of DenseNet, such as feature reuse, short paths, etc. In this framework, an agent evaluates the importance of each connection between any two block layers, and prunes the redundant connections. In addition, a novel reward-shaping trick is introduced to make DenseNet reach a better trade-off between accuracy and float point operations (FLOPs). Our experiments show that DenseNet with LWP is more compact and efficient than existing alternatives. Multi-scale DenseNet (a) and CondenseNet, have shown that 23 there exists high redundancy in DenseNet. Neural architecture search (NAS) has been successfully 24 applied to design model architectures for image classification and language models (; 25 ; ; a;). However, none of these 26 NAS methods are efficient for DenseNet due to the dense connectivity between layers. It is thus 27 interesting and important to develop an adaptive strategy for searching an on-demand neural network 28 structure for DenseNet such that it can satisfy both computational budget and inference accuracy 29 requirement. To this end, we propose a layer-wise pruning method for DenseNet based on reinforcement learning. Our scheme is that an agent learns to prune as many as possible weights and connections while 32 maintaining good accuracy on validation dataset. Our agent learns to output a sequence of actions 33 and receives reward according to the generated network structure on validation datasets. Additionally, 34 our agent automatically generates a curriculum of exploration, enabling effective pruning of neural 35 networks. Submitted to 32nd Conference on Neural Information Processing Systems (NIPS 2018). Do not distribute. Suppose the DenseNet has L layers, the controller needs to make K (equal to the number of layers in 38 dense blocks) decisions. For layer i, we specify the number of previous layers to be connected in the 39 range between 0 and n i (n i = i). All possible connections among the DenseNet constitute the action 40 space of the agent. However, the time complexity of traversing the action space is O(DISPLAYFORM0 which is NP-hard and unacceptable for DenseNet (b DISPLAYFORM1 where f denotes the controller parameterized with θ c . The j-th entry of the output vector o i, denoted by o ij ∈, represents the likelihood probability of the corresponding connection between the 53 i-th layer and the j-th layer being kept. The action a i ∈ {0, 1} ni is sampled from Bernoulli(o i). a ij = 1 means keeping the connection, otherwise dropping it. Finally, the probability distribution of 55 the whole neural network architecture is formed as: DISPLAYFORM0 The reward function is designed for each sample and not only considers the prediction correct or not, 57 but also encourages less computation: obtaining the feedback from the child network, we define the following expected reward: DISPLAYFORM1 DISPLAYFORM2 To maximize Eq and accelerate policy gradient training over θ c, we utilize the advantage actor-62 critic(A2C) with an estimation of state value function V (s; θ v) to derive the gradients of J(θ c) as: DISPLAYFORM3 The entire training procedure is divided into three stages: curriculum learning, joint training and 66 training from scratch and they are well defined in Appendix 4.1. Algorithm 1 shows the complete 67 recipe for layer-wise pruning. 3 Experiment and The on CIFAR are reported in which takes much search time complexity and needs more parameters but gets higher test error. We can also observe the on CIFAR-100 from the last t layers and keeps the policy of the remaining K − t layers consistent with the vanilla DenseNet. As t ≥ K, all block layers are involved in the decision making process. Training from scratch. After joint training, several child networks can be sampled from the policy 146 distribution π(a|s, θ c) and we select the child network with the highest reward to train from scratch, and thus better experimental have been produced. We summarize the entire process in Algorithm 1. The pseudo-code for layer-wise pruning. Input: Training dataset Dt; Validation dataset Dv; Pretrained DenseNet. Initialize the parameters θc of the LSTM controller and θv of the value network randomly. Set epochs for curriculum learning, joint training and training from scratch to M cl, M jt and M f s respectively and sample Z child networks. Output: The optimal child network 1: //Curriculum learning 2: DISPLAYFORM0 if t < K − t then 5: DISPLAYFORM1 end if 10:Sample a from Bernoulli(o) 11:DenseNet with policy makes predictions on the training dataset Dt 12:Calculate feedback R(a) with Eq 13:Update parameters θc and θv 14: end for 15: //Joint training 16: for t = 1 to M jt do 17:Simultaneously train DenseNet and the controller 18: end for 19: for t = 1 to Z do 20:Sample a child network from π(a|s, θc) 21:Execute the child network on the validation dataset Dv 22:Obtain feedback R (t) (a) with Eq (In this section, we argue that our proposed methods can learn more compact neural network architec- ture by analyzing the number of input channel in DenseNet layer and the connection dependency 152 between a convolution layer with its preceding layers. In FIG3 The input channel is 0 means this layer is dropped so that the block layers is reduced from 36 to FIG3 right, the x, y axis define the target layer t and source layer s. The small square at position (s, t) represents the connection dependency of target layer t on source that the values of these small square connecting the same target layer t are almost equal which means 167 the layer t almost has the same dependency on different preceding layers. Naturally, we can prove 168 that the child network learned from vanilla DenseNet is quite compact and efficient.

Represent the TLDR for finding the corresponding scientific article

Learning to Search Efficient DenseNet with Layer-wise Pruning

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

Propose first methods for exactly optimizing the softmax distribution using stochastic gradient with runtime independent on the number of classes or datapoints.


Question: Represent the next text

Question: We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world. We show that models trained to predict proprioceptive information about the agent's body come to represent objects in the external world. In spite of being trained with only internally available signals, these dynamic body models come to represent external objects through the necessity of predicting their effects on the agent's own body. That is, the model learns holistic persistent representations of objects in the world, even though the only training signals are body signals. Our dynamics model is able to successfully predict distributions over 132 sensor readings over 100 steps into the future and we demonstrate that even when the body is no longer in contact with an object, the latent variables of the dynamics model continue to represent its shape. We show that active data collection by maximizing the entropy of predictions about the body---touch sensors, proprioception and vestibular information---leads to learning of dynamic models that show superior performance when used for control. We also collect data from a real robotic hand and show that the same models can be used to answer questions about properties of objects in the real world. Videos with qualitative of our models are available at https://goo.gl/mZuqAV. Situation awareness is the perception of the elements in the environment within a volume of time and space, and the comprehension of their meaning, and the projection of their status in the near future. - As artificial intelligence moves off of the server and out into the world at large; be this the virtual world, in the form of simulated walkers, climbers and other creatures BID26, or the real world in the form of virtual assistants, self driving vehicles BID6, and household robots BID29; we are increasingly faced with the need to build systems that understand and reason about the world around them. When building systems like this it is natural to think of the physical world as breaking into two parts. The first part is the platform, the part we design and build, and therefore know quite a lot about; and the second part is everything else, which comprises all the strange and exciting situations that the platform might encounter. As designers, we have very little control over the external part of the world, and the variety of situations that might arise are too numerous to anticipate in advance. Additionally, while the state of the platform is readily accessible (e.g. through deployment of integrated sensors), the state of the external world is generally not available to the system. The platform hosts any sensors and actuators that are part of the system, and importantly it can be relied on to be the same across the wide variety situations where the system might be deployed. A virtual assistant can rely on having access to the camera and microphone on your smart phone, and the control system for a self driving car can assume it is controlling a specific make and model of vehicle, and that it has access to any specialized hardware installed by the manufacturer. These consistency assumptions hold regardless of what is happening in the external world. This same partitioning of the world occurs naturally for living creatures as well. As a human being your platform is your body; it maintains a constant size and shape throughout your life (or at least Figure 1 : Illustration of a preprogrammed grasp and release cycle of a single episode of the MPL hand. The target block is only perceivable to the agent through the constraints it imposes on the movement of the hand. Note that the shape of the object is correctly predicted even when the hand is not in contact with it. That is, the hand neural network sensory model has learned persistent representations of the external world, which enable it to be aware of object properties even when not touching the objects.these change vastly slower than the world around you), and you can hopefully rely on the fact that no matter what demands tomorrow might make of you, you will face them with the same number of fingers and toes. This story of partitioning the world into the self and the other, that exchange information through the body, suggests an approach to building models for reasoning about the world. If the body is a consistent vehicle through which an agent interacts with the world and proprioceptive and tactile senses live at the boundary of the body, then predictive models of these senses should in models that represent external objects, in order to accurately predict their future effects on the body. This is the approach we take in this paper. We consider two robotic hand bodies, one in simulation and one in reality. The hands are induced to grasp a variety of target objects (see Figure 1 for an example) and we build forward models of their proprioceptive signals. The target objects are perceivable only through the constraints they place on the movement of the body, and we show that this information is sufficient for the dynamics models to form holistic, persistent representations of the targets. We also show that we can use the learned dynamics models for planning, and that we can illicit behaviors from the planner that depend on external objects, in spite of those objects not being included in the observations directly (see Figure 7).Our simulated body is a model of the hand of the Johns Hopkins Modular Prosthetic Limb BID30, realized in MuJoCo BID58. The model is actuated by 13 motors each capable of exerting a bidirectional force on a single joint. The model is also instrumented with a series of sensors measuring angles and torques of the joints, as well as pressure sensors measuring contact forces at several locations across its surface. There are also inertial measurement units located at the end of each finger which measure translational and rotational accelerations. In total there are 132 sensor measurements whose values we predict using our dynamics model. Our real body is the Shadow Dexterous Hand, which is a real robotic hand with 20 degree of freedom control. This allows us to show that that our ideas apply not only in simulation, but succeed in the real world as well. The Shadow Hand is instrumented with sensors measuring the tension of the tendons driving the fingers, and also has pressure sensors on the pad of each fingertip that measure contact forces with objects in the world. We apply the same techniques used on the simulated model to data collected from this real platform and use the ing model to make predictions about states of external objects in the real world. Intrinsic motivation and exploration: Given our goal to gather information about the world and, and in particular to actively seek out information about external objects, our work is naturally related to work on intrinsic motivation. The literature on intrinsic motivation is vast and rich, and we do not attempt to review it fully here. Some representative works include BID42 BID61; BID53; BID4; BID39 BID49; BID40; BID25 a) Some of the ideas here, in particular the notion of choosing actions specifically to improve a model of the world, echo earlier speculative work of BID48 and BID54.Several authors have implemented intrinsic motivation, or curiosity based objectives in visual space, through predicting interactions with objects , or through predicting summary statistics of the future BID59 BID15. Other authors have also investigated using learned future predictions directly for control BID14.Many works formulate intrinsic motivation as a problem of learning to induce errors in a forward model, possibly regularized by an additional inverse model BID44 de ). However, since we use planning, rather than policies, for active control we cannot adapt their objectives directly. Objectives that depend on the observed error in a prediction cannot be rolled forward in time, and thus we are forced to work with similar, but different objectives in our planner. When stochastic transition and observation models are available, it is possible to use simulation to infer optimal plans for exploring environments BID38. Our setting uses predominantly deterministic distributed representations, and our models are learned from data. A sea of other methods have been proposed for exploration BID37 BID23 BID2 BID21 BID52 BID47 BID20. Our approach builds on this literature. Haptics: Humans use their hands to gather information in structured task driven ways BID34; and it will become clear from the experiments why hands are relevant to our work. Our interest in hands and touch brings us into contact with a vast literature on haptics BID63 BID22 BID9 BID36 BID16 BID56 BID41 BID0 BID57 BID11 BID31 BID55.There is also work in robotics on using the anticipation of sensation to guide actions BID28, and on showing how touch sensing can improve the performance of grasping tasks BID8. Model based planning has been very successful in these domains BID13.Sequence-to-sequence modelling: There has been a lot of recent interest in sequence-to-sequence modelling BID15 BID59 BID10 BID18 BID3 BID1 BID33, particularly in the context of predicting distributions and in dynamics modelling in reinforcement learning. In this paper we use a sequence to sequence variant that shares weights between the encoder and decoder portions of the model. The consciousness prior BID5 considers recurrent latent dynamics models similar to ours and suggests mapping from their hidden states to other spaces that aren't directly modelled. BID62 propose a method of learning control policies that operate under unknown dynamics models. They consider the dynamics model parameters as an unobserved part of the state, and train a system identification model to predict these parameters from a short history of observations. The predictions of the system identification model are used to augment the observations which are then fed to a universal policy, which has been trained to act optimally under an ensemble of dynamics models, when the dynamics parameters are observed. The key contribution of their work is a training procedure that makes this two stage modelling process robust. Although the high level motivation of BID62 is similar, there is an obvious analogy between their system identification model and our diagnostics, many of the specifics are quite different from the work presented here. They explicitly do not consider memory-based tasks (the system identification model looks only at a short window of the past) whereas one of our key interests is in how our models preserve information in time. They also use a two stage training process, where both stages of training require knowledge of the system parameters; it is only at test time where these are... unknown. In contrast, we use the system parameters only as an analysis strategy, and at no point require knowledge of them to train the system. Finally, the bodies we consider (robot hands) are substantially more complex than those of BID62, and we do not make use of an explicit parameterization of the system dynamics. The work of BID19 also fits dynamics models using neural networks and uses planning in these models to guide action selection. They train a global dynamics model on data from several tasks, and use this global model as a prior for fitting a much simpler local dynamics model within each episode. The global model captures course grained dynamics of the robot and its environment, while the local model accounts for the specific configuration of the environment within an episode. Although they do not probe for this explicitly, one might hypothesize that the type of awareness of the environment that we are after in this work could be encoded in the parameters of their local models. We consider an agent operating in a discrete-time setting where there is a stochastic unobservable global state s t ∈ S at each timestep t and the agent obtains a stochastic observation x t ∈ X where X ⊆ S and takes some action u t ∈ U. Our goal is to learn a predictive dynamics model of the agent's action-conditional future observations p(x t+1:t+k |u 1:t+k−1, x 1:t) for k timesteps into the future given all of the previous observations and actions it has taken. We assume that the dynamics model has some hidden state it uses to encode information in the observed trajectory. We will then use these models to reason about the global state s t even though no information about this state is available during training, which we refer to as awareness. FIG0 summarizes the notation and definitions we use throughout the rest of the paper. To show that information required for reasoning is present in the states of our dynamics models we use auxiliary models, which we call diagnostic models. A diagnostic model looks at the states of a dynamics model and uses them to to predict an interpretable unobserved state in the world y t ∈ Y, where Y ⊆ S and in most cases X ∩ Y = ∅. When training a diagnostic model we allow ourselves to use privileged information to define the loss, but we do not allow the diagnostic loss to influence the representations of the dynamics model. The diagnostic models are a post-hoc analysis strategy. The dynamics models are trained using only the observed states, and then frozen. After the dynamics models are trained we train diagnostic models on their states, and the claim is that if we can successfully predict properties of unobserved states using diagnostic models trained in this way then information about the external objects is available in the states of the dynamics model. This section introduces the Predictor-Corrector (PreCo) dynamics model we use for long-horizon multi-step predictions over the observation space p(x t+1:t+k |u 1:t+k−1, x 1:t). We first encode the observed trajectory {u 1:t, x 1:t} into a deterministic hidden state h t ∈ H using a recurrent model DISPLAYFORM0... DISPLAYFORM1... DISPLAYFORM2... parameterized by θ and then use this hidden state to predict distributions over the future observations x t+1:t+k. We show experimentally that even though the hidden states h t were only trained on observed states, they contain an awareness of unobserved states in the environment. Using a deterministic hidden state allows us to easily unroll the predictor without needing to approximate the distributions with sampling or other approximate methods. We assume that the observation predictions are independent of each other given the hidden state, and can be modeled as DISPLAYFORM3 This modelling is done with three deterministic components: 1. Predictor θ: H × U → H predicts the next hidden state after taking an action, 2. Corrector θ: H × X → H corrects the current hidden state after receiving an observation from the environment, and 3. Decoder θ: H → P X maps from the hidden state to a distribution over the observations. Separating the dynamics model into predictor and corrector components allows us to operate in single-step and multi-step prediction modes as FIG1 shows. The predictor can make action-conditional predictions using the hidden states from the corrector for single-step predictions as h p t,0 = Predictor θ (h c t−1, u t) or from itself for multi-step predictions as h p t,i+1 = Predictor θ (h p t,i, u t+i). In our notation, h p t,i denotes the predictor's hidden state prediction at time t + i starting from the corrector's state at time t − 1. The corrector then makes the updates h DISPLAYFORM4 To train PreCo models, we maximize the likelihood on a reference set of trajectories using the singlestep predictions as well as multi-step predictions stemming from every timestep. The structure of the ing graph of only the hidden states is shown on the right of FIG1, omitting the observed states, trajectories, and predicted distributions. We call this technique overshooting, and we call the number of steps predicted forward by the decoder the overshooting length. The predictor and corrector components use single layer LSTM cores. We embed the inputs with a separate embedding MLP for the controls and sensors. We predict independent mixtures of Gaus-sians at every step with a mixture density network . Each dimension of each prediction is an independent mixture. We use separate MLPs to produce the means, standard deviations and mixture weights. We use Adam for parameter optimization. Model predictive control (MPC), the strategy of controlling a system by repeatedly solving a modelbased optimization problem in a receding horizon fashion, is a powerful control technique when a dynamics model is known. Throughout this paper, we use MPC to achieve objectives based on predictions from our dynamics models. Formally, MPC requires that at each timestep after receiving an observation and correcting the hidden state, we solve the optimization problem DISPLAYFORM0 where the timesteps in this problem are offset from the actual timestep in the real system, the initial hidden state h init is from the most recent corrector's state, and the remaining hidden states are unrolled from the predictor. In our experiments we also add constraints to the actions U 1:T so that they lie in a box ||u t || ∞ ≤ 1 and we enforce slew rate constraints ||u t+1 − u t || ∞ ≤ 0.1. After solving this problem, we execute the first returned control u 1 on the real system, step forward in time, and repeat the process. This formulation allows us to express standard objectives defined over the observation space by using the decoder to map from the hidden state to a distribution over observations at each timestep. We can also use other learned models, such as diagnostic models, to map from the hidden state to other unobservable quantities in the world. Our MPC solver for uses a shooting method with a modified version of Adam BID32 to iteratively find an optimal control sequence from some initial hidden state. At every iteration, we unroll the predictor, compute the objective at each timestep, and use automatic differentiation to compute the gradient of the objective with respect to the control sequence. To handle control constraints, we project onto a feasible set after each Adam iteration. During an episode, we warm-start the nominal control and hidden state sequence to the appropriately time-shifted control and hidden state sequence from the previous optimal solution. Learning dynamics models such as the PreCo model in Section 4 requires a collection of trajectories. In this section, we discuss two ways of collecting trajectories for training dynamics models: passive collection does not use any input from the dynamics model while active collection seeks to actively improve the dynamics model. The simplest data collection strategy is to hand design a behavior that is independent of the dynamics model. Such strategies can be completely open loop, for example taking random actions driven by a noise process, and also encompass closed loop policies such as following a pre-programmed nominal trajectory. In these situations, we are only interested in learning a dynamics model to achieve an awareness of the unobserved states, not to make policy improvements. We use the term passive collection to emphasize that the the data collection behavior does not depend on the model being trained. Once collected, the maximum likelihood PreCo training procedure described in Section 4 can be used to fit the PreCo dynamics model to the trajectories, but there is no feedback between the state of the dynamics model and the data collection behavior. Beyond passive collection we can consider using using the dynamics model to guide exploration towards parts of the state space where the the model is poor. We call this process active collection to emphasize that the model being trained is also being used to guide the data collection process. This section describes the method of active collection we use in the experiments. In this paper, we consider environments that are entirely deterministic, except for a stochastic initial unobserved state that the observed state can gather information about. When our dynamics model over the observed state makes uncertain predictions, the source of that uncertainty stems from one of two places: the model is poor, as a consequence of there being too little data or from too small capacity, or properties of the external objects are not yet resolved by the observations seen so far. Our active exploration exploits this fact by choosing actions to maximize the uncertainty in the rollout predictions. An agent using this uncertainty maximization policy attempts to seek actions for which the outcome is not yet known. This uncertainty can then be resolved by executing these actions and observing their outcome, and the ing trajectory of observations, actions, and sensations can be used to refine the model. To choose actions to gather information we use MPC as described in Section 5 over an objective that maximizes the uncertainty in the predictions. Our predictions are Mixtures of Gaussians at each timestep, and the uncertainty over these distributions can be expressed in many ways. We use the Rényi entropy of our model predictions as our measure of uncertainty because it can be easily computed in closed form. Concretely, for a single Mixture of Gaussians prediction f (x) we can write DISPLAYFORM0 where i and j index the mixture components in the likelihood. A more complete derivation is shown in Appendix A, which extends the of to the case when the mixture components have different variances. We obtain an information seeking objective by summing the entropy of the predictions across observations and across time, which is expressed as the cost function in MPC as C(h t, u t) = − fi H 2 (f i) where, through a slight abuse of notation, f i ∈ Decoder θ (h t) is a distribution over the observation dimension i. We implement this information gathering policy to collect training data for the model in which it is planning. In our implementation these are two processes running in parallel: we have several actors each with a copy of the current model weights. These use MPC to plan and execute a trajectory of actions that maximizes the model's predicted uncertainty over a fixed horizon trajectory into the future. The observations and actions generated by the actors are collected into a large shared buffer and stored for the learner. While the actors are collecting data, a single learner process samples batches of the collected trajectories from the buffer being written to by the actors. The learner trains the PreCo model by maximum likelihood as described in Section 4, and the updated model propagates back to the actors who continue to plan using the updated model. We implemented this using the framework of BID27. Our simulated environment consists of a hand with a random object placed underneath of it in each episode. The observation state space consists of sensor readings from the hand, and the unobserved state space consists of properties of the object. Probability that baseline is better than dynamics model software and is available for download from the MuJoCo website.1 The hand is actuated by 13 motors and has sensors that provide a 132 dimensional observation, which we describe in more detail in Appendix C.In each episode the hand starts suspended above the table with its palm facing downwards. A random geometric object that we call the "target" is placed on the table, and the hand is free to move to grasp or manipulate the object. The shape of the target is randomly chosen in each episode to be a box, cylinder or ellipsoid and the size and orientation of the target are randomly chosen from reasonable ranges. Figures 1 and 7 show renderings of the environment. We begin by exploring awareness in the passive setting, as a pure supervised learning problem. We manually design a policy for the hand, which executes a simple grasping motion that closes the hand about the target it and then releases it. We generate data from the environment by running this graspand-release cycle three times for each episode. Using the dataset generated by the grasping policy, we train a PreCo model described in Section 4. The full set of hyperparameters for this model can be found in Appendix D.We evaluate the awareness of our model by measuring our ability to predict the shape of the target at each timestep. We are especially interested in the predictions at timesteps where the hand is not in direct contact with the target, since these are the points that allow us to measure the persistence of information in the dynamics model. We expect that even a naïve model should have enough information to identify the target shape at the peak of a grasp, but our model should do a better job of preserving that information once contact has been lost. Recall from Section 3 that the diagnostic model we use to predict the target shape is trained in a second phase after the dynamics model is fully trained. The identity of the target shape is used when training the diagnostic model, but is not available when training the dynamics model, and the Reward obtained by planning to achieve the max fingertip objective using dynamics models trained with different data collection strategies. Right: A frame from a planned max fingertip trajectory.training of the diagnostic does not modify the learned dynamics model, meaning that no information from the diagnostic loss is able to leak into the states of the dynamics model. FIG3 shows the of this experiment. We compare the diagnostic predictions trained on the features of our dynamics model to three different baselines that do not use the dynamics model features.1. The MLP baseline uses an MLP trained to directly classify the target from the sensor readings of the hand, ignoring all dependencies between timesteps within an episode. We expect this baseline to give a lower bound on performance of the diagnostic. This is an important baseline to have since as the hand opens there is still residual information about the shape of the target in the position of joints, which is identified by this baseline.2. The LSTM baseline uses an LSTM trained to directly classify the target from the sensor readings of the hand, taking full account of dependencies between timesteps within an episode. Since this baseline has access to the target class at training time, and is also able to take advantage of the temporal dependence within each episode, we expect it to give an upper bound on performance of the diagnostic model, which only has access to the states of the pre-trained PreCo model. RandLSTM baseline finds a middle ground between the MLP and LSTM models. We use the same architecture and training procedure as for the LSTM baseline, but we do not train the input or recurrent weights of the model. By comparing the performance of this baseline to the diagnostic model we can see that our success cannot be attributed merely to the existence of an arbitrary temporal dependence. The in FIG3 show that the dynamics PreCo model reliably preserves information about the identity of the target over time, even though this information is not available to the model directly either in the input or in the training loss. In this section we explore how different data collection strategies lead to models of different quality. We evaluate the quality of the trained models by using them in an MPC planner to execute a simple diagnostic control task. The diagnostic task we use in this section is to maximize the total pressure on the fingertip sensors on the hand. This is a good task to evaluate these models because the most straightforward way to maximize pressure on the fingers is to squeeze the target block, and demonstrating that we can achieve this objective through MPC shows that the models are able to anticipate the presence of the block, and reason about its effect on the body. Note that because we act through planning, the implication for the representations of the model are stronger than they would be if we trained a policy to achieve the same objective. A policy need only learn that when the hand is open it should be closed to reach reward. In contrast, a model must learn that closing the hand will lead the fingers to encounter contacts, and it is only later that this prediction is turned into a reward for evaluation. We compare several different data collection policies, and their performances on the diagnostic task are shown in FIG4.1. The IndNoise policy collects data by executing random actions sampled from a Normal distribution with standard deviation of 0.2.2. The CorNoise policy collects data by executing random actions sampled from a Ornstein Uhlenbeck process with damping of 0.2, driven by an independent normal noise source with standard deviation 0.2. Each of the 13 actions is sampled from an independent process, with correlation happening only over time.3. The AxEnt policy uses the MPC planner described in Section 5 to maximize the total entropy of the model predictions over a horizon of 100 steps. The objective for the planner is to maximize the Rényi entropy, as described in Section 6.2.4. The AxTask policy also uses the MPC planner of Section 5 to collect data, but here the planning objective for data collection is the same as for evaluation. For both of the planning policies we found that adding correlated noise (using the same parameters as the CorNoise policy) to the actions chosen by the planner lead to much better models. Without this source of noise the planners do not generate enough variety in the episodes and the models underperform. We evaluate each model by running several episodes where we plan to achieve maximum fingertip pressure, and show the ing rewards in FIG4. Note that the evaluation objective is different than the training objective for all models except AxTask. We do not add additional noise to planned actions when running the evaluation. We also evaluate the awareness of the AxEnt model using the shape diagnostic task from Section 7.2. FIG5 compares the performance of a diagnostic trained on the AxEnt dynamics model to the passive awareness diagnostic of Section 7.2. The model trained with actively collected data outperforms its passive counterpart in regions of the grasp trajectory where the hand is not in contact with the block. In this section we present qualitative of using a the AxEnt model to execute different objectives through planning. We do this with MPC as described in Section 5.1. Maximizing entropy of the predictions, as we did during training, leads to exploratory behavior. In Figure 7 we show a typical frame from an entropy maximizing trajectory, as well as typical frames from controlling for two different objectives.2. Optimizing for fingertip pressure tends to lead to grasping behavior, since the easiest way to achieve pressure on the fingertips is to push them against the target block. There is an alternative solution which is often found where the hand makes a tight fist, pushing its fingertips into its own palm. This is the same as the diagnostic task used in the previous section. Figure 7: Examples of the hand behaving to maximize uncertainty about the future (top) or minimize uncertainty (bottom). When the hand is trained to maximize uncertainty it engages in playful behavior with the object. The body models learned with this objective, can then be re-used with novel objectives, such as minimizing uncertainty. When doing so, we see that the hand avoids contact so as to minimize uncertainty about future proprioceptive and haptic predictions.3. Minimizing entropy of the predictions is also quite interesting. This is the negation of the information gathering objective, and it attempts to make future observations as uninformative as possible. Optimizing for this objective in behavior where the hand consistently pulls away from the target object. Qualitative from executing each of the above policies are shown in FIG4. The behavior when minimizing entropy of the predictions is particularly relevant. The ing behavior causes the hand to pull away from the target object, demonstrating that the model is aware not only of how to interact with the target, but also how to avoid doing so. Videos of the model in action are available online at https://goo.gl/mZuqAV. We have shown that our models work well in simulation. We now turn to demonstrating that they are effective in reality as well. We use the 24-joint Shadow Dexterous Hand 2 with 20-DOF tendon position control and set up a real life analog of our simulated environment, as shown in Figure 8. Since varying the spatial extents of an object in real life would be very labor intensive we instead use a single object fixed to a turntable that can rotate to any one of 255 orientations, and our diagnostic task in this environment is to recover the orientation of the grasped object. We built a turntable mechanism for orienting the object beneath the hand, and design some randomized grasp trajectories for the hand to close around the block. The object is a soft foam wedge (the shape is chosen to have an unambiguous orientation) and fixed to the turntable. At each episode we turn the table to a randomly chosen orientation and execute two grasp release cycles with the hand robot. Over the course of two days we collected 1140 grasp trajectories in three sessions of 47, 393 and 700 trajectories. We use the 47 trajectories from the initial session as test data, and use the remaining 1093 trajectories for training. Each trajectory is 81 frames long and consists of two grasp-release cycles with the target object at a fixed orientation. At each timestep we measure four different proprioceptive features from the robot:1. The actions, a set of 20 desired joint positions, sent to the robot for the current timestep. arately actuated, and the measured angles may not match the intended actions due to force limits imposed by the low level controller.3. The efforts, which provide 20 distinct torque readings. Each effort measurement is the signed difference in tension between tendons on the inside and outside of one of the actuated joints.4. The pressures are five scalar measurements that indicate the pressure experienced by the pads on the end of each finger. Joint ranges of the hand are limited to prevent fingers pushing each other, and the actuator strengths are limited for the safety of the robot and the apparatus. At each grasp-release cycle final grasped and released positions are sampled from handcrafted distributions. Position targets sent to the robot are calculated by interpolating between these two positions in 20 steps. There are multiple complexities the sensor model needs to deal with. First of all once a finger touches the object actual positions and target positions do not match, and the foam object bends and deforms. Also the hand can occasionally overcome the resistance in the turntable motor causing the target object to rotate during the episode (for about 10-20 degrees and rarely more). This creates extra unrecorded source of error in the data. We train a forward model on the collected data, and then treat prediction of the orientation of the block as a diagnostic task. Figure 8 shows that we can successfully predict the orientation of the block from the dynamics model state. In this paper we showed that learning a forward predictive model of proprioception we obtain models that can be used to answer questions and reason about objects in the external world. We demonstrated this in simulation with a series of diagnostic tasks where we use the model features to identify properties of external objects, and also with a control task where we show that we can plan in the model to achieve objectives that were not seen during training. We also showed that the same principles we applied to our simulated models are also successful in reality. We collected data from a real robotic platform and used the same modelling techniques to predict the orientation of a grasped block. A DERIVING THE RÉNYI ENTROPY OF A MIXTURE OF GAUSSIANS DISPLAYFORM0 where the last step can be computed with Mathematica, and is also given in: DISPLAYFORM1 Figures 9 and 10 show planned trajectories and model predictions when attempting to maximize fingertip pressure and to minimize predicted entropy, respectively. The MPL hand is actuated by 13 motors each capable of exerting a bidirectional force on a single degree of freedom of the hand model. Each finger is actuated by a motor that applies torque to the MCP joint, and the MCP joint of each finger is coupled by a tendon to the PIP and DIP joints of the same finger, causing a single action to flex all joints of the finger together. Abduction of the main digits (ABD) is controlled by two motors attached to the outside of the index and pinky fingers, respectively. Unlike the main digits, the thumb is fully actuated, with separate motors driving each joint. The thumb has its own abduction joint, and somewhat strangely the thumb is composed of three jointed segments (unlike a human thumb which has only two). Each segment is separately controlled for a total of four actuators controlling the thumb. Finally the hand is attached to the world by fully actuated three three degree of freedom wrist joint, for a total of 13 actuators. The hand model includes several sensors which we use as proprioceptive information. We observe the position and velocity of each joint in the model (three joints in the wrist and four in each finger except the middle which has no abduction joint, for a total of 22 joints), as well as the position, velocity and force of each of the 13 actuators. We also record from inertial measurement units (IMUs) located in the distal segment of each of the five fingers. Each IMU records three axis rotational and translational acceleration for a total of 30 acceleration measurements. Finally there are 19 pressure sensors placed throughout the inside of the hand that measure the magnitude of contact forces. Each finger including the thumb has three touch sensors, one on each segment (recall that the thumb has three segments in this model), and the palm of the hand has four different touch D HYPERPARAMETERS Table 1 shows hyperparameters for several of the models used in the experiments. Some of the hyperparameters (notably the Adam learning rates) are found through random search, so the numbers are quite particular, but the particularity should not be taken as a sign of delicacy. The meaning of each parameter is shown in FIG9. We do not count the output layer or the input layer in the depth parameter (so a depth of 0 is a single linear transform followed by an activation function). The output layers of the MLP parts of the model are all indicated separately in the diagrams. The three pieces shown here are attached together in various ways, as shown in FIG1 in the main body of the paper.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We train predictive models on proprioceptive information and show they represent properties of external objects.

Represent the TLDR for finding the corresponding scientific article

StrokeNet is a novel architecture where the agent is trained to draw by strokes on a differentiable simulation of the environment, which could effectively exploit the power of back-propagation.