Question: Represent the scientific article to find a short TLDR!

Question: To simultaneously capture syntax and semantics from a text corpus, we propose a new larger-context language model that extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond a conventional language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, we develop a hybrid of stochastic-gradient MCMC and recurrent autoencoding variational Bayes. Experimental on a variety of real-world text corpora demonstrate that the proposed model not only outperforms state-of-the-art larger-context language models, but also learns interpretable recurrent multilayer topics and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent. Both topic and language models are widely used for text analysis. Topic models, such as latent Dirichlet allocation (LDA) (; ;) and its nonparametric Bayesian generalizations , are well suited to extract document-level word concurrence patterns into latent topics from a text corpus. Their modeling power has been further enhanced by introducing multilayer deep representation (; ; ; ; ; . While having semantically meaningful latent representation, they typically treat each document as a bag of words (BoW), ignoring word order . Language models have become key components of various natural language processing (NLP) tasks, such as text summarization , speech recognition , machine translation , and image captioning (; ; ; ;). The primary purpose of a language model is to capture the distribution of a word sequence, commonly with a recurrent neural network (RNN) or a Transformer based neural network (; ; ; ; 2019). In this paper, we focus on improving RNN-based language models that often have much fewer parameters and are easier to perform end-to-end training. While RNN-based language models do not ignore word order, they often assume that the sentences of a document are independent to each other. This simplifies the modeling task to independently assigning probabilities to individual sentences, ignoring their orders and document context . Such language models may consequently fail to capture the long-range dependencies and global semantic meaning of a document (; . To relax the sentence independence assumption in language modeling, propose larger-context language models that model the context of a sentence by representing its preceding sentences as either a single or a sequence of BoW vectors, which are then fed directly into the sentence modeling RNN. An alternative approach attracting significant recent interest is leveraging topic models to improve RNN-based language models. use pre-trained topic model features as an additional input to the RNN hidden states and/or output. ; combine the predicted word distributions, given by both a topic model and a language model, under variational autoencoder . introduce an attention based convolutional neural network to extract semantic topics, which are used to extend the RNN cell. learn the global semantic coherence of a document via a neural topic model and use the learned latent topics to build a mixture-of-experts language model. further specify a Gaussian mixture model as the prior of the latent code in variational autoencoder, where each mixture component corresponds to a topic. While clearly improving the performance of the end task, these existing topic-guided methods still have clear limitations. For example, they only utilize shallow topic models with only a single stochastic hidden layer in their data generation process. Note several neural topic models use deep neural networks to construct their variational encoders, but still use shallow generative models (decoders) . Another key limitation lies in ignoring the sentence order, as they treat each document as a bag of sentences. Thus once the topic weight vector learned from the document context is given, the task is often reduced to independently assigning probabilities to individual sentences (; 2019). In this paper, as depicted in Fig. 1, we propose to use recurrent gamma belief network (rGBN) to guide a stacked RNN for language modeling. We refer to the model as rGBN-RNN, which integrates rGBN, a deep recurrent topic model, and stacked RNN , a neural language model, into a novel larger-context RNN-based language model. It simultaneously learns a deep recurrent topic model, extracting document-level multi-layer word concurrence patterns and sequential topic weight vectors for sentences, and an expressive language model, capturing both short-and long-range word sequential dependencies. For inference, we equip rGBN-RNN (decoder) with a novel variational recurrent inference network (encoder), and train it end-to-end by maximizing the evidence lower bound (ELBO). Different from the stacked RNN based language model in , which relies on three types of customized training operations (UPDATE, COPY, FLUSH) to extract multi-scale structures, the language model in rGBN-RNN learns such structures purely under the guidance of the temporally and hierarchically connected stochastic layers of rGBN. The effectiveness of rGBN-RNN as a new larger-context language model is demonstrated both quantitatively, with perplexity and BLEU scores, and qualitatively, with interpretable latent structures and randomly generated sentences and paragraphs. Notably, rGBN-RNN can generate a paragraph consisting of a sequence of semantically coherent sentences. Denote a document of J sentences as D = (S 1, S 2, . . ., S J), where S j = (y j,1, . . ., y j,Tj) consists of T j words from a vocabulary of size V. Conventional statistical language models often only focus on the word sequence within a sentence. Assuming that the sentences of a document are independent to each other, they often define Tj t=2 p (y j,t | y j,<t) p (y j,1). RNN based neural language models define the conditional probability of each word y j,t given all the previous words y j,<t within the sentence S j, through the softmax function of a hidden state h j,t, as where f (·) is a non-linear function typically defined as an RNN cell, such as long short-term memory (LSTM) and gated recurrent unit (GRU) . These RNN-based statistical language models are typically applied only at the word level, without exploiting the document context, and hence often fail to capture long-range dependencies.;; 2019) remedy the issue by guiding the language model with a topic model, they still treat a document as a bag of sentences, ignoring the order of sentences, and lack the ability to extract hierarchical and recurrent topic structures. We introduce rGBN-RNN, as depicted in Fig. 1(a), as a new larger-context language model. It consists of two key components: (i) a hierarchical recurrent topic model (rGBN), and (ii) a stacked RNN based language model. We use rGBN to capture both global semantics across documents and long-range inter-sentence dependencies within a document, and use the language model to learn the local syntactic relationships between the words within a sentence. Similar to;, we represent a document as a sequence of sentence-context pairs as + summarizes the document excluding S j, specifically (S 1, ..., S j−1, S j+1, ..., S J), into a BoW count vector, with V c as the size of the vocabulary excluding stop words. Note a naive way is to treat each sentence as a document, use a dynamic topic model to capture the temporal dependencies of the latent topic-weight vectors, which is fed to the RNN to model the word sequence of the corresponding sentence. However, the sentences are often too short to be well modeled by a topic model. In our setting, as d j summarizes the document-level context of S j, it is in general sufficiently long for topic modeling. Note during testing, we redefine d j as the BoW vector summarizing only the preceding sentences, i.e., S 1:j−1, which will be further clarified when presenting experimental . Fig. 1 (a), to model the time-varying sentence-context count vectors d j in document D, the generative process of the rGBN component, from the top to bottom hidden layers, is expressed as where θ l j ∈ R K l + denotes the gamma distributed topic weight vectors of sentence j at layer l, the transition matrix of layer l that captures cross-topic temporal dependencies, the loading matrix at layer l, K l the number of topics of layer l, and τ 0 ∈ R + a scaling hyperparameter. At j = 1, θ + can be factorized into the sum of Φ l+1 θ l+1 j, capturing inter-layer hierarchical dependence, and Π l θ l j−1, capturing intra-layer temporal dependence. rGBN not only captures the document-level word occurrence patterns inside the training text corpus, but also the sequential dependencies of the sentences inside a document. Note ignoring the recurrent structure, rGBN will reduce to the gamma belief network (GBN) of , which can be considered as a multi-stochastic-layer deep generalization of LDA (a). If ignoring its hierarchical structure (i.e., L = 1), rGBN reduces to Poisson-gamma dynamical systems . We refer to the rGBN-RNN without its recurrent structure as GBN-RNN, which no longer models sequential sentence dependencies; see Appendix A for more details. Different from a conventional RNN-based language model, which predicts the next word only using the preceding words within the sentence, we integrate the hierarchical recurrent topic weight vectors θ l j into the language model to predict the word sequence in the jth sentence. Our proposed language model is built upon the stacked RNN proposed in; , but with the help of rGBN, it no longer requires specialized training heuristics to extract multi-scale structures. As shown in Fig. 1 (b), to generate y j,t, the t th token of sentence j in a document, we construct the hidden states h l j,t of the language model, from the bottom to top layers, as where LSTM l word denotes the word-level LSTM at layer l, W e ∈ R V are word embeddings to be learned, and x j,t = y j,t−1. Note a, the conditional probability of y j,t becomes There are two main reasons for combining all the latent representations a 1:L j,t for language modeling. First, the latent representations exhibit different statistical properties at different stochastic layers of rGBN-RNN, and hence are combined together to enhance their representation power. Second, having "skip connections" from all hidden layers to the output one makes it easier to train the proposed network, reducing the number of processing steps between the bottom of the network and the top and hence mitigating the "vanishing gradient" problem . To sum up, as depicted in Fig. 1 (a), the topic weight vector θ l j of sentence j quantifies the topic usage of its document context d j at layer l. It is further used as an additional feature of the language model to guide the word generation inside sentence j, as shown in Fig. 1 (b). It is clear that rGBN-RNN has two temporal structures: a deep recurrent topic model to extract the temporal topic weight vectors from the sequential document contexts, and a language model to estimate the probability of each sentence given its corresponding hierarchical topic weight vector. Characterizing the word-sentencedocument hierarchy to incorporate both intra-and inter-sentence information, rGBN-RNN learns more coherent and interpretable topics and increases the generative power of the language model. Distinct from existing topic-guided language models, the temporally related hierarchical topics of rGBN exhibit different statistical properties across layers, which better guides language model to improve its language generation ability. For rGBN-RNN, given, the marginal likelihood of the sequence of sentence-context where e The inference task is to learn the parameters of both the topic model and language model components. One naive solution is to alternate the training between these two components in each iteration: First, the topic model is trained using a sampling based iterative algorithm provided in; Second, the language model is trained with maximum likelihood estimation under a standard cross-entropy loss. While this naive solution can utilize readily available inference algorithms for both rGBN and the language model, it may suffer from stability and convergence issues. Moreover, the need to perform a sampling based iterative algorithm for rGBN inside each iteration limits the scalability of the model for both training and testing. Algorithm 1 Hybrid SG-MCMC and recurrent autoencoding variational inference for rGBN-RNN. Set mini-batch size m and the number of layer L Initialize encoder and neural language model parameter parameter Ω, and topic model parameter Randomly select a mini-batch of m documents consisting of J sentences to form a subset X = {di,1:,j according to, and update Ω; Sample θ l i,j from and via and {Φ l} L l=1, will be described in Appendix C; end for To this end, we introduce a variational recurrent inference network (encoder) to learn the latent temporal topic weight vectors θ, the ELBO of the log marginal likelihood shown in can be constructed as which unites both the terms that are primarily responsible for training the recurrent hierarchical topic model component, and terms for training the neural language model component. Similar to, we define q(θ, a random sample from which can be obtained by transforming standard uniform variables To capture the temporal dependencies between the topic weight vectors, both k l j and λ l j, from the bottom to top layers, can be expressed as where h Rather than finding a point estimate of the global parameters of the rGBN, we adopt a hybrid inference algorithm by combining TLASGR-MCMC described in Cong et al. (2017a); and our proposed recurrent variational inference network. In other words, the global parameters can be sampled with TLASGR-MCMC, while the parameters of the language model and variational recurrent inference network, denoted by Ω, can be updated via stochastic gradient descent (SGD) by maximizing the ELBO in. We describe a hybrid variational/sampling inference for rGBN-RNN in Algorithm 1 and provide more details about sampling with TLASGR-MCMC in Appendix C. We defer the details on model complexity to Appendix E. To sum up, as shown in Fig. 1(c), the proposed rGBN-RNN works with a recurrent variational autoencoder inference framework, which takes the document context of the jth sentence within a document as input and learns hierarchical topic weight vectors θ 1:L j that evolve sequentially with j. The learned topic vectors in different layer are then used to reconstruct the document context input and as an additional feature for the language model to generate the jth sentence. We consider three publicly available corpora, including APNEWS, IMDB, and BNC. The links, preprocessing steps, and summary statistics for them are deferred to Appendix D. We consider a recurrent variational inference network for rGBN-RNN to infer θ l j, as shown in Fig. 1(c), whose number of hidden units in are set the same as the number of topics in the corresponding layer., which extracts the global semantic coherence of a document via a neural topic model, with the probability of each learned latent topic further adopted to build a mixture-of-experts language model; (vii) TGVAE , combining a variational auto-encoder based neural sequence model with a neural topic model; (viii) GBN-RNN, a simplified rGBN-RNN that removes the recurrent structure of its rGBN component. For rGBN-RNN, to ensure the information about the words in the jth sentence to be predicted is not leaking through the sequential document context vectors at the testing stage, the input d j in only summarizes the preceding sentences S <j. For GBN-RNN, following TDLM and TCNLM, all the sentences in a document, excluding the one being predicted, are used to obtain the BoW document context. As shown in Table 1, rGBN-RNN outperforms all baselines, and the trend of improvement continues as its number of layers increases, indicating the effectiveness of assimilating recurrent hierarchical topic information. rGBN-RNN consistently outperforms GBN-RNN, suggesting the benefits of exploiting the sequential dependencies of the sentence-contexts for language modeling. Moreover, comparing Table 1 and Table 4 of Appendix E suggests rGBN-RNN, with its hierarchical and temporal topical guidance, achieves better performance with fewer parameters than comparable RNN-based baselines. Note that for language modeling, there has been significant recent interest in replacing RNNs with Transformer , which consists of stacked multi-head self-attention modules, and its variants (; ; ; 2019). While Transformer based language models have been shown to be powerful in various natural language processing tasks, they often have significantly more parameters, require much more training data, and take much longer to train than RNN-based language models. For example, Transformer-XL with 12L and that with 24L , which improve Transformer to capture longer-range dependencies, have 41M and 277M parameters, respectively, while the proposed rGBN-RNN with three stochastic hidden layers has as few as 7.3M parameters, as shown in Table 4, when used for language modeling. From a structural point-of-view, we consider the proposed rGBN-RNN as complementary to rather than competing with Transformer based language models, and consider replacing RNN with Transformer to construct rGBN guided Transformer as a promising future extension., we use test-BLEU to evaluate the quality of generated sentences with a set of real test sentences as the reference, and self-BLEU to evaluate the diversity of the generated sentences . Given the global parameters of the deep recurrent topic model (rGBN) and language model, we can generate the sentences by following the data generation process of rGBN-RNN: we first generate topic weight vectors θ L j randomly and then downward propagate it through the rGBN as in to generate θ <L j. By assimilating the random draw topic weight vectors with the hidden states of the language model in each layer depicted in, we generate a corresponding sentence, where we start from a zero hidden state at each layer in the language model, and sample words sequentially until the end-of-the-sentence symbol is generated. Comparisons of the BLEU scores between different methods are shown in Fig. 2, using the benchmark tool in Texygen ; We show below BLEU-3 and BLEU-4 for BNC and defer the analogous plots for IMDB and APNEWS to Appendix G and H. Note we set the validation dataset as the ground-truth. For all datasets, it is clear that rGBN-RNN yields both higher test-BLEU and lower self-BLEU scores than related methods do, indicating the stacked-RNN based language model in rGBN-RNN generalizes well and does not suffer from mode collapse (i.e., low diversity). Hierarchical structure of language model: In Fig. 3, we visualize the hierarchical multi-scale structures learned with the language model of rGBN-RNN and that of GBN-RNN, by visualizing the L 2 -norm of the hidden states in each layer, while reading a sentence from the APNEWS validation set as "the service employee international union asked why cme group needs tax relief when it is making huge amounts of money?" As shown in Fig. 3(a), in the bottom hidden layer (h1), the L 2 norm sequence varies quickly from word to word, except within short phrases such "service employee", "international union," and "tax relief," suggesting layer h1 is in charge of capturing short-term local dependencies. By contrast, in the top hidden layer (h3), the L 2 norm sequence varies slowly and exhibits semantic/syntactic meaningful long segments, such as "service employee international union," "asked why cme group needs tax relief," "when it is," and "making huge amounts of," suggesting that layer h3 is in charge of capturing long-range dependencies. Therefore, the language model in 48 budget lawmakers gov. revenue vote proposal community legislation 57 lawmakers pay proposal legislation credit session meeting gambling 60 budget gov. revenue vote costs mayor california conservative 57 generated sentence: the last of the four companies and the mississippi inter national speedway was voted to accept the proposal. 60 generated sentence: adrian on thursday issued an officer a news release saying the two groups will take more than $ 40,000 for contacts with the private nonprofit. 48 generated sentence: looming monday, the assembly added a proposal to balance the budget medicaid plan for raising the rate to $ 142 million, whereas years later, to $ 200 million. lawmaker proposal legislation approval raising audit senate 75 generated sentence: the state senate would give lawmakers time to accept the retirement payment. 48-57-75 generated sentence: the proposal would give them a pathway to citizenship for the year before, but they don't have a chance to participate in the elections. inc gambling credit assets medicaid investment 62 generated sentence: the gambling and voting department says it was a chance of the game. 48-57-62 generated sentence: the a r k a n s a s s e n a t e h a s purchased a $ 500 million state bond for a proposed medicaid expansion for a new york city. 11 budget revenue loan gains treasury incentives profits 11 generated sentence: the office of north dakota has been offering a $ 22 million bond to a $ 68 m i l l i o n b u d g e t w i t h t h e proceeds from a escrow account. 48-60-11 generated sentence: a new report shows the state has been hit by a number of shortcomings in jindal's budget proposal for the past decade. 84 gov. vote months conservation ballot reform fundraising 84 generated sentence: the u.s. sen. joe mccoy in the democratic party says russ of the district, must take the rest of the vote on the issues in the first half of the year. it was partially the other in the republican caucus that raised significant amounts of spending last year and ended with cuts from a previous government shutdown. Figure 4: Topics and their temporal trajectories inferred by a three-hidden-layer rGBN-RNN from the APNEWS dataset, and the generated sentences under topic guidance (best viewed in color). Top words of each topic at layer 3, 2, 1 are shown in orange, yellow and blue boxes respectively, and each sentence is shown in a dotted line box labeled with the corresponding topic index. Sentences generated with a combination of topics in different layers are at the bottom of the figure. rGBN-RNN can allow more specific information to transmit through lower layers, while allowing more general higher level information to transmit through higher layers. Our proposed model have the ability to learn hierarchical structure of the sequence, despite without designing the multiscale RNNs on purpose like. We also visualize the language model of GBN-RNN in Fig. 3(b); with much less smoothly time-evolved deeper layers, GBN-RNN fails to utilize its stacked RNN structure as effectively as rGBN-RNN does. This suggests that the language model is much better trained in rGBN-RNN than in GBN-RNN for capturing long-range temporal dependencies, which helps explain why rGBN-RNN exhibits clearly boosted BLEU scores in comparison to GBN-RNN. We present an example topic hierarchy inferred by a three-layer rGBN-RNN from APNEWS. In Fig. 4, we select a large-weighted topic at the top hidden layer and move down the network to include any lower-layer topics connected to their ancestors with sufficiently large weights. Horizontal arrows link temporally related topics at the same layer, while top-down arrows link hierarchically related topics across layers. For example, topic 48 of layer 3 on "budget, lawmakers, gov., revenue," is related not only in hierarchy to topic 57 on "lawmakers, pay, proposal, legislation" and topic 60 of the lower layer on "budget, gov., revenue, vote, costs, mayor," but also in time to topic 35 of the same layer on "democratic, taxes, proposed, future, state." Highly interpretable hierarchical relationships between the topics at different layers, and temporal relationships between the topics at the same layer are captured by rGBN-RNN, and the topics are often quite specific semantically at the bottom layer while becoming increasingly more general when moving upwards. Sentence generation under topic guidance: Given the learned rGBN-RNN, we can sample the sentences both conditioning on a single topic of a certain layer and on a combination of the topics from different layers. Shown in the dotted-line boxes in Fig. 4, most of the generated sentences conditioned on a single topic or a combination of topics are highly related to the given topics in terms of their semantical meanings but not necessarily in key words, indicating the language model is successfully guided by the recurrent hierarchical topics. These observations suggest that rGBN-RNN has successfully captured syntax and global semantics simultaneously for natural language generation. Sentence/paragraph generation conditioning on a paragraph: Given the GBN-RNN and rGBN-RNN learned on APNEWS, we further present the generated sentences conditioning on a paragraph, as shown in Fig. 5. To randomly generate sentences, we encode the paragraph into a hierarchical latent representation and then feed it into the stacked-RNN. Besides, we can generate a paragraph with rGBN-RNN, using its recurrent inference network to encode the paragraph into a dynamic hierarchical latent representation, which is fed into the language model to predict the word sequence Document Generated Sentences with GBN-RNN Generated Sentences with rGBN-RNN the proposal would also give lawmakers with more money to protect public safety, he said. the proposal, which was introduced in the house on a vote on wednesday, has already passed the senate floor to the house. Generated temporal Sentences with rGBN-RNN (Paragraph) the senate sponsor (…), a house committee last week removed photo ids issued by public colleges and universities from the measure sponsored by republican rep. susan lynn, who said she agreed with the change. the house approved the bill on a 65-30 vote on monday evening. but republican sen. bill ketron in a statement noted that the upper chamber overwhelmingly rejected efforts to take student ids out of the bill when it passed 21-8 earlier this month. ketron said he would take the bill to conference committee if needed. if the house and senate agree, it will be the first time they'll have to seek their first meeting. the city commission voted last week to approve the law, which would have allowed the council to approve the new bill. senate president pro tem joe scarnati said the governor's office has never resolved the deadline for a vote in the house. the proposal is a new measure version of the bill to enact a senate committee to approve the emergency manager's emergency license. the house gave the bill to six weeks of testimony, but the vote now goes to the full house for consideration. jackson signed his paperwork wednesday with the legislature.the proposal would also give lawmakers with more money to protect public safety, he said. "a spokesman for the federal department of public safety says it has been selected for a special meeting for the state senate to investigate his proposed law. a new state house committee has voted to approve a measure to let idaho join a national plan to ban private school systems at public schools. the campaign also launched a website at the university of california, irvine, which are studying the current proposal. in each sentence of the input paragraph. It is clear that both the proposed GBN-RNN and rGBN-RNN can successfully capture the key textual information of the input paragraph, and generate diverse realistic sentences. Interestingly, the rGBN-RNN can generate semantically coherent paragraphs, incorporating contextual information both within and beyond the sentences. Note that with the topics that extract the document-level word cooccurrence patterns, our proposed models can generate semantically-meaningful words, which may not exist in the original document. We propose a recurrent gamma belief network (rGBN) guided neural language modeling framework, a novel method to learn a language model and a deep recurrent topic model simultaneously. For scalable inference, we develop hybrid SG-MCMC and recurrent autoencoding variational inference, allowing efficient end-to-end training. Experiments conducted on real world corpora demonstrate that the proposed models outperform a variety of shallow-topic-model-guided neural language models, and effectively generate the sentences from the designated multi-level topics or noise, while inferring interpretable hierarchical latent topic structure of document and hierarchical multiscale structures of sequences. For future work, we plan to extend the proposed models to specific natural language processing tasks, such as machine translation, image paragraph captioning, and text summarization. Another promising extension is to replace the stacked-RNN in rGBN-RNN with Transformer, i.e., constructing an rGBN guided Transformer as a new larger-context neural language model. GBN-RNN: {y 1:T, d} denotes a sentence-context pair, where d ∈ Z Vc + represents the document-level context as a word frequency count vector, the vth element of which counts the number of times the vth word in the vocabulary appears in the document excluding sentence y 1:T. The hierarchical model of a L-hidden-layer GBN, from top to bottom, is expressed as The stacked-RNN based language model described in is also used in GBN-RNN. Statistical inference: To infer GBN-RNN, we consider a hybrid of stochastic gradient MCMC (; ; ; ; a), used for the GBN topics φ l k, and auto-encoding variational inference , used for the parameters of both the inference network (encoder) and RNN. More specifically, GBN-RNN generalizes Weibull hybrid auto-encoding inference (WHAI) of: it uses a deterministic-downward-stochastic-upward inference network to encode the bag-of-words representation of d into the latent topic-weight variables θ l across all hidden layers, which are fed into not only GBN to reconstruct d, but also a stacked RNN in language model, as shown in, to predict the word sequence in y 1:T. The topics φ l k can be sampled with topic-layeradaptive stochastic gradient Riemannian (TLASGR) MCMC, whose details can be found in Cong et al. (2017a);, omitted here for brevity. Given the sampled topics φ l k, the joint marginal likelihood of {y 1:T, d} is defined as ) is used to provide a ELBO of the log joint marginal likelihood as and the training is performed by maximizing E pdata(, where both k l and λ l are deterministically transformed from d using neural networks. Distinct from a usual variational auto-encoder whose inference network has a pure bottom-up structure, the inference network here has a determisticupward-stoachstic-downward ladder structure . is the written portion of the British National Corpus . Following the preprocessing steps in , we tokenize words and sentences using Stanford CoreNLP , lowercase all word tokens, and filter out word tokens that occur less than 10 times. For the topic model, we additionally exclude stopwords 2 and the top 0.1% most frequent words. All these corpora are partitioned into training, validation, and testing sets, whose summary statistics are provided in Table 2 of the Appendix. in, and the parameters of the variational recurrent inference network (encoder), consisting of RNN The language model component is parameterized by LSTM l word in and the coupling vectors g l described in Appendix B. We summarize in Table 3 the complexity of rGBN-RNN (ignoring all bias terms), where V denotes the vocabulary size of the language model, E the dimension of word embedding vectors, V c the size of the vocabulary of the topic model that excludes stop words, H w l the number of hidden units of the word-level LSTM at layer l (stacked-RNN language model), H s l the number of hidden units of the sentence-level RNN at layer l (variational recurrent inference network), and K l the number of topics at layer l. Table 4 further compares the number of parameters between various RNN-based language models, where we follow the convention to ignore the word embedding layers. Some models in Table 1 are not included here, because we could not find sufficient information from their corresponding papers or code to accurately calculate the number of model parameters. Note when used for language generation at the testing stage, rGBN-RNN no longer needs its topics {Φ l}, whose parameters are hence not counted. Note the number of parameters of the topic model component is often dominated by that of the language model component. Left panel is BLEU-3 and right is BLEU-4, and a better BLEU score would fall within the lower right corner, where black point represents mean value and circles with different colors denote the elliptical surface of probability of BLEU in a two-dimensional space.

represent this text to find a much much longer article about it with say hundreds-thousands of words

------

Examples:

"We propose a new regularization technique based on the knowledge distillation." == "Deep neural networks with millions of parameters may suffer from poor generalizations due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label and augmented samples of the same source during training. In other words, we regularize the dark knowledge (i.e., the knowledge on wrong predictions) of a single network, i.e., a self-knowledge distillation technique, to force it output more meaningful predictions. We demonstrate the effectiveness of the proposed method via experiments on various image classification tasks: it improves not only the generalization ability, but also the calibration accuracy of modern neural networks. Deep neural networks (DNNs) have achieved state-of-the-art performance on many machine learning applications, e.g., computer vision , natural language processing , and reinforcement learning . As the scale of training dataset increases, the size of DNNs (i.e., the number of parameters) also scales up to handle such a large dataset efficiently. However, networks with millions of parameters may incur overfitting and suffer from poor generalizations (; . To address the issue, many regularization strategies have been investigated in the literature: early stopping, L 1 /L 2 -regularization , dropout , batch normalization and data augmentation Regularizing the predictive or output distribution of DNNs can be effective because it contains the most succinct knowledge of the model. On this line, several strategies such as entropy maximization and angular-margin based methods have been proposed in the literature. They can be also influential to solve related problems, e.g., network calibration , detection of out-of-distribution samples and exploration of the agent in reinforcement learning . In this paper, we focus on developing a new output regularizer for deep models utilizing the concept of dark knowledge , i.e., the knowledge on wrong predictions made by DNN. Its importance has been first evidenced by the so-called knowledge distillation and investigated in many following works (; ; ;). While the related works use the knowledge distillation (KD; Hinton et al. 2015) to transfer the dark knowledge learned by a teacher network to a student network, we regularize the dark knowledge itself during training a single network, i.e., self-knowledge distillation. Specifically, we propose a new regularization technique, coined class-wise self-knowledge distillation (CS-KD) that matches or distills the predictive distribution of DNNs between different samples of the same label (class-wise regularization) and augmented samples of the same source (sample-wise regularization) as shown in Figure 1. One can expect that the proposed regularization method forces DNNs to produce similar wrong predictions if samples are of the same class, while the conventional cross-entropy loss does not consider such consistency on the wrong predictions. We demonstrate the effectiveness of our regularization method using deep convolutional neural networks, such as ResNet and DenseNet trained for image classification tasks on various datasets including CIFAR-100 , TinyImageNet 1, CUB-200-2011 , Stanford Dogs , and MIT67 datasets. We compare or combine our method with prior regularizers. In our experiments, the top-1 error rates of our method are consistently smaller than those of prior output regularization methods such as angular-margin based methods and entropy regularization . In particular, the gain tends to be larger in overall for the top-5 error rates and the expected calibration errors , which confirms that our method indeed makes predictive distributions more meaningful. Moreover, we investigate a variant of our method by combining it with other types of regularization method for boosting performance, such as the mixup regularization and the original KD method. We improve the top-1 error rate of mixup from 37.09% to 31.95% and that of KD from 39.32% to 35.36% under ResNet trained by the CUB-200-2011 dataset. Our method is very simple to use, and would enjoy a broader usage in the future. In this section, we introduce a new regularization technique, named class-wise self-knowledge distillation (CS-KD). Throughout this paper, we focus on fully-supervised or classification tasks, and denote x ∈ X as an input and y ∈ Y = {1, ..., C} as its ground-truth label. Suppose that a softmax classifier is used to model a posterior distribution, i.e., given the input x, the predictive distribution is as follows:, where f = [f i] denotes the logit-vector of DNN, parameterized by θ and T > 0 is the temperature scaling parameter. We first consider matching the predictive distributions on samples of the same class, which distills their dark knowledge into the model itself. To this end, we propose a class-wise regularization loss that enforces consistent predictive distributions in the same class. Formally, given input x and another randomly sampled input x having the same label y, it is defined as follows: where KL denotes the Kullback-Leibler (KL) divergence and θ is a fixed copy of the parameters θ. As suggested by , the gradient is not propagated through θ to avoid Algorithm 1 Class-wise self-knowledge distillation (CS-KD) Initialize parameters θ. while θ has not converged do for (x, y) in a sampled batch do g θ ← 0 Get another sample x randomly which has the same label y from the training set. Generate x aug, x aug using data augmentation methods. Compute gradient: Update parameters θ using gradients g θ. end while the model collapsing issue. Similar to the knowledge distillation method (KD) by , L cls matches two predictions. While the original KD matches predictions of a sample from two networks, we do predictions of different samples from a single network. Namely, our method performs self-knowledge distillation. In addition to enforcing the intra-class consistency of predictive distributions, we apply this idea to the single-sample scenario by augmenting the input data. For a given training sample x, the proposed sample-wise regularization loss L sam is defined as follows: where x aug is an augmented input that is modified by some data augmentation methods, e.g., resizing, rotating, random cropping (; ;, cutout , and auto-augmentation . In our experiments, we use standard augmentation methods for ImageNet (i.e., flipping and random sized cropping) because they make training more stable. In summary, the total training loss L tot is defined as a weighted sum of the two regularization terms with cross-entropy loss as follows: where λ cls and λ sam are balancing weights for each regularization, respectively. Note that the first term is the cross-entropy loss of softmax outputs with temperature T = 1. In other words, we not only train the true label, but also regularize the wrong labels. The full training procedure with the proposed loss L tot is summarized in Algorithm 1. Datasets. To demonstrate our method under general situations of data diversity, we consider various image classification tasks including conventional classification and fine-grained classification tasks. We use CIFAR-100 and TinyImageNet 2 datasets for conventional classification tasks, and CUB-200-2011 , Stanford Dogs , and MIT67 datasets for fine-grained classification tasks. Note that fine-grained image classification tasks have visually similar classes and consist of fewer training samples per class compared to conventional classification tasks. We sample 10% of the training dataset randomly as a validation set for CIFAR-100 and TinyImageNet and report the test accuracy based on the validation accuracy. For the fine-grained datasets, we report the best validation accuracy. Network architecture. We consider two state-of-the-art convolutional neural network architectures: ResNet and DenseNet . We use standard ResNet-18 with 64 filters and DenseNet-121 with growth rate of 32 for image size 224 × 224. For CIFAR-100 and TinyImageNet, we modify the first convolutional layer 3 with kernel size 3 × 3, strides 1 and padding 1, instead of the kernel size 7 × 7, strides 2 and padding 3, for image size 32 × 32. Evaluation metric. For evaluation, we measure the following metrics: • Top-1 / 5 error rate. Top-k error rate is the fraction of test samples for which the correct label is amongst the top-k confidences. We measured top-1 and top-5 error rates to evaluate the generalization performance of the models. • Expected Calibration Error (ECE). ECE approximates the difference in expectation between confidence and accuracy, by partitioning predictions into M equally-spaced bins and taking a weighted average of bins' difference of confidence and accuracy, i.e., ECE = • Recall at k (R@k). Recall at k is the percentage of test samples that have at least one example from the same class in k nearest neighbors on the feature space. To measure the distance between two samples, we use L 2 -distance between their average-pooled features in the penultimate layer. We compare the recall at 1 scores to evaluate intra-class variations of learned features. Hyper-parameters. All networks are trained from scratch and optimized by stochastic gradient descent (SGD) with momentum 0.9, weight decay 0.0001 and an initial learning rate of 0.1. The learning rate is divided by 10 after epochs 100 and 150 for all datasets and total epochs are 200. We set batch size as 128 for conventional, and 32 for fine-grained classification tasks. We use standard flips, random resized crops, 32 for conventional and 224 for fine-grained classification tasks, overall experiments. Furthermore, we set T = 4, λ cls = 1 for all experiments and λ sam = 1 for experiments on fine-grained classification tasks, and λ sam = 0 on conventional classification tasks. To compute expected calibration error (ECE), we set the number of bins M as 20. Baselines. We compare our method with prior regularization methods such as the state-of-the-art angular-margin based methods and entropy regularization . They also regularize predictive distributions as like ours. • AdaCos . 4 AdaCos dynamically scales the cosine similarities between training samples and corresponding class center vectors to maximize angular-margin. • Virtual-softmax . Virtual-softmax injects an additional virtual class to maximize angular-margin. • Maximum-entropy . Maximum-entropy is a typical entropy regularization, which maximizes the entropy of the predictive distribution. Note that AdaCos and Virtual-softmax regularize the predictive or output distribution of DNN to learn feature representation by reducing intra-class variations and enlarging inter-class margins. Comparison with output regularization methods. We measure the top-1 error rates of the proposed method (denoted by CS-KD) by comparing with Virtual-softmax, AdaCos, and Maximumentropy on various image classification tasks. Table 1 shows that CS-KD outperforms other baselines consistently. In particular, CS-KD improves the top-1 error rate of cross-entropy loss from 46.00% to 33.50% in the CUB-200-2011 dataset, while the top-1 error rates of other baselines are even worse than the cross-entropy loss (e.g., AdaCos in the CIFAR-100, Virtual-softmax in the MIT67, and Maximum-entropy in the TinyImageNet and the MIT67 under DenseNet). The imply that our method is more effective and stable than other baselines. Compatibility with other types of regularization methods. We investigate orthogonal usage with other types of regularization methods such as mixup and knowledge distillation (KD). Mixup utilizes convex combinations of input pairs and corresponding label pairs for training. We combine our method with mixup regularization by applying the class-wise regularization loss L cls to mixed inputs and mixed labels, instead of standard inputs and labels. Table 2 shows the effectiveness of our method combined with mixup regularization. Interestingly, this simple idea significantly improves the performances of fine-grained classification tasks. In particular, our method improves the top-1 error rate of mixup regularization from 37.09% to 31.95%, where the top-1 error rate of cross-entropy loss is 46.00% in the CUB-200-2011. KD regularizes predictive distributions of student network to learn the dark knowledge of a teacher network. We combine our method with KD to learn dark knowledge from the teacher and itself simultaneously. Table 3 shows that the top-1 error rate under using our method solely is close to that of KD, although ours do not use additional teacher networks. Besides, learning knowledge from a teacher network improves the top-1 error rate of our method from 39.32% to 35.36% in the CUB-200-2011 dataset. The show a wide applicability of our method, compatible to use with other regularization methods. One can expect that our method forces DNNs to produce meaningful predictions by reducing the intra-class variations. To verify this, we analyze feature embedding and various evaluation metrics, including the top-1, top-5 error, expected calibration error and R@1. In Figure 2, we visualize feature embedding of the penultimate layer from ResNet-18 trained with various regularization techniques by t-SNE in the CIFAR-100 dataset. One can note that intra-class variations are significantly decreased by our method (Figure 2f), while Virtualsoftmax (Figure 2b) and AdaCos (Figure 2c) only reduce the angular-margin. We also provide quantitative analysis on the feature embedding by measuring the R@1 values, which are related to intra-class variations. Note that the larger value of R@1 means the more reduced intra-class variations on the feature embedding . As shown in Table 4, R@1 values can be significantly improved when ResNet-18 is trained with our methods. In particular, R@1 of our method is 59.22% in the CUB-200-2011 dataset, while R@1 of Virtual-softmax and Adacos are 55.56% and 54.86%, respectively. Moreover, Table 4 shows the top-5 error rates of our method significantly outperform other regularization methods. Figure 3 and Table 4 show that our method enhances model calibration significantly, which also confirm that ours forces DNNs to produce more meaningful predictions. Table 4: Top-1 / 5 error, ECE, and Recall at 1 rates (%) of ResNet-18. The arrow on the right side of the evaluation metric indicates ascending or descending order of the value. We reported the mean and standard deviation over 3 runs with different random seed, and the best are indicated in bold. &; ) which show accuracy as a function of confidence, for ResNet-18 trianed on CIFAR-100 using (a) Cross-entropy, (b) Virtual-softmax, (c) AdaCos, and (d) Maximum-entropy. All methods are compared with our proposed method, CS-KD. Regularization techniques. Numerous techniques have been introduced to prevent overfitting of neural networks, including early stopping, weight decay, dropout , and batch normalization . Alternatively, regularization methods for the output distribution also have been explored: showed that label-smoothing, which is a mixture of the ground-truth and the uniform distribution, improves generalization of neural networks. proposed penalizing low entropy output distributions, which improves exploration in reinforcement learning and supervised learning. proposed a powerful data augmentation method called mixup, which works as a regularizer that can be utilized with smaller weight decay. We remark that our method enjoys orthogonal usage with the prior methods, i.e., our methods can be combined with prior methods to further improve the generalization performance. Knowledge distillation. Knowledge distillation is an effective learning method to transfer the knowledge from a powerful teacher model to a student. This pioneering work showed that one can use softmax with temperature scaling to match soft targets for transferring dark knowledge, which contains the information of non-target labels. There are numerous follow-up studies to distill knowledge in the aforementioned teacher-student framework. FitNets tried to learn features of a thin deep network using a shallow one with linear transform. introduced a transfer method that matches attention maps of the intermediate features, and tried to maximize the mutual information between intermediate layers of teacher and student for enhanced performance. proposed a loss function for matching Jacobian of the networks output instead of the feature itself. We remark that our method and knowledge distillation have a similar component, i.e., using a soft target distribution, but our method utilizes the soft target distribution from itself. We also remark that joint usage of our method and the prior knowledge distillation methods is effective. Margin-based softmax losses. There have been recent efforts toward boosting the recognition performances via enlarging inter-class margins and reducing intra-class variation. Several approaches utilized metric-based methods that measure similarities between features using Euclidean distances, such as triplet and contrastive loss . To make the model extract discriminative features, center loss and range loss were proposed to minimize distances between samples belong to the same class. COCO loss (b) and NormFace optimized cosine similarities, by utilizing reformulations of softmax loss and metric learning with feature normalization. applied ring loss for soft normalization which uses a convex norm constraint. More recently, angular-margin based losses were proposed for further improvement. Lsoftmax and A-softmax (a) combined angular margin constraints with softmax loss to encourage the model to generate more discriminative features. CosFace, AM-softmax and ArcFace introduced angular margins for a similar purpose, by reformulating softmax loss. Different from L-Softmax and A-Softmax, Virtual-softmax encourages a large margin among classes via injecting additional virtual negative class. In this paper, we discover a simple regularization method to enhance generalization performance of deep neural networks. We propose two regularization terms which penalizes the predictive distribution between different samples of the same label and augmented samples of the same source by minimizing the Kullback-Leibler divergence. We remark that our ideas regularize the dark knowledge (i.e., the knowledge on wrong predictions) itself and encourage the model to produce more meaningful predictions. Moreover, we demonstrate that our proposed method can be useful for the generalization and calibration of neural networks. We think that the proposed regularization techniques would enjoy a broader range of applications, e.g., deep reinforcement learning and detection of out-of-distribution samples ."

We introduce a novel larger-context language model to simultaneously captures syntax and semantics, making it capable of generating highly interpretable sentences and paragraphs

represent the short summary for finding the corresponding scientific article (long):

For example, The paper describes a strategic intrinsically motivated learning algorithm which tackles the learning of complex motor policies. should be similar to We propose an active learning algorithmic architecture, capable of organizing its learning process in order to achieve a field of complex tasks by learning sequences of primitive motor policies: Socially Guided Intrinsic Motivation with Procedure Babbling (SGIM-PB). The learner can generalize over its experience to continuously learn new outcomes, by choosing actively what and how to learn guided by empirical measures of its own progress. In this paper, we are considering the learning of a set of interrelated complex outcomes hierarchically organized. We introduce a new framework called "procedures", which enables the autonomous discovery of how to combine previously learned skills in order to learn increasingly more complex motor policies (combinations of primitive motor policies). Our architecture can actively decide which outcome to focus on and which exploration strategy to apply. Those strategies could be autonomous exploration, or active social guidance, where it relies on the expertise of a human teacher providing demonstrations at the learner's request. We show on a simulated environment that our new architecture is capable of tackling the learning of complex motor policies, to adapt the complexity of its policies to the task at hand. We also show that our "procedures" increases the agent's capability to learn complex tasks. Recently, efforts in the robotic industry and academic field have been made for integrating robots in previously human only environments. In such a context, the ability for service robots to continuously learn new skills, autonomously or guided by their human counterparts, has become necessary. They would be needed to carry out multiple tasks, especially in open environments, which is still an ongoing challenge in robotic learning. Those tasks can be independent and self-contained but they can also be complex and interrelated, needing to combine learned skills from simpler tasks to be tackled efficiently. The range of tasks those robots need to learn can be wide and even change after the deployment of the robot, we are therefore taking inspiration from the field of developmental psychology to give the robot the ability to learn. Taking a developmental robotic approach BID13, we combine the approaches of active motor skill learning of multiple tasks, interactive learning and strategical learning into a new learning algorithm and we show its capability to learn a mapping between a continuous space of parametrized outcomes (sometimes referred to as tasks) and a space of parametrized motor policies (sometimes referred to as actions). Classical techniques based on Reinforcement Learning BID19 BID21 still need an engineer to manually design a reward function for each particular task. Intrinsic motivation, which triggers curiosity in human beings according to developmental psychology BID8, inspired knowledge-based algorithms BID17. The external reward is replaced by a surprise function: the learner is driven by unexpected outcomes BID4. However those approaches encounter limitations when the sensorimotor dimensionality increases. Another approach using competence progress measures successfully drove the learner's exploration through goal-babbling BID2 BID20.However when the dimension of the outcome space increases, these methods become less efficient BID3 due to the curse of dimensionality, or when the reachable space of the robot is small compared to its environment. To enable robots to learn a wide range of tasks, and even an infinite number of tasks defined in a continuous space, heuristics such as social guidance can help by driving its exploration towards interesting and reachable space fast. Combining intrinsically motivated learning and imitation BID16 has bootstrapped exploration by providing efficient human demonstrations of motor policies and outcomes. Also, such a learner has been shown to be more efficient if requesting actively a human for help when needed instead of being passive, both from the learner or the teacher perspective BID6. This approach is called interactive learning and it enables a learner to benefit from both local exploration and learning from demonstration. Information could be provided to the robot using external reinforcement signals , actions BID11, advice operators BID0, or disambiguation among actions BID7. Another advantage of introducing imitation learning techniques is to include non-robotic experts in the learning process BID7. One of the key element of these hybrid approaches is to choose when to request human information or learn in autonomy such as to diminish the teacher's attendance. Approaches enabling the learner to choose either what to learn (which outcome to focus on) or how to learn (which strategy to use such as imitation) are called strategic learning BID12. They aim at giving an autonomous learner the capability to self-organize its learning process. Some work has been done to enable a learner to choose on which task space to focus. The SAGG-RIAC algorithm BID2, by self-generating goal outcomes fulfills this objective. Other approaches focused on giving the learner the ability to change its strategy BID1 and showed it could be more efficient than each strategy taken alone. Fewer studies have been made to enable a learner to choose both its strategy and its target outcome. The problem was introduced and studied in BID12, and was implemented for an infinite number of outcomes and policies in continuous spaces by the SGIM-ACTS algorithm BID14. This algorithm is capable of organizing its learning process, by choosing actively both which strategy to use and which outcome to focus on. It relies on the empirical evaluation of its learning progress. It could choose among autonomous exploration driven by intrinsic motivation and low-level imitation of one of the available human teachers to learn more efficiently. It showed its potential to learn on a real high dimensional robot a set of hierarchically organized tasks BID9, which is why we consider it to learn complex motor policies. In this article, we tackle the learning of complex motor policies, which we define as combinations of simpler policies. We describe it more concretely as a sequence of primitive policies. A first approach to learning complex motor policies is to use via-points BID21. Via-points enable the definition of complex motor policies. Those via-points are in the robot motor policy space. When increasing the size of the complex policy (by chaining more primitive actions together), we can tackle more complex tasks. However, this would increase the difficulty for the learner to tackle simpler tasks which would be reachable using less complex policies. Enabling the learner to decide autonomously the complexity of the policy necessary to solve a task would allow the approach to be adaptive, and suitable to a greater number of problems. Options BID22 are a different way to tackle this problem by offering temporally abstract actions to the learner. However each option is built to reach one particular task and they have only been tested for discrete tasks and actions, in which a small number of options were used, whereas our new proposed learner is to be able to create an unlimited number of complex policies. As we aim at learning a hierarchical set of interrelated complex tasks, our algorithm could also benefit from this task hierarchy (as BID10 did for learning tool use with simple primitive policies only), and try to reuse previously acquired skills to build more complex ones. BID5 showed that building complex actions made of lower-level actions according to the task hierarchy can bootstrap exploration by reaching interesting outcomes more rapidly. In this paper, we would like to enable a robot learner to achieve a wide range of tasks that can be inter-related and complex. We allow the robot to use sequences of actions of undetermined length to achieve these tasks. We propose an algorithm for the robot to learn which sequence of actions to use to achieve any task in the infinite field of tasks. The learning algorithm has to face the problem of unlearnability of infinite task and policy spaces, and the curse of dimensionality of sequences of high-dimensionality policy spaces. Thus, we extend SGIM-ACTS so as to learn complex motor policies of unlimited size. We develop a new framework called "procedures" (see Section 2.2) which proposes to combine known policies according to their outcome. Combining these, we developed a new algorithm called Socially Guided Intrinsic Motivation with Procedure Babbling (SGIM-PB) capable of taking task hierarchy into account to learn a set of complex interrelated tasks using adapted complex policies. We will describe an experiment, on which we have tested our algorithm, and we will present and analyze the . Inspired by developmental psychology, we combine interactive learning and autonomous exploration in a strategic learner, which learning process is driven by intrinsic motivation. This learner also takes task hierarchy into account to reuse its previously learned skills while adapting the complexity of its policy to the complexity of the task at hand. In this section, we formalize our learning problem and explain the principles of SGIM-PB. In our approach, an agent can perform motions through the use of policies π θ, parametrized by θ ∈ Π, and those policies have an effect on the environment, which we call the outcome ω ∈ Ω. The agent is then to learn the mapping between Π and Ω: it learns to predict the outcome ω of each policy π θ (the forward model M), but more importantly, it learns which policy to choose for reaching any particular outcome (an inverse model L). The outcomes ω can be of different dimensionality and thus be split in subspaces Ω i ⊂ Ω. The policies do not only consist of one primitive but of a succession of primitives (each encoded by the parameters in Π) that are executed sequentially by the agent. Hence, policies also are of different dimensionality and are split in policy spaces Π i ⊂ Π (where i corresponds to the number of primitives used in each policy of Π i). Complex policies are represented by concatenating the parameters of each of its primitive policies in execution order. We take the trial and error approach, and suppose that Ω is a metric space, meaning the learner has a means of evaluating a distance between two outcomes d(ω 1, ω 2). As this algorithm tackles the learning of complex hierarchically organized tasks, exploring and exploiting this hierarchy could ease the learning of the more complex tasks. We define procedures as a way to encourage the robot to reuse previously learned skills, and chain them to build more complex ones. More formally, a procedure is built by choosing previously known outcomes (t 1, t 2, ..., t n ∈ Ω) and is noted t 1 t 1... t n.Executing a procedure t 1 t 1... t n means building the complex policy π θ corresponding to the succession of policies π θi, i ∈ 1, n (potentially complex as well) and execute it (where the π θi reach best the t i ∀i ∈ 1, n respectively). As the subtasks t i are generally unknown from the learner, the procedure is updated before execution (see Algo. 1) to subtasks t i which are the closest known by the learner according to its current skill set (by executing respectively π θ 1 to π θ n). When the agent selects a procedure to be executed, this latter is only a way to build the complex policy which will actually be executed. So the agent does not check if the subtasks are actually reached when executing a procedure. Algorithm 1 Procedure modification before execution DISPLAYFORM0 Outcome & The SGIM-PB algorithm (see Algo. 2, FIG0) learns by episodes, where an outcome ω g ∈ Ω to target and an exploration strategy σ have been selected. It is an extension of SGIM-ACTS BID14, which can perform complex motor policies and size 2 procedures (sequences of 2 subtasks only). It uses the same interest model and memory based inverse model. In an episode under the policy space exploration strategy, the learner tries to optimize the policy π θ to produce ω g by choosing between random exploration of policies and local optimization, following the SAGG-RIAC algorithm BID2 ) (Goal-Directed Policy Optimization(ω g)).Local optimization uses local linear regression. This is a slightly modified version of the SGIM-ACTS autonomous exploration strategy. In an episode under the procedural space exploration strategy, the learner builds a size 2 procedure t i t j such as to reproduce the goal outcome ω g the best (Goal-Directed Procedure Optimization(ω g)). It chooses either random exploration of procedures (which builds procedures by generating two subtasks at random) when the goal outcome is far from any previously reached one, or local procedure optimization, which optimizes a procedure using local linear regression. The procedure built is then modified and executed, following Algo. 1.In an episode under the mimicry of one policy teacher strategy, the learner requests a demonstration ξ d from the chosen teacher. ξ d is selected by the teacher as the closest from the goal outcome ω g in its demonstration repertoire. The learner then repeats the demonstrated policy (Mimic Policy(ξ d)). It is a strategy directly also available in the SGIM-ACTS algorithm. In an episode under the mimicry of one procedural teacher strategy, the learner requests a procedural demonstration of size 2 t di t dj which is built by the chosen teacher according to a preset function which depends on the target outcome ω g. Then the learner tries to reproduce the demonstrated procedure by refining and executing it, following Algo. 1 (Mimic Procedure(t di t dj)).In both autonomous exploration strategies, the learner uses a method, Goal-Directed Optimization, to optimize its input parameters (procedure for the procedure exploration and policy for the policy exploration) to reach ω g best. This generic method works similarly in both cases, by creating random inputs, if the goal outcome ω g is far from any previously reached one, or local optimization which uses linear regression. After each episode, the learner stores the policies and modified procedures executed along with their reached outcomes in its episodic memory. It computes its competence in reaching the goal outcome ω g by computing the distance d(ω r, ω g)) with the outcome ω r it actually reached. Then it updates its interest model by computing the interest interest(ω, σ) of the goal outcome and each outcome reached (including the outcome spaces reached but not targeted): DISPLAYFORM0 where K(σ) is the cost of the strategy used and the progress p(ω) is the derivate of the competence. The learning agent then uses these interest measures to partition the outcome space Ω into regions of high and low interest. For each strategey σ, the outcomes reached and the goal are added to their partition region. Over a fixed number of measures of interest in the region, it is then partitioned into 2 subregions so as maximise the difference in interest between the 2 subregions. The method used is detailed in BID15. Thus, the learning agent discovers by itself how to organise its learning process and partition its task space into unreachable regions, easy regions and difficult regions, based on empirical measures of competence and interest. The choice of strategy and goal outcome is based on the empirical progress measured in each region R n of the outcome space Ω. ω g, σ are chosen stochastically (with respectively probabilities p 1, p 2, p 3), by one of the sampling modes:• mode 1: choose σ and ω g ∈ Ω at random; • mode 2: choose an outcome region R n and a strategy σ with a probability proportional to its interest value. Then generate ω g ∈ R n at random; • mode 3: choose σ and R n like in mode 2, but generate a goal ω g ∈ R n close to the outcome with the highest measure of progress. When the learner computes nearest neighbours to select policies or procedures to optimize (when choosing local optimization in any of both autonomous exploration strategies and when refining procedures), it actually uses a performance metric which takes into account the complexity of the policy chosen: DISPLAYFORM1 where d(ω, ω g) is the normalized Euclidean distance between the target outcome ω g and the outcome ω reached by the policy, γ is a constant and n is equal to the size of the policy (the number of primitives chained).In this section, we have formalized the problem of learning an inverse model between an infinite space of outcomes and an infinite space of policies. We have introduced the framework of procedures to allow the learning agent to learn sequences of primitive policies as task compositions. We have then proposed SGIM-PB as a learning algorithm that leverages goal-babbling for autonomous exploration, sequences to learn complex policies, and social guidance to bootstrap the learning. SGIM-PB learns to reach an ensemble of outcomes, by mapping them with policies, but also with subgoal outcomes. The formalization and algorithmic architecture proposed are general and can apply to a high number of problems. The requirements for an experimental setup are:• to define the primitive policies of the robot• to define the different outcomes the user is interested in by defining the variables from the sensors needed and a rough range of their values (we do not need a precise estimation as the algorithm is robust to overestimations of these ranges, see Nguyen & Oudeyer FORMULA2).• a measure for the robot to assess its own performance such as a distance, as in all intrinsic motivation based algorithms. This measure is used as an internal reward function. Contrarily to classical reinforcement learning problems, this reward function is not fine tuned to the specific goal at hand, but is a generic function for all the goals in the outcome space.• the environment and robot can reset to an initial state, as in most reinforcement learning algorithms. In this study, we designed an experiment with a simulated robotic arm, which can move in its environment and interact with objects in it. It can learn an infinite number of tasks, organized as 6 hierarchically organized types of tasks. The robot is capable of performing complex policies of unrestricted size (i.e. consisting of any number of primitives), with primitive policies highly redundant and of high dimensionality. The FIG1 shows environmental setup (contained in a cube delimited by (x, y, z) ∈ [−1; 1] 3 ). The learning agent is a planar robotic arm of 3 joints with the base centred on the horizontal plane, able to rotate freely around the vertical axis (each link has a length of 0.33) and change its vertical position. The robot can grab objects in this environment, by hovering its arm tip (blue in the FIG1 close to them, which position is noted (x 0, y 0, z 0). The robot can interact with:• Floor (below z = 0.0): limits the motions of the robot, slightly elastic which enable the robot to go down to z = −0.2 by forcing on it;• Pen: can be moved around and draw on the floor, broken if forcing too much on the floor (when z <= −0.3);• Joystick 1 (the left one on the figure): can be moved inside a cube-shaped area (automatically released otherwise, position normalized for this area), its x-axis position control a video-game character x position on the screen when grabbed by the robot; DISPLAYFORM0 Figure 2: Experimental setup: a robotic arm, can interact with the different objects in its environment (a pen and two joysticks). Both joysticks enable to control a video-game character (represented in top-right corner). A grey floor limits its motions and can be drawn upon using the pen (a possible drawing is represented).• Joystick 2 (the right one on the figure): can be moved inside a cube-shaped area (automatically released otherwise, position normalized for this area), its y-axis position control a video-game character y position on the screen when grabbed by the robot;• Video-game character: can be moved on the screen by using the two joysticks, its position is refreshed only at the end of a primitive policy execution for the manipulated joystick. The robot grabber can only handle one object. When it touches a second object, it breaks, releasing both objects. The robot always starts from the same position before executing a policy, and primitives are executed sequentially without getting back to this initial position. Whole complex policies are recorded with their outcomes, but each step of the complex policy execution is recorded. The motions of each of the three joints of the robot are encoded using a one-dimensional Dynamic Movement Primitive BID18, defined by the system: DISPLAYFORM0 where x and v are the position and velocity of the system; s is the phase of the motion; x 0 and g are the starting and end position of the motion; τ is a factor used to temporally scale the system (set to fix the length of a primitive execution); K and D are the spring constant and damping term fixed for the whole experiment; α is also a constant fixed for the experiment; and f is a non-linear term used to shape the trajectory called the forcing term. This forcing term is defined as: DISPLAYFORM1 where ψ i (s) = exp(−h i (s − c i) 2 ) with centers c i and widths h i fixed for all primitives. There are 3 weights ω i per DMP.The weights of the forcing term and the end positions are the only parameters of the DMP used by the robot. The starting position of a primitive is set by either the initial position of the robot (if it is starting a new complex policy) or the end position of the preceding primitive. The robot can also set its position on the vertical axis z for every primitive. Therefore a primitive policy π θ is parametrized by: DISPLAYFORM2 where DISPLAYFORM3 ) corresponds to the DMP parameters of the joint i, ordered from base to tip, and z is the fixed vertical position. When combining two or more primitive policies (π θ0, π θ1, ...), in a complex policies π θ, the parameters (θ 0, θ 1, ...) are simply concatenated together from the first primitive to the last. The outcome subspaces the robot learns to reach are hierarchically organized and defined as:• Ω 0: the position (x 0, y 0, z 0) of the end effector of the robot in Cartesian coordinates at the end of a policy execution; • Ω 1: the position (x 1, y 1, z 1) of the pen at the end of a policy execution if the pen is grabbed by the robot; • Ω 2: the first (x a, y a) and last (x b, y b) points of the last drawn continuous line on the floor if the pen is functional (x a, y a, x b, y b); • Ω 3: the position (x 3, y 3, z 3) of the first joystick at the end of a policy execution if it is grabbed by the robot; • Ω 4: the position (x 4, y 4, z 4) of the second joystick at the end of a policy execution if it is grabbed by the robot; • Ω 5: the position (x 5, y 5) of the video-game character at the end of a policy execution if moved. To help the SGIM-PB learner, procedural teachers were available so as to provide procedures for every outcome subspaces but Ω 0. Each teacher was only giving procedures useful for its own outcome space, and was aware of the task representation. They all had a cost of 5. The rules used to provide procedures are the following:• ProceduralTeacher1 (Ω 1): t 1 t 0 with t 1 ∈ Ω 1 corresponding to the pen initial position and t 0 ∈ Ω 0 corresponding to the desired final pen position; • ProceduralTeacher2 (Ω 2): t 1 t 0 with t 1 ∈ Ω 1 corresponding to the point on the z = 1.0 plane above the first point of the desired drawing, and t 0 ∈ Ω 0 corresponding to the desired final drawing point; • ProceduralTeacher3 (Ω 3): t 3 t 0 with t 3 ∈ Ω 3 and t 3 =, t 0 ∈ Ω 0 corresponding to the end effector position corresponding to the desired final position of the first joystick; • ProceduralTeacher4 (Ω 4): t 4 t 0 with t 4 ∈ Ω 4 and t 4 =, t 0 ∈ Ω 0 corresponding to the end effector position corresponding to the desired final position of the second joystick;• ProceduralTeacher5 (Ω 5): t 3 t 4 with t 3 ∈ Ω 3 and t 3 = (x, 0, 0) with x corresponding to the desired x-position of the video-game character, t 4 ∈ Ω 4 and t 4 = (0, y, 0) with y corresponding to the desired y-position of the video-game character. We also added policy teachers corresponding to the same outcome spaces to bootstrap the robot early learning process. The strategy attached to each teacher has a cost of 10. Each teacher was capable to provide demonstrations (as policies executable by the robot) linearly distributed in its outcome space: To evaluate our algorithm, we created a benchmark dataset for each outcome space Ω i, linearly distributed across the outcome space dimensions, for a total of 27,600 points. The evaluation consists in computing the normalized Euclidean distance between each of the benchmark outcome and their nearest neighbour in the learner dataset. Then we compute the mean distance to benchmark for each outcome space. The global evaluation is the mean evaluation for the 6 outcome spaces. This process is then repeated across the learning process at predefined and regularly distributed timestamps. DISPLAYFORM0 Then to asses our algorithm efficiency, we compare its with 3 other algorithms:• SAGG-RIAC: performs autonomous exploration of the policy space Π guided by intrinsic motivation; • SGIM-ACTS: interactive learner driven by intrinsic motivation. Choosing between autonomous exploration of the policy space Π and mimicry of one of the available policy teachers; • IM-PB: performs both autonomous exploration of the procedural space and the policy space, guided by intrinsic motivation; • SGIM-PB: interactive learner driven by intrinsic motivation. Choosing between autonomous exploration strategies (either of the policy space or the procedural space) and mimicry of one of the available teachers (either policy or procedural teachers).Each algorithm was run 5 times on this setup. Each run, we let the algorithm performs 25,000 iterations (complex policies executions). The value of γ for this experiment is 1.2. The probabilities to choose either of the sampling mode of SGIM-PB are p 1 = 0.15, p 2 = 0.65, p 3 = 0.2. The code run for this experiment can be found here. The Fig. 3 shows the global evaluation of all the tested algorithms, which corresponds to the mean error made by each algorithm to reproduce the benchmarks with respect to the number of complete complex policies tried. The algorithms capable of performing procedures (IM-PB and SGIM-PB) have errors that drop to levels lower than the their non-procedure equivalents (respectively SAGG-RIAC and SGIM-ACTS), moreover since the beginning of the learning process (shown on Fig. 3). It seems that the procedures bootstrap the exploration, enabling the learner to progress further. Indeed, the autonomous learner SAGG-RIAC has significantly better performance when it can use procedures and is thus upgraded to IM-PB.We can also see that the SGIM-PB algorithm has a very quick improvement in global evaluation owing to the bootstrapping effect of the different teachers. It goes lower to the final evaluation of SAGG-RIAC (0.17) after only 500 iterations. This bootstrapping effect comes from the mimicry teachers, as it is also observed for SGIM-ACTS which shares the same mimicry teachers. If we look at the evaluation on each individual outcome space (Fig. 4), we can see that the learners with demonstrations (SGIM-PB and SGIM-ACTS) outperforms the other algorithms, except for the outcome space Ω 5 where IM-PB is better, due to the fact that IM-PB practiced much more on this outcome space (500 iterations where the goal was in Ω 5 against 160 for SGIM-PB) on this outcome space. SGIM-PB and SGIM-ACTS are much better than the other algorithms on the two joysticks outcome spaces (Ω 3 and Ω 4). This is not surprising given the fact that those outcome spaces require precise policies. Indeed, if the end-effector gets out of the area where it can control the joystick, the latter is released, thus potentially ruining the attempt. So on these outcome spaces working directly on carefully crafted policies can alleviate this problem, while using procedures might be tricky, as the outcomes used don't take into account the motion trajectory but merely its final state. SGIM-PB was provided with such policies by the policy teachers. Also if we compare the of the autonomous learner without procedures (SAGG-RIAC) with the one with procedures (IM-PB), we can see that it learn less on any outcome space but Ω 0 (which was the only outcome space reachable using only single primitive policies and that could not benefit from using the task hierarchy to be learned) and especially for Ω 1, Ω 2 and Ω 5 which were the most hierarchical in this setup. More generally, it seems than on this highly hierarchical Ω 5, the learners with procedures were better. So the procedures helped when learning any potentially hierarchical task in this experiment. We further analyzed the of our SGIM-PB learner. We looked in its learning process to see which pairs of teachers and target outcomes it has chosen (Fig. 5). It was capable to request demonstrations from the relevant teachers depending on the task at hand, except for the outcome space Ω 0 which had no human teachers and therefore could not find a better teacher to help it. Indeed, for the outcome space Ω 2, the procedural teacher (ProceduralTeacher2) specially built for this outcome space was greatly chosen. Figure 5: Choices of teachers and target outcomes of the SGIM-PB learnerWe wanted to see if our SGIM-PB learner adapts the complexity of its policies to the working task. We draw 1,000,000 goal outcomes for each of the Ω 0, Ω 1 and Ω 2 subspaces (chosen because they are increasingly complex) and we let the learner choose the known policy that would reach the closest outcome. Fig. 6 shows the of this analysis. Figure 6: Number of policies selected per policy size for three increasingly more complex outcome spaces by the SGIM-PB learnerAs we can see on those three interrelated outcome subspaces (Fig. 6), the learner is capable to adapt the complexity of its policies to the outcome at hand. It chooses longer policies for the Ω 1 subspace (policies of size 2 and 3 while using mostly policies of size 1 and 2 for Ω 0) and even longer for the Ω 2 subspace (using far more policies of size 3 than for the others). It shows that our learner is capable to correctly limit the complexity of its policies instead of being stuck into always trying longer and longer policies. With this experiment, we show the capability of SGIM-PB to tackle the learning of a set of multiple interrelated complex tasks. It successfully discovers the hierarchy between tasks and uses complex motor policies to learn a wider range of tasks. It is capable to correctly choose the most adapted teachers to the target outcome when available. Though it is not limited in the size of policies it could execute, the learner shows it could adapt the complexity of its policies to the task at hand. The procedures greatly improved the learning capability of autonomous learners, as shows the difference between IM-PB and SAGG-RIAC. Our SGIM-PB shows it is capable to use procedures to discover the task hierarchy and exploit the inverse model of previously learned skills. More importantly, it shows it can successfully combine the ability of SGIM-ACTS to progress quickly in the beginning (owing to the mimicry teachers) and the ability of IM-PB to progress further on highly hierarchical tasks (owing to the procedure framework).In this article, we aimed to enable a robot to learn sequences of actions of undetermined length to achieve a field of outcomes. To tackle this high-dimensionality learning between a continuous high-dimensional space of outcomes and a continuous infinite dimensionality space of sequences of actions, we used techniques that have proven efficient in previous studies: goal-babbling, social guidance and strategic learning based on intrinsic motivation. We extended them with the procedures framework and proposed SGIM-PB algorithm, allowing the robot to babble in the procedure space and to imitate procedural teachers. We showed that SGIM-PB can discover the hierarchy between tasks, learn to reach complex tasks while adapting the complexity of the policy. The study shows that:• procedures allow the learner to learn complex tasks, and adapt the length of sequences of actions to the complexity of the task • social guidance bootstraps the learning owing to demonstrations of primitive policy in the beginning, and then to demonstrations of procedures to learn how to compose tasks into sequences of actions • intrinsic motivation can be used as a common criteria for active learning for the robot to choose both its exploration strategy, its goal outcomes and the goal-oriented procedures. However a precise analysis of the impact of each of the different strategies used by our learning algorithm could give us more insight in the roles of the teachers and procedures framework. Also, we aim to illustrate the potency of our SGIM-PB learner on a real-world application. We are currently designing such an experiment with a real robotic platform. Besides, the procedures are defined as combinations of any number of subtasks but are used in the illustration experiment as only combinations of two subtasks. It could be a next step to see if the learning algorithm can handle the curse of dimensionality of a larger procedure space, and explore combinations of any number of subtasks. Moreover, the algorithm can be extended to allow the robot learner to decide on how to execute a procedure. In the current version, we have proposed the "refinement process" to infer the best policy. We could make this refinement process more recursive, by allowing the algorithm to select, not only policies, but also lower-level procedures as one of the policy components.

LSTMs learn long-range dependencies compositionally by building them from shorter constituents over the course of training.