Question: Represent the scientific article to find a short TLDR

Question: We propose an active learning algorithmic architecture, capable of organizing its learning process in order to achieve a field of complex tasks by learning sequences of primitive motor policies: Socially Guided Intrinsic Motivation with Procedure Babbling (SGIM-PB). The learner can generalize over its experience to continuously learn new outcomes, by choosing actively what and how to learn guided by empirical measures of its own progress. In this paper, we are considering the learning of a set of interrelated complex outcomes hierarchically organized. We introduce a new framework called "procedures", which enables the autonomous discovery of how to combine previously learned skills in order to learn increasingly more complex motor policies (combinations of primitive motor policies). Our architecture can actively decide which outcome to focus on and which exploration strategy to apply. Those strategies could be autonomous exploration, or active social guidance, where it relies on the expertise of a human teacher providing demonstrations at the learner's request. We show on a simulated environment that our new architecture is capable of tackling the learning of complex motor policies, to adapt the complexity of its policies to the task at hand. We also show that our "procedures" increases the agent's capability to learn complex tasks. Recently, efforts in the robotic industry and academic field have been made for integrating robots in previously human only environments. In such a context, the ability for service robots to continuously learn new skills, autonomously or guided by their human counterparts, has become necessary. They would be needed to carry out multiple tasks, especially in open environments, which is still an ongoing challenge in robotic learning. Those tasks can be independent and self-contained but they can also be complex and interrelated, needing to combine learned skills from simpler tasks to be tackled efficiently. The range of tasks those robots need to learn can be wide and even change after the deployment of the robot, we are therefore taking inspiration from the field of developmental psychology to give the robot the ability to learn. Taking a developmental robotic approach BID13, we combine the approaches of active motor skill learning of multiple tasks, interactive learning and strategical learning into a new learning algorithm and we show its capability to learn a mapping between a continuous space of parametrized outcomes (sometimes referred to as tasks) and a space of parametrized motor policies (sometimes referred to as actions). Classical techniques based on Reinforcement Learning BID19 BID21 still need an engineer to manually design a reward function for each particular task. Intrinsic motivation, which triggers curiosity in human beings according to developmental psychology BID8, inspired knowledge-based algorithms BID17. The external reward is replaced by a surprise function: the learner is driven by unexpected outcomes BID4. However those approaches encounter limitations when the sensorimotor dimensionality increases. Another approach using competence progress measures successfully drove the learner's exploration through goal-babbling BID2 BID20.However when the dimension of the outcome space increases, these methods become less efficient BID3 due to the curse of dimensionality, or when the reachable space of the robot is small compared to its environment. To enable robots to learn a wide range of tasks, and even an infinite number of tasks defined in a continuous space, heuristics such as social guidance can help by driving its exploration towards interesting and reachable space fast. Combining intrinsically motivated learning and imitation BID16 has bootstrapped exploration by providing efficient human demonstrations of motor policies and outcomes. Also, such a learner has been shown to be more efficient if requesting actively a human for help when needed instead of being passive, both from the learner or the teacher perspective BID6. This approach is called interactive learning and it enables a learner to benefit from both local exploration and learning from demonstration. Information could be provided to the robot using external reinforcement signals , actions BID11, advice operators BID0, or disambiguation among actions BID7. Another advantage of introducing imitation learning techniques is to include non-robotic experts in the learning process BID7. One of the key element of these hybrid approaches is to choose when to request human information or learn in autonomy such as to diminish the teacher's attendance. Approaches enabling the learner to choose either what to learn (which outcome to focus on) or how to learn (which strategy to use such as imitation) are called strategic learning BID12. They aim at giving an autonomous learner the capability to self-organize its learning process. Some work has been done to enable a learner to choose on which task space to focus. The SAGG-RIAC algorithm BID2, by self-generating goal outcomes fulfills this objective. Other approaches focused on giving the learner the ability to change its strategy BID1 and showed it could be more efficient than each strategy taken alone. Fewer studies have been made to enable a learner to choose both its strategy and its target outcome. The problem was introduced and studied in BID12, and was implemented for an infinite number of outcomes and policies in continuous spaces by the SGIM-ACTS algorithm BID14. This algorithm is capable of organizing its learning process, by choosing actively both which strategy to use and which outcome to focus on. It relies on the empirical evaluation of its learning progress. It could choose among autonomous exploration driven by intrinsic motivation and low-level imitation of one of the available human teachers to learn more efficiently. It showed its potential to learn on a real high dimensional robot a set of hierarchically organized tasks BID9, which is why we consider it to learn complex motor policies. In this article, we tackle the learning of complex motor policies, which we define as combinations of simpler policies. We describe it more concretely as a sequence of primitive policies. A first approach to learning complex motor policies is to use via-points BID21. Via-points enable the definition of complex motor policies. Those via-points are in the robot motor policy space. When increasing the size of the complex policy (by chaining more primitive actions together), we can tackle more complex tasks. However, this would increase the difficulty for the learner to tackle simpler tasks which would be reachable using less complex policies. Enabling the learner to decide autonomously the complexity of the policy necessary to solve a task would allow the approach to be adaptive, and suitable to a greater number of problems. Options BID22 are a different way to tackle this problem by offering temporally abstract actions to the learner. However each option is built to reach one particular task and they have only been tested for discrete tasks and actions, in which a small number of options were used, whereas our new proposed learner is to be able to create an unlimited number of complex policies. As we aim at learning a hierarchical set of interrelated complex tasks, our algorithm could also benefit from this task hierarchy (as BID10 did for learning tool use with simple primitive policies only), and try to reuse previously acquired skills to build more complex ones. BID5 showed that building complex actions made of lower-level actions according to the task hierarchy can bootstrap exploration by reaching interesting outcomes more rapidly. In this paper, we would like to enable a robot learner to achieve a wide range of tasks that can be inter-related and complex. We allow the robot to use sequences of actions of undetermined length to achieve these tasks. We propose an algorithm for the robot to learn which sequence of actions to use to achieve any task in the infinite field of tasks. The learning algorithm has to face the problem of unlearnability of infinite task and policy spaces, and the curse of dimensionality of sequences of high-dimensionality policy spaces. Thus, we extend SGIM-ACTS so as to learn complex motor policies of unlimited size. We develop a new framework called "procedures" (see Section 2.2) which proposes to combine known policies according to their outcome. Combining these, we developed a new algorithm called Socially Guided Intrinsic Motivation with Procedure Babbling (SGIM-PB) capable of taking task hierarchy into account to learn a set of complex interrelated tasks using adapted complex policies. We will describe an experiment, on which we have tested our algorithm, and we will present and analyze the . Inspired by developmental psychology, we combine interactive learning and autonomous exploration in a strategic learner, which learning process is driven by intrinsic motivation. This learner also takes task hierarchy into account to reuse its previously learned skills while adapting the complexity of its policy to the complexity of the task at hand. In this section, we formalize our learning problem and explain the principles of SGIM-PB. In our approach, an agent can perform motions through the use of policies π θ, parametrized by θ ∈ Π, and those policies have an effect on the environment, which we call the outcome ω ∈ Ω. The agent is then to learn the mapping between Π and Ω: it learns to predict the outcome ω of each policy π θ (the forward model M), but more importantly, it learns which policy to choose for reaching any particular outcome (an inverse model L). The outcomes ω can be of different dimensionality and thus be split in subspaces Ω i ⊂ Ω. The policies do not only consist of one primitive but of a succession of primitives (each encoded by the parameters in Π) that are executed sequentially by the agent. Hence, policies also are of different dimensionality and are split in policy spaces Π i ⊂ Π (where i corresponds to the number of primitives used in each policy of Π i). Complex policies are represented by concatenating the parameters of each of its primitive policies in execution order. We take the trial and error approach, and suppose that Ω is a metric space, meaning the learner has a means of evaluating a distance between two outcomes d(ω 1, ω 2). As this algorithm tackles the learning of complex hierarchically organized tasks, exploring and exploiting this hierarchy could ease the learning of the more complex tasks. We define procedures as a way to encourage the robot to reuse previously learned skills, and chain them to build more complex ones. More formally, a procedure is built by choosing previously known outcomes (t 1, t 2, ..., t n ∈ Ω) and is noted t 1 t 1... t n.Executing a procedure t 1 t 1... t n means building the complex policy π θ corresponding to the succession of policies π θi, i ∈ 1, n (potentially complex as well) and execute it (where the π θi reach best the t i ∀i ∈ 1, n respectively). As the subtasks t i are generally unknown from the learner, the procedure is updated before execution (see Algo. 1) to subtasks t i which are the closest known by the learner according to its current skill set (by executing respectively π θ 1 to π θ n). When the agent selects a procedure to be executed, this latter is only a way to build the complex policy which will actually be executed. So the agent does not check if the subtasks are actually reached when executing a procedure. Algorithm 1 Procedure modification before execution DISPLAYFORM0 Outcome & The SGIM-PB algorithm (see Algo. 2, FIG0) learns by episodes, where an outcome ω g ∈ Ω to target and an exploration strategy σ have been selected. It is an extension of SGIM-ACTS BID14, which can perform complex motor policies and size 2 procedures (sequences of 2 subtasks only). It uses the same interest model and memory based inverse model. In an episode under the policy space exploration strategy, the learner tries to optimize the policy π θ to produce ω g by choosing between random exploration of policies and local optimization, following the SAGG-RIAC algorithm BID2 ) (Goal-Directed Policy Optimization(ω g)).Local optimization uses local linear regression. This is a slightly modified version of the SGIM-ACTS autonomous exploration strategy. In an episode under the procedural space exploration strategy, the learner builds a size 2 procedure t i t j such as to reproduce the goal outcome ω g the best (Goal-Directed Procedure Optimization(ω g)). It chooses either random exploration of procedures (which builds procedures by generating two subtasks at random) when the goal outcome is far from any previously reached one, or local procedure optimization, which optimizes a procedure using local linear regression. The procedure built is then modified and executed, following Algo. 1.In an episode under the mimicry of one policy teacher strategy, the learner requests a demonstration ξ d from the chosen teacher. ξ d is selected by the teacher as the closest from the goal outcome ω g in its demonstration repertoire. The learner then repeats the demonstrated policy (Mimic Policy(ξ d)). It is a strategy directly also available in the SGIM-ACTS algorithm. In an episode under the mimicry of one procedural teacher strategy, the learner requests a procedural demonstration of size 2 t di t dj which is built by the chosen teacher according to a preset function which depends on the target outcome ω g. Then the learner tries to reproduce the demonstrated procedure by refining and executing it, following Algo. 1 (Mimic Procedure(t di t dj)).In both autonomous exploration strategies, the learner uses a method, Goal-Directed Optimization, to optimize its input parameters (procedure for the procedure exploration and policy for the policy exploration) to reach ω g best. This generic method works similarly in both cases, by creating random inputs, if the goal outcome ω g is far from any previously reached one, or local optimization which uses linear regression. After each episode, the learner stores the policies and modified procedures executed along with their reached outcomes in its episodic memory. It computes its competence in reaching the goal outcome ω g by computing the distance d(ω r, ω g)) with the outcome ω r it actually reached. Then it updates its interest model by computing the interest interest(ω, σ) of the goal outcome and each outcome reached (including the outcome spaces reached but not targeted): DISPLAYFORM0 where K(σ) is the cost of the strategy used and the progress p(ω) is the derivate of the competence. The learning agent then uses these interest measures to partition the outcome space Ω into regions of high and low interest. For each strategey σ, the outcomes reached and the goal are added to their partition region. Over a fixed number of measures of interest in the region, it is then partitioned into 2 subregions so as maximise the difference in interest between the 2 subregions. The method used is detailed in BID15. Thus, the learning agent discovers by itself how to organise its learning process and partition its task space into unreachable regions, easy regions and difficult regions, based on empirical measures of competence and interest. The choice of strategy and goal outcome is based on the empirical progress measured in each region R n of the outcome space Ω. ω g, σ are chosen stochastically (with respectively probabilities p 1, p 2, p 3), by one of the sampling modes:• mode 1: choose σ and ω g ∈ Ω at random; • mode 2: choose an outcome region R n and a strategy σ with a probability proportional to its interest value. Then generate ω g ∈ R n at random; • mode 3: choose σ and R n like in mode 2, but generate a goal ω g ∈ R n close to the outcome with the highest measure of progress. When the learner computes nearest neighbours to select policies or procedures to optimize (when choosing local optimization in any of both autonomous exploration strategies and when refining procedures), it actually uses a performance metric which takes into account the complexity of the policy chosen: DISPLAYFORM1 where d(ω, ω g) is the normalized Euclidean distance between the target outcome ω g and the outcome ω reached by the policy, γ is a constant and n is equal to the size of the policy (the number of primitives chained).In this section, we have formalized the problem of learning an inverse model between an infinite space of outcomes and an infinite space of policies. We have introduced the framework of procedures to allow the learning agent to learn sequences of primitive policies as task compositions. We have then proposed SGIM-PB as a learning algorithm that leverages goal-babbling for autonomous exploration, sequences to learn complex policies, and social guidance to bootstrap the learning. SGIM-PB learns to reach an ensemble of outcomes, by mapping them with policies, but also with subgoal outcomes. The formalization and algorithmic architecture proposed are general and can apply to a high number of problems. The requirements for an experimental setup are:• to define the primitive policies of the robot• to define the different outcomes the user is interested in by defining the variables from the sensors needed and a rough range of their values (we do not need a precise estimation as the algorithm is robust to overestimations of these ranges, see Nguyen & Oudeyer FORMULA2).• a measure for the robot to assess its own performance such as a distance, as in all intrinsic motivation based algorithms. This measure is used as an internal reward function. Contrarily to classical reinforcement learning problems, this reward function is not fine tuned to the specific goal at hand, but is a generic function for all the goals in the outcome space.• the environment and robot can reset to an initial state, as in most reinforcement learning algorithms. In this study, we designed an experiment with a simulated robotic arm, which can move in its environment and interact with objects in it. It can learn an infinite number of tasks, organized as 6 hierarchically organized types of tasks. The robot is capable of performing complex policies of unrestricted size (i.e. consisting of any number of primitives), with primitive policies highly redundant and of high dimensionality. The FIG1 shows environmental setup (contained in a cube delimited by (x, y, z) ∈ [−1; 1] 3 ). The learning agent is a planar robotic arm of 3 joints with the base centred on the horizontal plane, able to rotate freely around the vertical axis (each link has a length of 0.33) and change its vertical position. The robot can grab objects in this environment, by hovering its arm tip (blue in the FIG1 close to them, which position is noted (x 0, y 0, z 0). The robot can interact with:• Floor (below z = 0.0): limits the motions of the robot, slightly elastic which enable the robot to go down to z = −0.2 by forcing on it;• Pen: can be moved around and draw on the floor, broken if forcing too much on the floor (when z <= −0.3);• Joystick 1 (the left one on the figure): can be moved inside a cube-shaped area (automatically released otherwise, position normalized for this area), its x-axis position control a video-game character x position on the screen when grabbed by the robot; DISPLAYFORM0 Figure 2: Experimental setup: a robotic arm, can interact with the different objects in its environment (a pen and two joysticks). Both joysticks enable to control a video-game character (represented in top-right corner). A grey floor limits its motions and can be drawn upon using the pen (a possible drawing is represented).• Joystick 2 (the right one on the figure): can be moved inside a cube-shaped area (automatically released otherwise, position normalized for this area), its y-axis position control a video-game character y position on the screen when grabbed by the robot;• Video-game character: can be moved on the screen by using the two joysticks, its position is refreshed only at the end of a primitive policy execution for the manipulated joystick. The robot grabber can only handle one object. When it touches a second object, it breaks, releasing both objects. The robot always starts from the same position before executing a policy, and primitives are executed sequentially without getting back to this initial position. Whole complex policies are recorded with their outcomes, but each step of the complex policy execution is recorded. The motions of each of the three joints of the robot are encoded using a one-dimensional Dynamic Movement Primitive BID18, defined by the system: DISPLAYFORM0 where x and v are the position and velocity of the system; s is the phase of the motion; x 0 and g are the starting and end position of the motion; τ is a factor used to temporally scale the system (set to fix the length of a primitive execution); K and D are the spring constant and damping term fixed for the whole experiment; α is also a constant fixed for the experiment; and f is a non-linear term used to shape the trajectory called the forcing term. This forcing term is defined as: DISPLAYFORM1 where ψ i (s) = exp(−h i (s − c i) 2 ) with centers c i and widths h i fixed for all primitives. There are 3 weights ω i per DMP.The weights of the forcing term and the end positions are the only parameters of the DMP used by the robot. The starting position of a primitive is set by either the initial position of the robot (if it is starting a new complex policy) or the end position of the preceding primitive. The robot can also set its position on the vertical axis z for every primitive. Therefore a primitive policy π θ is parametrized by: DISPLAYFORM2 where DISPLAYFORM3 ) corresponds to the DMP parameters of the joint i, ordered from base to tip, and z is the fixed vertical position. When combining two or more primitive policies (π θ0, π θ1, ...), in a complex policies π θ, the parameters (θ 0, θ 1, ...) are simply concatenated together from the first primitive to the last. The outcome subspaces the robot learns to reach are hierarchically organized and defined as:• Ω 0: the position (x 0, y 0, z 0) of the end effector of the robot in Cartesian coordinates at the end of a policy execution; • Ω 1: the position (x 1, y 1, z 1) of the pen at the end of a policy execution if the pen is grabbed by the robot; • Ω 2: the first (x a, y a) and last (x b, y b) points of the last drawn continuous line on the floor if the pen is functional (x a, y a, x b, y b); • Ω 3: the position (x 3, y 3, z 3) of the first joystick at the end of a policy execution if it is grabbed by the robot; • Ω 4: the position (x 4, y 4, z 4) of the second joystick at the end of a policy execution if it is grabbed by the robot; • Ω 5: the position (x 5, y 5) of the video-game character at the end of a policy execution if moved. To help the SGIM-PB learner, procedural teachers were available so as to provide procedures for every outcome subspaces but Ω 0. Each teacher was only giving procedures useful for its own outcome space, and was aware of the task representation. They all had a cost of 5. The rules used to provide procedures are the following:• ProceduralTeacher1 (Ω 1): t 1 t 0 with t 1 ∈ Ω 1 corresponding to the pen initial position and t 0 ∈ Ω 0 corresponding to the desired final pen position; • ProceduralTeacher2 (Ω 2): t 1 t 0 with t 1 ∈ Ω 1 corresponding to the point on the z = 1.0 plane above the first point of the desired drawing, and t 0 ∈ Ω 0 corresponding to the desired final drawing point; • ProceduralTeacher3 (Ω 3): t 3 t 0 with t 3 ∈ Ω 3 and t 3 =, t 0 ∈ Ω 0 corresponding to the end effector position corresponding to the desired final position of the first joystick; • ProceduralTeacher4 (Ω 4): t 4 t 0 with t 4 ∈ Ω 4 and t 4 =, t 0 ∈ Ω 0 corresponding to the end effector position corresponding to the desired final position of the second joystick;• ProceduralTeacher5 (Ω 5): t 3 t 4 with t 3 ∈ Ω 3 and t 3 = (x, 0, 0) with x corresponding to the desired x-position of the video-game character, t 4 ∈ Ω 4 and t 4 = (0, y, 0) with y corresponding to the desired y-position of the video-game character. We also added policy teachers corresponding to the same outcome spaces to bootstrap the robot early learning process. The strategy attached to each teacher has a cost of 10. Each teacher was capable to provide demonstrations (as policies executable by the robot) linearly distributed in its outcome space: To evaluate our algorithm, we created a benchmark dataset for each outcome space Ω i, linearly distributed across the outcome space dimensions, for a total of 27,600 points. The evaluation consists in computing the normalized Euclidean distance between each of the benchmark outcome and their nearest neighbour in the learner dataset. Then we compute the mean distance to benchmark for each outcome space. The global evaluation is the mean evaluation for the 6 outcome spaces. This process is then repeated across the learning process at predefined and regularly distributed timestamps. DISPLAYFORM0 Then to asses our algorithm efficiency, we compare its with 3 other algorithms:• SAGG-RIAC: performs autonomous exploration of the policy space Π guided by intrinsic motivation; • SGIM-ACTS: interactive learner driven by intrinsic motivation. Choosing between autonomous exploration of the policy space Π and mimicry of one of the available policy teachers; • IM-PB: performs both autonomous exploration of the procedural space and the policy space, guided by intrinsic motivation; • SGIM-PB: interactive learner driven by intrinsic motivation. Choosing between autonomous exploration strategies (either of the policy space or the procedural space) and mimicry of one of the available teachers (either policy or procedural teachers).Each algorithm was run 5 times on this setup. Each run, we let the algorithm performs 25,000 iterations (complex policies executions). The value of γ for this experiment is 1.2. The probabilities to choose either of the sampling mode of SGIM-PB are p 1 = 0.15, p 2 = 0.65, p 3 = 0.2. The code run for this experiment can be found here. The Fig. 3 shows the global evaluation of all the tested algorithms, which corresponds to the mean error made by each algorithm to reproduce the benchmarks with respect to the number of complete complex policies tried. The algorithms capable of performing procedures (IM-PB and SGIM-PB) have errors that drop to levels lower than the their non-procedure equivalents (respectively SAGG-RIAC and SGIM-ACTS), moreover since the beginning of the learning process (shown on Fig. 3). It seems that the procedures bootstrap the exploration, enabling the learner to progress further. Indeed, the autonomous learner SAGG-RIAC has significantly better performance when it can use procedures and is thus upgraded to IM-PB.We can also see that the SGIM-PB algorithm has a very quick improvement in global evaluation owing to the bootstrapping effect of the different teachers. It goes lower to the final evaluation of SAGG-RIAC (0.17) after only 500 iterations. This bootstrapping effect comes from the mimicry teachers, as it is also observed for SGIM-ACTS which shares the same mimicry teachers. If we look at the evaluation on each individual outcome space (Fig. 4), we can see that the learners with demonstrations (SGIM-PB and SGIM-ACTS) outperforms the other algorithms, except for the outcome space Ω 5 where IM-PB is better, due to the fact that IM-PB practiced much more on this outcome space (500 iterations where the goal was in Ω 5 against 160 for SGIM-PB) on this outcome space. SGIM-PB and SGIM-ACTS are much better than the other algorithms on the two joysticks outcome spaces (Ω 3 and Ω 4). This is not surprising given the fact that those outcome spaces require precise policies. Indeed, if the end-effector gets out of the area where it can control the joystick, the latter is released, thus potentially ruining the attempt. So on these outcome spaces working directly on carefully crafted policies can alleviate this problem, while using procedures might be tricky, as the outcomes used don't take into account the motion trajectory but merely its final state. SGIM-PB was provided with such policies by the policy teachers. Also if we compare the of the autonomous learner without procedures (SAGG-RIAC) with the one with procedures (IM-PB), we can see that it learn less on any outcome space but Ω 0 (which was the only outcome space reachable using only single primitive policies and that could not benefit from using the task hierarchy to be learned) and especially for Ω 1, Ω 2 and Ω 5 which were the most hierarchical in this setup. More generally, it seems than on this highly hierarchical Ω 5, the learners with procedures were better. So the procedures helped when learning any potentially hierarchical task in this experiment. We further analyzed the of our SGIM-PB learner. We looked in its learning process to see which pairs of teachers and target outcomes it has chosen (Fig. 5). It was capable to request demonstrations from the relevant teachers depending on the task at hand, except for the outcome space Ω 0 which had no human teachers and therefore could not find a better teacher to help it. Indeed, for the outcome space Ω 2, the procedural teacher (ProceduralTeacher2) specially built for this outcome space was greatly chosen. Figure 5: Choices of teachers and target outcomes of the SGIM-PB learnerWe wanted to see if our SGIM-PB learner adapts the complexity of its policies to the working task. We draw 1,000,000 goal outcomes for each of the Ω 0, Ω 1 and Ω 2 subspaces (chosen because they are increasingly complex) and we let the learner choose the known policy that would reach the closest outcome. Fig. 6 shows the of this analysis. Figure 6: Number of policies selected per policy size for three increasingly more complex outcome spaces by the SGIM-PB learnerAs we can see on those three interrelated outcome subspaces (Fig. 6), the learner is capable to adapt the complexity of its policies to the outcome at hand. It chooses longer policies for the Ω 1 subspace (policies of size 2 and 3 while using mostly policies of size 1 and 2 for Ω 0) and even longer for the Ω 2 subspace (using far more policies of size 3 than for the others). It shows that our learner is capable to correctly limit the complexity of its policies instead of being stuck into always trying longer and longer policies. With this experiment, we show the capability of SGIM-PB to tackle the learning of a set of multiple interrelated complex tasks. It successfully discovers the hierarchy between tasks and uses complex motor policies to learn a wider range of tasks. It is capable to correctly choose the most adapted teachers to the target outcome when available. Though it is not limited in the size of policies it could execute, the learner shows it could adapt the complexity of its policies to the task at hand. The procedures greatly improved the learning capability of autonomous learners, as shows the difference between IM-PB and SAGG-RIAC. Our SGIM-PB shows it is capable to use procedures to discover the task hierarchy and exploit the inverse model of previously learned skills. More importantly, it shows it can successfully combine the ability of SGIM-ACTS to progress quickly in the beginning (owing to the mimicry teachers) and the ability of IM-PB to progress further on highly hierarchical tasks (owing to the procedure framework).In this article, we aimed to enable a robot to learn sequences of actions of undetermined length to achieve a field of outcomes. To tackle this high-dimensionality learning between a continuous high-dimensional space of outcomes and a continuous infinite dimensionality space of sequences of actions, we used techniques that have proven efficient in previous studies: goal-babbling, social guidance and strategic learning based on intrinsic motivation. We extended them with the procedures framework and proposed SGIM-PB algorithm, allowing the robot to babble in the procedure space and to imitate procedural teachers. We showed that SGIM-PB can discover the hierarchy between tasks, learn to reach complex tasks while adapting the complexity of the policy. The study shows that:• procedures allow the learner to learn complex tasks, and adapt the length of sequences of actions to the complexity of the task • social guidance bootstraps the learning owing to demonstrations of primitive policy in the beginning, and then to demonstrations of procedures to learn how to compose tasks into sequences of actions • intrinsic motivation can be used as a common criteria for active learning for the robot to choose both its exploration strategy, its goal outcomes and the goal-oriented procedures. However a precise analysis of the impact of each of the different strategies used by our learning algorithm could give us more insight in the roles of the teachers and procedures framework. Also, we aim to illustrate the potency of our SGIM-PB learner on a real-world application. We are currently designing such an experiment with a real robotic platform. Besides, the procedures are defined as combinations of any number of subtasks but are used in the illustration experiment as only combinations of two subtasks. It could be a next step to see if the learning algorithm can handle the curse of dimensionality of a larger procedure space, and explore combinations of any number of subtasks. Moreover, the algorithm can be extended to allow the robot learner to decide on how to execute a procedure. In the current version, we have proposed the "refinement process" to infer the best policy. We could make this refinement process more recursive, by allowing the algorithm to select, not only policies, but also lower-level procedures as one of the policy components.

The paper describes a strategic intrinsically motivated learning algorithm which tackles the learning of complex motor policies.

Represent the TLDR for finding the corresponding scientific article

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.


Question: Represent the article for finding a single-sentence TLDR

Question: Monte Carlo Tree Search (MCTS) has achieved impressive on a range of discrete environments, such as Go, Mario and Arcade games, but it has not yet fulfilled its true potential in continuous domains. In this work, we introduceTPO, a tree search based policy optimization method for continuous environments. TPO takes a hybrid approach to policy optimization. Building the MCTS tree in a continuous action space and updating the policy gradient using off-policy MCTS trajectories are non-trivial. To overcome these challenges, we propose limiting tree search branching factor by drawing only few action samples from the policy distribution and define a new loss function based on the trajectories’ mean and standard deviations. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of samples and simulations. However, we observed that bootstrappingtree search with a pre-trained policy allows us to achieve high quality with a low MCTS branching factor and few number of simulations. Without the proposed policy bootstrapping, continuous MCTS would require a much larger branching factor and simulation count, rendering it computationally and prohibitively expensive. In our experiments, we use PPO as our baseline policy optimization algorithm. TPO significantly improves the policy on nearly all of our benchmarks. For example, in complex environments such as Humanoid, we achieve a 2.5×improvement over the baseline algorithm. Fueled by advances in neural representation learning, the field of model-free reinforcement learning has rapidly evolved over the past few years. These advances are due in part to the advent of algorithms capable of navigating larger action spaces and longer time horizons, as well as the distribution of data collection and training across massive-scale computing resources. While learning algorithms have been continuously improving, it is undeniable that tree search methods have played a large role in some of the most successful applications of RL (e.g., AlphaZero, Mario and Arcade games ). Tree search methods enable powerful explorations of the action space in a way which is guided by the topology of the search space, focusing on branches (actions) that are more promising. Although tree search methods have achieved impressive on a range of discrete domains, they have not yet fulfilled their true potential in continuous domains. Given that the number of actions is inherently unbounded in continuous domains, traditional approaches to building the search tree become intractable from a computational perspective. In this paper, we introduce TPO, a Tree Search Policy Optimization for environments with continuous action spaces. We address the challenges of building the tree and running simulations by adopting a hybrid method, in which we first train a policy using existing model-free RL methods, and then use the pre-trained policy distribution to draw actions with which to build the tree. Once the tree has been constructed, we run simulations to generate experiences using an Upper Confidence Bounds for Trees (UCT) approach. Populating the tree with the action samples drawn from a pre-trained policy enables us to perform a computationally feasible search. TPO is a variation of the policy iteration method. Broadly, in these methods, the behavior of policy is iteratively updated using the trajectories generated by an expert policy. Then, the newly updated policy in return guides the expert to generate higher quality samples. In TPO, we use tree search as an expert to generate high quality trajectories. Later, we employ the updated policy to re-populate the tree search. For tree search, we use the Monte Carlo Tree Search (MCTS) expansion and selection methods. However, it is challenging to directly infer the probability of selected actions for rollout; unlike in discrete domains where all actions can be exhaustively explored, in continuous domains, we cannot sample more than a subset of the effectively innumerable continuous action space. Furthermore, to use the trajectories generated by MCTS, we must perform off-policy optimization. To address this challenge, we define a new loss function that uses the weighted mean and standard deviation of the tree search statistics to update the pre-trained policy. For ease of implementation and scalability, we use Proximal Policy Optimization (PPO) and choose as our policy optimization baseline. In phase 1, we perform a policy gradient based optimization training to build a target policy. In phase 2, we iteratively build an MCTS tree using the pre-trained target policy and update the target policy using roll-out trajectories from MCTS. Both training and data collection are done in a distributed manner. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of branches and simulations. For example, AlphaGo uses 1600 simulations per tree search and a branching factor of up to 362. However, we observed that if we pre-train the policy, we require far fewer simulations to generate high quality trajectories. While we do benefit from exploring a greater number of branches, especially for higher dimensional action spaces (e.g. Humanoid), we observed diminishing returns after only a small number of branches (e.g., 32) across all of the evaluated environments. Furthermore, performance quickly plateaued as we increased the number of simulations past 32. This property did not hold when we initialized tree search with an untrained policy. This is a critical advantage of our method as it would otherwise be computationally infeasible to generate high quality trajectories using tree search. The main contributions of TPO are summarized as follows: 1. Tree search policy optimization for continuous action spaces. TPO is one of the very first techniques that integrates tree search into policy optimization for continuous action spaces. This unique integration of tree search into policy optimization yields a superior performance compared to baseline policy optimization techniques for continuous action spaces. 2. Policy bootstrapping. We propose a policy bootstrapping technique that significantly improves the sample efficiency of the tree search and enables us to discretize continuous action spaces into only a few number of highly probable actions. More specifically, TPO only performs 32 tree searches compared to substantially larger number of tree searches (1600, 50× more) in AlphaGo. In addition, TPO narrows down the number of tree expansion (actions) compared to discretization techniques such as Tang et al. which requires 7-11 bins per action dimension. This number of bins translates to a prohibitively large number of actions even in discrete domain for complex environments such as Humanoid which has a 17 dimensional action space. In contrast, TPO only samples 32 actions at each simulation step across all the environments. 3. Infrastructure and . On the infrastructure side, we developed a distributed system (shown in Figure 1), in which both policy optimization and data collection are performed on separate distributed platforms. The policy optimization is done on a TPU-v2 using multiple cores, and MCTS search is performed on a rack of CPU nodes. A synchronous policy update and data collection approach is used to train the policy and generate trajectories. TPO readily extends to challenging and high-dimensional tasks, such the Humanoid benchmark. Our empirical indicate that TPO significantly improves the performance of the baseline policy optimization algorithm, achieving up to 2.5× improvement. Policy iterations methods have shown promising in reinforcement learning. AlphaZero, a major breakthrough in reinforcement learning, uses MCTS as an expert to train a discrete policy model for the game of Go. EXiT established theoretical grounds for policy iterations using MCTS in discrete domains. A recent work proposes some theoretical extensions to AlphaZero's MCTS method for environments with continuous action spaces. However, the authors only evaluate their method on the Pendulum OpenAI Gym environment which has only a one dimensional action space. It is not obvious how the approach can scale to more complex environments with much higher dimensions. While TPO also belongs to the category of policy iteration methods, it studies a rather less explored territory of using MCTS as expert for complex environments with continuous action spaces. Policy optimization falls into two major categories: on-policy and off-policy optimization. In on-policy methods, the policy is optimized using samples generated from the current policy, whereas with off-policy algorithms, we can also train the policy using experiences from other sources, such as an expert. Although on-policy reinforcement learning methods, such as TRPO, PPO, PGQL, and A3C have been extremely successful, they are known to be sample inefficient, as they must generate fresh trajectories between each gradient update and cannot reuse these trajectories. This effect becomes more pronounced as the complexity of the task grows, requiring millions of steps to learn a policy. On the other hand, off-policy methods (e.g., DQN, DDPG ) are more sample efficient, as each example is used multiple times for training. However, these methods require more extensive hyper-parameter tuning, suffering from instability and slow convergence. A few recent methods have been developed to embed differentiable planning procedures in computation graphs. MCTSnets is a deep learning model that incorporates simulation-based search with a vector embedding for expansion, evaluation, and backup. The three policies are jointly optimized with an end-to-end gradient based approach. The authors only report on the classic planning benchmark, Sokoban, which is a simple discrete optimization environment. Extension of this approach to continuous domains is non-obvious. SoRB is another approach that combines planning with policy optimization. However, the target problem space of SoRB is only goal-reaching tasks. Although there has been previous work on continuous MCTS, none has been conducted in the context of deep reinforcement learning. More sophisticated approaches to node expansion improve the quality of search, but TPO only uses a simple scheme in which a fixed number of actions are sampled from the policy. We found our approach both simple to implement and effective, without the need for extensive fine-tuning or hyper-parameter optimization. For example, we found that across the evaluated OpenAI Gym environments, a sample size of 32 yields strong . In our experiments, we chose PPO, which has been shown to yield superior performance in various large-scale problems such as Hide and Seek and Dota, to pre-train the policy used to populate the MCTS tree. However, our framework is independent of the specific choice of baseline policy and can be readily applied to other model-free RL algorithms, such as SAC or TD3. Preliminaries and notations. Policy optimization in continuous environments can be well modeled by a Markov Decision Process (MDP), defined by the tuple (S,A,p,r,γ), where S(A) is a set of states (actions) in continuous space. The state transition probability p(s t+1,s t,a k) represents the probability desnsity of observing the next state in time, s t+1 ∈ S, given the current s t ∈ S and the current action a k ∈ A. Taking action a k at state s t, the environment moves to a new state s t+1 and returns a bounded reward r(s t,a k): S ×A → [r min,r max]. Finally, γ ∈ represents the discount factor, used to calculate the discounted cumulative expected reward. We use π θ (a k |s t) to represent the neural network policy with trainable parameters θ. We use v φ (s t) to represent the value from a neural network with parameters φ. Previous work on Monte Carlo Tree Search (MCTS) was mostly focused on zero-sum games with discrete action spaces. In this section, we will highlight our adaptation of MCTS for environments with continuous action spaces and non-zero immediate rewards. For the complete definition of MCTS, we invite the readers to see. To make MCTS possible, we assume the environment can be set to an arbitrary state. This assumption can be relaxed using a learned model of the environment (See §6). In this work, we use a variant of Polynomial Upper Confident Tree (PUCT) algorithm in TPO for MCTS. Nodes of the Monte Carlo tree are states of the environment s t ∈ S and edges are actions, a k ∈ A. We store a set of statistics for each edge of Monte Carlo tree: the number of visits N, the cumulative sum of action value W, and the average of the action value Q. We update these statistics after each MCTS simulation for the traversed edges. Each MCTS simulation starts from a root state and consists of four steps: select, expand, evaluate, and backup. Here we only overview the select and expand steps, as they are the contributions of TPO to enable tree search for continuous action spaces. 1 Select. We follow a variant of Polynomial Upper Confident Tree (PUCT) algorithm to select actions. For each node, s t, we select an action: where c puct is a constant exploration coefficient and π θ (a k |s t) represents the prior probability assigned to each action by the policy network. To further promote exploration, we add a Dirichlet noise, Dir(α) 1, to the prior probabilities of the actions at the root node. To support environments with immediate rewards, we propose to use the following equation for estimating the value of an action: where W (s t,a k) is the cumulative sum over the backed up values from its children. 2 Expand. In discrete environments such as the game of Go or Chess/Shogi, all possible actions are considered in MCTS expansion. However, in environments with continuous action spaces, the set of possible actions is innumerable. Hence, enumerating all the possible actions for each node is impossible. To circumvent this challenge, we propose to draw samples from the policy distribution. These sampled actions will become the edges of the leaf node. After a fixed number of simulations, we select an action with the largest visit counts from the current root node and advances the environment. We use Q(s t,a k)+U (s t,a k) as a tie-breaker between the actions with the same number of visit counts. We also defineμ T S (s t) as an unbiased estimate for the mean value of the tree policy at a given node computed across each action space dimension: Later in Section 4, we use this unbiased mean estimate to optimize the policy distribution. All MCTS trajectories are added to a replay buffer, which is then used to optimize the policy and value networks. Neural network architecture. Both value, v φ , and policy networks, π θ , are represented by a fully-connected multi-layer perceptron (MLP) with two hidden layers of 64 units. The output of each layer is passed through a tanh(·) non-linearity function. The weights of value and policy networks are not shared. The value network outputs a single scalar value v(s t), whereas the policy network outputs a vector of means µ(s t), and variances σ(s t) for a number of independent Normal distributions for each action dimension. Loss functions. We first describe the loss functions for the on-policy phase and then delve into the loss functions of the proposed off-policy tree search optimization. For ease of implementation, we used PPO for on-policy optimization. The policy loss function of PPO is: where r t (θ) is the probability ratio between the new policy, π θt (a k |s t), and the old policy, π θ old (a k |s t), PPO clips the probability ratio to be within the range [1−, 1+], which penalizes large updates to the policy. Â t is the truncated generalized advantage estimate, computed for a trajectory of length T. To update the value network, we use a variant of PPO 2 that also applies clipping to the value function: 1 Dir(α) returns a noise vector of {η1,η2,...,ηn} where n i=1 ηi = 1. α ∈ R + is Dirichlet noise scaling parameter that controls the shape of the generated noise vector. Roll out trajectories {τ1, τ2, ..., τκ} for T timesteps each using πc Finally, the total loss combines the policy surrogate and value function terms with a regularization coefficients c 1: In the off-policy phase, we use the same value function as PPO for the value updates, as shown in Eq. 5 and Eq. 6. For the policy updates, we define a new loss function, derived from the collected MCTS statistics: L PF (θ) is the sum of the squared-error loss between the unbiased mean estimate and the standard deviation estimate, calculated from the MCTS trajectories, and the mean and standard deviation of the policy distribution at a given state s t. The intuition for choosing this loss term for policy updates is to incentivize moving the mean of the policy distribution towards an unbiased estimate computed from the tree search. Finally, we combine the defined TPO policy loss and value loss to obtain the following objective, which is maximized at each training step: If the policy and value networks do not share parameters, the regularization coefficients c 1 and c 3 become irrelevant. In neither L On-π or L Off-π losses, we do not use any entropy bonus term. Algorithm. Algorithm 1 fleshes out the proposed TPO algorithm, which uses a combination of policy gradient and MCTS. In the first phase of training, we use the policy network to roll out trajectories and optimize the PPO loss function (See Eq. 7). In the second phase, we switch to an off-policy mode, in which we only use trajectories from MCTS to optimize the policy. That is, we construct the loss (See Eq. 9.) on the roll out trajectories generated by the tree search. We treat ρ, which defines the switching ratio between policy gradient and tree search, as a hyperparameter of the TPO algorithm. The intuition behind bootstrapping tree search with a pre-trained policy is that continuous action spaces require a relatively stable policy to encourage the exploration of trajectories with higher returns, and an untrained policy cannot adequately guide the tree search. Our experiments support this hypothesis; if we start from a lightly trained policy (i.e. small switching ratio, ρ), the final performance is severely diminished. The same reasoning applies to choosing to start with policy gradient optimization in the first phase of training followed by tree search policy optimization. We evaluate TPO on a variety of continuous environments from OpenAI Gym (Table 1). Table 2 show the hyperparameters used for training the value and policy networks and those used for tree search. Our policy training is done on a TPU-v2 platform with eight cores, and MCTS is performed on CPU. We chose PPO for the on-policy optimization update. For each environment, we run TPO for four million time steps to estimate the gradients. To efficiently perform MCTS sampling for trajectory rollouts, we use BatchEnv that extends the OpenAI Gym interface to multiple parallel environments. Using BatchEnv, we evaluate up to 32 actions in parallel and obtain their associated rewards. We use 32 Intel CPUs, each assigned to one environment instance, in order to parallelize simulation. We found that 32 is a reasonable choice for the maximum number of MCTS simulations per each action selection, as it strikes a good balance between search latency and policy performance. Using our parallel infrastructure, the latency of action selection is somewhere between 0.2-0.3 sec when we use 32 MCTS simulations per action. Number of MCTS simulations 32 Switching ratio (ρ) 0. Unless stated otherwise, in all of the experiments, we report the average return of the last 100 episodes across five runs (each with a different random seed). To make a fair comparison with the baseline algorithm (PPO), we reproduce the reported in using our infrastructure. In all the environments, we use the best obtained for PPO using our infrastructure, which in some environments (HalfCheetah-v2 and Walker2d-v2) outperform the reported in. In all the experiments, we use Adam as our optimizer and run training for 10 epochs per policy update. Summary of . Table 3 depicts the performance comparison between TPO and PPO across six environments with continuous action spaces. In all the environments, TPO yields a better total return compared to PPO, generally by a significant margin. Using TPO, we observe a maximum improvement of roughly 2,900 in Ant-v2 and a minimum improvement of only 7 in Swimmer-v2. Swimmer-v2 is the simplest environment in the set of evaluated benchmarks with an action space of only two dimensions. On the other hand, Ant-v2 is a much more complex benchmark with an action-space of eight dimensions. The delta in performance improvement suggests that TPO is better able to leverage tree search exploration to improve policies in complex environments with large action spaces. Figure 2 shows the training progress of TPO and PPO across six environments. To conduct this experiment, we start by pre-training with PPO. At ∼ 2M time steps, TPO restores checkpoints from PPO training and continues training using tree search. As such, the total return curve for both TPO and PPO overlap before reaching 2M time steps. In all the environments, we observe a sharp spike in the total return curve after switching to tree search, which can be attributed to the effectiveness of the proposed tree search method for policy optimization. For complex environments, such as Humanoid-v2, we observe continuing improvement after switching to MCTS-based policy optimization. Ablation Studies To better understand the impact of key components of TPO, we perform a set of ablation studies. Concretely, we show the for Number of Timesteps three environments with different levels of action space complexity, namely Hopper-v2, Ant-v2, and Humanoid-v2 with action spaces of three, eight, and 17, respectively. Switching ratio. First, we study the impact of varying the switching ratio (ρ in Algorithm 1) between 10%, 50% (default value in TPO), and 90% across three tasks. As shown in Figure 3, prematurely switching (ρ = 10%, blue line in Figure 3) to tree search optimization severely harms policy performance, showing no further improvement after switching to MCTS optimization. This behavior can be attributed to the fact that in tree search policy optimization for environments with continuous action spaces, it is of utmost importance to have a reasonably well-trained policy to guide the tree search toward trajectories with potentially higher return values. Switching to tree search at the very end of training (ρ = 10%, green line in Figure 3) can be damaging as well. While switching to tree search at the 90% of the total training steps in Hopper-v2 does not improve the average return compared to switching after 50% of the total training steps, it slightly improves the average return in Ant-v2. That is, the point at which to switch to tree search is a hyperparameter that should be adjusted carefully for better policy performance. Number of actions per node. space, Humanoid-v2, has significant benefit for policy performance. As shown in Figure 4c, changing the number of actions per node from eight to 32 increases the average total return by ∼1100. Number of MCTS simulations. Figure 5 shows the impact of the number of MCTS simulations (See Section 3) on the performance of the policy. While increasing the number of MCTS simulations generally improves performance, we found that increasing the branching factor (number of actions per node) had more impact. For example, for Humanoid-v2, the performance gap between 8 and 32 simulations (shown in Figure 5c), is less pronounced than the improvement from increasing the number of actions per node from 8 and 32 (shown in Figure 4c). These studies suggest that it is more beneficial to devote the tree search budget to exploring more actions per node, especially for more complex environments, rather than conducting a larger number of tree search simulations. In this paper, we have studied Monte Carlo Tree Search in continuous space for improving the performance of a baseline on-policy algorithm. Our show that MCTS policy optimization can indeed improve the quality of policy in choosing better actions during policy evaluation at the cost of more samples during MCTS rollout. We show that bootstrapping tree search with a pretrained policy enables us to achieve high performance with a low MCTS branching factor and few simulations. On the other hand, without pre-training, we require a much larger branching factor and simulation count, rendering MCTS computationally infeasible. One of the future research direction is to explore techniques for improving the sample efficiency and removing the need for having a reset-able environment. To achieve these goals, we can use a trained model of the environment similar to model-based reinforcement learning approaches, instead of interacting directly with the environment in MCTS. Recently, MBPO showed that they can train a model of Mujoco environments that is accurate enough for nearly 200-step rollouts in terms of accumulated rewards. This level of accuracy horizon is more than enough for the shallow MCTS simulations (32 simulations) that is employed in TPO. As mentioned earlier, TPO assumes access to an environment that can be restarted from an arbitrary state for MCTS simulations. While this assumption can be readily satisfied in some RL problems such as playing games, it may be harder to achieve for physical RL problems like Robotics. This assumption can also be relaxed using a modeled environment to replace the interactions with the real environment during MCTS simulations.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We use MCTS to further optimize a bootstrapped policy for continuous action spaces under a policy iteration setting.

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. In this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer. Furthermore, when switching the latent factor (e.g., topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way -- a capability that has never been attempted by previous methods. High-dimensional data, such as images and text, are often causally generated through the interaction of many complex factors, such as lighting and pose in images or style and content in texts. Recently, VAEs and other unsupervised generative models have found successes in modelling the manifold of natural images (; ;). These models often discover controllable latent factors that allow manipulation of the images through conditional generation from interpolated or extrapolated latent codes, often with impressive quality. On the other hand, while various attributes of text such as sentiment and topic can be discovered in an unsupervised way, manipulating the text by changing these learned factors have not been possible with unsupervised generative models to the best of our knowledge. Cífka et al.; observed that text manipulation is generally more challenging compared to images, and the successes of these models cannot be directly transferred to texts. Controllable text generation aims at generating realistic text with control over various attributes including sentiment, topic and other high-level properties. Besides being a scientific curiosity, the possibility of unsupervised controllable text generation could help in a wide range of application, e.g., dialogues systems . Existing promising progress (; ; ;) all relies on supervised learning from annotated attributes to generate the text in a controllable fashion. The high cost of labelling large training corpora with attributes of interest limits the usage of these models, as pre-existing annotations often do not align with some downstream goal. Even if cheap labels are available, for example, review scores as a proxy for sentiment, the control is limited to the variation defined by the attributes. In this work, we examine the obstacles that prevent sequence VAEs from performing well in unsupervised controllable text generation. We empirically discover that manipulating the latent factors for typical semantic variations often leads to latent codes that reside in some low-density region of the aggregated posterior distribution. In other words, there are vacant regions in the latent code space not being considered by the decoding network, at least not at convergence. As a , the decoding network is unable to process such manipulated latent codes, yielding unpredictable generation of low quality. In order to mitigate the latent vacancy problem, we propose to constrain the posterior mean to a learned probability simplex and only perform manipulation within the probability simplex. Two regularizers are added to the original objective of VAE. The first enforces an orthogonal structure of the learned probability simplex; the other encourages this simplex to be filled without holes. Besides confirming that latent vacancy is indeed a cause of failure in previous sequence VAEs', it is also the first successful attempt towards unsupervised learning of controllable representations for text to the best of our knowledge. Experimental on text style transfer show that our approach significantly outperforms unsupervised baselines, and is competitive with strong supervised approaches across a wide range of evaluation metrics. Our proposed framework also enables finer-grained and more flexible control over text generation. In particular, we can switch the topic in the middle of sentence generation, and the model will often still find a way to complete the sentence in a natural way. The variational autoencoder (VAE) is a generative model defined by a prior p(z z z) and a conditional distribution p θ θ θ (x x x|z z z). The VAE is trained to optimize a tractable variational lower bound of log p θ θ θ (x x x): where q φ φ φ (z z z|x x x) is a variational distribution parameterized by an encoding network with parameters φ φ φ, and p θ θ θ (x x x|z z z) denotes the decoding network with parameters θ θ θ. This objective tries to minimize the reconstruction error to generate the data, and at the same time regularizes q φ φ φ (z z z|x x x) towards the prior p(z z z). In this paper, p(z z z) is chosen as N (0, I 0, I 0, I). For text modelling, the input x x x is some observed text. Both the encoding and decoding network are usually recurrent neural networks. Note that during learning, the decoding network p θ θ θ only learns to decode conditioned on z z z that are sampled from q φ φ φ (z z z|x x x). In other words, the decoding network only learns to process z z z sampled from the aggregated posterior distribution q φ φ φ (z z z) = E x x x∼p d (x x x) q φ φ φ (z z z|x x x), where p d (x x x) is the data distribution. If q φ φ φ (z z z) has regions of low density, there is no guarantee that p θ θ θ would decode well in such regions. This is an important intuition that will become central to our analysis in Sec. 3. In this section, we take a deeper look into the aggregated posterior latent space of sequence VAE trained on text, and provide justification for the alternative solution we propose in Section 4. As pointed out by , one of the motivations to apply VAEs on text is to allow generation of the sentences conditioned on extrinsic features by controlling the latent codes. Without annotated labels, no previous methods have successfully learned controllable latent factors as mentioned in Sec. 1. To understand what is missing, we conduct exploratory experiments to use VAE for unsupervised sentiment manipulation. We use the Yelp restaurant reviews dataset and the same data split following. We train a β-VAE 1 with a latent space of 80 dimensions, an LSTM encoder, and an LSTM decoder. Details about this experiment are described in Appendix A.1. By inspecting the accuracy on the validation set, we find that there exists one dimension of latent code achieving higher than 90% sentiment classification accuracy by its value alone, while other latent codes get accuracy around 50%. Further details can be found in Appendix A.2. It means that this latent dimension is an effective sentiment indicator. Similar phenomena have been observed in large-scale language models . However, the direct influence on the generative process of the model observed in does not apply on the VAE. When we try to perform sentiment manipulation by modifying this latent dimension 2, the decoding network fails to generate the desired outputs most of the time, as evidenced by the poor quantitative evaluation in Table. 1, and poor samples shown in Appendix A.3. One possible reason for the failure is that the decoding network is never trained on codes like the manipulated ones. This is the case if the aggregated posterior has holes or regions of low density, and the manipulated codes fall into such vacant regions. Supposing the aggregated posterior latent space possesses a shape as shown in Fig. 1, the direct manipulated latent codes will fall out of the aggregated posterior latent space for most input samples. Such latent codes are never seen by the model during training and possess a low density under the aggregated posterior distribution, leading to unpredictable behaviours during decoding. To verify our hypothesis demonstrated in Fig. 1, we empirically estimate the density of sentimentmanipulated codes under the aggregated posterior distribution of our trained VAE. Here, we approximate the data distribution p d (x x x) with the empirical distribution over all the training samples. As a , the estimated aggregated posterior distribution is a large mixture of Gaussian distribution. For all 1000 test samples, we move the dimension of code capturing sentiment from µ − 2σ to µ + 2σ where µ and σ are the mean and the standard deviation estimated on all the training samples and measure the averaged negative log-likelihood (NLL) under the aggregated posterior distribution. As depicted in Fig. 3 (a), the NLL plotted in blue dot curve rises sharply when moving away from µ even if there is only one dimension of code is changing, indicating the existence of the vacancy in the aggregated posterior latent space. In addition, we draw the histogram of all the test samples' NLL considering their original latent codes and modified ones in Fig. 3 (b). The histogram shows that there is a large divergence in NLL between the original latent codes and the modified ones. Also, the modified latent codes have two separate modes, confirming the irregular shape of the aggregated posterior latent space. In order to resolve this issue, the approach proposed in this work is to constrain the posterior in a way that the manipulation only happens in a learned simplex, as depicted in Fig. 2. In this constrained subspace, the phenomenom of low density holes of aggregated posterior is significantly reduced, as Fig. 3 (a) and (c) empirically show that there is little change in NLL of original versus modified codes. The details of our method is presented in the next section. The experiments conducted in Sec. 3 validates the existence of vacancy in the aggregated posterior latent space. One potential way to resolve the problem is to better match the aggregated posterior with the prior (; ;). However, in terms of unsupervised learning of controllable representation for text, these previous methods have not shown successes; only attempted supervised text style transfer, and also reported negative from the AAE . Another way to resolve the vacancy issue is to directly enforce that the aggregated posterior itself has no vacant region anywhere where we would like to perform latent code manipulation. We propose to map the posterior Gaussian mean to a constrained space, more specifically a learned probability simplex, where we can encourage the constrained latent space to be filled without vacancy, and perform manipulation to be within this simplex. As illustrated in Fig. 2, we add an additional mapping function as part of the encoding network which maps the mean of the Gaussian posterior to a constrained space. Two regularization terms are introduced later to ensure the learned simplex is not degenerate and that this subspace is well filled. In addition, we separately model the relevant factors that we wish to control and the irrelevant factors by splitting z z z into two parts, z z z and z z z, following prior work . The first part captures the relevant factors that are dominant in the data without an inductive bias from external signals, while the second part learns to encode the remaining local information that is useful for reconstructing the source sentences. As a , q φ φ φ (z z z|x x x) is decomposed into q φ φ φ1 (z z z |x x x)q φ φ φ2 (z z z |x x x) where φ φ φ = φ φ φ 1 ∪ φ φ φ 2. With diagonal covariances the KL divergence term in Eq. 1 splits into two separate KL terms. In practice, we use a MLP encoding network to parametrize z z z with some sentence representations as the input (e.g., averaging GloVe embeddings over the input tokens) and a LSTM encoding network to parametrize z z z. We only constrain the posterior of z z z and z z z is optimized the same way as the traditional VAE. We now describe how to map the mean µ µ µ of the Gaussian posterior for z z z ∈ R N to a constrained latent space. We would like to constrain the mean µ µ µ to have a structure as follows: e e e i, e e e j = 0, where e e e i are vectors representing the relevant factors, p i is the proportion of ith relevant factor encoded in z z z and K is a hyperparameter indicating the number of relevant factors to discover. In other words, the mean of the Gaussian posterior of z z z is constrained to be inside a K-dimension probability simplex in R N whose vertices are represented by the orthogonal basis vectors e e e i, i = 1,..., K. Given the outputs of the MLP encoder h h h and log σ σ σ 2, we learn an additional mapping function π which maps h h h to the constrained posterior space, which can be treated as part of the encoding network: where E E E = [e e e 1, . . ., e e e K] is a learnable embedding matrix representing the bases, W W W is the learnable weight matrix, and b b b is the learnable bias vector. As a , the constrained posterior is parametrized by µ µ µ and log σ σ σ 2 as a Gaussian distribution N (µ µ µ, diag(σ σ σ 2)). With the mapping function alone, the proposed VAE suffers from posterior collapse , a well-known problem where the model ignores the latent code z z z during the training. Further complicating matters is the fact that there is an abundance of signals for predicting the next token in the text, but the signals indicating high-level semantics are quite sparse. It is thus unlikely that the VAEs can capture useful relevant factors from raw text without collapse. For these reasons, we enforce orthogonality in the learnt basis vectors as defined in Eq. 2, which introduces a natural recipe to prevent posterior collapse for z z z. Note that the KL divergence between q φ φ φ1 (z z z |x x x) and p(z z z ) is With orthogonality in the basis vectors, the first term in the above equation can be factorized into i e e e i e e e i. To encourage orthogonality in the basis vectors, a regularization term is added to the objective function: where I I I is the identity matrix and α is a hyperparamter. When L REG = 0, e e e i e e e i = α. In this case, µ µ µ µ µ µ = α i p 2 i reaches its minimum α K when p p p is a uniform distribution. The proof can be found in Appendix C. In practice, L REG will quickly decrease to around 0, ensuring that the KL term will never fully collapse with the structural constraint. When it comes to controlled generation, one can choose a vertex or any desired point in the probability simplex, as illustrated in Fig. 2. Note that the constrained posterior also means that the aggregated posterior can never match the isotropic Gaussian prior. In other word, we achieve good controlled text generation potentially at the cost of poor uncontrolled generation from the prior, but such is not the focus of this current work, and could potentially be resolved by selecting or learning a better prior as in. Constraining the posterior inside a certain space does not guarantee that this space will be filled after training. In order to prevent this, we want the probability distribution over the relevant factors p p p to cover as much of the constrained latent space as possible. We introduce a reconstruction error of the structured latent code in order to push p p p away from a uniform distribution. For each input sentence, we randomly sample m sentences from the training data as negative samples. By applying the same encoding process, we get the structured latent code µ µ µ (−) i for each negative sample. Our goal is to make the raw latent code h h h similar to the restructured latent code µ µ µ while different from latent codes µ µ µ (−) i of the negative samples, so that p p p is generally different for each input sample. The structured reconstruction loss is formulated as a margin loss as follows: Our final objective function is defined as follows: 5 RELATED WORK Learning disentangled representations is an important step towards better representation learning which can be useful for (semi-)supervised learning of downstream tasks, transfer and few-shot learning . VAEs have achieved promising for unsupervised learning of disentangled representations. Several variations of VAEs have been proposed to achieve better disentanglement (; ; ;). However, most recent progress in this direction has been restricted to the domain of images. In order to perform controllable text generation, previous methods either assume annotated attributes or multiple text datasets with different known styles (; ; ; ; ; ; ;). The requirement of labelled data largely restricts the capabilities and the applications of these models. Instead, all our proposed framework needs is raw text without any annotated attribute. The dominant underlying relevant factors in the given corpus will be discovered and disentangled by our unsupervised method, which can in turn be used for controlled generation. To demonstrate the effectiveness of our approach, we compare it to unsupervised baselines with traditional VAEs, considering the density under the aggregated posterior distribution and the performance on sentiment manipulation. Following evaluation protocols in text style transfer, we also compare our method to strong supervised approaches. Furthermore, we showcase the ability of finer-grained style discovery and transition possessed by our system, which has not been attempted in the literature. In this section, our proposed framework is referred as CP-VAE (Constrained Posterior VAE). Detailed configurations including the hyperparameters, model architecture, training regimes, and decoding strategy are found in Appendix B. Experimental setup: We use the same experimental setting and dataset as mentioned in Sec. 3. The 80D latent code is split into 16 and 64 dimensions for z z z and z z z respectively. The sentence representations used for z z z is the averaged GloVe embeddings over the input tokens and K is chosen as 3. To decide which basis vector corresponds to which sentiment, we sample 10 positive and 10 negative sentences respectively in the development set, pass them to the encoder, and choose the basis vector with the highest average p i in p p p = softmax(W W Wh h h + b b b), yielding v p as the positive basis and v n as the negative basis. If v p and v n are chosen to be the same vector, we choose the index with the second highest p i for v p. To perform sentiment manipulation, we fix z z z to be the chosen basis vector; that is, v p or v n. Comparisons on density under the aggregated posterior distribution: First, we do linear interpolation between the two discovered basis vectors v p and v n and estimate the averaged NLL under the aggregated posterior distribution the same way as introduced in Sec. 3. The green solid curve in Fig. 3 (a) shows that the NLL of CP-VAE is relatively stable for the whole range of the interpolation. In Fig. 3 (c), the original latent codes and the modified ones largely overlap with each other. Both observations validate the effectiveness of CP-VAE in resolving the latent vacancy problem, leading to significant improvements on unsupervised sentiment manipulation, as seen later. Comparsions with metrics on text style transfer: For quantitative evaluation, we adopt automatic evaluation metrics used in text style transfer including classification accuracy (AC), BLEU score (BL), GLEU score (GL) and language model perplexity (PL), whose definitions are elaborated in the next section. We also report D KL (q φ φ φ1 (z z z |x x x) p(z z z)) (KL) for z z z of CP-VAE. As shown in Tab. 1, CP-VAE performs significantly better than β-VAE in terms of accuracy, BLEU and GLEU. The lower perplexity of β-VAE is due to mode collapse, which produces very short pivot sentences such as "great!". The match our observations from the experiments on density under the aggregated posterior distribution, confirming that latent vacancy prevents effective manipulation of the latent codes. We also conduct an ablation study by removing L REG and L S-REC from the objective. The demonstrate that both terms are crucial to the success of CP-VAE. Without L REG, CP-VAE experiences posterior collapse for z z z. As a , v p and v n collide with each other, leading to failure in disentangled representation learning. Since we choose K as 3, it is convenient to visualize the samples during training with p p p in the learnt probability simplex, as shown in Fig. 4. We can see that the whole simplex is mostly covered with samples with the help of L S-REC. Without L S-REC, the decoding network fails to recognize the basis vectors due to the poor coverage of the probability simplex, causing the model to lose most of its transferring ability. , denoted as CP-G(loVe) and CP-B(ert) respectively. The remaining settings are as described in the above section. On the two datasets, we compare to three adversarially trained models: StyleEmbedding (SE) , MultiDecoder (MD) , CrossAligned (CA) and two state-of-the-art models based on a "delete, transform, and generate" framework: DeleteAndRetrieve (D&R) and Blind-GenerativeStyleTransformer (B-GST) . Evaluation protocols: Four different automatic evaluation metrics are used to measure the different perspectives of the transferring quality, following. To measure transferring ability, we use pre-trained CNN based classifiers achieving 98% and 84% accuracies on the test sets of Yelp and Amazon respectively. To measure content preservation, we use the BLEU score between the transferred sentences and the source sentences. To measure fluency, we finetune OpenAI GPT-2 with 345 million parameters on the same trainingdev-test split to obtain the perplexity of generated sentences. The fine-tuned language models achieve perplexities of 26.6 and 34.5 on the test sets of Yelp and Amazon respectively. In addition, argued that the Generalized Language Evaluation Understanding Metric (GLEU) has a better correlation with the human judgement. Here, we use the implementation of GLEU 3 provided by to calculate the GLEU score. Result Analysis: As observed by and , accuracy, BLEU score and perplexity do not correlate well with human evaluations. Therefore, it is important to not consider them in isolation. Tab. 2 shows that our proposed approaches get similar scores on these metrics with human reference sentences on the second row, indicating that the generated sentences of our proposed approaches is reasonable considering the combination of these metrics. As seen by and verified in Sec. 6.1, GLEU strike a balance between target style match and content retention and correlate well with the human evaluations. From Tab. 2, CP-VAE consistently outperforms the three adversarially trained models on GLEU by a noticeable margin and achieve competitive as compared to the recent state-of-the-art models. By checking the samples generated from the models as shown in Tab. 3, B-GST, the current state-of-the-art, is more they are cheerios, afterall, and we love the original kind. HUMAN because it is made of cast iron, scorching is maximized. they are cheerios, and we love them. B-GST because it s cheaply made of cast iron, is useless. they are sturdy, afterall, sturdy and we love the original. because it s made of cast iron, vomitting. they are ripe, tastier, and we love them. because it s made of cast iron, limp. they are divine, fluffier, and we love them. consistent to the source sentence, which can be expected, since it only makes necessary edits to flip the sentiment. CP-VAE tends to generate more diverse contents which may not be relevant sometimes, but the overall quality is reasonable considering it is trained without the label information. More samples can be found in Appendix E. Table 4: Two pairs of samples generated without and with topic transition. The first sentence in the pair is generated with a topic fixed throughout the generation; while the second sentence is generated with topic transition, the generated outputs after switching are marked as bold. World throughout A federal judge on Friday ordered a federal appeals court to overturn a federal appeals court ruling that the Visa and MasterCard credit card associations violated federal antitrust law by barring the names of the state. A federal judge on Friday ordered a federal appeals court to overturn a decision by the Supreme Court to overturn a decision by the Federal Communications Commission to block the company's antitrust case against Microsoft Corp. Sports throughout NEW YORK (Reuters) -Roger Federer, the world's No. 1 player, will miss the rest of the season because of a sore quadriceps. Sports to Business NEW YORK (Reuters) -Roger Federer, the world's No. 1 player, will miss the rest of the year because of a bid-rigging scandal. To further explore the potential of CP-VAE, we conduct the following exploratory experiments. We use the AG news dataset constructed by , which contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. Here, we drop the title and just use the description field to train CP-VAE and set K = 10. All four topics are automatically discovered by CP-VAE and identified as described in Sec. 6.1. We also compare the of our identified topics to standard baselines for unsupervised topic modelling, the details can be found in Appendix D. We choose a basis vector discovered by our model and generate a few tokens. Then, we switch the basis vector and continue the generation until the end-of-seq token is generated. Generated samples are shown in Table 4. We see that our model learns to transition from one topic to another in a natural and fluent way within the same sentence. Several observations can be made based on these samples: it is good at detecting name entities and replacing them with the name entities related to the chosen topic; there is no hard restriction on when to switch the topic; the model will determine an appropriate way to do the transition by itself. Such observations confirm that CP-VAE possesses a filled constrained latent space which make the latent code robust to manipulation across different time steps, which can be effectively reflected in the generation process. Due to space limitations, we put more samples in Appendix F. In this work, we investigate latent vacancy as an important problem in unsupervised learning of controllable representations when modelling text with VAEs. To mitigate this, we propose to constrain the posterior within a learned probability simplex, achieving the first success towards controlled text generation without supervision. For the β-VAE we used for the exploratory experiments, we use a LSTM encoding network and a LSTM decoding network. For the encoding network, the input size is 256, and the hidden size is 1,024. For the decoding network, the input size is 256, the hidden size is 1,024, and dropouts with probability 0.5 are applied on after the embedding layer and the LSTM layer in the decoding network. β is chosen as 0.35, the dimension for the latent code is 80, and the batch size is 32. We use SGD with learning rate 1.0 to update the parameters for both the encoding and the decoding network. We train the model until the reconstruction loss stops decreasing. First, we normalize the value of each latent code by subtracting the mean estimated over all the training samples. Then we use the polarity of each latent code to classify the sentiment in the validation set. The one with the highest accuracy is identified as the latent factor indicating the sentiment. A.3 SAMPLES GENERATED FROM β-VAE Table 5: Samples of generated sentences from β-VAE on Yelp. Positive to Negative Negative to Positive SRC this place is super yummy! but it probably sucks too! β-VAE this place is perfect for all of us or so long and over priced! thank you! SRC i will be going back and enjoying this great place there is definitely not enough room in that part of the venue. β-VAE i will be going back and recommending this place to anyone who lives in the valley! there is great. Following manipulation strategies have been attempted: fixing the relevant factor to µ + 2σ and µ − 2σ; fixing the relevant factor to µ − σ and µ − σ; fixing the relevant factor to the maximum value and the minimum value of the relevant factor appearing in the training samples; calculating a latent vector based on 10 manually constructed parallel sentences with opposite sentiment while keeping other factors unchanged. However, none of these four strategies is effective considering the generation . We report the with the first strategy in the paper, since it performs the best considering the accuracy and the BLEU score. Across all the datasets, we use Adam with learning rate 0.001 to update the parameters for the encoding network, while SGD with learning rate 1.0 to update the parameters for the decoding network. The batch size is chosen to be 32. Dropouts with drop probability 0.5 are applied on applied on after the embedding layer and the LSTM layer in the decoding network. We train the model until the reconstruction loss stops decreasing. For the structured part z z z, we use β-VAE setting β as 0.2 across all the datasets. For the unstructured part z z z, different strategies are employed for each dataset: • Yelp: β-VAE setting β as 0.35. • Amazon: β-VAE setting β as 0.35. • AG-News: KL annealing, from 0.1 to 1.0 in 10 epochs. The hyperparameters are chosen by checking L VAE, KL, and the generated outputs on the development set for Yelp and AG-News. Amazon follows the same setting as Yelp without extra tuning. For decoding, we use beam search with a beam size of 5. C PROOF OF MINIMALIZATION OF EQ. 5 The problem can be formulated as an optimization problem as follows: By introducing a Lagrange multiplier λ, the Lagrange function is defined as In order to find the optimal point, we require that which shows that all p i are equal. By using the constraint i p i = 1, we find p i = Experimental setup: We use the AG news dataset for this task constructed by . It contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. For each category, there are 30, 000 training samples and 1, 900 test samples. In this paper, we drop the title and just use the description field. We compare our approach to two standard baselines for unsupervised topic modelling: LDA , a standard implementation of LDA is used for this baseline 4; k-means. To show the power of our approach beyond the pre-trained sentence representations, we perform k-means clustering directly on the sentence representations. Following , we assign each inferred topic to one of the gold-standard topics with the optimal mapping and report the precision (a.k.a. purity), recall (a.k.a. collocation) and F 1 score. The number of topics is chosen to be 10. The reported for the baselines and our model are the average over 10 runs. Quantitative : The are shown in Table 7. We can see that our approach achieves comparable to LDA while significantly outperforming k-means in all four categories, indicating that our approach can go beyond just clustering on pre-trained sentence representations. Table 8: Sentiment manipulation from positive to negative SRC this was the best i have ever had! B-GST this was the worst place i have ever had! CP-G this was the worst pizza i have ever had! CP-B this was the worst i have ever had! SRC friendly and welcoming with a fun atmosphere and terrific food. B-GST the hummus is ridiculously bland and bland. CP-G rude and unorganized with a terrible atmosphere and coffee. CP-B the hummus is ridiculously greasy and tasteless. SRC i ordered the carne asada steak and it was cooked perfectly! B-GST i ordered the carne asada steak and it was just as bad! CP-G i ordered the carne asada steak and it was n't cooked and it was lacking. CP-B i ordered the carne asada burrito and it was mediocre. SRC the owner is a hoot and the facility is very accommodating. B-GST the owner is a jerk and the facility is very outdated. CP-G the owner is a hoot and the facility is empty and the layout is empty. the owner is a riot and the facility is very clean. SRC i will be going back and enjoying this great place! B-GST i wo n't be going back and this place is horrible! CP-G i will be going back and eat this pizza hut elsewhere. CP-B i will be going back and hated the worst dining experience. Table 9: Sentiment manipulation from negative to positive SRC there is definitely not enough room in that part of the venue. B-GST there is plenty enough seating in that part of the venue. there is definitely an authentic dinner in that part. there is definitely a nice theatre in that part. SRC but it probably sucks too! B-GST but it tastes great too! CP-G but it's truly fun and insanely delicious. CP-B but it's probably wonderful when u! SRC always rude in their tone and always have shitty customer service! B-GST always in tune with their tone and have great customer service. CP-G always great with their birthdays and always excellent music. CP-B always accommodating and my dog is always on family. SRC i was very sick the night after. B-GST i was very happy the night after. CP-G i was very pleased with the night. CP-B i was very happy with the night. SRC this is a horrible venue. B-GST this is a wonderful venue. CP-G this is a great place for celebrating friends. this is a great place for beginners. Table 10: Sentiment manipulation from positive to negative SRC most pizza wheels that i ve seen are much smaller. B-GST most pizza dough that i ve seen are much better. CP-G most pizza wheels that i ve seen are much more good and are much quality. CP-B most pizza wheels that i ve seen are much better than are much better SRC however, this is an example of how rosle got it right. B-GST however, this game is an example of how rosle loves it. CP-G however, this is an example of how toxic... sad... obviously. however, this is an example of how cheap. similar. cheap advice. cheap advice. similar. SRC auto shut off after num num hours, which is a good feature. B-GST auto shuts off after num num hours, which is a shame. CP-G whipped mask off after num num hours, which is slimy, which is disgusting. auto shut off after num num hours, which is a stupid idea, which seems to be bad. SRC that said, the mic did pic up everything it could. B-GST that said, the game took up everything it could. CP-G that said, the shampoo did nt smell him well. stopped cleaning everything. ended up smelling sick CP-B that said, the mic did not fit everything on well, let me down it weren t cleaning SRC i also prefered tha blade weight and thickness of the wustof! B-GST i also like the blade weight and of the wustof. CP-G i also disliked the blade weight and thickness of the materials. i also slammed the blade weight and thickness of the wide. Table 11: Sentiment manipulation from negative to positive SRC the quality is declined quickly by heat exposure. B-GST the water is quickly drained by head exposure. CP-G the quality is utilitarian so grinding or sandwiches. the quality is priceless quickly by heat rises. SRC the directions were easy to follow but the quality of the easel was pathetic. B-GST the directions were easy to follow but the quality of the product was excellent. CP-G the directions were easy to follow but the quality is good for the quality and is CP-B the directions were easy to follow but the quality is what the quality is like the best quality of SRC multiplayer is just as bad, though thankfully not worse. B-GST quality is just as good, though thankfully not perfect. besides it is just good, though. those.. usually... usually... multiplayer is just as bad, though somebody s also so far not so far but no problem. SRC another energy product that simply wastes our money. B-GST another energy product that simply saves our money. another energy product that simply glides your pasta. another energy product that simply wastes this money. SRC i received the wrong color and it shreds easily. B-GST i received the color and it works easily. CP-G i low the new color and it closes easily. CP-B i received the wrong color and it pours easily from dishwasher and dries easily on garlic easily. said on Tuesday it had developed a prototype micro fuel cell aimed at protecting the world's largest oil producer.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

why previous VAEs on text cannot learn controllable latent representation as on images, as well as a fix to enable the first success towards controlled text generation without supervision

Represent the short summary for finding the corresponding scientific article (long)

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.