Question: Represent the scientific article to find a short TLDR

Question: Imitation learning algorithms provide a simple and straightforward approach for training control policies via standard supervised learning methods. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator -- typically a person -- to provide the demonstrations. In this paper, we ask: can we use imitation learning to train effective policies without any expert demonstrations? The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks. In particular, in the setting where the tasks correspond to different goals, every trajectory is a successful demonstration for the state that it actually reaches. Informed by this observation, we propose a very simple algorithm for learning behaviors without any demonstrations, user-provided reward functions, or complex reinforcement learning methods. Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached. Although related variants of this approach have been proposed previously in imitation learning settings with example demonstrations, we present the first instance of this approach as a method for learning goal-reaching policies entirely from scratch. We present a theoretical linking self-supervised imitation learning and reinforcement learning, and empirical showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems. Reinforcement learning (RL) algorithms hold the promise of providing a broadly-applicable tool for automating control, and the combination of high-capacity deep neural network models with RL extends their applicability to settings with complex observations and that require intricate policies. However, RL with function approximation, including deep RL, presents a challenging optimization problem. Despite years of research, current deep RL methods are far from a turnkey solution: most popular methods lack convergence guarantees or require prohibitive numbers of samples . Moreover, in practice, many commonly used algorithms are extremely sensitive to hyperparameters . Besides the optimization challenges, another usability challenge of RL is reward function design: although RL automatically determines how to solve the task, the task itself must be specified in a form that the RL algorithm can interpret and optimize. These challenges prompt us to consider whether there might exist a general method for learning behaviors without the need for complex, deep RL algorithms. Imitation learning is an alternative paradigm to RL that provides a simple and straightforward approach for training control policies via standard supervised learning methods. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of RL. Supervised learning algorithms in deep learning have matured to the point of being robust and reliable, and imitation learning algorithms have demonstrated success in acquiring behaviors robustly and reliably from high-dimensional sensory data such as images . The catch is that imitation learning methods require an expert demonstrator -typically a human -to provide a number of demonstrations of optimal behavior. Obtaining expert demonstrations can be challenging; the large number of demonstrations required limits the scalability of such algorithms. In this paper, we ask: can we use ideas from imitation learning to train effective policies without any expert demonstrations, retaining the benefits of imitation learning, but making it possible to learn goal-directed behavior autonomously from scratch? The key observation for making progress on this problem is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can serve as optimal examples for other tasks. In particular, in the setting where the tasks correspond to reaching different goal states, every trajectory is a successful demonstration for the state that it actually reaches. Similar observations have been made in prior works as well (; ; ; ;), but have been used to motivate data reuse in off-policy RL or semiparametric methods. Our approach will leverage this idea to obtain near-optimal goal-conditioned policies without RL or reward functions. The algorithm that we study is, at its core, very simple: at each iteration, we run our latest goalconditioned policy, collect data, and then use this data to train a policy with supervised learning. Supervision is obtained by noting that each action that is taken is a good action for reaching the states that actually occurred in future time steps along the same trajectory. This algorithm resembles imitation learning, but is self-supervised. This procedure combines the benefits of goal-conditioned policies with the simplicity of supervised learning, and we theoretically show that this algorithm corresponds to a convergent policy learning procedure. While several prior works have proposed training goal-conditioned policies via imitation learning based on a superficially similar algorithm , to our knowledge no prior work proposes a complete policy learning algorithm based on this idea that learns from scratch, without expert demonstrations. This procedure reaps the benefits of off-policy data re-use without the need for learning complex Q functions or value functions. Moreover, we can bootstrap our algorithm with a small number of expert demonstrations, such that it can continue to improve its behavior self supervised, without dealing with the challenges of combining imitation learning with off-policy RL. The main contribution of our work is a complete algorithm for learning policies from scratch via goal-conditioned imitation learning, and to show that this algorithm can successfully train goalconditioned policies. Our theoretical analysis of self-supervised goal-conditioned imitation learning shows that this method optimizes a lower bound on the probability that the agent reaches the desired goal. Empirically, we show that our proposed algorithm is able to learn goal reaching behaviors from scratch without the need for an explicit reward function or expert demonstrations. Our work addresses the same problem statement as goal conditioned reinforcement learning (RL) (; ; ;), where we aim to learn a policy via RL that can reach different goals. Learning goal-conditioned policies is quite challenging, especially when provided only sparse rewards. This challenge can be partially mitigated by hindsight relabeling approaches that relabel goals retroactively (; ;). However, even with relabelling, the goalconditioned optimization problem still uses unstable off-policy RL methods. In this work, we take a different approach and leverage ideas from supervised learning and data relabeling to build offpolicy goal reaching algorithms which do not require any explicit RL. This allows GCSL to inherit the benefits of supervised learning without the pitfalls of off-policy RL. While, in theory, on-policy algorithms might be used to solve goal reaching problem as well, their inefficient use of data makes it challenging to apply these approaches to real-world settings. Our algorithm is based on ideas from imitation learning via behavioral cloning but it is not an imitation learning method. While it is built on top of ideas from supervised learning, we are not trying to imitate externally provided expert demonstrations. Instead, we build an algorithm which can learn to reach goals from scratch, without explicit rewards. A related line of work has explored how agents can leverage expert demonstrations to bootstrap the process of reinforcement learning. While GCSL is an algorithm to learn goal-reaching policies from scratch, it lends itself naturally to bootstrapping from demonstrations. As we show in Section 5.4, GCSL can easily incorporate demonstrations into off-policy learning and continue improving, avoiding many of the challenges described in Kumar et al. (2019b). Recent imitation learning algorithms propose methods that are closely related to GCSL. aim to learn general goal conditioned policies from "play" data collected by a human demonstrator, and perform goal-conditioned imitation learning where expert goal-directed demonstrations are relabeled for imitation learning. However, neither of these methods are iterative, and both require human-provided expert demonstrations. Our method instead iteratively performs goal-conditioned behavioral cloning, starting from scratch. Our analysis shows that performing such iterated imitation learning on the policy's own sampled data actually optimizes a lower bound on the probability of successfully reaching goals, without the need for any expert demonstrations. The cross-entropy method , self-imitation learning , rewardweighted regression , path-integral policy improvement , reward-augmented maximum likelihood ), and proportional cross-entropy method selectively weight policies or trajectories by their performance during learning, as measured by then environment's reward function. While these may appear procedurally similar to GCSL, our method is fully self-supervised, as it does not require a reward function, and is applicable in the goal-conditioned setting. Additionally, our algorithm continues to perform well in the purely off-policy setting, where no new data is collected, a key difference from other algorithms . A few works similar to ours in spirit study the problem of learning goal-conditioned policy without external supervision. Zero-shot visual imitation uses an inverse model with forward consistency to learn from novelty seeking behavior, but lacks convergence guarantees and requires learning a complex inverse model. Semi-parametric methods learn a policy similar to ours but do so by building a connectivity graph over the visited states in order to navigate environments, which requires large memory storage and computation time that increases with the number of states. Goal Reaching We consider the goal reaching in an environment defined by the tuple S, A, T, ρ(s 0), T, p(g). S and A correspond to the state and action spaces respectively, T (s |s, a) to the transition kernel, ρ(s 0) to the initial state distribution, T the horizon length, and p(g) to the distribution over goal states g ∈ S. We aim to find a time-varying goal-conditioned policy π(·|s, g, h): S × S × [T] → ∆(A), where ∆(A) is the probability simplex over the action space A and h is the remaining horizon. We will say that a policy is optimal if it maximizes the probability the specified goal is reached at the end of the episode: It is important to note here that we are not considering the notion of optimality to be finding the shortest path to the goal, but merely saying that the trajectory must reach the goal at the end of T time-steps. This problem can equivalently be cast in reinforcement learning. The modified state space S = S × S × [T] contains the current state, goal, and the remaining horizon; the modified appropriately handles the modified state space; and the reward function r((s, g, h)) = 1(s = g, h = 0) depends on both the goal and the time step. Because of the special structure of this formulation, off-policy RL methods can relabel an observed transition ((s, g, h), a, (s, g, h − 1)) to that of a different goal g and different horizon h like ((s, g, h), a, (s, g, h − 1)). A common approach is to relabel trajectories with the goal they actually reached instead of the commanded goal, and often referred to as hindsight experience replay . We consider algorithms for goal-reaching that use behavior cloning, a standard method for imitation learning. In behavior cloning for goal-conditioned policies, an expert policy provides demonstrations for reaching some target goals at the very last timestep, and we aim to find a policy that best predicts the expert actions from the observations. More formally, Goal-conditioned imitation learning investigate a similar formalism that makes an additional assumption on the quality of the expert demonstrations: that the expert is optimal not just for reaching s T, but also optimal for reaching all the states s 1,... s T −1 preceding it. This corresponding policy is for t, h > 0 and t + h ≤ T. Note that implement a special case of this objective where the policy is independent of the horizon. In the next section, we will discuss how repeatedly alternating between data collection and goal-conditioned imitation learning can be used to learn a goal-reaching policy. Perhaps surprisingly, this procedure optimizes the objective in Equation 1, without relying on expert demonstrations. The goal-conditioned imitation learning in prior work show that expert demonstrations can provide supervision not only for the task the expert was aiming for, but also for reaching any state along the expert's trajectory . Can we design a procedure that uses goal-conditioned behavior cloning as a subroutine, that does not need any expert demonstrations, but that nonetheless optimizes a well-defined reward function? In this work, we show how the idea of imitation learning with data relabeling can be re-purposed to construct a learning algorithm that is able to learn how to reach goals from scratch without any expert demonstrations. We shed light on the reasons that imitation learning with data relabeling is so powerful, and how building an iterative algorithm out of this procedure gives rise to a method that optimizes a lower bound on a reinforcement learning objective, while providing a number of benefits over standard RL algorithms. It is important to note here that we are not proposing an imitation learning algorithm, but an algorithm for learning how to reach goals from scratch without any expert demonstrations. We are simply leveraging ideas from imitation learning to build such a goal reaching algorithm. Figure 1: Goal conditioned supervised learning: We can learn how to reach goals by simply sampling trajectories, relabeling them to be optimal in hindsight and treating them as expert data, and then performing supervised learning via behavior cloning. First, consider goal conditioned imitation learning via behavioral cloning with demonstrations (Equation 2). This scheme works well given expert data D, but expert data is unavailable when we are learning to reach goals from scratch. Can we use goal conditioned behavior cloning to learn how to reach goals from scratch, without the need for any expert demonstrations? To leverage behavior cloning when learning from scratch, we use the following insight: while an arbitrary trajectory from a sub-optimal policy may be suboptimal for reaching the intended goal, it may be optimal for reaching some other goal. In the goal-reaching formalism defined in Equation 1, recall a policy is optimal if it maximizes the probability that the goal is reached at the last timestep of an episode. This notion of optimality doesn't have to take the direct or shortest path to the goal, it simply has to eventually reach the goal. Under this notion of optimality, we can use a simple data relabeling scheme to construct an expert dataset from an arbitrary set of trajectories. Consider a trajectory τ = {s 1, a 1, s 2, a 2, . . ., s T, a T} obtained by commanding the policy π θ (a|s, g, h) to reach some goal g. Although the actions may be suboptimal for reaching the commanded goal g, they do succeed at reaching the states s t+1, s t+2,... that occur later in the observed trajectory. More precisely, for any timestep t and horizon h, the action a t in state s t is likely to be a good action for reaching s t+h in h timesteps, and thus useful supervision for π θ (·|s t, s t+h, h). This step of autonomous relabeling allows us to convert suboptimal trajectories into optimal goal reaching trajectories for different goals, without the need for any human intervention. To obtain a concrete algorithm, we can relabel all such timesteps and horizons in a trajectoryto create an expert dataset according to Because the relabelling procedure is valid for any horizon h ≤ T, we can relabel every such combination to create T 2 optimal tuples of (s, a, g, h) from a single trajectory. This relabeled dataset can then be used to perform goal-conditioned behavioral cloning to update the policy π θ. While performing one iteration of goal conditioned behavioral cloning on the relabeled dataset is not immediately sufficient to reach all desired goals, we will show that this procedure does in fact optimize a lower bound on a well-defined reinforcement learning objective. As described concretely in Algorithm 1, the algorithm proceeds as follows: Sample a goal from a target goal distribution p(g). Execute the current policy π(a|s, g, h) for T steps in the environment to collect a potentially suboptimal trajectory τ. Relabel the trajectory according to the previous paragraph to add T 2 new expert tuples (s t, a t, s t+h, h) to the training dataset. Perform supervised learning on the entire dataset to update the policy π(a|s, g, h) via maximum likelihood. We term this iterative procedure of sampling trajectories, relabelling them, and training a policy until convergence goalconditioned supervised learning (GCSL). Initialize policy π 1 (· | s, g, h) and dataset D((s, a, g, h)) 3: Sample g ∼ p(g) and execute π k in environment trying to reach g 5: Optimize the policy end for 9: end procedure The GCSL algorithm (described above) provides us with an algorithm that can learn to reach goals from the target distribution p(g) simply using iterated behavioral cloning. The ant goal reaching algorithm is off-policy, uses low variance gradients, and is simple to implement and tune without the need for any explicit reward function engineering or demonstrations. Additionally, since this algorithm is off-policy and does not require a value function estimator, it is substantially easier to bootstrap from demonstrations when real demonstrations are available, as our experiments will show in Section 5.4. While the GCSL algorithm is simple to implement, does this algorithm actually solve a well-defined policy learning problem? In this section, we argue that GCSL maximizes a lower bound on the probability for a policy to reach commanded goals. We start by writing the probability that policy π conditioned on goal g produces trajectory τ as π(τ |g) = p(s 0) T h=0 π(a t |s t, g, h)T (s t+1 |s t, a t). We define G(τ) = s T as final state of a trajec-tory. Recalling Equation 1, the target goal-reaching objective we wish to maximize is the probability of reaching a commanded goal: In the language of reinforcement learning, we are optimizing a multi-task problem where the reward in each task is an indicator that a goal was reached. The distribution over tasks (goals) of interest is assumed to be pre-specified as p(g). In the on-policy setting, GCSL performs imitation learning on trajectories commanded by the goals that were reached by the current policy, an objective that can be written as Here, π old corresponds to a copy of π through which gradients do not propagate, following the notation of. Our main , which is derived in the on-policy data collection setting, shows that optimizing J GCSL (π) optimizes a lower bound on the desired objective, J(π) (proof in Appendix B): Theorem 4.1. Let J GCSL and J be as defined above. Then, J(π) ≥ J GCSL (π) + C, Where C is a constant that does not depend on π. Note that to prevent J(π) and J GCSL (π) from being zero, the probability of reaching a goal under π must be nonzero -in scenarios where such a condition does not hold, the bound remains true, albeit vacuous. The tightness of this bound can be controlled by the effective error in the GCSL objective -we present this, alongside a technical analysis of the bound in Appendix B.1. This indicates that in the regime with expressive policies where the loss function can be minimized well, GCSL will improve the expected reward. In our experimental evaluation, we aim to answer the following questions: 1. Does GCSL effectively learn goal-conditioned policies from scratch? 2. Does the performance of GCSL improve over successive iterations? 3. Can GCSL learn goal-conditioned policies from high-dimensional image observations? 4. Can GCSL incorporate demonstration data more effectively than standard RL algorithms? We consider a number of simulated control environments: 2-D room navigation, object pushing with a robotic arm, and the classic Lunar Lander game, shown in Figure 2. The tasks allow us to study the performance of our method under a variety of system dynamics, both low-dimensional state inputs and high-dimensional image observations, and in settings with both easy and difficult exploration. For each task, the target goal distribution corresponds to a uniform distribution over all reachable configurations. Performance of a method is quantified by the distance of the agent to the goal at the last timestep (not by the number of time-steps to the goal as is sometimes considered in goal reaching). We present full details about the environments, evaluation protocol, and hyperparameter choices in Appendix A. For the practical implementation of GCSL, we parametrize the policy as a neural network that takes in state, goal, and horizon as input and outputs a parametrized action distribution. We find that omitting the horizon from the input to the policy still provides good , despite the formulation suggesting that the optimal policy is most likely non-Markovian. We speculate that this is due to optimal actions changing only mildly with different horizons in our tasks. Full details about the implementation for GCSL are presented in Appendix A.1. Figure 3: State-based tasks: GCSL is competitive with state-of-the-art off-policy value function RL algorithms for goal-reaching from low-dimensional sensory observations. Shaded regions denote the standard deviation across 3 random seeds (lower is better). We evaluate the effectiveness of GCSL for reaching goals on the domains visualized in Figure 2, both from low-dimensional proprioception and from images. To better understand the performance of our algorithm, we compare to two families of reinforcement learning algorithms for solving goalconditioned tasks. First, we consider off-policy temporal-difference RL algorithms, particular TD3-HER , which uses hindsight experience replay to more efficiently learn goal-conditioned value functions. TD3-HER requires significantly more machinery than our simple procedure: it maintains a policy, a value function, a target policy, and a target value function, all which are required to prevent degradation of the learning procedure. We also compare with on-policy reinforcement learning algorithms such as TRPO that cannot leverage data relabeling, but often provide more stable optimization than off-policy methods. Because these methods cannot relabel data, we provide an epsilon-ball reward corresponding to reaching the goal. Details for the training procedure for these comparisons, along with hyperpa- rameter and architectural choices, are presented in Appendix A.2. Videos and further details can be found at https://sites.google.com/view/gcsl/. We first investigate the learning performance of these algorithms from low-dimensional sensor observations, as shown in Figure 3. We find that on the pushing and lunar lander domains, GCSL is able to reach a larger set of goals consistently than either RL algorithm. Although TD3 is able to fully solve the navigation task, on the other domains which require synthesis of more challenging control behavior, the algorithm makes slow, if any, learning progress. Given a limited amount of data, TRPO performs poorly as it cannot relabel or reuse data, and so cannot match the performance of the other two algorithms. When scaling these algorithms to image-based domains, which we evaluate in Figure 4, we find that GCSL is still able to learn goal-reaching behaviors on several of these tasks. albeit slower than from state. For most tasks, from both state and images, GCSL is able to reach within 80% of the desired goals and learn at a rate comparable to or better than previously proposed off-policy RL methods. This evaluation demonstrates that simple iterative self-imitation is a competitive scheme for reaching goals in challenging environments which scales favorably with dimensionality of state and complexity of behaviors. To better understand the learning behaviors of the algorithm, we investigate how GCSL performs as we vary the quality and quantity of data, the policy class we optimize over, and the relabelling technique (Figure 5). Full details for these scenarios can be found in Appendix A.4. First, we consider how varying the policy class can affect the performance of GCSL. In Section 5.1, we hypothesized that optimizing over a Markovian policy class would be performant over maintaining a non-Markovian policy. We find that allowing policies to be time-varying ("Time-Varying Policy" in Figure 5) can drastically speed up training on small domains, as these non-Markovian optimal policies can be fit more closely. However, on domains with active exploration challenges such as the Pusher, exploration using time-varying policies is ineffective, and degrades performance. We investigate how the quality of the data in the dataset used to train the policy affects the learned policy. We consider two variations of GCSL: one which collects data using a fixed policy ("Fixed Data Collection" in Figure 5) and another which limits the size of the dataset to be small, forcing all the data to be on-policy ("On-Policy" in Figure 5). When collecting data using a fixed policy, the learning progress of the algorithm demonstratedly decreases, which indicates that the iterative loop of collecting data and training the policy is crucial for converging to a performant solution. By forcing the data to be all on-policy, the algorithm cannot utilize the full set of experiences seen thus far and must discard data. Although this on-policy process remains effective on simple domains, the technique leads to slower learning progress on tasks requiring more challenging control. Because GCSL can perform self-imitation from arbitrary data sources, the algorithm is amenable to initialization from prior exploration or from demonstrations. In this section, we study how GCSL performs when incorporating expert demonstrations as initializations. Our comparing GCSL and TD3 in this setting corroborate the existing hypothesis that off-policy value function RL algorithms are challenging to integrate with initialization from demonstrations Kumar et al. (2019a). We consider the setting where an expert provides a set of demonstration trajectories, each for reaching a different goal. GCSL requires no modifications to incorporate these demonstrations -it simply adds the expert data to the initial dataset, and begins the training procedure. In contrast, multiple prior works have proposed additional algorithmic changes to off-policy TD-based methods to in-corporate data from expert demonstrations (a). We compare the performance of GCSL to one such variant of TD3-HER in incorporating expert demonstrations on the robotic pushing environment in Figure 6 (Details in Appendix A.5). Although TD3 achieves better performance with the demonstrations than when learning from scratch, it drops in performance at the beginning of training, which means TD3 regresses from the initial behavior-cloned policy, an undesirable characteristic for initializing from demonstrations. In contrast, GCSL scales favorably, learns faster than from scratch, and effectively incorporates the expert demonstrations. We believe this benefit largely comes from not needing to train an explicit critic, which can be unstable when trained using highly off-policy data such as demonstrations (b). 6 DISCUSSION AND FUTURE WORK Pusher with Demos: Average Final Distance Figure 6: Initializing from Demonstrations: GCSL is more amenable to initializing using expert demonstrations than value-function RL methods. In this work, we proposed GCSL, a simple algorithm for learning goal-conditioned policies that uses imitation learning, while still learning autonomously from scratch. This method is exceptionally simple, relying entirely on supervised learning to learn policies by relabeling its own previously collected data. This method can easily utilize off-policy data, seamlessly incorporate expert demonstrations when they are available, and can learn directly from image observations. Although several prior works have explored similar algorithm designs in an imitation learning setting , to our knowledge our work is the first to derive a complete iterated algorithm based on this principle for learning from scratch, and the first to theoretically show that this method optimizes a lower bound on a well-defined reinforcement learning objective. While our proposed method is simple, scalable, and readily applicable, it does have a number of limitations. The current instantiation of this approach provides limited facilities for effective exploration, relying entirely on random noise during the rollouts to explore. More sophisticated exploration methods, such as exploration bonuses , are difficult to apply to our method, since there is no explicit reward function that is used during learning. However, a promising direction for future work would be to reweight the sampled rollouts based on novelty to effectively incorporate a novelty-seeking exploration procedure. A further direction for future work is to study whether the simplicity and scalability of our method can make it possible to perform goal-conditioned reinforcement learning on substantially larger and more varied datasets. This can in principle enable wider generalization, and realize a central goal in goal-conditioned reinforcement learning -universal policies that can succeed at a wide range of tasks in diverse environments. A EXPERIMENTAL DETAILS GCSL iteratively performs maximum likelihood estimation using a dataset of relabelled trajectories that have been previously collected by the agent. Here we present details about the policy class, data collection procedure, and other design choices. We parametrize a time-invariant policy using a neural network which takes as input state and goal, and returns probabilities for a discretized grid of actions of the action space. For the state-based domains, the neural network is a feedforward network with two hidden layers of size 400 and 300 respectively. For the image-based domains, both the observation image and the goal image are first preprocessed through three convolutional layers, with kernel size 5, 3, 3 and channels 16, 32, 32 respectively. When executing in the environment, data is sampled according to an exploratory policy which increases the temperature of the current policy: π explore (a|s, g) ∝ π(a|s, g) α. The replay buffer stores trajectories and relabels on the fly, with the size of the buffer subject only to memory constraints. We perform experimental comparisons with TD3-HER . We relabel transitions as ((s, g), a, (s, g)) gets relabelled to ((s, g), a, (s, g)), where g = g with probability 0.1, g = s with probability 0.5, and g = s t for some future state in the trajectory s t with probability 0.4. As described in Section 3, the agent receives a reward of 1 and the trajectory ends if the transition is relabelled to g = s, and 0 otherwise. Under this formalism, the optimal Qfunction, Q * (s, a, g) = exp(−T (s, g)), where T (s, g) is the minimum expected time to go from s to g. Both the Q-function and the actor for TD3 are parametrized as neural networks, with the same architecture (except final layers) for state-based domains and image domains as those for GCSL. We also compare to TRPO , an on-policy RL algorithm. Because TRPO is on-policy, we cannot relabel goals, and so we provide a surrogate -ball indicator reward function: r(s, g) = 1(d(s, g) < ), where is chosen appropriately for each environment. To maximize the data efficiency of TRPO, we performed a coarse hyperparameter sweep over the batch size for the algorithm. Just as with TD3, we mimic the same neural network architecture for the parametrizations of the policies as GCSL. For each environment, the goal space is identical to the state space. For the image-based experiments, images were rendered at resolution 84 × 84 × 3. 2D Room Navigation This environment requires an agent to navigate to points in an environment with four rooms that connect to adjacent rooms. The state space has two dimensions, consisting of the cartesian coordinates of the agent. The agent has acceleration control, and the action space has two dimensions. The distribution of goals p(g) is uniform on the state space, and the agent starts in a fixed location in the bottom left room. Robotic Pushing This environment requires a Sawyer manipulator to move a freely moving block in an enclosed play area with dimensions 40 cm × 20 cm. The state space is 4-dimensional, consistsing of the cartesian coordinates of the end-effector of the sawyer agent and the cartesian coordinates of the block. The Sawyer is controlled via end-effector position control with a threedimensional action space. The distribution of goals p(g) is uniform on the state space (uniform block location and uniform end-effector location), and the agent starts with the block and end-effector both in the bottom-left corner of the play area. Lunar Lander This environment requires a rocket to land in a specified region. The state space includes the normalized position of the rocket, the angle of the rocket, whether the legs of the rocket are touching the ground, and velocity information. Goals are sampled uniformly along the landing region, either touching the ground or hovering slightly above, with zero velocity. In Section 5.3, we analyzed the performance of the following variants of GCSL (Figure 7). 1. Inverse Model -This model relabels only states and goals that are one step apart: 2. On-Policy Only the most recent 10000 transitions are stored and trained on. 3. Fixed Data Collection Data is collected according to a uniform policy over actions. 4. Time-Varying Policy Policies are are conditioned on the remaining horizon. Alongside the state and goal, the policy gets a reverse temperature encoding of the remaining horizon as input. We train an expert policy for robotic pushing using TRPO with a shaped dense reward function, and collect a dataset of 200 trajectories, each corresponding to a different goal. To train GCSL using these demonstrations, we simply populate the replay buffer with these trajectories at the beginning of training, and optimize the GCSL objective using these trajectories to warm-start the algorithm. Initializing a value function method using demonstrates requires significantly more attention: we perform the following procedure. First, we perform goal-conditioned behavior cloning to learn an initial policy π BC. Next, we collect 200 new trajectories in the environment using a uniform data collection scheme. Using this dataset of 400 trajectories, we perform policy evaluation on π BC to learn Q π BC using policy evaluation via bootstrapping. Having trained such an estimate of the Qfunction, we initialize the policy and Q-function to these estimates, and run the appropriate value function RL algorithm. B PROOF OF THEOREM 4.1 We will assume a discrete state space in this proof, and denote a trajectory as τ = {s 0, a 0, . . ., s T, a T}. Let the notation G(τ) = s T denote the final state of a trajectory, which represents the goal that the trajectory reached. As there can be multiple paths to a goal, we let τ g = {τ : G(τ) = g} denote the set of trajectories that reach a particular goal g. We abbreviate a policy's trajectory distribution as π(τ |g) = p(s 0) T t=0 π(a t |s t, g)T (s t+1 |s t, a t). The target goal-reaching objective we wish to optimize is the probability of reaching a commanded goal, The distribution over tasks (goals) is assumed to be pre-specified as p(g). GCSL optimizes the following objective, where the log-likelihood of the actions conditioned on the goals actually reached by the policy, G(τ): Here, using notation from , π old is a copy of the policy π through which gradients do not propagate. To analyze how this objective relates to J(π), we first analyze the relationship between J(π) and a surrogate objective, given by As J surr (π) and J(π) have the same gradient for all π, the differ by some π-independent constant C 1, i.e. J(π) = J surr (π) + C 1. We can now lower-bound the surrogate objective via the following: The final line is our goal-relabeling objective: we train the policy to reach goals we reached g. The inequality holds since log π(τ) is always negative. The inequality is loose by a term related to the probability of not reaching the commanded goal. Since the initial state and transition probabilities do not depend on the policy, we can simplify log π(τ |G(τ)) as (by absorbing non π-dependent terms into C 2): Combining this with the bound on the expected return completes the proof, namely that J(π) ≥ J GCSL (π) + C 1 + C 2. Note that in order for J(π) and J GCSL (π) to not be degenerate, the probability of reaching a goal under π old must be non-zero. This assumption is reasonable, and matches the assumptions on "exploratory data-collection" and full-support policies that are required by Q-learning and policy gradient convergence guarantees. We now seek to better understand the gap introduced by Equation 3 in the analysis above. We define P π old (G(τ) = g) to be the probability of failure under π old and p(g). We overload the notation π old (τ) = E g [π old (τ |g)], and additionally define p wrong (τ) and p right (τ)the conditional distribution of trajectories under π old given that it did not reach and did the commanded goal respectively. In the following section, we show that the gap can be controlled by the probability of making a mistake, P π old (G(τ) = g), and D T V (p wrong (τ), p right (τ)), a measure of the difference between the distribution of trajectories that must be relabelled and those not. We rewrite Equation 3 as follows: Define D to be the Radon-Nikodym derivative of p wrong (τ) wrt π old (τ) = E τ ∼π old [log π(τ |G(τ))] − P π old (G(τ) = g)) E τ ∼π old [D log π(τ |G(τ))] = (1 − P π old (G(τ) = g))E τ ∼π old [log π(τ |G(τ))] + P π old (G(τ) = g)) E τ ∼π old [(1 − D) log π(τ |G(τ))] Relevant Gap The first term is affine with respect to the GCSL loss, so the second term is the error we seek to understand. The inequality is maintained because of the nonpositivity of log π(τ), and the final step holds because π old (τ) is a mixture of p wrong (τ) and p right (τ). This derivation shows that the gap between J surr and J GCSL (up to affine consideration) can be controlled by 1) the probability of reaching the wrong goal and 2) the divergence between the conditional distribution of good trajectories and those which must be relabelled. As either term goes to 0, the bound becomes tight. We now show that sufficiently optimizing the GCSL objective causes the probability of reaching the wrong goal to be bounded close to 0, and thus bounds the gap close to 0. Suppose we collect trajectories from a policy π data. Mimicking the notation from before, we define π data (τ) = E g∼p(g) [π data (τ |g)]. For convenience, we define π * data (a t |s t, g) ∝ τ \at π data (τ)1(G(τ) = g)1(s t (τ) = s t ) to be the conditional distribution of actions for a given state given that the goal g is reached at the end of the trajectory. If this conditional distribution is not defined, we let π * data (a t |s t, g) be uniform, so that π * data (a t |s t, g) is well-defined for all states, goals, and timesteps. Lemma B.1. Consider an environment with deterministic dynamics in which all goals are reachable in T timesteps, and a behavior policy π data which is exploratory: π data (a t |s t, g) > 0 for all (a, s, g) (for example epsilon-greedy exploration). Suppose the GCSL objective is sufficiently optimized, so that for all s, g ∈ S, and time-steps t ∈ {1, . . ., T}, D T V (π(a t |s t, g), π * data (a t |s t, g)) ≤. Then, the probability of making a mistake P π (G(τ) = g) can be bounded above by T Proof. We show the through a coupling argument, similar to;;. Because D T V (π(a t |s t, g), π * data (a t |s t, g)) ≤, we can define a (1 −)-coupled policy pair (π, π * data), which takes differing actions with probability. By a union bound over all timesteps, the probability that π and π * data take any different actions throughout the trajectory is bounded by T, and under assumptions of deterministic dynamics, take the same trajectory with probability 1 − T. Under assumption of deterministic dynamics and all goals being reachable from the initial state distribution in T timesteps, the policy π * data (a t |s t, g) satisfies P π * data (G(τ) = g) = 0. Because π * data reaches the goal with probability 1, this implies that π must also reach the goal with probability at least 1 − T. Thus, P π (G(τ) = g) ≤ T. Figure 8 below shows parts of the state along trajectories produced by GCSL. In Lunar Lander, this state is captured by the rocket's position, and in 2D Room Navigation it is the agent's position. While these trajectories do not always take the shortest path to the goal, they do often take fairly direct paths to the goal from the initial position avoiding very roundabout trajectories.

Represent the TLDR for finding the corresponding scientific article.

Learning how to reach goals from scratch by using imitation learning with data relabeling

Represent this text to find a much much longer article about it with say hundreds-thousands of words

DCEM learns latent domains for optimization problems and helps bridge the gap between model-based and model-free RL --- we create a differentiable controller and fine-tune parts of it with PPO


Question: Represent the scientific article to find a short TLDR

Question: Neural networks have recently shown excellent performance on numerous classi- fication tasks. These networks often have a large number of parameters and thus require much data to train. When the number of training data points is small, however, a network with high flexibility will quickly overfit the training data, ing in a large model variance and a poor generalization performance. To address this problem, we propose a new ensemble learning method called InterBoost for small-sample image classification. In the training phase, InterBoost first randomly generates two complementary datasets to train two base networks of the same structure, separately, and then next two complementary datasets for further training the networks are generated through interaction (or information sharing) between the two base networks trained previously. This interactive training process continues iteratively until a stop criterion is met. In the testing phase, the outputs of the two networks are combined to obtain one final score for classification. Detailed analysis of the method is provided for an in-depth understanding of its mechanism. Image classification is an important application of machine learning and data mining. Recent years have witnessed tremendous improvement in large-scale image classification due to the advances of deep learning BID15 BID17 BID7 BID4. Despite recent breakthroughs in applying deep networks, one persistent challenge is classification with a small number of training data points BID12. Small-sample classification is important, not only because humans learn a concept of class without millions or billions of data but also because many kinds of real-world data have a small quantity. Given a small number of training data points, a large network will inevitably encounter the overfitting problem, even when dropout BID16 and weight decay are applied during training BID19. This is mainly because a large network represents a large function space, in which many functions can fit a given small-sample dataset, making it difficult to find the underlying true function that is able to generalize well. As a , a neural network trained with a small number of data points usually exhibits a large variance. Ensemble learning is one way to reduce the variance. According to bias-variance dilemma BID2, there is a trade-off between the bias and variance contributions to estimation or classification errors. The variance is reduced when multiple models or ensemble members are trained with different datasets and are combined for decision making, and the effect is more pronounced if ensemble members are accurate and diverse BID3.There exist two classic strategies of ensemble learning BID21 BID13. The first one is Bagging BID20 and variants thereof. This strategy trains independent classifiers on bootstrap re-samples of training data and then combines classifiers based on some rules, e.g. weighted average. Bagging methods attempt to obtain diversity by bootstrap sampling, i.e. random sampling with replacement. There is no guarantee to find complementary ensemble members and new datasets constructed by bootstrap sampling will contain even fewer data points, which can potentially make the overfitting problem even more severe. The second strategy is Boosting BID14 BID10 and its variants. This strategy starts from a classifier trained on the available data and then sequentially trains new member classifiers. Taking Adaboost BID20 as an example, a classifier in Adaboost is trained according to the training error rates of previous classifiers. Adaboost works well for weak base classifiers. If the base classifier is of high complexity, such as a large neural network, the first base learner will overfit the training data. Consequently, either the Adaboost procedure is stopped or the second classifier has to be trained on data with original weights, i.e. to start from the scratch again, which in no way is able to ensure the diversity of base networks. In addition, there also exist some "implicit" ensemble methods in the area of neural networks. Dropout BID16, DropConnect BID18 and Stochastic Depth techniques BID5 create an ensemble by dropping some hidden nodes, connections (weights) and layers, respectively. Snapshot Ensembling BID6 ) is a method that is able to, by training only one time and finding multiple local minima of objective function, get many ensemble members, and then combines these members to get a final decision. Temporal ensembling, a parallel work to Snapshot Ensembling, trains on a single network, but the predictions made on different epochs correspond to an ensemble prediction of multiple sub-networks because of dropout regularization BID8. These works have demonstrated advantages of using an ensemble technique. In these existing "implicit" ensemble methods, however, achieving diversity is left to randomness, making them ineffective for small-sample classification. Therefore, there is a need for new ensemble learning methods able to train diverse and complementary neural networks for small-sample classification. In this paper, we propose a new ensemble method called InterBoost for training two base neural networks with the same structure. In the method, the original dataset is first re-weighted by two sets of complementary weights. Secondly, the two base neural networks are trained on the two re-weighted datasets, separately. Then we update training data weights according to prediction scores of the two base networks on training data, so there is an interaction between the two base networks during the training process. When base networks are trained interactively with the purpose of deliberately pushing each other in opposite directions, they will be complementary. This process of training network and updating weights is repeated until a stop criterion is met. In this paper, we present the training and test procedure of the proposed ensemble method and evaluate it on the UIUC-Sports dataset BID9 ) and the LabelMe dataset BID11 with a comparison to Bagging, Adaboost, SnapShot Ensembling and other existing methods. In this section, we present the proposed method in detail, followed by discussion. We are given a training dataset {x d, y d}, d∈ {1, 2, ..., D}, where y d is the true class label of x d. We assign a weight to the point {x d, y d}, which is used for re-weighting the loss of the data point in the loss function of neural network. It is equivalent to changing the distribution of training dataset and thus changing the optimization objective of neural network. We randomly assign a weight 0 < W 1d < 1 to {x d, y d} for training the first base network, and then assign a complementary weight W 2d = 1 − W 1d to {x d, y d} for training the second base network. The core idea of the InterBoost method is to train two base neural networks interactively. This is in contrast to Boosting, where base networks are typically trained in sequence, namely the subsequent network or learner is trained on a dataset with new data weights that are updated using the error rate performance of the previous base network. As shown in Figure 1, the procedure contains multiple iterations. It first trains a number of epochs for two base networks using two complementary datasets DISPLAYFORM0, 2,..., D}, separately, and then iteratively update data weights based on the probabilities that the two base networks classify DISPLAYFORM1 2 ), where θ 1 and θ 2 are parameters of the two base networks. During the iterative process, the weights always have the DISPLAYFORM2 Figure 1: InterBoost training procedure. n is the number of iteration. W 1d and W 2d are the weights of data point {x d, y d}, d∈ {1, 2, ..., D} for two base networks. θ 1 and θ 2 are the parameters of two base neural networks. DISPLAYFORM3, 2} is the probability that the ith base network can classify x d correctly after nth iteration.constraints W 1d + W 2d = 1 and 0 < W 1d, W 2d < 1. That is, they are always kept complementary to ensure the trained networks are complementary. Training networks and updating data weights run alternately until a stop condition is met. To compute θ DISPLAYFORM4 in the nth iteration, we minimize weighted loss functions are follows. DISPLAYFORM5 DISPLAYFORM6 where DISPLAYFORM7 are loss functions of of x d for two base networks, respectively. To update W 1d and W 2d, we devise the following updating rule: If the prediction probability of a data point in one base network is higher than that in another, its weight in next iteration for training this network will be smaller than its weight for training another base network. In this way, a base network will be assigned a larger weight for a data point on which it does not perform well. Hence the interaction make it be trained on diverse datasets in sequence, which can be considered as "implicit" Adaboost. Moreover, considering the fact that the two networks are always trained based on loss functions with different data weights, this interaction makes them diverse and complementary. Figure 2: Function w 1 = p 2 /(p 1 + p 2) (left) and function w 1 = ln(p 1)/(ln(p 1) + ln(p 2)) (right), where 0 < p 1, p 2 < 1.To implement the rule of updating data weights, a simple method is to use function w 1 = p 2 /(p 1 + p 2), and then assign W 1d = w 1 and W 2d = 1 − W 1d. Here, for convenience, we use p 1 and p 2 to represent the probabilities that the point x d is classified by the two base networks correctly. Moreover, this is problematic, as illustrated on the left side of Figure 2. For example, when both p 1 and p 2 are large and close to each other, w 1 will be close to 0.5. In this situation, there will be no big difference between W 1d and W 2d. In addition, this situation will occur frequently as neural networks with high flexibility will fit the data well. As a , the function have difficulty to make a data point have different weights in two base networks. Instead, we use function w 1 = ln(p 1)/(ln(p 1) + ln(p 2)), as shown on the right side of Figure 2, to update data weights. It is observed that the function is more sensitive to the small differences between p 1 and p 2 when they are both large. Specifically, for {x d, y d}, d ∈ {1, 2, ..., D}, we update its weights W1d and W2d by Equation and. DISPLAYFORM8 DISPLAYFORM9 The training procedure of InterBoost is described in Algorithm 1. First, two base networks are trained by minimizing loss functions L 1 and L 2, respectively. Secondly, weights of data point on training data are recalculated using Equation and on the basis of the prediction from two base networks. We repeat the two steps until the proposed ensemble network achieves a predefined performance on the validation dataset or the maximum iteration number is reached. Input: DISPLAYFORM0 and maximum number of iterations N. Initialize weights for each data point, W1d, W2d, and parameters of two base neural networks θ, d ∈ {1, 2, ..., D}, according to and Computing accuracy on V, temp acc, by if temp acc ≥ val acc then val acc ← temp acc DISPLAYFORM0 end if until val acc == 1 or n == N return Parameters of two base neural networks, θ 1 and θ 2 Through the interactive and iterative training process, the two networks are expected to be well trained over various regions of the problem space, represented by the data. In other words, they become "experts" with different knowledge. Therefore, we adopt a simple fusion strategy of linearly combining the prediction of two networks with equal weights and choose the index class with a maximum prediction value as the final label, as detailed in. DISPLAYFORM0 where P (c | x new, θ i), i ∈ {1, 2} is the cth class probability of the unseen data point x new from the ith network, and O(x new) is the final classification label of the point x new. Because base networks or "experts" have different knowledge, in the event that one base network makes a wrong decision, it is quite possible that another network will correct it. Complementary datasets DISPLAYFORM0 2d } Figure 3: Generated training datasets during the InterBoost training process. The datasets on the left side are for the first base network, and the datasets on the right side are for the second base network. During the training process, we always keep the constraints W 1d +W 2d = 1 and 0 < W 1d, W 2d < 1, to ensure the base networks diverse and complementary. Equation FORMULA10 and FORMULA11 are designed for updating weights of data points, so that the weight updating rule is sensitive to small differences between prediction probabilities from two base networks to prevent premature training. Furthermore, if the prediction of a data point in one network is more accurate than another network, its weight in next round will be smaller than its weight for another network, thus making the training of individual network on more different regions. The training process generates many diverse training dataset pairs, as shown in Figure 3. That is, each base network will be trained on these diverse datasets in sequence, which is equivalent to that an "implicit" ensemble is applied on each base network. Therefore, the base network will get more and more accurate during training process. At the same time, the two networks are complementary to each other. In each iteration, determination of the number of epochs for training base networks is also crucial. If the number is too large, the two base networks will fit training data too well, making it difficult to change data weights of to generate diverse datasets. If it is too small, it is difficult to obtain accurate base classifiers. In experiments, we find that a suitable epoch number in each iteration is the ones that make the classification accuracy of the base network fall in the interval of (0.9, 0.98).Similar to Bagging and Adaboost, our method has no limitation on the type of neural networks. In addition, it is straightforward to extend the proposed ensemble method for multiple networks, just by keeping DISPLAYFORM0.., D}, in which H is the number of base networks and 0 < W id < 1. Considering our focus on small-sample image classification, we choose the following datasets.• LabelMe datase (LM): A subset of scene classification dataset from BID11.The dataset contains 8 classes of natural scene images: coast, forest, highway, inside city, mountain, open country, street and tall building. We randomly select 210 images for each class, so the total number of images is 1680.• UIUC-Sports dataset (UIUC): A 8 class sports events classification dataset 1 from BID9. The dataset contains 8 classes of sports scene images. The total number of images is 1579. The numbers of images for different classes are: bocce, polo, rowing, sailing, snowboarding, rock climbing, croquet and badminton.For the LM dataset, we split the whole dataset into three parts: training, validation and test datasets. Both training and test datasets contain 800 data points, in which each class contains 100 data points. The validation dataset contains 8 classes, and each class contains 10 data points. For the UIUC dataset, we also split the whole dataset into three parts as above. In this dataset, the number of data points in each class, however, is different. We first randomly choose 10 data points for every class to form validation dataset, ing in 80 data points in total. The remaining parts of the dataset are divided equally into training and test datasets. For small-sample classification, good discriminative features are crucial. For both LM and UIUC datasets, we first resize the images into the same size of 256 × 256 and then directly use the VGG16 BID15 network trained on the ImageNet dataset without any additional tunning, to extract image features. Finally, we only reserve the features of last convolutional layer and simply flatten them. Hence, final feature dimensions for each image is 512 × 8 × 8 = 32768. Considering the small number of data points in the two datasets, we only use fully connected network with two layers. In the first layer, the activation function is Rectified Linear Unit function (Relu). In the second layer, the activation function is Softmax. We tried different numbers of hidden units, from 1024 to 32, and found overfitting is more severe if the number of hidden units is larger. Finally, we set the number of hidden layer units as 32. We did not adopt the dropout technique, simply because we found there was no difference between with and without dropout, and we set the parameter of L 2 norm about network weights as 0.01. We used minibatch gradient descent to minimize the softmax cross entropy loss. The optimization algorithm is RMSprop, the initial learning rate is 0.001, and the batch size is 32. In order to evaluate the classification performance of the proposed InterBoost method on LM and UIUC datasets, we use the training, validation and test datasets described above, and compare it with SVM with a polynomial kernel (SVM), Softmax classifier BID0, Fully connected network (FC), Bagging, Adaboost and SnapShot Ensembling (SnapShot) BID6.For SnapShot, we adopt the code published by the author of it. For Softmax, FC, Bagging, Adaboost and our method, we implement them based on Keras framework BID1, in which FC is the base network of Bagging, Adaboost, SnapShot and the proposed method. The code of the proposed method and the codes of the other referred methods and the features of two datasets used in the experiment can be found on an anonymous webpage 2 on DropBox.On the two datasets, we test different epochs ranging from 50 to 800 for Adaboost. It is found that the performance is similar when the epoch number of training base networks is set as 800 and 500. Hence, epoch numbers for FC, Softmax and the base network of bagging are also set as 800. For SnapShot, we get a snapshot network every 800 epochs, and the number of snapshot is 2. For the base networks of our method, we choose 8 iterations, each iteration has 100 epochs, and thus the total epoch number remains the same as that of Bagging, Adaboost and Snapshot. SnapShot does not use a validation dataset, so we merge the train and validation datasets into a new training dataset while the test dataset remains unchanged. We run these methods on the two datasets 60 rounds each. The average accuracies are reported in TAB0. Meanwhile, to further evaluate the robustness and stability of the proposed method and other referred methods, we show box plots of accuracies obtained by FC, Bagging, Adaboost, SnapShot and our method on the two datasets in FIG2.From TAB0, it can be seen that our method obtains performance with an accuracy of 89.0% on the UIUC dataset, and with an accuracy of 86.4% on the LM dataset. On UIUC dataset, our method performs better than Bagging and Softmax, similarly to FC and worse than Adaboost, Snapshot and SVM. On the LM dataset, our method performs better than FC, Bagging and Softmax, but worse than Adaboost, Snapshot and SVM. FIG2, our method does not have overall superior performance to other baseline methods on both LM and UIUC datasets. In the paper, we have proposed an ensemble method called InterBoost for training neural networks for small-sample classification and detailed the training and test procedures. In the training procedure, the two base networks share information with each other in order to push each other optimized in different directions. At the same time, each base network is trained on diverse datasets iteratively. Experimental on UIUC-Sports (UIUC) and LabelMe (LM) datasets showed that our ensemble method does not outperform other ensemble methods. Future work includes improving the proposed method, increasing the number of networks, experimenting on different types of network as well as different kinds of data to evaluate the effectiveness of the InterBoost method.

In the paper, we proposed an ensemble method called InterBoost for training neural networks for small-sample classification. The method has better generalization performance than other ensemble methods, and reduces variances significantly.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.