Question: Represent the scientific article to find a short TLDR

Question: The success of popular algorithms for deep reinforcement learning, such as policy-gradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difficultly in credit assignment. Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, but have been found to suffer from high sample complexity by completing forgoing the temporal nature of the problem. Improving the efficiency of RL algorithms in real-world problems with sparse or episodic rewards is therefore a pressing need. In this work, we introduce a self-imitation learning algorithm that exploits and explores well in the sparse and episodic reward settings. We view each policy as a state-action visitation distribution and formulate policy optimization as a divergence minimization problem. We show that with Jensen-Shannon divergence, this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from experience replays. Experimental indicate that our algorithm works comparable to existing algorithms in environments with dense rewards, and significantly better in environments with sparse and episodic rewards. We then discuss limitations of self-imitation learning, and propose to solve them by using Stein variational policy gradient descent with the Jensen-Shannon kernel to learn multiple diverse policies. We demonstrate its effectiveness on a challenging variant of continuous-control MuJoCo locomotion tasks. Deep reinforcement learning (RL) has demonstrated significant applicability and superior performance in many problems outside the reach of traditional algorithms, such as computer and board games BID28, continuous control BID25, and robotics. Using deep neural networks as functional approximators, many classical RL algorithms have been shown to be very effective in solving sequential decision problems. For example, a policy that selects actions under certain state observation can be parameterized by a deep neural network that takes the current state observation as input and gives an action or a distribution over actions as output. Value functions that take both state observation and action as inputs and predict expected future reward can also be parameterized as neural networks. In order to optimize such neural networks, policy gradient methods BID29 BID37 BID38 and Q-learning algorithms BID28 capture the temporal structure of the sequential decision problem and decompose it to a supervised learning problem, guided by the immediate and discounted future reward from rollout data. Unfortunately, when the reward signal becomes sparse or delayed, these RL algorithms may suffer from inferior performance and inefficient sample complexity, mainly due to the scarcity of the immediate supervision when training happens in single-timestep manner. This is known as the temporal credit assignment problem BID44. For instance, consider the Atari Montezuma's revenge game -a reward is received after collecting certain items or arriving at the final destination in the lowest level, while no reward is received as the agent is trying to reach these goals. The sparsity of the reward makes the neural network training very inefficient and also poses challenges in exploration. It is not hard to see that many of the real-world problems tend to be of the form where rewards are either only sparsely available during an episode, or the rewards are episodic, meaning that a non-zero reward is only provided at the end of the trajectory or episode. In addition to policy-gradient and Q-learning, alternative algorithms, such as those for global-or stochastic-optimization, have recently been studied for policy search. These algorithms do not decompose trajectories into individual timesteps, but instead apply zeroth-order finite-difference gradient or gradient-free methods to learn policies based on the cumulative rewards of the entire trajectory. Usually, trajectory samples are first generated by running the current policy and then the distribution of policy parameters is updated according to the trajectory-returns. The cross-entropy method and evolution strategies BID36 are two nominal examples. Although their sample efficiency is often not comparable to the policy gradient methods when dense rewards are available from the environment, they are more widely applicable in the sparse or episodic reward settings as they are agnostic to task horizon, and only the trajectorybased cumulative reward is needed. Our contribution is the introduction of a new algorithm based on policy-gradients, with the objective of achieving better performance than existing RL algorithms in sparse and episodic reward settings. Using the equivalence between the policy function and its state-action visitation distribution, we formulate policy optimization as a divergence minimization problem between the current policy's visitation and the distribution induced by a set of experience replay trajectories with high returns. We show that with the Jensen-Shannon divergence (D JS), this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped, dense rewards learned from these experience replays. This algorithm can be seen as self-imitation learning, in which the expert trajectories in the experience replays are self-generated by the agent during the course of learning, rather than using some external demonstrations. We combine the divergence minimization objective with the standard RL objective, and empirically show that the shaped, dense rewards significantly help in sparse and episodic settings by improving credit assignment. Following that, we qualitatively analyze the shortcomings of the self-imitation algorithm. Our second contribution is the application of Stein variational policy gradient (SVPG) with the Jensen-Shannon kernel to simultaneously learn multiple diverse policies. We demonstrate the benefits of this addition to the self-imitation framework by considering difficult exploration tasks with sparse and deceptive rewards. Related Works. Divergence minimization has been used in various policy learning algorithms. Relative Entropy Policy Search (REPS) BID33 restricts the loss of information between policy updates by constraining the KL-divergence between the state-action distribution of old and new policy. Policy search can also be formulated as an EM problem, leading to several interesting algorithms, such as RWR BID32 and PoWER BID20. Here the M-step minimizes a KL-divergence between trajectory distributions, leading to an update rule which resembles return-weighted imitation learning. Please refer to BID7 for a comprehensive exposition. MATL BID47 uses adversarial training to bring state occupancy from a real and simulated agent close to each other for efficient transfer learning. In Guided Policy Search (GPS, BID21), a parameterized policy is trained by constraining the divergence between the current policy and a controller learnt via trajectory optimization. Learning from Demonstrations (LfD). The objective in LfD, or imitation learning, is to train a control policy to produce a trajectory distribution similar to the demonstrator. Approaches for self-driving cars BID4 and drone manipulation BID34 have used human-expert data, along with Behavioral Cloning algorithm to learn good control policies. Deep Q-learning has been combined with human demonstrations to achieve performance gains in Atari and robotics tasks BID46 BID30. Human data has also been used in the maximum entropy IRL framework to learn cost functions under which the demonstrations are optimal. BID17 use the same framework to derive an imitation-learning algorithm (GAIL) which is motivated by minimizing the divergence between agent's rollouts and external expert demonstrations. Besides humans, other sources of expert supervision include planningbased approaches such as iLQR and MCTS. Our algorithm departs from prior work in forgoing external supervision, and instead using the past experiences of the learner itself as demonstration data. Exploration and Diversity in RL. Count-based exploration methods utilize state-action visitation counts N (s, a), and award a bonus to rarely visited states BID42. In large statespaces, approximation techniques BID45, and estimation of pseudo-counts by learning density models BID3 BID13 has been researched. Intrinsic motivation has been shown to aid exploration, for instance by using information gain or prediction error BID41 as a bonus. Hindsight Experience Replay adds additional goals (and corresponding rewards) to a Q-learning algorithm. We also obtain additional rewards, but from a discriminator trained on past agent experiences, to accelerate a policy-gradient algorithm. Prior work has looked at training a diverse ensemble of agents with good exploratory skills BID27 BID6 BID12. To enjoy the benefits of diversity, we incorporate a modification of SVPG BID27 in our final algorithm. In very recent work, BID31 propose exploiting past good trajectories to drive exploration. Their algorithm buffers (s, a) and the corresponding return for each transition in rolled trajectories, and reuses them for training if the stored return value is higher than the current state-value estimate. Our approach presents a different objective for self-imitation based on divergence-minimization. With this view, we learn shaped, dense rewards which are then used for policy optimization. We further improve the algorithm with SVPG. Reusing high-reward trajectories has also been explored for program synthesis and semantic parsing tasks BID23 BID0. We start with a brief introduction to RL in Section 2.1, and then introduce our main algorithm of self-imitating learning in Section 2.2. Section 2.3 further extends our main method to learn multiple diverse policies using Stein variational policy gradient with Jensen-Shannon kernel. A typical RL setting involves an environment modeled as a Markov Decision Process with an unknown system dynamics model p(s t+1 |s t, a t) and an initial state distribution p 0 (s 0). An agent interacts sequentially with the environment in discrete time-steps using a policy π which maps the an observation s t ∈ S to either a single action a t (deterministic policy), or a distribution over the action space A (stochastic policy). We consider the scenario of stochastic policies over high-dimensional, continuous state and action spaces. The agent receives a per-step reward r t (s t, a t) ∈ R, and the RL objective involves maximization of the expected discounted sum of rewards, η(π θ) = E p0,p,π ∞ t=0 γ t r(s t, a t), where γ ∈ is the discount factor. The action-value function is DISPLAYFORM0. We define the unnormalized γ-discounted statevisitation distribution for a policy π by ρ π (s) = ∞ t=0 γ t P (s t = s|π), where P (s t = s|π) is the probability of being in state s at time t, when following policy π and starting state s 0 ∼ p 0. The expected policy return η(π θ) can then be written as E ρπ(s,a) [r(s, a)], where ρ π (s, a) = ρ π (s)π(a|s) is the state-action visitation distribution. Using the policy gradient theorem BID43, we can get the direction of ascent ∇ θ η(π θ) = E ρπ(s,a) ∇ θ log π θ (a|s)Q π (s, a). Although the policy π(a|s) is given as a conditional distribution, its behavior is better characterized by the corresponding state-action visitation distribution ρ π (s, a), which wraps the MDP dynamics and fully decides the expected return via η(π) = E ρπ [r(s, a)]. Therefore, distance metrics on a policy π should be defined with respect to the visitation distribution ρ π, and the policy search should be viewed as finding policies with good visitation distributions ρ π that yield high reward. Suppose we have access to a good policy π *, then it is natural to consider finding a π such that its visitation distribution ρ π matches ρ π *. To do so, we can define a divergence measure D(ρ π, ρ π *) that captures the similarity between two distributions, and minimize this divergence for policy improvement. Assume there exists an expert policy π E, such that policy optimization can be framed as minimizing the divergence min π D(ρ π, ρ π E), that is, finding a policy π to imitate π E. In practice, however, we do not have access to any real guiding expert policy. Instead, we can maintain a selected subset M E of highly-rewarded trajectories from the previous rollouts of policy π, and optimize the policy π to minimize the divergence between ρ π and the empirical state-action pair distribution {(s i, a i)} M E: DISPLAYFORM0 Since it is not always possible to explicitly formulate ρ π even with the exact functional form of π, we generate rollouts from π in the environment and obtain an empirical distribution of ρ π. To measure the divergence between two empirical distributions, we use the Jensen-Shannon divergence, with the following variational form (up to a constant shift) as exploited in GANs BID15: DISPLAYFORM1 where d(s, a) and d E (s, a) are empirical density estimators of ρ π and ρ π E, respectively. Under certain assumptions, we can obtain an approximate gradient of D JS w.r.t the policy parameters, thus enabling us to optimize the policy. Gradient Approximation: Let ρ π (s, a) and ρ π E (s, a) be the state-action visitation distributions induced by two policies π and π E respectively. Let d π and d π E be the surrogates to ρ π and ρ π E, respectively, obtained by solving Equation 2. Then, if the policy π is parameterized by θ, the gradient of D JS (ρ π, ρ π E) with respect to policy parameters (θ) can be approximated as: DISPLAYFORM2 where DISPLAYFORM3.. The derivation of the approximation and the underlying assumptions are in Appendix 5.1. Next, we introduce a simple and inexpensive approach to construct the replay memory M E using highreturn past experiences during training. In this way, ρ π E can be seen as a mixture of deterministic policies, each representing a delta point mass distribution in the trajectory space or a finite discrete visitation distribution of state-action pairs. At each iteration, we apply the current policy π θ to sample b trajectories {τ} b 1. We hope to include in M E, the top-k trajectories (or trajectories with returns above a threshold) generated thus far during the training process. For this, we use a priorityqueue list for M E which keeps the trajectories sorted according to the total trajectory reward. The reward for each newly sampled trajectory in {τ} b 1 is compared with the current threshold of the priority-queue, updating M E accordingly. The frequency of updates is impacted by the exploration capabilities of the agent and the stochasticity in the environment. We find that simply sampling noisy actions from Gaussian policies is sufficient for several locomotion tasks (Section 3). To handle more challenging environments, in the next sub-section, we augment our policy optimization procedure to explicitly enhance exploration and produce an ensemble of diverse policies. In the usual imitation learning framework, expert demonstrations of trajectories-from external sources-are available as the empirical distribution of ρ π E of an expert policy π E. In our approach, since the agent learns by treating its own good past experiences as the expert, we can view the algorithm as self-imitation learning from experience replay. As noted in Equation 3, the gradient estimator of D JS has a form similar to policy gradients, but for replacing the true reward function with per-timestep reward defined as log(DISPLAYFORM4). Therefore, it is possible to interpolate the gradient of D JS and the standard policy gradient. We would highlight the benefit of this interpolation soon. The net gradient on the policy parameters is: DISPLAYFORM5 where Q r is the Q function with true rewards, and π E is the mixture policy represented by the DISPLAYFORM6. r φ (s, a) can be computed using parameterized networks for densities d π and d π E, which are trained by solving the D JS optimization (Eq 2) using the current policy rollouts and M E, where φ includes the parameters for d π and d π E. Using Equation 3, the interpolated gradient can be further simplified to: DISPLAYFORM7 where DISPLAYFORM8 is the Q function calculated using − log r φ (s, a) as the reward. This reward is high in the regions of the S × A space frequented more by the expert than the learner, and low in regions visited more by the learner than the expert. The effective Q in Equation 5 is therefore an interpolation between Q r obtained with true environment rewards, and Q r φ obtained with rewards which are implicitly shaped to guide the learner towards expert behavior. In environments with sparse or deceptive rewards, where the signal from Q r is weak or sub-optimal, a higher weight on Q r φ enables successful learning by imitation. We show this empirically in our experiments. We further find that even in cases with dense environment rewards, the two gradient components can be successfully combined for policy optimization. The complete algorithm for self-imitation is outlined in Appendix 5.2 (Algorithm 1).Limitations of self-imitation. We now elucidate some shortcomings of the self-imitation approach. Since the replay memory M E is only constructed from the past training rollouts, the quality of the trajectories in M E is hinged on good exploration by the agent. Consider a maze environment where the robot is only rewarded when it arrives at a goal G placed in a far-off corner. Unless the robot reaches G once, the trajectories in M E always have a total reward of zero, and the learning signal from Q r φ is not useful. Secondly, self-imitation can lead to sub-optimal policies when there are local minima in the policy optimization landscape; for example, assume the maze has a second goal G in the opposite direction of G, but with a much smaller reward. With simple exploration, the agent may fill M E with below-par trajectories leading to G, and the reinforcement from Q r φ would drive it further to G. Thirdly, stochasticity in the environment may make it difficult to recover the optimal policy just by imitating the past top-k rollouts. For instance, in a 2-armed bandit problem with reward distributions Bernoulli (p) and Bernoulli (p+), rollouts from both the arms get conflated in M E during training with high probability, making it hard to imitate the action of picking the arm with the higher expected reward. We propose to overcome these pitfalls by training an ensemble of self-imitating agents, which are explicitly encouraged to visit different, non-overlapping regions of the state-space. This helps to discover useful rewards in sparse settings, avoids deceptive reward traps, and in environments with reward-stochasticity like the 2-armed bandit, increases the probability of the optimal policy being present in the final trained ensemble. We detail the enhancements next. One approach to achieve better exploration in challenging cases like above is to simultaneously learn multiple diverse policies and enforce them to explore different parts of the high dimensional space. This can be achieved based on the recent work by BID27 on Stein variational policy gradient (SVPG). The idea of SVPG is to find an optimal distribution q(θ) over the policy parameters θ which maximizes the expected policy returns, along with an entropy regularization that enforces diversity on the parameter space, i.e. DISPLAYFORM0 Without a parametric assumption on q, this problem admits a challenging functional optimization problem. Stein variational gradient descent (SVGD, BID26) provides an efficient solution for solving this problem, by approximating q with a delta measure q = n i=1 δ θi /n, where DISPLAYFORM1 is an ensemble of policies, and iteratively update {θ i} with DISPLAYFORM2 where k(θ j, θ i) is a positive definite kernel function. The first term in ∆θ i moves the policy to regions with high expected return (exploitation), while the second term creates a repulsion pressure between policies in the ensemble and encourages diversity (exploration). The choice of kernel is critical. BID27 used a simple Gaussian RBF kernel k(θ j, θ i) = exp(− θ j − θ i 2 2 /h), with the bandwidth h dynamically adapted. This, however, assumes a flat Euclidean distance between θ j and θ i, ignoring the structure of the entities defined by them, which are probability distributions. A statistical distance, such as D JS, serves as a better metric for comparing policies BID1 BID19. Motivated by this, we propose to improve SVPG using JS kernel k(θ j, θ i) = exp(−D JS (ρ π θ j, ρ π θ i)/T ), where ρ π θ (s, a) is the state-action visitation distribution obtained by running policy π θ, and T is the temperature. The second exploration term in SVPG involves the gradient of the kernel w.r.t policy parameters. With the JS kernel, this requires estimating gradient of D JS, which as shown in Equation 3, can be obtained using policy gradients with an appropriately trained reward function. Our full algorithm is summarized in Appendix 5.3 (Algorithm 2). In each iteration, we apply the SVPG gradient to each of the policies, where the ∇ θ η(π θ) in Equation 6 is the interpolated gradient from self-imitation (Equation 5). We also utilize state-value function networks as baselines to reduce the variance in sampled policy-gradients. Our goal in this section is to answer the following questions: 1) How does self-imitation fare against standard policy gradients under various reward distributions from the environment, namely episodic, noisy and dense? 2) How far does the SVPG exploration go in overcoming the limitations of selfimitation, such as susceptibility to local-minimas?We benchmark high-dimensional, continuous-control locomotion tasks based on the MuJoCo physics simulator by extending the OpenAI Baselines BID8 framework. Our control policies (θ i) are modeled as unimodal Gaussians. All feed-forward networks have two layers of 64 hidden units each with tanh non-linearity. For policy-gradient, we use the clipped-surrogate based PPO algorithm BID39. Further implementation details are in the Appendix. Table 1: Performance of PPO and Self-Imitation (SI) on tasks with episodic rewards, noisy rewards with masking probability pm, and dense rewards. All runs use 5M timesteps of interaction with the environment. ES performance at 5M timesteps is taken from BID36. Missing entry denotes that we were unable to obtain the 5M timestep performance from the paper. DISPLAYFORM0 We evaluate the performance of self-imitation with a single agent in this sub-section; combination with SVPG exploration for multiple agents is discussed in the next. We consider the locomotion tasks in OpenAI Gym under 3 separate reward distributions: Dense refers to the default reward function in Gym, which provides a reward for each simulation timestep. In episodic reward setting, rather than providing r(s t, a t) at each timestep of an episode, we provide t r(s t, a t) at the last timestep of the episode, and zero reward at other timesteps. This is the case for many practical settings where the reward function is hard to design, but scoring each trajectory, possibly by a human BID5, is feasible. In noisy reward setting, we probabilistically mask out each out each per-timestep reward r(a t, s t) in an episode. Reward masking is done independently for every new episode, and therefore, the agent receives non-zero feedback at different-albeit only few-timesteps in different episodes. The probability of masking-out or suppressing the rewards is denoted by p m.In FIG0, we plot the learning curves on three tasks with episodic rewards. Recall that ν is the hyper-parameter controlling the weight distribution between gradients with environment rewards and the gradients with shaped reward from r φ (Equation 5). The baseline PPO agents use ν = 0, meaning that the entire learning signal comes from the environment. We compare them with selfimitating (SI) agents using a constant value ν = 0.8. The capacity of M E is fixed at 10 trajectories. We didn't observe our method to be particularly sensitive to the choice of ν and the capacity value. For instance, ν = 1 works equally well. Further ablation on these two hyper-parameters can be found in the Appendix. In FIG0, we see that the PPO agents are unable to make any tangible progress on these tasks with episodic rewards, possibly due to difficulty in credit assignment -the lumped rewards at the end of the episode can't be properly attributed to the individual state-action pairs during the episode. In case of Self-Imitation, the algorithm has access to the shaped rewards for each timestep, derived from the high-return trajectories in M E. This makes credit-assignment easier, leading to successful learning even for very high-dimensional control tasks such as Humanoid. Table 1 summarizes the final performance, averaged over 5 runs with random seeds, under the various reward settings. For the noisy rewards, we compare performance with two different reward masking values -suppressing each reward r(s t, a t) with 90% probability (p m = 0.9), and with 50% probability (p m = 0.5). The density of rewards increases across the reward settings from left to right in Table 1. We find that SI agents (ν = 0.8) achieve higher average score than the baseline PPO agents (ν = 0) in majority of the tasks for all the settings. This indicates that not only does self-imitation vastly help when the environment rewards are scant, it can readily be incorporated with the standard policy gradients via interpolation, for successful learning across reward settings. For completion, we include performance of CEM and ES since these algorithms depend only on the total trajectory rewards and don't exploit the temporal structure. CEM perform poorly in most of the cases. ES, while being able to solve the tasks, is sample-inefficient. We include ES performance from BID36 after 5M timesteps of training for a fair comparison with our algorithm. We now conduct experiments to show how self-imitation can lead to sub-optimal policies in certain cases, and how the SVPG objective, which trains an ensemble with an explicit D JS repulsion between policies, can improve performance.2D-Navigation. Consider a simple Maze environment where the start location of the agent (blue particle) is shown in the figure on the right, along with two regions -the red region is closer to agent's starting location but has a per-timestep reward of only 1 point if the agent hovers over it; the green region is on the other side of the wall but has a per-timestep reward of 10 points. We run 8 independent, non-interacting, self-imitating (with ν = 0.8) agents on this task. This ensemble is denoted as SI-independent. Figures 2a plots the state-visitation density for SI-independent after training, from which it is evident that the agents get trapped in the local minima. The red-region is relatively easily explored and trajectories leading to it fill the M E, causing sub-optimal imitation. We contrast this with an instantiation of our full algorithm, which is referred to as SI-interact-JS. It is composed of 8 self-imitating agents which share information for gradient calculation with the SVPG objective (Equation 6). The temperature T = 0.5 is held constant, and the weight on exploration-facilitating repulsion term (α) is linearly decayed over time. FIG2 depicts the state-visitation density for this ensemble. SI-interact-JS explores wider portions of the maze, with multiple agents reaching the green zone of high reward. Figures 2c and 2d show the kernel matrices for the two ensembles after training. Cell (i, j) in the matrix corresponds to the kernel value k(θ i, θ j) = exp(−JS(ρ i, ρ j)/T ). For SI-independent, many darker cells indicate that policies are closer (low JS). For SI-interact-JS, which explicitly tries to decrease k(θ i, θ j), the cells are noticeably lighter, indicating dissimilar policies (high JS). Behavior of PPO-independent (ν = 0) is similar to SI-independent (ν = 0.8) for the Maze task. Locomotion. To explore the limitations of self-imitation in harder exploration problems in highdimensional, continuous state-action spaces, we modify 3 MuJoCo tasks as follows -SparseHalfCheetah, SparseHopper and SparseAnt yield a forward velocity reward only when the centerof-mass of the corresponding bot is beyond a certain threshold distance. At all timesteps, there is an energy penalty to move the joints, and a survival bonus for bots that can fall over causing premature episode termination (Hopper, Ant). FIG3 plots the performance of PPO-independent, SI-independent, SI-interact-JS and SI-interact-RBF (which uses RBF-kernel from BID27 instead of the JS-kernel) on the tasks. Each of these 4 algorithms is an ensemble of 8 agents using the same amount of simulation timesteps. The are averaged over 3 separate runs, where for each run, the best agent from the ensemble after training is selected. The SI-independent agents rely solely on action-space noise from the Gaussian policy parameterization to find high-return trajectories which are added to M E as demonstrations. This is mostly inadequate or slow for sparse environments. Indeed, we find that all demonstrations in M E for SparseHopper are with the bot standing upright (or tilted) and gathering only the survival bonus, as action-space noise alone can't discover hopping behavior. Similarly, for SparseHalfCheetah, M E has trajectories with the bot haphazardly moving back and forth. On the other hand, in SI-interact-JS, the D JS repulsion term encourages the agents to be diverse and explore the state-space much more effectively. This leads to faster discovery of quality trajectories, which then provide good reinforcement through self-imitation, leading to higher overall score. SI-interact-RBF doesn't perform as well, suggesting that the JS-kernel is more formidable for exploration. PPO-independent gets stuck in the local optimum for SparseHopper and SparseHalfCheetah -the bots stand still after training, avoiding energy penalty. For SparseAnt, the bot can cross our preset distance threshold using only action-space noise, but learning is slow due to naïve exploration. We approached policy optimization for deep RL from the perspective of JS-divergence minimization between state-action distributions of a policy and its own past good rollouts. This leads to a self-imitation algorithm which improves upon standard policy-gradient methods via the addition of a simple gradient term obtained from implicitly shaped dense rewards. We observe substantial performance gains over the baseline for high-dimensional, continuous-control tasks with episodic and noisy rewards. Further, we discuss the potential limitations of the self-imitation approach, and propose ensemble training with the SVPG objective and JS-kernel as a solution. Through experimentation, we demonstrate the benefits of a self-imitating, diverse ensemble for efficient exploration and avoidance of local minima. An interesting future work is improving our algorithm using the rich literature on exploration in RL. Since ours is a population-based exploration method, techniques for efficient single agent exploration can be readily combined with it. For instance, parameter-space noise or curiosity-driven exploration can be applied to each agent in the SI-interact-JS ensemble. Secondly, our algorithm for training diverse agents could be used more generally. In Appendix 5.6, we show preliminary for two cases: a) hierarchical RL, where a diverse group of Swimmer bots is trained for downstream use in a complex Swimming+Gathering task; b) RL without environment rewards, relying solely on diversity as the optimization objective. Further investigation is left for future work. Let d * π (s, a) and d * E (s, a) be the exact state-action densities for the current policy (π θ) and the expert, respectively. Therefore, by definition, we have (up to a constant shift): a) is a local surrogate to ρ π θ (s, a). By approximating it to be constant in an −ball neighborhood around θ, we get the following after taking gradient of the above equation w.r.t θ: DISPLAYFORM0 DISPLAYFORM1 where DISPLAYFORM2 The last step follows directly from the policy gradient theorem (Calculate g 1 = ∇ θ η r1 (π θ) with PPO objective using r 1 reward Calculate g 2 = ∇ θ η r2 (π θ) with PPO objective using r 2 reward 9Update θ with (1 − ν)g 1 + νg 2 using ADAM We show the sensitivity of self-imitation to ν and the capacity of M E, denoted by C. The experiments in this subsection are done on Humanoid and Hopper tasks with episodic rewards. The tables show the average performance over 5 random seeds. For ablation on ν, C is fixed at 10; for ablation on C, ν is fixed at 0.8. With episodic rewards, a higher value of ν helps boost performance since the RL signal from the environment is weak. With ν = 0.8, there isn't a single best choice for C, though all values of C give better than baseline PPO (ν = 0). The diversity-promoting D JS repulsion can be used for various other purposes apart from aiding exploration in the sparse environments considered thus far. First, we consider the paradigm of hierarchical reinforcement learning wherein multiple sub-policies (or skills) are managed by a highlevel policy, which chooses the most apt sub-policy to execute at any given time. In FIG7, we use the Swimmer environment from Gym and show that diverse skills (movements) can be acquired in a pre-training phase when D JS repulsion is used. The skills can then be used in a difficult downstream task. During pre-training with SVPG, exploitation is done with policy-gradients calculated using the norm of the velocity as dense rewards, while the exploration term uses the JS-kernel. As before, we compare an ensemble of 8 interacting agents with 8 independent agents. Figures 4a and 4b depict the paths taken by the Swimmer after training with independent and interacting agents, respectively. The latter exhibit variety. FIG7 is the downstream task of Swimming+Gathering where the bot has to swim and collect the green dots, whilst avoiding the red ones. The utility of pre-training a diverse ensemble is shown in FIG7, which plots the performance on this task while training a higher-level categorical manager policy (|A| = 8). Diversity can sometimes also help in learning a skill without any rewards from the environment, as observed by BID10 in recent work. We consider a Hopper task with no rewards, but we do require weak supervision in form of the length of each trajectory L. Using policy-gradient Let the policy parameters be parameterized by θ. To achieve diverse, high-return policies, we seek to obtain the distribution q * (θ) which is the solution of the optimization problem: max q E θ∼q [η(θ)] + αH(q), where H(q) = E θ∼q [− log q(θ)] is the entropy of q. Solving the above equation by setting derivative to zero yields the an energy-based formulation for the optimal policy-parameter distribution: q * (θ) ∝ exp(η(θ) α ). Drawing samples from this posterior using traditional methods such as MCMC is computationally intractable. Stein variational gradient descent (SVGD; BID26) is an efficient method for generating samples and also converges to the posterior of the energy-based model. Let {θ} n 1 be the n particles that constitute the policy ensemble. SVGD provides appropriate direction for perturbing each particle such that induced KL-divergence between the particles and the target distribution q * (θ) is reduced. The perturbation (gradient) for particle θ i is given by (please see BID26 for derivation): DISPLAYFORM0 DISPLAYFORM1 where k(θ j, θ i) is a positive definite kernel function. Using q * (θ) ∝ exp(DISPLAYFORM2 α) as target distribution, and k(θ j, θ i) = exp(−D JS (ρ π θ j, ρ π θ i)/T ) as the JS-kernel, we get the gradient direction for ascent: DISPLAYFORM3 where ρ π θ (s, a) is the state-action visitation distribution for policy π θ, and T is the temperature. Also, for our case, ∇ θj η(π θj) is the interpolated gradient from self-imitation (Equation 5). The −∇ θj D JS (ρ π θ j, ρ π θ i) gradient in the above equation is the repulsion factor that pushes π θi away from π θj. Similar repulsion can be achieved by using the gradient +∇ θi D JS (ρ π θ j, ρ π θ i); note that this gradient is w.r.t θ i instead of θ j and the sign is reversed. Empirically, we find that the latter in slightly better performance. DISPLAYFORM0 This can be done in two ways -using implicit and explicit distributions. In the implicit method, we could train a parameterized discriminator network (φ) using state-actions pairs from π i and π j to implicitly approximate the ratio r s, a) ]. We could then use the policy gradient theorem to obtain the gradient of D JS as explained in Section 2.2. This, however, requires us to learn O(n 2) discriminator networks for a population of size n, one for each policy pair (i, j). To reduce the computational and memory resource burden to O(n), we opt for explicit modeling of ρ πi. Specifically, we train a network ρ ψi to approximate the state-action visitation density for each policy π i. The ρ ψ1... ρ ψn networks are learned using the D JS optimization (Equation 2), and we can easily obtain the ratio r ij (s, a) = ρ ψi (s, a)/[ρ ψi (s, a) + ρ ψj (s, a)]. The agent then uses log r ij (s, a) as the SVPG exploration rewards in the policy gradient theorem. DISPLAYFORM1 We use state-value function networks as baselines to reduce the variance in sampled policy-gradients. Each agent θ i in a population of size n trains n + 1 state-value networks corresponding to real environment rewards r(s, a), self-imitation rewards − log r φ (s, a), and n − 1 SVPG exploration rewards log r ij (s, a). In this section, we provide evaluation for a recently proposed method for self-imitation learning (SIL; BID31). The SIL loss function take the form: DISPLAYFORM0 In words, the algorithm buffers (s, a) and the corresponding return (R) for each transition in rolled trajectories, and reuses them for training if the stored return value is higher than the current statevalue estimate V θ (s).We use the code provided by the authors 2. As per our understanding, PPO+SIL does not use a single set of hyper-parameters for all the MuJoCo tasks (Appendix A; BID31). We follow their methodology and report numbers for the best configuration for each task. This is different from our experiments since we run all tasks on a single fix hyper-parameter set (Appendix 5.5), and therefore a direct comparison of the average scores between the two approaches is tricky. Table 3: Performance of PPO+SIL BID31 on tasks with episodic rewards, noisy rewards with masking probability pm, and dense rewards. All runs use 5M timesteps of interaction with the environment. Table 3 shows the performance of PPO+SIL on MuJoCo tasks under the various reward distributions explained in Section 3.1 -dense, episodic and noisy. We observe that, compared to the dense rewards setting (default Gym rewards), the performance suffers under the episodic case and when the rewards are masked out with p m = 0.9. Our intuition is as follows. PPO+SIL makes use of the cumulative return (R) from each transition of a past good rollout for the update. When rewards are provided only at the end of the episode, for instance, cumulative return does not help with the temporal credit assignment problem and hence is not a strong learning signal. Our approach, on the other hand, derives dense, per-timestep rewards using an objective based on divergence-minimization. This is useful for credit assignment, and as indicated in Table 1. (Section 3.1) leads to learning good policies even under the episodic and noisy p m = 0.9 settings. Our approach makes use of replay memory M E to store the past good rollouts of the agent. Offpolicy RL methods such as DQN BID28 also accumulate agent experience in a replay buffer and reuse them for learning (e.g. by reducing TD-error). In this section, we evaluate the performance of one such recent algorithm -Twin Delayed Deep Deterministic policy gradient (TD3; BID14) on tasks with episodic and noisy rewards. TD3 builds on DDPG BID25 and surpasses its performance on all the MuJoCo tasks evaluated by the authors. Table 4: Performance of TD3 BID14 on tasks with episodic rewards, noisy rewards with masking probability pm, and dense rewards. All runs use 5M timesteps of interaction with the environment. Table 4 shows that the performance of TD3 suffers appreciably with the episodic and noisy p m = 0.9 reward settings, indicating that popular off-policy algorithms (DDPG, TD3) do not exploit the past experience in a manner that accelerates learning when rewards are scarce during an episode. * For 3 tasks used in our paper-Swimmer and the high-dimensional Humanoid, HumanoidStandup-the TD3 code from the authors 3 is unable to learn a good policy even in presence of dense rewards (default Gym rewards). These tasks are also not included in the evaluation by BID14. We run a new exploration baseline -EX 2 BID13 and compare its performance to SIinteract-JS on the hard exploration MuJoCo tasks considered in Section 3.2. The EX 2 algorithm does implicit state-density ρ(s) estimation using discriminative modeling, and uses it for noveltybased exploration by adding − log ρ(s) as the bonus. We used the author provided code 4 and hyperparameter settings. TRPO is used as the policy gradient algorithm. BID13 and SI-interact-JS on the hard exploration MuJoCo tasks from Section 3.2. SparseHalfCheetah, SparseHalfCheetah, SparseAnt use 1M, 1M and 2M timesteps of interaction with the environment, respectively. Results are averaged over 3 separate runs.

Represent the TLDR for finding the corresponding scientific article:

Policy optimization by using past good rollouts from the agent; learning shaped rewards via divergence minimization; SVPG with JS-kernel for population-based exploration.

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the scientific article to find a short TLDR

Question: We study the precise mechanisms which allow autoencoders to encode and decode a simple geometric shape, the disk. In this carefully controlled setting, we are able to describe the specific form of the optimal solution to the minimisation problem of the training step. We show that the autoencoder indeed approximates this solution during training. Secondly, we identify a clear failure in the generalisation capacity of the autoencoder, namely its inability to interpolate data. Finally, we explore several regularisation schemes to resolve the generalisation problem. Given the great attention that has been recently given to the generative capacity of neural networks, we believe that studying in depth simple geometric cases sheds some light on the generation process and can provide a minimal requirement experimental setup for more complex architectures. Autoencoders are neural networks, often convolutional neural networks, whose purpose is twofold. Firstly, to compress some input data by transforming it from the input domain to another space, known as the latent, or code, space. The second goal of the autoencoder is to take this latent representation and transform it back to the original space, such that the output is similar, with respect to some criterion, to the input. One of the main objectives of this learning process being to reveal important structure in the data via the latent space, and therefore to represent this data in a more meaningful fashion or one that is easier to model. Autoencoders have been proven to be extremely useful in many tasks ranging from image compression to synthesis. Many variants on the basic idea of autoencoders have been proposed, the common theme being how to impose useful properties on the learned latent space. However, very little is known about the actual inner workings and mechanisms of the autoencoder. The goal of this work is to investigate these mechanisms and describe how the autoencoder functions. Many applications of autoencoders or similar networks consider relatively high-level input objects, ranging from the MNIST handwritten digits to abstract sketches of conceptual objects BID18; BID7 ). Here, we take a radically different approach. We consider, in depth, the encoding/decoding processes of a simple geometric shape, the disk, and investigate how the autoencoder functions in this case. There are several important advantages to such an approach. Firstly, since the class of objects we consider has an explicit parametrisation, it is possible to describe the "optimal" performance of the autoencoder, ie. can it compress and uncompress a disk to and from a code space of dimensionality 1? Secondly, the setting of this study fixes certain architecture characteristics of the network, such as the number of layers, leaving fewer free parameters to tune. This means that the which we obtain are more likely to be robust than in the case of more high-level applications. Finally, it is easier to identify the roles of different components of the network, which enables us to carry out an instructive ablation study. Using this approach, we show that the autoencoder approximates the theoretical solution of the training problem when no biases are involved in the network. Secondly, we identify certain limitations in the generalisation capacity of autoencoders when the training database is incomplete with respect to the underlying manifold. We observe the same limitation using the architecture of BID18, which is considerably more complex and is proposed to encode natural images. Finally, we analyse several regularisation schemes and identify one in particular which greatly aids in overcoming this generalisation problem. The concept of autoencoders has been present for some time in the learning community (; BID3). The objective is to train two networks, an "encoder" and a "decoder", which transform the input data to and from a code, or latent, space which is learned by the algorithm. In many applications, the dimensionality d of the latent space is smaller than that of the original data, so that the autoencoder is encouraged to discover useful features of the data. In practice, we obviously do not know the exact value of d, but we would still like to impose as much structure in the latent space as possible. This idea lead to the regularisation in the latent space of autoencoders, which comes in several flavours. The first is the sparse autoencoder BID14 ), which attempts to have as few active (non-zero) neurons as possible in the network. This can be done either by modifying the loss function to include sparsity-inducing penalisations, or by acting directly on the values of the code z. In the latter option, one can use rectified linear units (ReLUs) to encourage zeros in the code ) or simply specifying a maximum number of non-zero values as in the "k-sparse" autoencoder BID12 ). Another approach, taken by the variational autoencoder, is to specifying the a priori distribution of the code z. BID9 use the Kullback-Leibler divergence to achieve this goal, and the authors suppose a Gaussian distribution of z. The "contractive" autoencoder BID16 ) encourages the derivatives of the code with respect to the input image to be small, meaning that the representation of the image should be robust to small changes in the input. Autoencoders can be applied to a variety of problems, such as denoising ("denoising autoencoder") or image compression BID1 ). For a good overview of autoencoders, see the book of Goodfellow et al. ). Recently, a great deal of attention has been given to the capacity of CNNs, and in particular generative adversarial networks (GANs) BID13 ) or autoencoders, to generate new images. It is well-known that these networks have important limitations, such as the tendency to produce low quality images or to reproduce images from the training set because of mode collapse. But despite these limitations, many works have investigated the generative capacity of such networks, see for instance BID4; BID17; BID15; BID18 careful study of the autoencoder are the main goals of the paper, and structure our work throughout. Before continuing, we describe our autoencoder in a more formal fashion. We denote input images with x ∈ R m×n and z ∈ R d, where m and n are the height and the width of the image, respectively, and d is the dimension of z. The autoencoder consists of the couple (E, D), the encoder and decoder which transform to and from the "code" space, with E: DISPLAYFORM0 As mentioned, the goal of the auto-encoder is to compress and uncompress a signal into a representation with a smaller dimensionality, while losing as little information as possible. Thus, we search for the parameters of the encoder and the decoder, which we denote with Θ E and Θ D respectively, by minimising DISPLAYFORM1 The autoencoder consists of a series of convolutions with filters of small compact support, subsampling/up-sampling, biases and non-linearities. The values of the filters are termed the weights of the network, and we denote the encoding filters with w,i, where is the layer number and i the number of the filter. Similarly, we denote the decoding filters w,i, the encoding and decoding biases b,i and b,i. We choose leaky ReLUs for the non-linearities: DISPLAYFORM2 with parameter α = 0.2. Thus, the output of a given encoding layer is given by DISPLAYFORM3 and similarly for the decoding layers (except for an zero-padding upsampling prior to the convolution), with weights and biases w and b, respectively. We consider images of a fixed (square) spatial support DISPLAYFORM4 and also that the subsampling rate s is fixed. In the encoder, subsampling is carried out until and z is a single scalar. Thus, the number of layers in our encoder and decoder is not an independent parameter. We set the support of all the convolutional filters in our network to 3 × 3. The architecture of our autoencoder remains the same throughout the paper, and is shown in FIG0. We summarise our parameters in Table 1. We now investigate the inner mechanics of autoencoders in the case of a simple geometric shape: the disk. Our training set consists of binary images of centred disks of random radii, with one disk per image in the test database. Each disk image is determined by the indicator function of a disk of radius r, and is therefore binary. Theoretically, an optimal encoder would only need one scalar to represent the image. Therefore the architecture in FIG0 is set up to ensure a code size d = 1. Our first important observation (see FIG1) is that not only can the network learn to encode/decode On the left side, we have interpolated z in the latent space between two encoded input disks (one small and one large), and show the decoded, output image. It can be seen that the training works well, with the ing code space being meaningful. On the right, we plot the radii of the input disks against their codes z ∈ R. The autoencoder appears to represent the disks with their area.disks, but that the code z which is learned can be interpolated and the corresponding decoding is meaningful. Thus, in this case, the autoencoder is able to encode/decode the data in an optimal fashion. We now proceed to see how the autoencoder actually works on a detailed level, starting with the encoding step. Encoding a centred disk of a certain radius to a scalar z can be done in several ways, the most intuitive being integrating over the area of the disk (encoding a scalar proportionate to its area) or integrating over the perimeter of the disk (encoding a scalar proportionate to its radius). The empirical evidence given by our experiments points towards the first option, since z seems to represent the area and not the radius of the input disks (see FIG1). If this is the case, the integration operation can be done by means of a simple cascade of linear filters. As such, we should be able to encode the disks with a network containg only convolutions and sub-sampling, and no having non-linearities. We have verified experimentally this with such an encoder. A more difficult question is how does the autoencoder convert a scalar, z, to an output disk of a certain size (the decoding process). One approach to understanding the inner workings of autoencoders, and indeed any neural network, is to remove certain elements of the network and to see how it responds, otherwise known as an ablation study. We found that removing the biases of the autoencoder leads to very interesting observations. While, as we have shown, the encoder is perfectly able to function without these biases, this is not the case for the decoder. FIG2 shows the of this ablation. The decoder learns to spread the energy of z in the output according to a certain function g. Thus, the goal of the biases is to shift the intermediary (hidden layer) images such that a cut-off can be carried out to create a satisfactory decoding. We have investigated the behaviour of the decoder without biases in detail. In particular, we will derive an explicit form for the energy minimized by the network, for which a closed form solution can be found (see Appendix A), but more importantly for which we will show experimentally that the network finds the right solution. We first make a general observation about this configuration (without biases). Consider a decoder, without biases DISPLAYFORM0, where U stands for upsampling with zero-padding. In this case, the decoder acts multiplicatively on z, meaning that ∀z, ∀λ ∈ R +, D(λz) = λD(z). Proof. For a fixed z and for any λ > 0. We have DISPLAYFORM1 This reasoning can be applied successively to each layer up to the output y. When the code z is one dimensional, the decoder can be summarized as two linear functions, one for positive codes and a second one for the negative codes. However, in all our experiments, the autoencoder without bias has chosen to use only one possible sign for the code, ing in a linear decoder. Furthermore, the profiles in FIG2 suggest that a single function is learned, and that this function is multiplied by a factor which is constant for each radius. In light of Proposition 1, this means that the decoder has chosen a fixed sign for the code and that the decoder is linear. This can be expressed as DISPLAYFORM2 where t is a spatial variable and r ∈ (0, m 2] is the radius of the disk. This is checked experimentally in Figure 7 in Appendix A. In this case, we can write the optimisation problem of the decoder aŝ DISPLAYFORM3 where R is the maximum radius observed in the training set, DISPLAYFORM4 is the image domain, and B r is the disk of radius r. Note that we have expressed the minimisation problem for continuous functions f. This is not strictly the case, especially for images of small disk radii, however for our purposes the approximation is good. In this case, we have the following proposition. Proposition 2 (Decoding Energy for an autoencoder without Biases). The decoding training problem of the autoencoder without biases has an optimal solutionf that is radially symmetric and Input Output Figure 4: Autoencoding of disks with a database with limited radii. The autoencoder is not able to extrapolate further than the largest observed radius. The images with a green border represent disks whose radii have been observed during training, while those in red have not been observed.maximises the following energy: DISPLAYFORM5 under the (arbitrary) normalization f 2 2 = 1.Proof. When f is fixed, the optimal h for Equation FORMULA8 is given bŷ DISPLAYFORM6 where f, 1 Br = Ω f (t)1 Br (t) dt. After replacing this in Equation, we find that DISPLAYFORM7 where we have chosen the arbitrary normalisation f 2 2 = 1. The form of the last equation shows that the optimal solution is obviously radially symmetric 1. Therefore, after a change of variables, the energy maximised by the decoder can be written as DISPLAYFORM8 such that f 2 2 = 1. In Appendix A, we compare the numerical solution of this problem with the actual profile learned by the network, yielding a very close match. This is very interesting, since it shows that the training process has achieved the optimal solution, in spite of the fact that the loss is non convex. As we have recalled in Section2, many works have recently investigated the generative capacity of autoencoders or GANs. Nevertheless, it is not clear that these architectures truly invent or generalize some visual content. A simpler question is: to what extent is the network able to generalise a simple geometric notion? In this section, we address this issue in our restricted but interpretable case. 1 If not, then consider its mean on every circle, which decreases the L 2 norm of f while maintaining the scalar product with any disk. We then can increase back the energy by deviding by this smaller L 2 norm according to f 2 = 1. Figure 5: Input and output of our network when autoencoding examples of disks when the database contains a "hole". Disks of radii between 11 and 18 pixels (out of 32) were not observed in the database. In green, the disks whose radii have been observed in the database, in red those which have not. For this, we study the behaviour of our autoencoder when examples are removed from the training dataset. In Figure 4, we show the autoencoder when the disks with radii above a certain threshold R are removed. The radii of the left three images (with a green border) are present in the training database, whereas the radii of the right three (red border) have not been observed. It is clear that the network lacks the capacity to extrapolate further than this radius. Indeed, the autoencoder seems to project these disks onto smaller, observed, disks, rather than learning the abstraction of a disk. Again by removing the biases from the network, we may explain why the autoencoder fails to extrapolate when a maximum radius R is imposed. In Appendix B, we show experimental evidence that in this situation, the autoencoder learns a function f whose support is restricted by the value of R, leading to the autoencoder's failure. However, a fair criticism of the previous experiment is simply that the network (and deep learning in general) is not designed to work on data which lie outside of the domain observed in the training data set. Nevertheless, it is reasonable to expect the network to be robust to such "holes" inside the domain. Therefore, we have also analysed the behaviour of the autoencoder when we removed training datapoints whose disks' radii lie within a certain range, between 11 and 18 pixels (out of a total of 32). We then attempt to reconstruct these points in the test data. Figure 5 shows the of this experiment. Once again, in the unknown regions the network is unable to recreate the input disks. (page 521) and BID2 propose several explanations in the deep learning literature of this phenomenon, such as a high curvature of the underlying data manifold, noisy data or high intrinsic dimensionality of the data. In our setting, none of these explanations is sufficient. Thus we conclude that, even in the simple setting of disks, the "classic" autoencoder cannot generalise correctly when a database contains holes. This behavior is potentially problematic for applications which deal with more complex natural images, lying on a high-dimensional manifold, as these are likely to contain such holes. We have therefore carried out the same experiments using the state-of-the-art "iGAN" approach of BID18, which is in turn based on the work of BID13, "DCGAN". The visual of their algorithm are displayed in Appendix C. We trained their network using both a code size of d = 100 (as proposed by the authors), and d = 1 in order to ensure fair comparisons. Indeed, in our case, not only the dimension of the latent space should be d = 1, but also the amount of training data is not enough to work with d = 100. Type-3 regularisation Figure 6: Result of different types of regularisation on autoencoding in an "unknown region" of the training database. We have encoded/decoded a disk which was not observed in the training dataset. We show the of four experiments: no regularisation, 2 regularisation in the latent space ("Type 1"), 2 weight penalisation of the encoder and decoder ("Type 2") and 2 weight penalisation of the encoder only ("Type 3").both cases the network fails to correctly autoencode the disks belonging to the unobserved region. This shows that the generalisation problem is likely to be ubiquitous, and indeed observed in more sophisticated networks, designed to learn natural images manifolds, even in the simple case of disks. We therefore believe that this issue deserves careful attention. Actually this experiment suggets that the capacity to generate new and simple geometrical shapes could be taken as a minimal requirement for a given architecture. In order to address the problem, we now investigate several regularisation techniques whose goal is to aid the generalisation capacity of neural networks. We would like to impose some structure on the latent space in order to interpolate correctly in the case of missing datapoints. This is often achieved via some sort of regularisation. This regularisation can come in many forms, such as imposing a certain distribution in the latent space, as in variational autoencoders BID9 ), or by encouraging z to be sparse, as in sparse auto-encoders BID14; BID12 ). In the present case, the former is not particularly useful, since a probabilistic approach will not encourage the latent space to correctly interpolate. The latter regularisation does not apply, since we already have d = 1. Another commonly used approach is to impose an 2 penalisation of the weights of the filters in the network. The idea behind this bears some similarity to sparse regularisation; we wish for the latent space to be as "simple" as possible, and therefore hope to avoid over-fitting. We have implemented several regularisation techniques on our network. Firstly, we attempt a simple regularisation of the latent space by requiring a "locality-preservation" property as suggested in BID8; BID0; BID11, namely that the 2 distance between two images (x,x) be maintained in the latent space. This is done by randomly selecting a neighbour of each element in the training batch. Secondly, we regularise the weights of the encoder and/or the decoder. Thus, our training attempts to minimise the sum of the data term, x − D(E(x)) 2 2, and a regularisation term λψ(x, θ), which can take one of the following forms: DISPLAYFORM0 • Type 2: DISPLAYFORM1 • Type 3: DISPLAYFORM2 2 2; Figure 6 shows the of these experiments. First of all, we observe that the type 1 regularisation does not work satisfactorily. One interpretation of this is that the manifold in the training data is "discontinuous", and therefore there are no close neighbours for the disks on the edge of the unobserved region. Therefore, this regularisation is to be avoided in cases where there are significant holes in the sampling of the data manifold. The second type of regularisation, minimising the 2 norm of the encoder and decoder weights, produces an interesting effect. Indeed, while the manifold seems reasonable, upon closer inspection, the code z increases in amplitude during the training. Thus, the network cannot converge to a stable solution, which worsens the quality of the . Finally, we observe that regularising the weights of the encoder works particularly well, and that the ing manifold is continuous and correctly represents the area of the disks. Consequently, this asymmetrical regularisation approach is to be encouraged in other applications of autoencoders. At this point, we take the opportunity to note that the clear, marked effects seen with the different regularisation approaches are consistently observed in different training runs. This is due in large part to the controlled, simple setting of autoencoding with disks. Indeed, many other more sophisticated networks, especially GANs, are known to be very difficult to trainSalimans et al. FORMULA2, leading to unstable or poor reproducibility. We believe that our approach can be of use to more high-level applications, by making it easier to clearly identify which components and regularisations schemes best help in processing complex input data. We have investigated in detail the specific mechanisms which allow autoencoders to encode image information in an optimal manner in the specific case of disks. We have shown that, in this case, the encoder functions by integrating over disk, and so the code z represents the area of the disk. In the case where the autoencoder is trained with no bias, the decoder learns a single function which is multiplied by scalar depending on the input. We have shown that this function corresponds to the optimal function. The bias is then used to induce a thresholding process applied to ensure the disk is correctly decoded. We have also illustrated certain limitations of the autoencoder with respect to generalisation when datapoints are missing in the training set. This is especially problematic for higher-level applications, whose data have higher intrinsic dimensionality and therefore are more likely to include such "holes". Finally, we identify a regularisation approach which is able to overcome this problem particularly well. This regularisation is asymmetrical as it consists of regularizing the encoder while leaving more freedom to the decoder. An important future goal is to extend the theoretical analyses obtained to increasingly complex visual objects, in order to understand whether the same mechanisms remain in place. We have experimented with other simple geometric objects such as squares and ellipses, with similar in an optimal code size. Another question is how the decoder functions with the biases included. This requires a careful study of the different non-linearity activations as the radius increases. Finally, the ultimate goal of these studies is to determine the capacity of autoencoders to encode and generate images representing more complex objects or scenes. As we have seen, the proposed framework can help identifying some limitations of complex networks such as the one from BID18 and future works should investigate whether this framework can help developing the right regularization scheme or architecture. Value of < f, 1 Br >, plotted against z Figure 7: Verification of the hypothesis that y(t, r) = h(r)f (t) for decoding in the case where the autoencoder contains no bias.. We have determined the average profile of the output of the autoencoder when no biases are involved. On the left, we have divided several random experimental profiles y by the function h, and plotted the , which is close to constant (spatially) for a fixed radius of the input disk. On the right, we plot z against the theoretically optimal value of h (C f, 1 Br, where C is some constant accounting for the arbitrary normalization of f). This experimental sanity check confirms our theoretical derivations. During the training of the autoencoder for the case of disks (with no bias in the autoencoder), the objective of the decoder is to convert a scalar into the image of a disk with the 2 distance as a metric. Given the profiles of the output of the autoencoder, we have made the hypothesis that the decoder approximates a disk of radius r with a function y(t; r) = h(r)f (t), where f is a continuous function. We show that this is true experimentally in Figure 7 by determining f experimentally by taking the average of all output profiles, and showing the pointwise division of f by randomly selected output profiles. We see that h is approximately constant for varying t and fixed r. Please note that we have removed the last spatial coordinate of the profile which suffers from border effects. We now compare the numerical optimisation of the energy in Equation using a gradient descent approach with the profile obtained by the autoencoder without biases. The ing comparison can be seen in FIG5. One can also derive a closed form solution of Equation FORMULA10 by means of the Euler-Lagrange equation and see that the optimal f for Equation FORMULA10 is the solution of the differential equation y = −kty with initial state (y, y) =, where k is a free positive constant that accommodates for the position of the first zero of y. This gives a closed form of the f in terms of Airy functions. In Figure 9, we see the grey-levels of the input/output of an autoencoder trained (without biases) on a restricted database, that is to say a database whose disks have a maximum radius R which is smaller than the image width. We have used R = 18 for these experiments. We see that the decoder learns a useful function f which only extends to this maximum radius. Beyond this radius, another function is used corresponding to the other sign of codes (see proposition 1) that is not tuned. C AUTOENCODING DISKS WITH THE IGAN ZHU In FIG0, we show the autoencoding of the IGAN network of Zhu et al. We trained their network with a code size of both z = 100 and z = 1. Although the IGAN works better in the latter case, in both experiments the network fails to correctly autoencode disks in the missing radius region which has not been observed in the training database. Disk profile Output profile Figure 9: Profile of the encoding/decoding of centred disks, with a restricted database. The decoder learns a profile f which only extends to the largest observed radius R = 18. Beyond this radius, another profile is learned that has is obviously not tuned to any data. Output, d = 100Output, d = 1 FIG0: Input and output of the network of Zhu et al. BID18 ("IGAN") for disks when the database is missing disks of certain radii. We have applied the IGAN with a code size of d = 100, as in the original paper, and d = 1 as in our autoencoder. In both cases the IGAN interpolates incorrectly in the unknown region. Outlined in green are the images with observed radii and in red the unobserved radii.

We study the functioning of autoencoders in a simple setting and advise new strategies for their regularisation in order to obtain bettre generalisation with latent interpolation in mind for image sythesis.

represent the short summary for finding the corresponding scientific article (long)

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.