Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments. These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces. However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy. In this work, we propose an approach using deep representation learning algorithms to learn an adequate goal space. This is a developmental 2-stage approach: first, in a perceptual learning stage, deep learning algorithms use passive raw sensor observations of world changes to learn a corresponding latent space; then goal exploration happens in a second stage by sampling goals in this latent space. We present experiments with a simulated robot arm interacting with an object, and we show that exploration algorithms using such learned representations can closely match, and even sometimes improve, the performance obtained using engineered representations. Spontaneous exploration plays a key role in the development of knowledge and skills in human children. For example, young children spend a large amount of time exploring what they can do with their body and external objects, independently of external objectives such as finding food or following instructions from adults. Such intrinsically motivated exploration BID6 BID17 BID31 ) leads them to make ratcheting discoveries, such as learning to locomote or climb in various styles and on various surfaces, or learning to stack and use objects as tools. Equipping machines with similar intrinsically motivated exploration capabilities should also be an essential dimension for lifelong open-ended learning and artificial intelligence. In the last two decades, several families of computational models have both contributed to a better understanding of such exploration processes in infants, and how to apply them efficiently for autonomous lifelong machine learning. One general approach taken by several research groups BID1 BID4 BID16 has been to model the child as intrinsically motivated to make sense of the world, exploring like a scientist that imagines, selects and runs experiments to gain knowledge and control over the world. These models have focused in particular on three kinds of mechanisms argued to be essential and complementary to enable machines and animals to efficiently explore and discover skill repertoires in the real world BID10: embodiment 1, intrinsic motivation and social guidance 3. This article focuses on challenges related to learning goal representations for intrinsically motivated exploration, but also leverages models of embodiment, through the use of parameterized Dynamic Movement Primitives controllers BID20 and social guidance, through the use of observations of another agent. Given an embodiment, intrinsically motivated exploration 4 consists in automatically and spontaneously conducting experiments with the body to discover both the world dynamics and how it can be controlled through actions. Computational models have framed intrinsic motivation as a family of mechanisms that self-organize agents exploration curriculum, in particular through generating and selecting experiments that maximize measures such as novelty BID0 BID48, predictive information gain BID25, learning progress BID43 BID21, compression progress BID44, competence progress, predictive information BID26 or empowerment BID41. When used in the Reinforcement Learning (RL) framework (e.g. BID48 BID43 BID21 BID4), these measures have been called intrinsic rewards, and they are often applied to reward the "interestingness" of actions or states that are explored. They have been consistently shown to enable artificial agents or robots to make discoveries and solve problems that would have been difficult to learn using a classical optimization or RL approach based only on the target reward (which is often rare or deceptive) BID11 BID47. Recently, they have been similarly used to guide exploration in difficult deep RL problems with sparse rewards, e.g. BID5 BID19 BID49 BID35.However, many of these computational approaches have considered intrinsically motivated exploration at the level of micro-actions and states (e.g. considering low-level actions and pixel level perception). Yet, children's intrinsically motivated exploration leverages abstractions of the environments, such as objects and qualitative properties of the way they may move or sound, and explore by setting self-generated goals BID53, ranging from objects to be reached, toy towers to be built, or paper planes to be flown. A computational framework proposed to address this higher-level form of exploration has been Intrinsically Motivated Goal Exploration Processes (IMGEPs) BID2 BID15, which is closely related to the idea of goal babbling BID39. Within this approach, agents are equipped with a mechanism enabling them to sample a goal in a space of parameterized goals 5, before they try to reach it by executing an experiment. Each time they sample a goal, they dedicate a certain budget of experiments time to improve the solution to reach this goal, using lower-level optimization or RL methods for example. Most importantly, in the same time, they take advantage of information gathered during this exploration to discover other outcomes and improve solutions to other goals 6.This property of cross-goal learning often enables efficient exploration even if goals are sampled randomly in goal spaces containing many unachievable goals. Indeed, generating random goals (including unachievable ones) will very often produce goals that are outside the convex hull of already discovered outcomes, which in turn leads to exploration of variants of known corresponding policies, pushing the convex hull further. Thus, this fosters exploration of policies that have a high probability to produce novel outcomes without the need to explicitly measure novelty. This explains why forms of random goal exploration are a form of intrinsically motivated exploration. However, more powerful goal sampling strategies exist. A particular one consists in using meta-learning algorithms to monitor the evolution of competences over the space of goals and to select the next goal to try, according to the expected competence progress ing from practicing it. This enables to automate curriculum sequences of goals of progressively increasing complexity, which has been shown to allow high-dimensional real world robots to acquire efficiently repertoires of locomotion skills or soft object manipulation, or advanced forms of nested tool use BID15 Here a goal is not necessarily an end state to be reached, but can characterize certain parameterized properties of changes of the world, such as following a parameterized trajectory. 6 E.g. while learning how to move an object to the right, they may discover how to move it to the left.sampling them randomly BID9 BID27 or adaptively BID13 ).Yet, a current limit of existing algorithms within the family of Intrinsically Motivated Goal Exploration Processes is that they have assumed that the designer 7 provides a representation allowing the autonomous agent to generate goals, together with formal tools used to measure the achievement of these goals (e.g. cost functions). For example, the designer could provide a representation that enables the agent to imagine goals as potential continuous target trajectories of objects BID15, or reach an end-state starting from various initial states defined in Euclidean space BID13, or realize one of several discrete relative configurations of objects BID9 ), which are high-level abstractions from the pixels. While this has allowed to show the power of intrinsically motivated goal exploration architectures, designing IMGEPs that sample goals from a learned goal representation remains an open question. There are several difficulties. One concerns the question of how an agent can learn in an unsupervised manner a representation for hypothetical goals that are relevant to their world before knowing whether and how it is possible to achieve them with the agent's own action system. Another challenge is how to sample "interesting" goals using a learned goal representation, in order to remain in regions of the learned goal parameters that are not too exotic from the underlying physical possibilities of the world. Finally, a third challenge consists in understanding which properties of unsupervised representation learning methods enable an efficient use within an IMGEP architecture so as to lead to efficient discovery of controllable effects in the environment. In this paper, we present one possible approach named IMGEP-UGL where aspects of these difficulties are addressed within a 2-stage developmental approach, combining deep representation learning and goal exploration processes:Unsupervised Goal space Learning stage (UGL): In the first phase, we assume the learner can passively observe a distribution of world changes (e.g. different ways in which objects can move), perceived through raw sensors (e.g. camera pixels or other forms of low-level sensors in other modalities). Then, an unsupervised representation learning algorithm is used to learn a lower-dimensional latent space representation (also called embedding) of these world configurations. After training, a Kernel Density Estimator (KDE) is used to estimate the distribution of these observations in the latent space. In the second phase, the embedding representation and the corresponding density estimation learned during the first stage are reused in a standard IMGEP. Here, goals are iteratively sampled in the embedding as target outcomes. Each time a goal is sampled, the current knowledge (forward model and meta-policy, see below) enables to guess the parameters of a corresponding policy, used to initialize a time-bounded optimization process to improve the cost of this policy for this goal. Crucially, each time a policy is executed, the observed outcome is not only used to improve knowledge for the currently selected goal, but for all goals in the embedding. This process enables the learner to incrementally discover new policy parameters and their associated outcomes, and aims at learning a repertoire of policies that produce a maximally diverse set of outcomes. A potential limit of this approach, as it is implemented and studied in this article, is that representations learned in the first stage are frozen and do not evolve in the second stage. However, we consider here this decomposition for two reasons. First, it corresponds to a well-known developmental progression in infant development: in their first few weeks, motor exploration in infants is very limited (due to multiple factors), while they spend a considerable amount of time observing what is happening in the outside world with their eyes (e.g. observing images of social peers producing varieties of effects on objects). During this phase, a lot of perceptual learning happens, and this is reused later on for motor learning (infant perceptual development often happens ahead of motor development in several important ways). Here, passive perceptual learning from a database of visual effects observed in the world in the first phase can be seen as a model of this stage where infants learn by passively observing what is happening around them 8. A second reason for this decomposi-tion is methodological: given the complexity of the underlying algorithmic components, analyzing the dynamics of the architecture is facilitated when one decomposes learning in these two phases (representation learning, then exploration).Main contribution of this article. Prior to this work, and to our knowledge, all existing goal exploration process architectures used a goal space representation that was hand designed by the engineer, limiting the autonomy of the system. Here, the main contribution is to show that representation learning algorithms can discover goal spaces that lead to exploration dynamics close to the one obtained using an engineered goal representation space. The proposed algorithmic architecture is tested in two environments where a simulated robot learns to discover how to move and rotate an object with its arm to various places (the object scene being perceived as a raw pixel map). The objective measure we consider, called KL-coverage, characterizes the diversity of discovered outcomes during exploration by comparing their distribution with the uniform distribution over the space of outcomes that are physically possible (which is unknown to the learner). We even show that the use of particular representation learning algorithms such as VAEs in the IMGEP-UGL architecture can produce exploration dynamics that match the one using engineered representations. • We show that the IMGEP-UGL architecture can be successfully implemented (in terms of exploration efficiency) using various unsupervised learning algorithms for the goal space learning component: AutoEncoders (AEs) BID8, Variational AE (VAE) BID38 BID22, VAE with Normalizing Flow , Isomap BID50, PCA BID36, and we quantitatively compare their performances in terms of exploration dynamics of the associated IMGEP-UGL architecture.• We show that specifying more embedding dimensions than needed to capture the phenomenon manifold does not deteriorate the performance of these unsupervised learning algorithms.• We show examples of unsupervised learning algorithms (Radial Flow VAEs) which produce less efficient exploration dynamics than other algorithms in our experiments, and suggest hypotheses to explain this difference. In this section, we first present an outline of intrinsically motivated goal exploration algorithmic architectures (IMGEPs) as originally developed and used in the field of developmental robotics, and where goal spaces are typically hand crafted. Then, we present a new version of this architecture (IMGEP-UGL) that includes a first phase of passive perceptual learning where goal spaces are learned using a combination of representation learning and density estimation. Finally, we outline a list of representation learning algorithms that can be used in this first phase, as done in the experimental section. Intrinsically Motivated Goal Exploration Processes (IMGEPs), are powerful algorithmic architectures which were initially introduced in BID2 and formalized in BID15. They can be used as heuristics to drive the exploration of high-dimensional continuous action spaces so as to learn forward and inverse control models in difficult robotic problems. To clearly understand the essence of IMGEPs, we must envision the robotic agent as an experimenter seeking information about an unknown physical phenomenon through sequential experiments. In this perspective, the main elements of an exploration process are:• A context c, element of a Context Space C. This context represents the initial experimental factors that are not under the robotic agent control. In most cases, the context is considered fully observable (e.g. state of the world as measured by sensors).have considered how stronger forms of social guidance, such as imitation learning BID42, could accelerate intrinsically motivated goal exploration BID28 ), but they did not consider the challenge of learning goal representations.• A parameterization θ, element of a Parameterization Space Θ. This parameterization represents the experimental factors that can be controlled by the robotic agent (e.g. parameters of a policy).• An outcome o, element of an Outcome Space O. The outcome contains information qualifying properties of the phenomenon during the execution of the experiment (e.g. measures characterizing the trajectory of raw sensor observations during the experiment).• A phenomenon dynamics D: C, Θ → O, which in most interesting cases is unknown. If we take the example of the Arm-Ball problem 9 in which a multi-joint robotic arm can interact with a ball, the context could be the initial state of the robot and the ball, the parameterization could be the parameters of a policy that generate a sequence of motor torque commands for N time steps, and the outcome could be the position of the ball at the last time step. Developmental roboticists are interested in developing autonomous agents that learn two models, the forward modelD: C × Θ → O which approximates the phenomenon dynamics, and the inverse modelĨ: C × O → Θ which allows to produce desired outcomes under given context by properly setting the parameterization. Using the aforementioned elements, one could imagine a simple strategy that would allow the agent to gather tuples {c, θ, o} to train those models, by uniformly sampling a random parameterization θ ∼ U(θ) and executing the experiment. We refer to this strategy as Random Parameterization Exploration. The problem for most interesting applications in robotics, is that only a small subspace of Θ is likely to produce interesting outcomes. Indeed, considering again the Arm-Ball problem with time-bounded action sequences as parameterizations, very few of those will lead the arm to touch the object and move it. In this case, a random sampling in Θ would be a terrible strategy to yield interesting samples allowing to learn useful forward and inverse models for moving the ball. To overcome this difficulty, one must come up with a better approach to sample parameterizations that lead to informative samples. Intrinsically Motivated Goal Exploration Strategies propose a way to address this issue by giving the agent a set of tools to handle this situation:• A Goal Space T whose elements τ represent parameterized goals that can be targeted by the autonomous agent. In the context of this article, and of the IMGEP-UGL architecture, we consider the simple but important case where the Goal Space is equated with the Outcome space. Thus, goals are simply vectors in the outcome space that describe target properties of the phenomenon that the learner tries to achieve through actions.• A Goal Policy γ(τ), which is a probability distribution over the Goal Space used for sampling goals (see Algorithmic Architecture 2). It can be stationary, but in most cases, it will be updated over time following an intrinsic motivation strategy. Note that in some cases, this Goal Policy can be conditioned on the context γ(τ |c).• A set of Goal-parameterized Cost Functions C τ: O → R defined over all O, which maps every outcome with a real number representing the goodness-of-fit of the outcome o regarding the goal τ. As these cost functions are defined over O, this enables to compute the cost of a policy for a given goal even if the goal is imagined after the policy roll-out. Thus, as IMGEPs typically memorize the population of all executed policies and their outcomes, this enables reuse of experimentations across multiple goals.• A Meta-Policy Π: T, C → Θ which is a mechanism to approximately solve the minimization problem Π(τ, c) = arg min θ C τ (D(θ, c)), whereD is a running forward model (approximating D), trained on-line during exploration. In some applications, a de-facto ensemble of such tools can be used. For example, in the case where O is an Euclidean space, we can allow the agent to set goals in the Outcome Space T = O, in which case for every goal τ we can consider a Goal-parameterized cost function DISPLAYFORM0. is a similarity metric. In the case of the Arm-Ball problem, the final position of the ball can be used as Outcome Space, hence the Euclidean distance between the goal position and the final ball position at the end of the episode can be used as Goal-parameterized cost function (but one could equally choose the full trajectories of the ball as outcomes and goals, and an associated similarity metric).Algorithmic architecture 2 describes the main steps of Intrinsically Motivated Goal Exploration Processes using these tools 10:Bootstrapping phase: Sampling a few policy parameters (called Random Parametrization Exploration, RPE), observing the starting context and the ing outcome, to initialize a memory of experiments (H = {(c i, θ i, o i)}) and a regressorD running approximating the phenomenon dynamics. Goal exploration phase: Stochastically mixing random policy exploration with goal exploration. In goal exploration, one first observes the context c and then samples a goal τ using goal policy γ (this goal policy can be a random stationary distribution, as in experiments below, or a contextual multi-armed bandit maximizing information gain or competence progress, see). Then, a meta-policy algorithm Π is used to search the parameterization θ minimizing the Goal-parameterized cost function C τ, i.e. it computes θ = arg min θ C τ (D running (θ, c)). This process is typically initialized by searching the parameter θ init in H such that the corresponding c init is in the neighborhood of c and C τ (o init) is minimized. Then, this initial guess is improved using an optimization algorithm (e.g. L-BFGS) over the regressorD running. The ing policy θ is executed, and the outcome o is observed. The observation (c, θ, o) is then used to update H andD running.This procedure has been experimentally shown to enable sample efficient exploration in highdimensional continuous action robotic setups, enabling in turn to learn repertoires of skills in complex physical setups with object manipulations using tools BID14 BID15 or soft deformable objects BID28.Nevertheless, two issues arise when it comes to using these algorithms in real-life setups, and within a fully autonomous learning approach. First, there are many real world cases where providing an Outcome Space (in which to make observations and sample goals, so this is also the Goal Space) to the agent is difficult, since the designer may not himself understand well the space that the robot is learning about. The approach taken until now BID15, was to create an external program which extracted information out of images, such as tracking all objects positions. This information was presented to the agent as a point in n, which was hence considered as an Outcome Space. In such complex environments, the designer may not know what is actually feasible or not for the robot, and the Outcome space may contain many unfeasible goals. This is the reason why advanced mechanisms for sampling goals and discovering which ones are actually feasible have been designed BID15. Second, a system where the engineer designs the representation of an Outcome Space space is limited in its autonomy. A question arising from this is: can we design a mechanism that allows the agent to construct an Outcome Space that leads to efficient exploration by the mean of examples? Representation Learning methods, in particular Deep Learning algorithms, constitute a natural approach to this problem as it has shown outstanding performances in learning representations for images. In the next two sections, we present an update of the IMGEP architecture that includes a goal space representation learning stage, as well as various Deep Representation Learning algorithms tested: Autoencoders along with their more recent Variational counter-parts. In order to enable goal space representation learning within the IMGEP framework, we propose to add a first stage of unsupervised perceptual learning (called UGL) before the goal exploration stage, leading to the new IMGEP-UGL architecture described in Algorithmic Architecture 1. In the passive perceptual learning stage (UGL, lines 2-8), the learner passively observes the unknown phenomenon by collecting samples x i of raw sensor values as the world changes. The architecture is neutral with regards to how these world changes are produced, but as argued in the introduction, one can see them as coming from actions of other agents in the environment. Then, this database of x i observations is used to train an unsupervised learning algorithm (e.g. VAE, Isomap) to learn an embedding functionR which maps the high-dimensional raw sensor observations onto a lower-10 IMGEPs characterize an architecture and not an algorithm as several of the steps of this architecture can be implemented in multiple ways, for e.g. depending on which regression or meta-policy algorithms are implemented dimensional representation o. Also, a kernel density estimator KDE estimates the distribution p kde (o) of observed world changes projected in the embedding. Then, in the goal exploration stage (lines 9-26), this lower-dimensional representation o is used as the outcome and goal space, and the distribution p kde (o) is used as a stochastic goal policy, within a standard IMGEP process (see above). Learn an embedding functionR: DISPLAYFORM0 Estimate the outcome distribution p kde (o) from {R(x i)} i∈ using algorithm KDE 9 Set the Goal Policy γ = p kde to be the estimated outcome distribution As IMGEP-UGL is an algorithmic architecture, it can be implemented with several algorithmic variants depending on which unsupervised learning algorithm is used in the UGL phase. We experimented over different deep and classical Representation Learning algorithms for the UGL phase. We rapidly outline these algorithms here. For a more in-depth introduction to those models, the reader can refer to Appendix B which contains details on the derivations of the different Cost Functions and Architectures of the Deep Neural Networks based models. Auto-Encoders (AEs) are a particular type of Feed-Forward Neural Networks that were introduced in the early hours of neural networks BID8. They are trained to output a reconstructionx of the input vector x of dimension D, through a representation layer of size d < D. They can be trained in an unsupervised manner using a large dataset of unlabeled samples D = {x (i) } i∈{0... N}. Their main interest lies in their ability to model the statistical regularities existing in the data. Indeed, during training, the network learns the regularities allowing to encode most of the information existing in the input in a more compact representation. Put differently, AEs can be seen as learning a non-linear compression for data coming from an unknown distribution. Those models can be trained using different algorithms, the most simple being Stochastic Gradient Descent (SGD), to minimize a loss function J (D) that penalizes differences betweenx and x for all samples in D.Variational Auto-Encoders (VAEs) are a recent alternative to classic AEs BID38 BID22, that can be seen as an extension to a stochastic encoding. The argument underlying this model is slightly more involved than the simple approach taken for AEs, and relies on a statistical standpoint presented in Appendix B. In practice, this model simplifies to an architecture very similar to an AE, differing only in the fact that the encoder f θ outputs the parameters µ and σ of a multivariate Gaussian distribution N (µ, diag(σ 2)) with diagonal covariance matrix, from which the representation z is sampled. Moreover, an extra term is added to the Cost Function, to condition the distribution of z in the representation space. Under the restriction that a factorial Gaussian is used, the neural network can be made fully differentiable thanks to a reparameterization trick, making it possible to use SGD for training. In practice VAEs tend to yield smooth representations of the data, and are faster to converge than AEs from our experiments. Despite these interesting properties, the derivation of the actual cost function relies mostly on the assumption that the factors can be described by a factorial Gaussian distribution. This hypothesis can be largely erroneous, for example if one of the factors is periodic, multi-modal, or discrete. In practice our experiments showed that even if training could converge for non-Gaussian factors, it tends to be slower and to yield poorly conditioned representations. Normalizing Flow proposes a way to overcome this restriction on distribution, by allowing more expressive ones . It uses the classic rule of change of variables for random variables, which states that considering a random variable z 0 ∼ q(z 0), and an invertible transformation t: DISPLAYFORM1 Using this, we can chain multiple transformations t 1, t 2,..., t K to produce a new random variable z K = t K • · · · • t 2 • t 1 (z 0). One particularly interesting transformation is the Radial Flow, which allows to radially contract and expand a distribution as can be seen in FIG5 in Appendix. This transformation seems to give the required flexibility to encode periodic factors. Isomap is a classical approach of Multi-Dimensional Scaling BID24 a procedure allowing to embed a set of N -dimensional points in a n dimensional space, with N > n, minimizing the Kruskal Stress, which measures the distortion induced by the embedding in the pairwise Euclidean distances. This algorithm in an embedding whose pairwise distances are roughly the same as in the initial space. Isomap BID50 goes further by assuming that the data lies in the vicinity of a lower dimensional manifold. Hence, it replaces the pairwise Euclidean distances in the input space by an approximate pairwise geodesic distance, computed by the Dijkstra's Shortest Path algorithm on a κ nearest-neighbors graph. Principal Component Analysis is an ubiquitous procedure BID36 which, for a set of data points, allows to find the orthogonal transformation that yields linearly uncorrelated data. This transformation is found by taking the principal axis of the covariance matrix of the data, leading to a representation whose variance is in decreasing order along dimensions. This procedure can be used to reduce dimensionality, by taking only the first n dimensions of the transformed data. Estimation of sampling distribution: Since the Outcome Space O was learned by the agent, it had no prior knowledge of p(o) for o ∈ O. We used a Gaussian Kernel Density Estimation (KDE) BID34 BID40 to estimate this distribution from the projection of the images observed by the agent, into the learned goal space representation. Kernel Density Estimation allows to estimate the continuous density function (cdf) f (o) out of a discrete set of samples {o i} i∈{1,...,n} drown from distribution p(o). The estimated cdf is computed using the following equation: DISPLAYFORM2 with K(·) a kernel function and H a bandwidth d × d matrix (d the dimension of O). In our case, we used a Gaussian Kernel: DISPLAYFORM3 with the bandwidth matrix H equaling the covariance matrix of the set of points, rescaled by factor n − 1 d+4, with n the number of samples, as proposed in. We conducted experiments to address the following questions in the context of two simulated environments:• Is it possible for an IMGEP-UGL implementation to produce a Goal Space representation yielding an exploration dynamics as efficient as the dynamics produced by an IMGEP implementation using engineered goal space representations? Here, the dynamics of exploration is measured through the KL Coverage defined thereafter.• What is the impact of the target embedding dimensionality provided to these algorithms? We now present in depth the experimental campaign we performed 11.Environments: We experimented on two different Simulated Environments derived from the Arm-Ball benchmark represented in FIG1, namely the Arm-Ball and the Arm-Arrow environments, in which a 7-joint arm, controlled by a 21 continuous dimension Dynamic Movement Primitives (DMP) BID20 controller, evolves in an environment containing an object it can handle and move around in the scene. In the case of IMGEP-UGL learners, the scene is perceived as a 70x70 pixel image. For the UGL phase, we used the following mechanism to generate the distribution of samples x i: the object was moved randomly uniformly over [−1, 1] 2 for ArmBall, and over [−1, 1] 2 × [0, 2π] for ArmArrow, and the corresponding images were generated and provided as an observable sample to IMGEP-UGL learners. Note that the physically reachable space (i.e. the largest space the arm can move the object to) is the disk centered on 0 and of radius 1: this means that the distribution of object movements observed by the learner is slightly larger than the actual space of moves that learners can produce themselves (and learners have no knowledge of which subspace corresponds to physically feasible outcomes). The environments are presented in depth in Appendix C.Algorithmic Instantiation of the IMGEP-UGL Architecture: We experimented over the following Representation Learning Algorithms for the UGL component: Auto-Encoders with KDE (RGE-AE), Variational Auto-Encoders with KDE (RGE-VAE), Variational Auto-Encoders using the associated Gaussian prior for sampling goal instead of KDE (RGE-VAE-GP), Radial Flow Variational Auto-Encoders with KDE (RGE-RFVAE), Radial Flow Variational Auto-Encoders using the associated Gaussian prior for sampling goal (RGE-RFVAE-GP), Isomap (RGE-Isomap) BID50 and Principal Component Analysis (RGE-Isomap).Regarding the classical IMGEP components, we considered the following elements:• Context Space C = ∅: In the implemented environments, the initial positions of the arm and the object were reset at each episode 12. Consequently, the context was not observed nor accounted for by the agent. • Parameterization Space Θ = 21: During the experiments, we used DMP controllers as parameterized policies to generate time-bounded motor actions sequences. Since the DMP controller was parameterized by 3 basis functions for each joint of the arm FORMULA12, the parameterization of the controller was represented by a point in DISPLAYFORM0 The Outcome Space is the subspace of R l spanned by the embedding representations of the ensemble of images observed in the first phase of learning. For the RGE-EFR algorithm, l = 2 in ArmBall and l = 3 in ArmArrow. For IMGEP-UGL algorithms, as the representation learning algorithms used in the UGL stage require a parameter specifying the maximum dimensionality of the target embedding, we considered two cases in experiments: 1) l = 10, which is 5 times larger than the true manifold dimension for ArmBall, and 3.3 times larger for ArmArrow (the algorithm is not supposed to know this, so testing the performance with larger embedding dimension is key); 2) l = 2 for ArmBall, and l = 3 for ArmArrow, which is the same dimensionality as the true dimensions of these manifolds.• Goal Space T = O: The Goal Space was taken to equate the Outcome Space.• Goal-Parameterized Cost function C τ (·) = τ − · 2: Sampling goals in the Outcome Space allows us to use the Euclidean distance as Goal-parameterized cost function. Considering those elements, we used the instantiation of the IMGEP architecture represented in Appendix D in Algorithm 3. We implemented a goal sampling strategy known as Random Goal Exploration (RGE), which consists, given a stationary distribution over the Outcome Space p(o), in sampling a random goal o ∼ p(o) each time (note that this stationary distribution p(o) is learnt in the UGL stage for IMGEP-UGL implementations). We used a simple k-neighbors regressor to implement the running forward modelD, and the Meta-Policy mechanism consisted in returning the nearest achieved outcome in the outcome space, and taking the same parameterization perturbed by an exploration noise (which has proved to be a very strong baseline in IMGEP architectures in previous works BID14).Exploration Performance Measure: In this article, the central property we are interested in is the dynamics and quality of exploration of the outcome space, characterizing the evolution of the distribution of discovered outcomes, i.e. the diversity of effects that the learner discovers how to produce. In order to characterize this exploration dynamics quantitatively, we monitored a measure which we refer to as Kullback-Leibler Coverage (KLC). At a given point in time during exploration, this measure computes the KL-divergence between the distribution of the outcomes produced so far, with a uniform distribution of outcomes in the space of physically possible outcomes (which is known by the experimenter, but unknown by the learner). To compute it, we use a normalized histogram of the explored outcomes, with 30 bins per dimension, which we refer to as E, and we compute its Kullback Leibler Divergence with the normalized histogram of attainable points which we refer to as A: DISPLAYFORM1 We emphasize that, when computed against a uniform distribution, the KLC measure is a proxy for the (opposite) Entropy of the E distribution. Nevertheless, we prefer to keep it under the divergence form, as the A distribution allows to define what the experimenter considers to be a good exploration distribution. In the case of this study, we consider a uniform distribution of explored locations over the attainable domain, to be the best exploration distribution achievable. Baseline algorithms: We are using two natural baseline algorithms for evaluating the exploration dynamics of our IMGEP-UGL algorithmic implementations: This is an IMGEP implementation using a goal/outcome space with handcrafted features that directly encode the underlying structure of environments: for Arm-Ball, this is the 2D position of the ball in 2, and for Arm-Arrow this is the 2D position and the 1D orientation of the arrow in 3. This algorithm is also given the prior knowledge of p(o) = U(O). All other aspects of the IMGEP (regressor, meta-policy, other parameters) are identical to IMGEP-UGL implementations. This algorithm is known to provide highly efficient exploration dynamics in these environments BID14.• Random Parameterization Exploration (RPE): The Random Parameterization Exploration approach does not use an Outcome Space, nor a Goal Policy, and only samples a random parameterization θ ∼ U(Θ) at each episode. We expected this algorithm to lower bound the performances of our novel architecture. We first study the exploration dynamics of all IMGEP-UGL algorithms, comparing them to the baselines and among themselves. Then, we study specifically the impact of the target embedding dimension (latent space) for the UGL implementations, by observing what exploration dynamics is produced in two cases:• Using a target dimension larger than the true dimension (l = 10)• Providing the true embedding dimension to the UGL implementations (l = 2, 3)Finally, we specifically study RGE-VAE, using the intrinsic Gaussian prior of these algorithms to replace the KDE estimator of p(O) in the UGL part. Exploration Performances: In FIG2, we can see the evolution of the KLC through exploration epochs (one exploration epoch is defined as one experimentation/roll-out of a parameter θ). We can see that for both environments, and all values of latent spaces, all IMGEP-UGL algorithms, except RGE-RFVAE, achieve similar or better performance (both in terms of asymptotic KLC and speed to reach it) than the RGE-EFR algorithm using engineered Goal Space features, and much better performance than the RPE algorithm. Figure 3 (see also Figure 8 and 9 in Appendix) show details of the evolution of discovered outcomes in ArmBall (final ball positions after the end of a policy roll-out) and corresponding KLC measures for individual runs with various algorithms. It also shows the evolution of the number of times learners managed to move the ball, which is considered in the KLC measure but not easily visible in the displayed set of outcomes in FIG3. For instance, we observe that both RPE FIG3 ) and RGE-RFVAE FIG3 ) algorithms perform poorly: they discover very few policies moving the ball at all (pink curves), and these discovered ball moves cover only a small part of the physically possible outcome space. On the contrary, both RGE-EFR (handcrafted features) and RGE-VAE (learned goal space representation with VAE) perform very well, and the KLC of RGE-VAE is even better than the KLC of RGE-EFR, due to the fact that RGE-VAE has discovered more policies (around 2400) that move the ball than RGE-EFR (around 1600, pink curve). Impact of target latent space size in IMGEP-UGL algorithms On the ArmBall problem, we observe that if one provides the true target embedding dimension (l = 2) to IMGEP-UGL implementations, RGE-Isomap is slightly improving (getting quasi-identical to RGE-EFR), RGE-AE does not change (remains quasi-identical to RGE-EFR), but the performance of RGE-PCA and RGE-VAE is degraded. For ArmArrow, the effect is similar: IMGEP-UGL algorithms with a larger target embedding dimension (l = 10) than the true dimensionality all perform better than RGE-EFR (except RGE-RFVAE which is worse in all cases), while when l = 2 only RGE-VAE is significantly better than RGE-EFR. In Appendix F, more examples of exploration curves with attached exploration scatters are shown. For most example runs, increasing the target embedding dimension enables learners to discover more policies moving the ball and, in these cases, the discovered outcomes are more concentrated towards the external boundary of the discus of physically possible outcomes. This behavior, where increasing the target embedding dimension improves the KLC while biasing the discovered outcome towards the boundary the feasible goals, can be understood as a consequence of the following well-known general property of IMGEPs: if goals are sampled outside the convex hull of outcomes already discovered, this has the side-effect of biasing exploration towards policies that will produce outcomes beyond this convex hull (until the boundary of feasible outcomes is reached). Here, as observations in the UGL phase were generated by uniformly moving the objects on the square [−1, 1] 2, while the feasible outcome space was the smaller discus of radius 1, goal sampling happened in a distribution of outcomes larger than the feasible outcome space. As one increases the embedding space dimensionality, the ratio between the volume of the corresponding hyper-cube and hyper-discus increases, in turn increasing the probability to sample goals outside the feasible space, which has the side effect of fostering the discovery of novel outcomes and biasing exploration towards the boundaries. Impact of Sampling Kernel Density Estimation Another factor impacting the exploration assessed during our experiments was the importance of the distribution used as stationary Goal Policy. If, in most cases, the representation algorithm gives no particular prior knowledge of p(o), in the case of Variational Auto-Encoders, it is assumed in the derivation that p(o) = N (0, I). Hence, the isotropic Gaussian distribution is a better candidate stationary Goal Policy than Kernel Density Estimation. FIG4 shows a comparison between exploration performances achieved with RGE-VAE using a KDE distribution or an isotropic Gaussian as Goal Policy. The performance is not significantly different from the isotropic Gaussian case. Our experiments showed that convergence on the KL term of the loss can be more or less quick depending on the initialization. Since we used a number of iterations as stopping criterion for training (based on early experiments), we found that sometimes, at stop, the divergence was still pretty high despite achieving a low reconstruction error. In those cases the representation was not be perfectly matching an isotropic Gaussian, which could lead to a goal sampling bias when using the isotropic Gaussian Goal Policy. In this paper, we proposed a new Intrinsically Motivated Goal Exploration architecture with Unsupervised Learning of Goal spaces (IMGEP-UGL). Here, the Outcome Space (also used as Goal Space) representation is learned using passive observations of world changes through low-level raw sensors (e.g. movements of objects caused by another agent and perceived at the pixel level).Within the perspective of research on Intrinsically Motivated Goal Exploration started a decade ago, and considering the fundamental problem of how AI agents can autonomously explore environments and skills by setting their own goals, this new architecture constitutes a milestone as it is to our knowledge the first goal exploration architecture where the goal space representation is learned, as opposed to hand-crafted. Furthermore, we have shown in two simulated environments (involving a high-dimensional continuous action arm) that this new architecture can be successfully implemented using multiple kinds of unsupervised learning algorithms, including recent advanced deep neural network algorithms like Variational Auto-Encoders. This flexibility opens the possibility to benefit from future advances in unsupervised representation learning research. Yet, our experiments have shown that all algorithms we tried (except RGE-RFVAE) can compete with an IMGEP implementation using engineered feature representations. We also showed, in the context of our test environments, that providing to IMGEP-UGL algorithms a target embedding dimension larger than the true dimensionality of the phenomenon can be beneficial through leveraging exploration dynamics properties of IMGEPs. Though we must investigate more systematically the extent of this effect, this is encouraging from an autonomous learning perspective, as one should not assume that the learner initially knows the target dimensionality. Limits and future work. The experiments presented here were limited to a fairly restricted set of environments. Experimenting over a larger set of environments would improve our understanding of IMGEP-UGL algorithms in general. In particular, a potential challenge is to consider environments where multiple objects/entities can be independently controlled, or where some objects/entities are not controllable (e.g. animate entities). In these cases, previous work on IMGEPs has shown that random Goal Policies should be either replaced by modular Goal Policies (considering a modular goal space representation, see BID15), or by active Goal Policies which adaptively focus the sampling of goals in subregions of the Goal Space where the competence progress is maximal. For learning modular representations of Goal Spaces, an interesting avenue of investigations could be the use of the Independently Controllable Factors approach proposed in BID51.Finally, in this paper, we only studied a learning scenario where representation learning happens first in a passive perceptual learning stage, and is then fixed during a second stage of autonomous goal exploration. While this was here motivated both by analogies to infant development and to facilitate evaluation, the ability to incrementally and jointly learn an outcome space representation and explore the world is a stimulating topic for future work. Bernouilli distribution of ξ parameters 13, and the log likelihood of the dataset D is expressed as: DISPLAYFORM0 with DISPLAYFORM1. For a binary valued input vector x (i), the unitary Cost Function to minimize is: DISPLAYFORM2 provided that f θ is the encoder part of the architecture and g φ is the decoding part of the architecture. This Cost Function can be minimized using Stochastic Gradient Descent BID7, or more advanced optimizers such as Adagrad BID12 ) or Adam (.Depending on the depth of the network 14, those architectures can prove difficult to train using vanilla Stochastic Gradient Descent. A particularly successful procedure to overcome this difficulty is to greedily train each pairs of encoding-decoding layers and stacking those to sequentially form the complete network. This procedure, known as stacked AEs, accelerates convergence. But it has shown bad with our problem, and thus was discarded for the sake of clarity. Variational Auto-Encoders (VAEs) If we assume that the observed data are realizations of a random variable x ∼ p(x|ψ), we can hypothesize that they are conditioned by a random vector of independent factors z ∼ p(z|ψ). In this setting, learning the model would amount to searching the parameters ψ of both distributions. We might use the same principle of maximum likelihood as before to find the best parameters by computing the likelihood log L(D) = N i=1 log p(x (i) |ψ) by using the fact that p(x|ψ) = p(x, z|ψ)dz = p(x|z, ψ)p(z|ψ)dz. Unfortunately, in most cases, this integral is intractable and cannot be approximated by Monte-Carlo sampling in reasonable time. To overcome this problem, we can introduce an arbitrary distribution q(z|x, χ) and remark that the following holds: DISPLAYFORM3 with the Evidence Lower Bound being: DISPLAYFORM4 Looking at Equation FORMULA10, we can see that since the KL divergence is non-negative, L(q, ψ) ≤ log p(x|ψ) − D KL ([q(z|x, χ) p(z|x, ψ)] whatever the q distribution, hence the name of Evidence Lower Bound (ELBO). Consequently, maximizing the ELBO have the effect to maximize the log likelihood, while minimizing the KL-Divergence between the approximate q(z|x) distribution, and the true unknown posterior p(z|x, ψ). The approach taken by VAEs is to learn the parameters of both conditional distributions p(x|z, ψ) and q(z|x, χ) as non-linear functions. Under some restricted conditions, Equation can be turned into a valid cost function to train a neural network. First, we hypothesize that q(z|x, χ) and p(z|ψ) follow Multivariate Gaussian distributions with diagonal covariances, which allows us to compute the b term in closed form. Second, using the Gaussian assumption on q, we can reparameterize the inner sampling operation by z = µ + σ 2 with ∼ N (0, I). Using this trick, the Path-wise Derivative estimator can be used for the a member of the ELBO. Under those conditions, and assuming that p(x|ψ) follows a Multivariate Bernouilli distribution, we can write the cost function used to train the neural network as: DISPLAYFORM5 where f χ represents the encoding and sampling part of the architecture and g ψ represents the decoding part of the architecture. In essence, this derivation simplifies to the initial cost function used in AEs augmented by a term penalizing the divergence between q(z|x, χ) and the assumed prior that p(x|ψ) = N (0, I).Normalizing Flow overcomes the problem stated earlier, by permitting more expressive prior distributions . It is based on the classic rule of change of variables for random variables. Considering a random variable z 0 ∼ q(z 0), and an invertible transformation t: DISPLAYFORM6 We can then directly chain different invertible transformations t 1, t 2,..., t K to produce a new ran- DISPLAYFORM7. In this case, we have: DISPLAYFORM8 This formulation is interesting because the Law Of The Unconscious Statistician allows us to compute expectations over q(z k) without having a precise knowledge of it: DISPLAYFORM9 provided that h does not depends on q(z k). Using this principle on the ELBO allows us to derive the following: DISPLAYFORM10 This is nothing more than the regular ELBO with an additional term concerning the log-determinant of the transformations. In practice, as before, we use p(z 0) = N (z 0 ; 0, I), and q(z 0 |x) = N (z 0 ; µ(x), diag(σ(x)2 )). We only have to find out parameterized transformations t, whose parameters can be learned and have a defined log-determinant. Using radial flow, which is expressed as: DISPLAYFORM11 where r = |z − c|, h(α, r) = 1 α+r and α, β, c are learnable parameters of the transformation, our cost function can be written as: DISPLAYFORM12 (1 + log(σ( DISPLAYFORM13 provided that f χ represents the encoding, sampling ad transforming part of the architecture, g ψ represents the decoding part of the architecture, and β k, α k, c k are the parameters of the different transformations. Other types of transformations have been proposed lately. The Householder flow BID52) is a volume preserving transformation, meaning that its log determinant equals 1, with the consequence that it can be used with no modifications of the loss function. A more convoluted type of transformations based on a masked autoregressive auto-encoder, the Inverse Autoregressive Flow, was proposed in BID23. We did not explore those two last approaches. The following environments were considered:• Arm-Ball: A 7 joints arm, controlled in angular position, can move around in an environment containing a ball. The environment state is perceived visually as a 50x50 pixels image. The arm has a sticky arm tip: if the tip of the arm touches the ball, the ball sticks to the arm until the end of the movement. The underlying state of the environment is hence parameterized by two bounded continuous factors which represent the coordinates of the ball. A situation can be sampled by the experimenter by taking a random point in 2.• Arm-Arrow: The same arm can manipulate an arrow in a plane, an arrow being considered as an object with a single symmetry that can be oriented in space. Consequently, the underlying state of the environment is parameterized by two bounded continuous factors representing the coordinates of the arrow, and one periodic continuous factor representing its orientation. A particular situation can hence be sampled by taking a random point in 3.The physical situations were represented by small 70x70 images very similar to the dSprites dataset proposed by BID18 15. The arm was not depicted in the field of view of the (virtual) camera used to gather images for representation learning. We used a robotic arm composed of 7 joints, whose motions were parameterized by DMPs using 3 basis functions (hence action policies have 21 continuous parameters), during 50 time-steps. An example of such a DMP executed in the environment is represented in FIG6. The first phase, where the learner observes changes of the environment (= ball moves) caused by another agent, is modeled by a process which samples iteratively a random state in the underlying state space, e.g. in the case of Arm-Ball s ∼ U ( 2 ), and then generating the corresponding image x = f (s) that is observed by the learner. For the experiments, we instantiated the Algorithmic Architecture 1 into Algorithm 3.In the text, Algorithm 3 is denoted (RGE-), where denotes any representation learning algorithm: (RGE-AE) for Auto-Encoders, (RGE-VAE) for Variational Auto-Encoders, (RGE-RFVAE) Image, 70x70

Represent the TLDR for finding the corresponding scientific article

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: One of the main challenges of deep learning methods is the choice of an appropriate training strategy. In particular, additional steps, such as unsupervised pre-training, have been shown to greatly improve the performances of deep structures. In this article, we propose an extra training step, called post-training, which only optimizes the last layer of the network. We show that this procedure can be analyzed in the context of kernel theory, with the first layers computing an embedding of the data and the last layer a statistical model to solve the task based on this embedding. This step makes sure that the embedding, or representation, of the data is used in the best possible way for the considered task. This idea is then tested on multiple architectures with various data sets, showing that it consistently provides a boost in performance. One of the main challenges of the deep learning methods is to efficiently solve the highly complex and non-convex optimization problem involved in the training step. Many parameters influence the performances of trained networks, and small mistakes can drive the algorithm into a sub-optimal local minimum, ing into poor performances BID0. Consequently, the choice of an appropriate training strategy is critical to the usage of deep learning models. The most common approach to train deep networks is to use the stochastic gradient descent (SGD) algorithm. This method selects a few points in the training set, called a batch, and compute the gradient of a cost function relatively to all the layers parameter. The gradient is then used to update the weights of all layers. Empirically, this method converges most of the time to a local minimum of the cost function which have good generalization properties. The stochastic updates estimate the gradient of the error on the input distribution, and several works proposed to use variance reduction technique such as Adagrap BID2, RMSprop BID8 or Adam , to achieve faster convergence. While these algorithms converge to a local minima, this minima is often influenced by the properties of the initialization used for the network weights. A frequently used approach to find a good starting point is to use pre-training BID12 BID5. This method iteratively constructs each layer using unsupervised learning to capture the information from the data. The network is then fine-tuned using SGD to solve the task at hand. Pretraining strategies have been applied successfully to many applications, such as classification tasks BID0 BID17, regression BID6, robotics BID4 or information retrieval BID19. The influence of different pre-training strategies over the different layers has been thoroughly studied in BID13. In addition to improving the training strategies, these works also shed light onto the role of the different layers BID3 BID15. The first layers of a deep neural network, qualified as general, tend to learn feature extractors which can be reused in other architectures, independently of the solved task. Meanwhile, the last layers of the network are much more dependent of the task and data set, and are said to be specific. Deep Learning generally achieves better than shallow structures, but the later are generally easier to train and more stable. For convex models such as logistic regression, the training problem is also convex when the data representation is fixed. The separation between the representation and the model learning is a key ingredient for the model stability. When the representation is learned simultaneously, for instance with dictionary learning or with EM algorithms, the problem often become non-convex. But this coupling between the representation and the model is critical for end-to-end models. For instance, showed that for networks trained using pretraining, the fine-tuning step -where all the layers are trained together -improves the performances of the network. This shows the importance of the adaptation of the representation to the task in end-to-end models. Our contribution in this chapter is an additional training step which improves the use of the representation learned by the network to solve the considered task. This new step is called post-training. It is based on the idea of separating representation learning and statistical analysis and it should be used after the training of the network. In this step, only the specific layers are trained. Since the general layers -which encode the data representation -are fixed, this step focuses on finding the best usage of the learned representation to solve the desired task. In particular, we chose to study the case where only the last layer is trained during the post-training, as this layer is the most specific one BID22. In this setting, learning the weights of the last layer corresponds to learning the weights for the kernel associated to the feature map given by the previous layers. The post-training scheme can thus be interpreted in light of different from kernel theory. To summarize our contributions:• We introduce a post-training step, where all layers except the last one are frozen. This method can be applied after any traditional training scheme for deep networks. Note that this step does not replace the end-to-end training, which co-adapts the last layer representation with the solver weights, but it makes sure that this representation is used in the most efficient way for the given task.• We show that this post-training step is easy to use, that it can be effortlessly added to most learning strategies, and that it is computationally inexpensive.• We highlight the link existing between this method and the kernel techniques. We also show numerically that the previous layers can be used as a kernel map when the problem is small enough.• We experimentally show that the post-training does not overfit and often produces improvement for various architectures and data sets. The rest of this article is organized as follows: Section 2 introduces the post-training step and discusses its relation with kernel methods. Section 4 presents our numerical experiments with multiple neural network architectures and data sets and Section 5 discusses these . In this section, we consider a feedforward neural network with L layers, where X 1,..., X L denote the input space of the different layers, typically R d l with d l > 0 and Y = X L+1 the output space of our network. Let φ l: X l → X l+1 be the applications which respectively compute the output of the l-th layer of the network, for 1 ≤ l ≤ L, using the output of the l−1-th layer and Φ L = φ L • · · · • φ 1 be the mapping of the full network from X 1 to Y. Also, for each layer l, we denote W W W l its weights matrix and ψ l its activation function. The training of our network is done using a convex and continuous loss function: Y × Y → R +. The objective of the neural network training is to find weights parametrizing Φ L that solves the following problem: DISPLAYFORM0, drawn from this input distribution. Using these notations, the training objective can then be rewritten DISPLAYFORM1 This reformulation highlights the special role of the last layer in our network compared to the others. When Φ L−1 is fixed, the problem of finding W W W L is simple for several popular choices of activation function ψ L and loss. For instance, when the activation function ψ L is the softmax function and the loss is the cross entropy, is a multinomial logistic regression. In this case, training the last layer is equivalent to a regression of the labels y using the embedding of the data x in X L by the mapping Φ L−1. Since the problem is convex in W W W L (see Appendix A), classical optimization techniques can efficiently produce an accurate approximation of the optimal weights W W W L -and this optimization given the mapping Φ L−1 is the idea behind post-training. Indeed, during the regular training, the network tries to simultaneously learn suitable representation for the data in the space X L through its L − 1 first layer and the best use of this representation with W W W L. This joint minimization is a strongly non-convex problem, therefore ing in a potentially sub-optimal usage of the learned data representation. The post-training is an additional step of learning which takes place after the regular training and proceeds as follows:1. Regular training: This step aims to obtain interesting features to solve the initial problem, as in any usual deep learning training. Any training strategy can be applied to the network, optimizing the empirical loss DISPLAYFORM2 The stochastic gradient descent explores the parameter space and provides a solution for Φ L−1 and W W W L. This step is non restrictive: any type of training strategy can be used here, including gradient bias reduction techniques, such as Adagrad BID2, or regularization strategies, for instance using Dropout BID1. Similarly, any type of stopping criterion can be used here. The training might last for a fixed number of epochs, or can stop after using early stopping BID16. Different combinations of training strategies and stopping criterion are tested in Section 4.2. Post-training: During this step, the first L − 1 layers are fixed and only the last layer of the network, φ L, is trained by minimizing over W W W L the following problem DISPLAYFORM3 where˜ (x, y):= (ψ L (x), y). This extra learning step uses the mapping Φ L−1 as an embedding of the data in X L and learn the best linear predictor in this space. This optimization problem takes place in a significantly lower dimensional space and since there is no need for back propagation, this step is computationally faster. To reduce the risk of overfitting with this step, a 2 -regularization is added. FIG0 illustrates the post-training step. We would like to emphasize the importance of the 2 -regularization used during the post-training. This regularization is added regardless of the one used in the regular training, and for all the network architectures. The extra term improves the strong convexity of the minimization problem, making post-training more efficient, and promotes the generalization of the model. The choice of the 2 -regularization is motivated from the comparison with the kernel framework discussed in Section 3 and from our experimental . Remark 1 (Dropout.). It is important to note that Dropout should not be applied on the previous layers of the network during the post-training, as it would lead to changes in the feature function Φ L−1. In this section, we show that for the case where DISPLAYFORM0 can be approximated using kernel methods. We define the kernel k as follows, DISPLAYFORM1 Then k is the kernel associated with the feature function Φ L−1. It is easy to see that this kernel is continuous positive definite and that for DISPLAYFORM2 belongs by construction to the Reproducing Kernel Hilbert Space (RKHS) H k generated by k. The post-training problem is therefore related to the problem posed in the RKHS space H k, defined by DISPLAYFORM3 This problem is classic for the kernel methods. With mild hypothesis on˜, the generalized representer theorem can be applied BID20. As a consequence, there exists α * ∈ R N such that DISPLAYFORM4 Rewriting FORMULA8 with g * of the form, we have that g * = g W W W *, with DISPLAYFORM5 We emphasize that W W W * gives the optimal solution for the problem and should not be confused with W W W * L, the optimum of. However, the two problems differ only in their regularization, which are closely related (see the next paragraph). Thus W W W * can thus be seen as an approximation of the optimal value W W W * L. It is worth noting that in our experiments, W W W * appears to be a nearly optimal estimator of W W W * L (see Subsection 4.3).Relation between · H and · 2. The problems and only differ in the choice of the regularization norm. By definition of the RKHS norm, we have DISPLAYFORM6 Consequently, we have that g W H ≤ W 2, with equality when Vect(Φ L−1 (X 1)) spans the entire space X L. In this case, the norm induced by the RKHS is equal to the 2 -norm. This is generally the case, as the input space is usually in a far higher dimensional space than the embedding space, and since the neural network structure generally enforces the independence of the features. Therefore, while both norms can be used in, we chose to use the 2 -norm for all our experiments as it is easier to compute than the RKHS norm. Close-form Solution. In the particular case where (y 1, y 2) = y 1 − y 2 2 and f (x) = x, can be reduced to a classical Kernel Ridge Regression problem. In this setting, W * can be computed by combining FORMULA9 and DISPLAYFORM7 where DISPLAYFORM8 Y is the matrix of the output data y 1,..., y N and I I I N is the identity matrix in R N. This is experimentally illustrated in Subsection 4.3. Although data sets are generally too large for to be computed in practice, it is worth noting that some kernel methods, such as Random Features BID18, can be applied to compute approximations of the optimal weights during the post-training. Multidimensional Output. Most of the previously discussed related to kernel theory hold for multidimensional output spaces, i.e. dim(X L+1) = d > 1, using multitask or operator valued kernels BID9. Hence the previous remarks can be easily extended to multidimensional outputs, encouraging the use of post-training in most settings. This section provides numerical arguments to study post-training and its influence on performances, over different data sets and network architectures. All the experiments were run using python and Tensorflow. The code to reproduce the figures is available online 1. The of all the experiments are discussed in depth in Section 5. The post-training method can be applied easily to feedforward convolutional neural network, used to solve a wide class of real world problems. To assert its performance, we apply it to three classic benchmark datsets: CIFAR10 BID11, MNIST and FACES BID5.CIFAR10. This data set is composed of 60, 000 images 32 × 32, representing objects from 10 classes. We use the default architecture proposed by Tensorflow for CIFAR10 in our experiments, based on the original architecture proposed by BID11. It is composed of 5 layers described in FIG1. The first layers use various common tools such as local response normalization (lrn), max pooling and RELU activation. The last layer have a softmax activation function Test Error (-0.1) Figure 3: Evolution of the performances of the neural network on the CIFAR10 data set, (dashed) with the usual training and (solid) with the post-training phase. For the post-training, the value of the curve at iteration q is the error for a network trained for q − 100 iterations with the regular training strategy and then trained for 100 iterations with post-training. The top figure presents the classification error on the training set and the bottom figure displays the loss cost on the test set. The curves have been smoothed to increase readability. and the chosen training loss was the cross entropy function. The network is trained for 90k iterations, with batches of size 128, using stochastic gradient descent (SGD), dropout and an exponential weight decay for the learning rate. Figure 3 presents the performance of the network on the training and test sets for 2 different training strategies. The dashed line present the classic training with SGD, with performance evaluated every 100 iterations and the solid line present the performance of the same network where the last 100 iterations are done using post-training instead of regular training. To be clearer, the value of this curve at iteration q is the error of the network, trained for q − 100 iterations with the regular training strategy, and then trained for 100 iterations with post-training. The regularization parameter λ for post-training is set to 1 × 10 −3.The show that while the training cost of the network mildly increases due to the use of post-training, this extra step improves the generalization of the solution. The gain is smaller at the end of the training as the network converges to a local minimum, but it is consistent. Also, it is interesting to note that the post-training iterations are 4× faster than the classic iterations, due to their inexpensiveness. Additional Data Sets. We also evaluate post-training on the MNIST data set (65000 images 27 × 27, with 55000 for train and 10000 for test; 10 classes) and the pre-processed FACES data set (400 images 64 × 64, from which 102400 sub-images, 32 × 32, are extracted, with 92160 for training and 10240 for testing; 40 classes). For each data set, we train two different convolutional neural networks -to assert the influence of the complexity of the network over post-training:• a small network, with one convolutional layer (5 × 5 patches, 32 channels), one 2 × 2 max pooling layer, and one fully connected hidden layer with 512 neurons,• a large network, with one convolutional layer (5 × 5 patches, 32 channels), one 2 × 2 max pooling layer, one convolutional layer (5 × 5 patches, 64 channels), one 2 × 2 max pooling layer and one fully connected hidden layer with 1024 neurons. We use dropout for the regularization, and set λ = 1 × 10 −2. We compare the performance gain ing of the application of post-training (100 iterations) at different epochs of each of these networks. The are reported in TAB0.As seen in TAB0, post-training improves the test performance of the networks with as little as 100 iterations -which is negligible compared to the time required to train the network. While the improvement varies depending on the complexity of the network, of the data set, and of the time spent training the network, it is important to remark that it always provides an improvement. While the kernel framework developed in Section 2 does not apply directly to Recurrent Neural Network, the idea of post-training can still be applied. In this experiment, we test the performances of post-training on Long Short-Term Memory-based networks (LSTM), using PTB data set BID14.Penn Tree Bank (PTB). This data set is composed of 929k training words and 82k test word, with a 10000 words vocabulary. We train a recurrent neural network to predict the next word given the word history. We use the architecture proposed by , composed of 2 layers of 1500 LSTM units with tanh activation, followed by a fully connected softmax layer. The network is trained to minimize the average per-word perplexity for 100 epochs, with batches of size 20, using gradient descent, an exponential weight decay for the learning rate, and dropout for regularization. The performances of the network after each epoch are compared to the obtained if the 100 last steps (i.e. 100 batches) are done using post-training. The regularization parameter for posttraining, λ, is set to 1 × 10 −3. The are reported in Figure 4, which presents the evolution of the training and testing perplexity. Similarly to the previous experiments, post-training improves the test performance of the networks, even after the network has converged. In this subsection we aim to empirically evaluate the close-form solution discussed in Section 2 for regression tasks. We set the activation function of the last layer to be the identity f L (x) = x, and consider the loss function to be the least-squared error (x, y) = x − y 2 2 in. In in each experiment, and FORMULA9 are used to compute W * for the kernel learned after the regular training of Test Perplexity (-80.5) Figure 4: Evolution of the performances of the Recurrent network on the PTB data set. The top figure presents the train perplexity and the bottom figure displays the test perplexity. For the posttraining, the value of the curve at iteration q is the error for a network trained for q − 100 iterations with the regular training strategy and then trained for 100 iterations with post-training.the neural network, which learn the embedding Φ L−1 and an estimate W L. In order to illustrate this , and to compare the performances of the weights W * with respect to the weights W L, learned either with usual learning strategies or with post-training, we train a neural network on two regression problems using a real and a synthetic data set. 70% of the data are used for training, and 30% for testing. Real Data Set Regression. For this experiment, we use the Parkinson Telemonitoring data set BID21. The input consists in 5, 875 instances of 17 dimensional data, and the output are one dimensional real number. For this data set, a neural network made of two fully connected hidden layers of size 17 and 10 with respectively tanh and RELU activation, is trained for 250, 500 and 750 iterations, with batches of size 50. The layer weights are all regularized with the 2 -norm and a fixed regularization parameter λ = 10 −3. Then, starting from each of the trained networks, 200 iterations of post-training are used with the same regularization parameter λ and the performances are compared to the closed-form solutions computed using for each saved network. The are presented in TAB1.Simulated Data Set Regression. For this experiment, we use a synthetic data set. The inputs were generated using a uniform distribution on 0, 1 10. The outputs are computed as follows: DISPLAYFORM0 where W 1 ∈ −1, 1 10×5 and W 2 ∈ −1, 1 5 are randomly generated using a uniform law. In total, the data set is composed of 10, 000 pairs (x i, y j). For this data set, a neural network with two fully connected hidden layers of size 10 with activation tanh for the first layer and RELU for the second layer is trained for 250, 500 and 750 iterations, with batches of size 50. We use the same protocol with 200 extra post-training iterations. The are presented in TAB1.For these two experiments, the post-training improves the performances toward these of the optimal solution, for several choices of stopping times. It is worth noting that the performance of the optimal solution is better when the first layers are not fully optimized with Parkinson Telemonitoring data set. This effect denotes an overfitting tendency with the full training, where the first layers become overly specified for the training set. The experiments presented in Section 4 show that post-training improves the performances of all the networks considered -including recurrent, convolutional and fully connected networks. The gain is significant, regardless of the time at which the regular training is stopped and the posttraining is done. In both the CIFAR10 and the PTB experiment, the gap between the losses with and without post-training is more pronounced if the training is stopped early, and tends to be smaller as the network converges to a better solution (see Figure 4 and Figure 3). The reduction of the gap between the test performances with and without post-training is made clear in TAB0. For the MNIST data set, with a small-size convolutional neural network, while the error rate drops by 1.5% when post-training is applied after 5000 iterations, this same error rate only drops by 0.2% when it is applied after 20000 iterations. This same observation can be done for the other reported in TAB0. However, while the improvement is larger when the network did not fully converge prior to the post-training, it is still significant when the network has reached its minimum: for example in PTB the final test perplexity is 81.7 with post-training and 82.4 without; in CIFAR10 the errors are respectively 0.147 and 0.154.If the networks are allowed to moderately overfit, for instance by training them with regular algorithm for a very large number of iterations, the advantage provided by post-training vanishes: for example in PTB the test perplexity after 2000 iterations (instead of 400) is 83.2 regardless of posttraining. This is coherent with the intuition behind the post-training: after overfitting, the features learned by the network become less appropriate to the general problem, therefore their optimal usage obtained by post-training no longer provide an advantage. It is important to note that the post-training computational cost is very low compared to the full training computations. For instance, in the CIFAR10 experiment, each iteration for post-training is 4× faster on the same GPU than an iteration using the full gradient. Also, in the different experiments, post-training produces a performance gap after using as little as 100 batches. There are multiple reasons behind this efficiency: first, the system reaches a local minimum relatively rapidly for post-training as the problem FORMULA3 has a small number of parameters compared to the dimensionality of the original optimization problem. Second, the iterations used for the resolution of are computationally cheaper, as there is no need to chain high dimensional linear operations, contrarily to regular backpropagation used during the training phase. Finally, since the post-training optimization problem is generally convex, the optimization is guaranteed to converge rapidly to the optimal weights for the last layer. Another interesting point is that there is no evidence that the post-training step leads to overfitting. In CIFAR10, the test error is improved by the use of post-training, although the training loss is similar. The other experiments do not show signs of overfitting either as the test error is mostly improved by our additional step. This stems from the fact that the post-training optimization is much simpler than the original problem as it lies in a small-dimensional space -which, combined with the added 2 -regularization, efficiently prevents overfitting. The regularization parameter λ plays an important role in post-training. Setting λ to be very large reduces the explanatory capacity of the networks whereas if λ is too small, the capacity can become too large and lead to overfitting. Overall, our experiments highlighted that the post-training produces significant for any choice of λ reasonably small (i.e 10 −5 ≤ λ ≤ 10 −2). This parameter is linked to the regularization parameter of the kernel methods, as stated in Section 3.Overall, these show that the post-training step can be applied to most trained networks, without prerequisites about how optimized they are since post-training does not degrade their performances, providing a consistent gain in performances for a very low additional computational cost. In Subsection 4.3, numerical experiments highlight the link between post-training and kernel methods. As illustrated in TAB1, using the optimal weights derived from kernel theory immediately a performance boost for the considered network. The post-training step estimate numerically this optimal layer with the gradient descent optimizer. However, computing the optimal weights for the last layer is only achievable for small data set due to the required matrix inversion. Moreover, the closed form solution is known only for specific problems, such as kernelized least square regression. But post-training approaches the same performance in these cases solving with gradient-based methods. The post-training can be linked very naturally to the idea of pre-training, developed notably by BID12, and BID5. The unsupervised pre-training of a layer is designed to find a representation that captures enough information from the data to be able to reconstruct it from its embedding. The goal is thus to find suitable parametrization of the general layers to extract good features, summarizing the data. Conversely, the goal of the posttraining is, given a representation, to find the best parametrization of the last layer to discriminate the data. These two steps, in contrast with the usual training, focus on respectively the general or specific layers. In this work, we studied the concept of post-training, an additional step performed after the regular training, where only the last layer is trained. This step is intended to take fully advantage of the data representation learned by the network. We empirically shown that post-training is computationally inexpensive and provide a non negligible increase of performance on most neural network structures. While we chose to focus on post-training solely the last layer -as it is the most specific layer in the network and the ing problem is strongly convex under reasonable prerequisites -the relationship between the number of layers frozen in the post-training and the ing improvements might be an interesting direction for future works. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.arXiv preprint, arXiv:1409, 2014. We show here, for the sake of completeness, that the post-training problem is convex for the softmax activation in the last layer and the cross entropy loss. This is proved showing that the hessian of the function is positive semidefinite, as it is a diagonally dominant matrix. Proposition 2 (convexity). ∀N, M ∈ N, ∀X ∈ R N, ∀j ∈ 1, M, the following function F is convex: DISPLAYFORM0 δ ij log exp(XW i).where δ is the Dirac function, and W i denotes the i-th row of a W. Proof 1. Let DISPLAYFORM1. DISPLAYFORM2 we have DISPLAYFORM3 ∂W m,n ∂W p,q = x n ∂P m ∂W p,q, = x n x q P m (W) δ m,p − P p (W). where ⊗ is the Kronecker product, and the matrix P(W) is defined by P m,p = P m (W) δ m,p − P p (W). Now since ∀1 ≤ m ≤ M, and thus P(W) is positive semidefinite. Since XX T is positive semidefinite too, their Kronecker product is also positive semidefinite, hence the .

We propose an additional training step, called post-training, which computes optimal weights for the last layer of the network.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.