Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we focus on two challenges which offset the promise of sparse signal representation, sensing, and recovery. First, real-world signals can seldom be described as perfectly sparse vectors in a known basis, and traditionally used random measurement schemes are seldom optimal for sensing them. Second, existing signal recovery algorithms are usually not fast enough to make them applicable to real-time problems. In this paper, we address these two challenges by presenting a novel framework based on deep learning. For the first challenge, we cast the problem of finding informative measurements by using a maximum likelihood (ML) formulation and show how we can build a data-driven dimensionality reduction protocol for sensing signals using convolutional architectures. For the second challenge, we discuss and analyze a novel parallelization scheme and show it significantly speeds-up the signal recovery process. We demonstrate the significant improvement our method obtains over competing methods through a series of experiments. High-dimensional inverse problems and low-dimensional embeddings play a key role in a wide range of applications in machine learning and signal processing. In inverse problems, the goal is to recover a signal X ∈ R N from a set of measurements Y = Φ(X) ∈ R M, where Φ is a linear or non-linear sensing operator. A special case of this problem is compressive sensing (CS) which is a technique for efficiently acquiring and reconstructing a sparse signal BID12 BID6 BID1. In CS Φ ∈ R M ×N (M N) is typically chosen to be a random matrix ing in a random low-dimensional embedding of signals. In addition, X is assumed to be sparse in some basis Γ, i.e., X = ΓS, where S 0 = K N.While sparse signal representation and recovery have made significant real-world impact in various fields over the past decade , arguably their promise has not been fully realized. The reasons for this can be boiled down to two major challenges: First, real-world signals are only approximately sparse and hence, random/universal sensing matrices are sub-optimal measurement operators. Second, many existing recovery algorithms, while provably statistically optimal, are slow to converge. In this paper, we propose a new framework that simultaneously takes on both these challenges. To tackle the first challenge, we formulate the learning of the dimensionality reduction (i.e., signal sensing operator) as a likelihood maximization problem; this problem is related to the Infomax principle BID24 asymptotically. We then show that the simultaneous learning of dimensionality reduction and reconstruction function using this formulation gives a lower-bound of the objective functions that needs to be optimized in learning the dimensionality reduction. This is similar in spirit to what Vincent et al. show for denoising autoencoders in the non-asymptotic setting BID38. Furthermore, we show that our framework can learn dimensionality reductions that preserve specific geometric properties. As an example, we demonstrate how we can construct a data-driven near-isometric low-dimensional embedding that outperforms competing embedding algorithms like NuMax BID18. Towards tackling the second challenge, we introduce a parallelization (i.e., rearrangement) scheme that significantly speeds up the signal sensing and recovery process. We show that our framework can outperform state-of-the-art signal recovery methods such as DAMP BID26 and LDAMP BID25 both in terms of inference performance and computational efficiency. We now present a brief overview of prior work on embedding and signal recovery. Beyond random matrices, there are other frameworks developed for deterministic construction of linear (or nonlinear) near-isometric embeddings BID18 BID16 BID0 BID35 BID39 BID5 BID37 BID32. However, these approaches are either computationally expensive, not generalizable to outof-sample data points, or perform poorly in terms of isometry. Our framework for low-dimensional embedding shows outstanding performance on all these aspects with real datasets. Algorithms for recovering signals from undersampled measurements can be categorized based on how they exploit prior knowledge of a signal distribution. They could use hand-designed priors BID7 BID13 BID9 BID29, combine hand-designed algorithms with data-driven priors BID25 BID3 BID20 BID8 BID17, or take a purely data-driven approach BID28 BID22 BID41. As one moves from hand-designed approaches to data-driven approaches, models lose simplicity and generalizability while becoming more complex and more specifically tailored for a particular class of signals of interest. Our framework for sensing and recovering sparse signals can be considered as a variant of a convolutional autoencoder where the encoder is linear and the decoder is nonlinear and specifically designed for CS application. In addition, both encoder and decoder contain rearrangement layers which significantly speed up the signal sensing and recovery process, as we discuss later. Convolutional autoencoder has been previously used for image compression; however, our work is mainly focused on the CS application rather than image compression. In CS, measurements are abstract and linear whereas in the image compression application measurements are a compressed version of the original image and are nonlinear. Authors in have used bicubic interpolation for upscaling images; however, our framework uses a data-driven approach for upscaling measurements. Finally, unlike the image compression application, when we deploy our framework for CS and during the test phase, we do not have high-resolution images beforehand. In addition to image compression, there have been previous works BID34 BID22 to jointly learn the signal sensing and reconstruction algorithm in CS using convolutional networks. However, the problem with these works is that they divide images into small blocks and recover each block separately. This blocky reconstruction approach is unrealistic in applications such as medical imaging (e.g. MRI) where the measurement operator is a Fourier matrix and hence we cannot have blocky reconstruction. Since both papers are designed for block-based recovery whereas our method senses/recovers images without subdivision, we have not compared against them. Note that our method could be easily modified to learn near-optimal frequency bands for medical imaging applications. In addition, BID34 and BID22 use an extra denoiser (e.g. BM3D, DCN) for denoising the final reconstruction while our framework does not use any extra denoiser and yet outperforms state-of-the-art as we show later. Beside using convolutional autoencoders, authors in BID40 have introduced the sparse recovery autoencoder (SRA). In SRA, the encoder is a fully-connected layer while in this work, the encoder has a convolutional structure and is basically a circulant matrix. For large-scale problems, learning a fully-connected layer (as in the SRA encoder) is significantly more challenging than learning convolutional layers (as in our encoder). In SRA, the decoder is a T -step projected subgradient. However, in this work, the decoder is several convolutional layers plus a rearranging layer. It should also be noted that the optimization in SRA is solely over the measurement matrix and T (which is the number of layers in the decoder) scalar values. However, here, the optimization is performed over convolution weights and biases that we have across different layers of our network. In this section, we describe our framework for sparse signal representation and recovery and demonstrate how we can learn (near-)optimal projections and speed up signal recovery using parallelization along with convolutional layers. We call our framework by DeepSSRR, which stands for Deep Sparse Signal Representation and Recovery. DeepSSRR consists of two parts: A linear dimensionality reduction Φ: R N → R M for taking undersampled measurements and a nonlinear inverse mapping f Λ : R M → R N for recovering signals from their undersampled measurements. We learn both Φ and f Λ from training data. DeepSSRR FIG0 ) is based primarily on deep convolutional networks (DCN) as this gives us two advantages: (a) sparse connectivity of neurons, and (b) having shared weights which increases learning speed compared to fully-connected networks. Therefore, we impose a convolutional network architecture on both Φ and f Λ while learning them. Please note that we assume that measurements are linear; however, it is easy to extend DeepSSRR to adopt nonlinear measurements, i.e., allowing for Φ to be nonlinear by adding nonlinear units to convolutional layers. Given that the intervening layers are linear, one might argue that one convolutional layer (i.e., a single circulant matrix) is enough since we can merge kernel matrices into a single matrix. However, we consider a multi-layer architecture for learning Φ for two reasons. First, computationally it is cheaper to have separate and smaller kernels and second, it makes the implementation suitable for adding the aforementioned nonlinearities. We previously mentioned that in order to speed up the sensing and recovery process, we add a parallelization scheme in learning both Φ and f Λ that we describe in the following. Our original sensing model was Y = ΦX where X ∈ R N and Y ∈ R M. Assume that the undersampling ratio, i.e., M N is equal to 1 r. The left vector-matrix multiplication in FIG1 (a) denotes a convolution of zero-padded input signal with size N = rM = r(M + q − 1), filter size rq, stride (i.e., filter shift at every step) of size r, and output size of M. If we denote the input signal by X (in) and output by X (out) and filter by W we can write DISPLAYFORM0 If we concatenate the sub-filters and sub-signals denoted in orange in the left vector-matrix multiplication of FIG1 (a), we derive a new vector-matrix multiplication shown on the right side of FIG1 (a). There the input size is M = (M + q − 1), filter size is q, stride size is 1, and output size is M. Equation FORMULA0 states that the left convolution in FIG1 (a) can be written as the summation of r separate and parallel convolutions shown on the right side. Much like in the sensing part (i.e., learning Φ), as shown in FIG1 (b), a large strided deconvolution can be chopped into several parallel smaller deconvolutions for the recovery part (i.e., learning f Λ ). Because of these parallelizations, the computational complexity of calculating the outputs of layers in DeepSSRR is O(M) which is much less than the one for typical iterative and unrolled algorithms O(M N) (e.g. DAMP and LDAMP BID26) or previous recovery algorithms based on deep learning O(N) (e.g. DeepInverse). DISPLAYFORM1 Input: Training Dataset D, Number of Epochs n epochs, Network Parameters Ω e Output: A near-isometric embedding Φ: R N → R M for i = 1 to n epochs do -generate a randomly permuted training set → P(D) for every batch B j ∈ P(D) do -compute embedding Φ(X) for every x ∈ B j -compute the loss function corresponding to B j as the maximum deviation from isometry DISPLAYFORM0 As DeepSSRR architecture is shown in FIG0, For learning Φ, we first divide the input signal (of size N) into r (r = N M) sub-signals (of size M) such that all the congruent entries (modulo r) are in the same sub-signal. Then we run parallel convolutions on r sub-signals and stack the outputs (of size M), deriving a tensor of length M and depth r. Through several convolutional layers, we turn this tensor into a vector of size M which is the measurements vector Y and this completes construction of Φ. Similarly and for learning f Λ , through several convolutional layers, we turn vector Y into a tensor of length M and depth r. We then unstack channels similar to the sub-pixel layer architecture BID33 and derive the final reconstruction X = f Λ (Y) = f Λ (ΦX). We use MSE as a loss function and ADAM BID21 to learn the convolution kernels and biases. where M N. Therefore, an important question is how does one design Φ? Conventional CS is based on random projections of a signal which means that Φ is a random matrix in conventional CS. However, since signals are usually structured, random projections are not optimal for successfully recovering the corresponding signals. In many applications (e.g. medical imaging), we know a lot about the signals we are acquiring. Hence, given a large-scale dataset of the same type of signals of interest, we can learn (near-)optimal measurement matrices. As in the usual CS paradigm, if we assume that the measurement matrix Φ is fixed, each DISPLAYFORM0 } consists of pairs of signals and their corresponding measurements. Accordingly, we define the optimal measurement operator Φ as the one which maximizes the probability of training data given the undersampled projections, Φ = arg max DISPLAYFORM1 According to the law of large numbers, notice that we can write DISPLAYFORM2 where in (a) I denotes the mutual information, and the equality follows since the Shannon entropy H(X) is constant for every Φ. According to, in the asymptotic setting, the measurement matrix which maximizes the probability of training data given its measurements, maximizes the mutual information between the input signal and undersampled measurements as well. Equation FORMULA4 is the same as infomax principle first introduced in BID24.Now, suppose that we have a function f : DISPLAYFORM3 and reconstructs input signals as DISPLAYFORM4 We define the best reconstruction as the one which generates training data with the highest probability. In other words, we define DISPLAYFORM5 Therefore, in the asymptotic setting and similar to we can write DISPLAYFORM6 = arg max DISPLAYFORM7 In practice and since we do not know the true underlying probability distribution of P(X| X), we maximize a parametric distribution q(X| X) instead. In this case, in the asymptotic setting we can write DISPLAYFORM8 = arg max DISPLAYFORM9 Therefore, since Kullback-Leibler divergence is bounded above zero we have DISPLAYFORM10 [log(P(X|Y = ΦX; Λ))], meaning that learning a parametric distribution for reconstructing X from Y is equivalent to maximizing a lower-bound of true conditional entropy and accordingly, mutual information between the input signal X and undersampled measurements Y. Hence, although we are not maximizing the mutual information between X and Y, we are maximizing a lower-bound of it through learning Φ and Λ. If we assume X = X +, where and has an isotropic Gaussian distribution, then, since q(X| X = x) = N (x, λI), the above maximization may be performed by minimizing the mean squared error (MSE). DeepSSRR is mainly designed for jointly sensing and recovering sparse signals for CS applications. However, we can specifically train the sensing part of DeepSSRR (without using the recovery part) for several important dimensionality reduction tasks. The sensing part of DeepSSRR (i.e., the encoder or matrix Φ) is a linear low-dimensional embedding that we can apply it to learn a mapping from a subset of R N to R M (M < N) that is a near-isometry, i.e., a mapping that nearly preserves all inter-point distances. This problem has a range of applications, from approximate nearest neighbor search to the design of sensing matrices for CS. Recall that, for a set Q ⊂ R N and > 0, the (linear or nonlinear) mapping Φ: Q → R M is an -isometry w.r.t the 2 -norm if for every x and x in Q we have DISPLAYFORM11 Algorithm 1 demonstrates the use of the low-dimensional embedding matrix Φ of DeepSSRR to construct a near-isometric embedding. We achieve this by penalizing the maximum deviation from isometry in several batches of data that are created by permuting the original training data in every training epoch. In Section 3 we will show how our proposed algorithm works compared to competing methods. We now illustrate the performance of DeepSSRR against competing methods in several problems. We first study the quality of the linear embeddings produced by DeepSSRR and its comparison with two other linear algorithms -NuMax BID18 and random Gaussian projections. To show the price of linearity, we also pit these against the nonlinear version of DeepSSRR and a DCN (eight nonlinear convolutional layers + a max-pooling layer). We use the grayscale version of CIFAR-10 dataset (50,000 training + 10,000 test 32 × 32 images). We train DeepSSRR and DCN according to Algorithm 1 by using filters of size 5 × 5. For DeepSSRR, depending on the size of the embedding we use five to seven layers to learn Φ in Algorithm 1.Figure 3(a) shows the size of embedding M as a function of the isometry constant for different methods. For the random Gaussian projections we have considered 100 trials and the horizontal error bars represent the deviation from average value. As we can see, the nonlinear version of DeepSSRR low-dimensional embedding outperforms almost all the other methods by achieving a given isometry constant with fewer measurements. The only exception is when > 0.6 (i.e., a regime where we are not demanding a good isometry), where the DCN outperforms the nonlinear version of DeepSSRR; though, with more number of parameters. A convolutional layer is equivalent to the product of a circulant matrix and the vectorized input. The number of nonzero elements in a circulant matrix depends on the size of the convolution filter. As the number of such layers grows, so does the number of nonzero elements in the final embedding matrix. There are lower bounds BID30 on the number of nonzero elements in a matrix to ensure it is near-isometric. TAB0 shows the isometry constant value of DeepSSRR's low-dimensional embedding with different number of layers and different filter sizes. As we can see, gets smaller as the final embedding matrix has more nonzero elements (more layers, larger filters).Approximate Nearest Neighbors. Finding the closest k points to a given query datapoint is challenging for high-dimensional datasets. One solution is to create a near-isometric embedding that maps datapoints from R N to R M (M < N) and solving the approximate nearest neighbors (ANN) problem in the embedded space. FIG2 compares the performance of different methods in the ANN problem. It shows the fraction of k-nearest neighbors that are retained when embedding datapoints in a low-dimensional space. We have considered two separate embedding problems: First M = 65 for random embedding and NuMax and M = 64 for DCN and DeepSSRR's low-dimensional embedding. Second, M = 289 for random embedding and NuMax and M = 256 for DCN and DeepSSRR's lowdimensional embedding. Since the size of the embedding for DCN and DeepSSRR's low-dimensional embedding is smaller in both settings, they have a more challenging task to find the nearest neighbors. As shown in FIG2 (b) DeepSSRR's low-dimensional embedding outperforms other approaches. We divide the discussion of this section into two parts. In the first part, we study the performance of DeepSSRR in the sparse signal recovery problem. The discussion of this part along with experimental showing the effect of learning a sparse representation and parallelization on different criteria (e.g. phase transition, recovery accuracy and speed) are provided in Appendix A. In the second part that we provide in the following, we study the performance of DeepSSRR for the compressive image recovery problem. Compressive Image Recovery. In this part, we study the compressive image recovery problem by comparing DeepSSRR with two state-of-the-art algorithms DAMP BID26 and LDAMP BID25. Both DAMP and LDAMP use random Gaussian Φ while DeepSSRR learns a Φ. Here we run DAMP for 10 iterations and use a BM3D denoiser at every iteration. We also run LDAMP for 10 layers and use a 20-layer DCN in every layer as a denoiser. For DeepSSRR, we use 7 layers to learn the Φ and 7 layers to learn the f Λ . DeepSSRR is trained with an initial learning rate of 0.001 that is changed to 0.0001 when the validation error stops decreasing. For training, we have used batches of 128 images of size 64 × 64 from ImageNet BID31. Our training and validation sets include 10,000 and 500 images, respectively. On the other hand, DeepSSRR uses only 7 convolutional layers to recover the Man image which is significantly smaller compared to LDAMP's number of layers. Iterative recovery algorithms and their unrolled versions such as DAMP and LDAMP typically involve a matrix vector multiplication in every iteration or layer, and hence their computational complexity is O(M N). In DeepSSRR, the length of feature maps in every convolutional layer is equal to the size of embedding M. Therefore, computing the output of typical middle layers will cost O(M) that is significantly cheaper than the one for iterative or unrolled methods such as DAMP and LDAMP.Effect of the Number of Layers. Our experiments indicate that having more number of layers does not necessarily in a better signal recovery performance. This phenomenon is also observed in BID11 for the image super-resolution problem. The reason for this problem is the increased non-convexity and non-smoothness of loss function as we add more layers. One way to mitigate this problem is to add skip connections between layers. As shown in, skip connections smooth the loss surface of deep networks and make the optimization problem simpler. In this paper we introduced DeepSSRR, a framework that can learn both near-optimal sensing schemes, and fast signal recovery procedures. Our findings set the stage for several directions for future exploration including the incorporation of adversarial training and its comparison with other methods BID2 BID14 BID10 ). Furthermore, a major question arising from our work is quantifying the generalizability of a DeepSSRR-learned model based on the richness of training data. We leave the exploration of this for future research. In this section we study the problem of sparse signal recovery by comparing DeepSSRR to another DCN called DeepInverse and to the LASSO BID36 1 -solver implemented using the coordinate descent algorithm of BID15. We assume that the optimal regularization parameter of the LASSO is given by an oracle in order to obtain its best possible performance. Also, both training and test sets are wavelet-sparsified versions of 1D signals of size N = 512 extracted from rows of CIFAR-10 images and contain 100,000 and 20,000 signals, respectively. While DeepSSRR learns how to take undersampled measurements of data through its low-dimensional embedding Φ, DeepInverse uses random undersampling (i.e., a random Φ). DeepSSRR in this section has 3 layers for learning Φ and 3 layers for learning f Λ with filter size 25 × 1 while DeepInverse has five layers for learning the inverse mapping with filter size 125 × 1.Figure 5(a) shows the 1 phase transition plot BID13 ). This plot associates each grid point to an ordered pair (δ, ρ) ∈ 2, where δ = M N denotes the undersampling ratio and ρ = K M denotes the normalized sparsity level. Each grid point (δ, ρ) represents the probability of an algorithm's success in signal recovery for that particular problem configuration. As the name suggests, there is a sharp phase transition between values of (δ, ρ) where recovery fails with high probability to when it succeeds with high probability. In FIG4 (a), the blue curve is the 1 phase transition curve. The circular points denote the problem instances on which we study the performance of DeepInverse and the LASSO. The square points denote the problem instances on which we have trained and tested DeepSSRR. By design, all these problem instances are on the "failure" side of the 1 phase transition. For DeepSSRR (square points), we have made recovery problems harder by reducing δ and increasing ρ. The arrows between the square points and circular points in FIG4 (a) denote correspondence between problem instances in DeepSSRR and DeepInverse. TAB1 shows the average normalized MSE (NMSE) for the test signals. While DeepSSRR recovers the same signals from fewer measurements, it outperforms DeepInverse and the LASSO. DeepSSRR outperforms DeepInverse while having significantly fewer number of parameters (less than 70,000 vs. approximately 200,000 parameters). This is mainly due to the fact that DeepSSRR learns Φ instead of using a random Φ as is the case in DeepInverse and conventional CS. While training and test sets are the same, the configuration for DeepInverse (and LASSO) is (δ, ρ) = (0.7, 0.72) and for DeepSSRR is (δ, ρ) = (0.5, 1.003) which means we have given DeepSSRR a more challenging problem. As shown in FIG4, due to the extra parallelization scheme (i.e., rearrangement layer) convergence is significantly faster for DeepSSRR compared to DeepInverse. DeepSSRR outperforms the LASSO after only 4 training epochs while DeepInverse takes 138 epochs. This fast convergence has two major reasons: First, DeepSSRR has fewer number of parameters to learn. Second, DeepSSRR learns adaptive measurements (i.e., low-dimensional embedding) instead of using random measurements (i.e., random embedding). ≤ 0.01, where X (j) is the j-th sample,X (j) is the recovered signal from measurements of j-th sample, and I is the indicator function. We denote empirical successful recovery probability by P δ = 1 q q j=1 ϕ δ,j. In FIG4, our test samples are k-sparse where k = 34, and we have considered three different configurations: M = 64, 128, 256 that correspond to above, on, and below the 1 phase transition, respectively. As we can see in FIG4, DeepSSRR significantly outperforms LASSO when the problem configuration lies above (failure phase) or on the 1 phase transition and LASSO slightly outperforms when the problem configuration lies below the 1 phase transition (success phase). For a setting below the 1 phase transition, we expect 1 minimization to behave the same as 0 minimization. However, DeepSSRR should learn a transformation for transforming measurements back to the original signals. Furthermore, FIG4 shows the price we pay for using a linear low-dimensional embedding ΦX instead of a nonlinear one Φ(X). The main message of FIG4 (c) is that by using DeepSSRR we can have a significantly better phase transition compared to 1 -minimization. In this section we study another example of the compressive image recovery problem. The settings we have used in here is exactly the same as Section 3.2. FIG8 shows the reconstruction of the mandrill image (M N = 0.25). FIG8 shows the reconstruction of whole face and FIG8 shows the reconstruction of nose and cheeks. As we can see, although LDAMP slightly outperforms our method in FIG8, our method does a significantly better job in recovering the texture of nose and cheeks in FIG8. Not only our method outperforms LDAMP by 0.9 dB, but also it has a better visual quality and fewer artifacts (e.g. less over-smoothing). In this section we compare the running time of different algorithms. We consider the reconstruction of a 512 × 512 image with an undersampling ratio of M N = 0.25. Table 3 shows the comparison between different algorithms. We should note that authors in BID25 have used coded diffraction pattern in DAMP and LDAMP which simplifies the computational complexity of vector-matrix multiplications in DAMP and LDAMP to O(N log(N)) instead of O(M N). In addition, we should note that LDAMP uses filters of size 3 × 3 in its convolutional layers while we use filters of size 5 × 5 in the convolutional layers of our architecture. Table 3 shows that our method is almost 4 times faster than the LDAMP method.

We use deep learning techniques to solve the sparse signal representation and recovery problem.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent

Question: To select effective actions in complex environments, intelligent agents need to generalize from past experience. World models can represent knowledge about the environment to facilitate such generalization. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks purely by latent imagination. We efficiently learn behaviors by backpropagating analytic gradients of learned state values through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance. Intelligent agents can achieve goals in complex environments even though they never encounter the exact same situation twice. This ability requires building representations of the world from past experience that enable generalization to novel situations. World models offer an explicit way to represent an agent's knowledge about the world in a parametric model learned from experience that can make predictions about the future. When the sensory inputs are high-dimensional images, latent dynamics models can abstract observations to predict forward in compact state spaces (; ;). Compared to predictions in image space, latent states have a small memory footprint and enable imagining thousands of trajectories in parallel. Learning effective latent dynamics models is becoming feasible through advances in deep learning and latent variable models (; ; ;). Behaviors can be derived from learned dynamics models in many ways. Often, imagined rewards are maximized by learning a parametric policy (; ;) or by online planning . However, considering only rewards within a fixed imagination horizon in shortsighted behaviors. Moreover, prior work commonly resorts to derivative-free optimization for robustness to model errors (; ;), rather than leveraging the analytic gradients offered by neural network dynamics models . We present Dreamer, an agent that learns long-horizon behaviors from images purely by latent imagination. A novel actor critic algorithm accounts for rewards beyond the planning horizon while making efficient use of the neural network dynamics. For this, we predict state values and actions in the learned latent space as summarized in Figure 1. The values optimize Bellman consistency for imagined rewards and the policy maximizes the values by propagating their analytic gradients back through the dynamics. In comparison to actor critic algorithms that learn online or by experience replay;;, world models enable interpolating between past experience and offer analytic gradients of multi-step returns for efficient policy optimization. Figure 2: Agent observations for 5 of the 20 control tasks used in our experiments. These pose a variety of challenges including contact dynamics, sparse rewards, many degrees of freedom, and 3D environments that exceed the difficult to tasks previously solved through world models. The agent observes the images as 64 × 64 × 3 pixel arrays. The key contributions of this paper are summarized as follows: • Learning long-horizon behaviors in imagination Purely model-based agents can be shortsighted due to finite imagination horizons. We approach this limitation in latenby predicting both actions and state values. Training purely by latent imagination lets us efficiently learn the policy by propagating analytic gradients of the value function back through latent state transitions. • Empirical performance for visual control We pair Dreamer with three representation learning objectives to evaluate it on the DeepMind Control Suite with image inputs, shown in Figure 2. Using the same hyper parameters for all tasks, Dreamer exceeds existing model-based and model-free agents in terms of data-efficiency, computation time, and final performance. Reinforcement learning We formulate visual control as a partially observable Markov decision process (POMDP) with discrete time step t ∈ [1; T], continuous vector-valued actions a t ∼ p(a t | o ≤t, a <t) generated by the agent, and high-dimensional observations and scalar rewards o t, r t ∼ p(o t, r t | o <t, a <t) generated by the unknown environment. The goal is to develop an agent that maximizes the expected sum of rewards E p T t=1 r t. Figure 2 shows a selection of our tasks. Agent components The classical components of agents that learn in imagination are dynamics learning, behavior learning, and environment interaction . In the case of Dreamer, the behavior is learned by predicting hypothetical trajectories in the compact latent space of the world model. As outlined in Figure 3 and detailed in Algorithm 1, Dreamer performs the following operations throughout the agent's life time, either sequentially interleaved or in parallel: • Learn the latent dynamics model from the dataset of past experience to predict future rewards from actions and past observations. Any learning objective for the world model can be incorporated with Dreamer. We review existing methods for learning latent dynamics in Section 4. • Learn action and value models from predicted latent trajectories, as described in Section 3. The value model optimizes Bellman consistency for imagined rewards and the action model is updated by propagating gradients of value estimates back through the neural network dynamics. • Execute the learned action model in the world to collect new experience for growing the dataset. Latent dynamics Dreamer uses a latent dynamics model that consists of three components. The representation model encodes observations and actions to create continuous vector-valued model states s t with Markovian transitions (; ;). The transition model predicts future model states without seeing the corresponding observations that will cause them. The reward model predicts the rewards given the model states, The model mimics a non-linear Kalman filter , latent state space model, or HMM with real-valued states. However, it is conditioned on actions and predicts rewards, allowing the agent to imagine the outcomes of potential action sequences without executing them in the environment. Dreamer learns long-horizon behaviors in the compact latent space of a learned world model. For this, we propagate stochastic gradients of multi-step returns through neural network predictions of actions, states, rewards, and values using reparameterization. This section describes our core contribution. The latent dynamics define a Markov decision process (MDP;) that is fully observed since the compact model states s t are Markovian. We denote imagined quantities with τ as the time index. Imagined trajectories start at the true model states s t of observation sequences drawn from the agent's past experience. They follow predictions of the transition model, and a policy a τ ∼ q(a τ | s τ). The objective is to maximize expected imagined rewards E q ∞ τ =t γ τ −t r τ with respect to the policy. Action and value models Consider imagined trajectories with a finite horizon H. Dreamer uses an actor critic approach to learn behaviors that consider rewards beyond the horizon. We learn an action model and a value model in the latent space of the world model. The action model implements the policy and aims to predict actions that solve the imagination environment. The value model estimates the state values V(s τ) E q(·|sτ) t+H τ =t γ τ −t r τ for the action model, the expected sum of imagined rewards that it achieves in each state s τ, Action model: The action and value models are trained cooperatively as typical in policy iteration: the action model aims to maximize an estimate of the value, while the value model aims to match an estimate of the value that changes as the action model changes. We use dense neural networks for the action and the value model with parameters φ and ξ, respectively. The action model outputs a tanh-transformed Gaussian with sufficient statistics predicted by the neural network. This allows for reparameterized sampling that lets sampled actions depend deterministically on the neural network output, allowing to backpropagate analytic gradients through the sampling operation, Value estimation To learn the action and value models, we need to estimate the state values of imagined trajectories {s τ, a τ, r τ} t+H τ =t. These trajectories branch off of the model states s t of sequence batches drawn from the agent's dataset of experience and predict forward for the imagination horizon H using actions sampled from the action model. State values can be estimated in multiple Figure 4: Imagination horizons. We compare the final performance of Dreamer to learning an action model without value prediction and to online planning using PlaNet. Learning a state value model to estimate rewards beyond the imagination horizon makes Dreamer more robust to the horizon length. The agents use reconstruction for representation learning and an action repeat of R = 2. ways that trade off bias and variance , where the expectations are estimated with the imagined trajectories. V R simply sums the rewards from τ until the horizon and ignores rewards beyond it. This allows learning the action model without value model, an ablation we compare to in our experiments. V k N estimates rewards beyond k steps with the learned value model. Dreamer uses V λ, which computes an exponentially-weighted average of the estimates for different k to balance bias and variance. Figure 4 shows that learning a value function in imagination enables Dreamer to solve long-horizon tasks while being robust to the imagination horizon. The experimental details and on all tasks are described in Section 5. Learning objective To update the action and value models, we first compute the value estimates V λ (s τ) for states s τ along the imagined trajectories. The objective for the action model q φ (a τ | s τ) is to output actions that in state trajectories with high value estimates. The objective for the value model v ξ (s τ), in turn, is to regress the value estimates, The value model is simply updated to regress the targets, around which we stop the gradient as typical in temporal difference learning . The action model uses analytic gradients through the learned dynamics to maximize the value estimates. To understand this, we note that the value estimates depend on the reward and value predictions, which depend on the imagined states, which in turn depend on the imagined actions. Since these steps are all implemented as neural networks with reparameterized sampling, we analytically compute . The world model is fixed while learning the action and value models. Comparison to actor critic methods Agents using Reinforce gradients employ a value baseline to reduce gradient variance, such as A3C and PPO , while Dreamer backpropagates through the value model. This is similar to analytic actor critics , such as DDPG and SAC . However, these do not leverage gradients through the state transitions and only maximize immediate Q-values. MVE and STEVE extend these to multi-step Q-learning using learned dynamics to help rewards propagate faster into the value estimates. We simply predict state values, which is sufficient for policy optimization since we backpropagate through the dynamics. We empirically compare learning action and value models from V λ, learning the action model from V R which does not require a value model, and online planning in our experiments in Learning behaviors in imagination requires a world model that generalizes well. We focus on latent dynamics models that predict forward in a compact latent space, facilitating long-term predictions and allowing to imagine thousands of trajectories in parallel. Several objectives for learning representations for control have been proposed (; ; ; . We review three approaches for learning representations to use with Dreamer: image reconstruction, contrastive estimation, and reward prediction. Reward prediction Latent imagination requires a representation model p(s t | s t−1, a t−1, o t), transition model q(s t | s t−1, a t−1,), and reward model q(r t | s t), as described in Section 2. In principle, this could be achieved by simply learning to predict future rewards given actions and past observations . Given a large and diverse dataset, such representations should be sufficient for solving a given control problem. However, while the agent is still exploring and especially when the reward signal is limited, additionally learning about observations is likely to improve the world model . Representation learning The world model is learned from sequences {(o t, a t, r t)} T t=1 drawn from the agent's dataset of experience. To learn representations that generalize, the model states s 1:T should be predictive of observations o 1:T and rewards r 1:T while not overfitting to individual examples in the dataset. At a high level, this is formalized by an information bottleneck , max The first terms encourages mutual information between the model states and the observations and rewards. The second term penalizes information between model states and dataset indices i 1:T by an amount 0 ≤ β ≤ 1. The dataset indices relate to the images by a Dirac delta p(o t | i t) as in. The information bottleneck poses the representation learning problem in a generic way and provides a common view on pixel reconstruction and contrastive estimation. While the two information terms are difficult to estimate, they are easy to bound an optimize . Reconstruction We first describe the world model used by PlaNet , shown in Figure 3a. It bounds the objective by predicting observations and rewards from the model states, where the expectation samples sequences from the dataset and states from the representation model. The bound includes a reconstruction term, a reward prediction term, and a KL regularizer. We refer to Appendix C for the derivation. The bound uses four distributions that we implement as neural networks and optimize jointly to increase the bound, Representation model: Episode Return n/a n/a n/a n/a Dreamer (2e6) PlaNet (2e6) D4PG (1e9 steps) A3C (1e9 steps, proprio) Figure 6: Performance comparison to existing methods. Dreamer exhibits the data-efficiency of PlaNet while exceeding the asymptotic performance of the best model-free agents. After 2 * 10 6 environment steps, Dreamer reaches an average performance of 802 across tasks, compared to PlaNet at 312 and the top model-free D4PG agent at 786 after 10 9 steps. Results are averages over 3 seeds. We implement the transition model as recurrent state space model (RSSM;), the representation model by combining the RSSM with a convolutional neural network (CNN;) applied to the image observation, the observation model as a transposed CNN, and the reward model as dense network. The combined parameter vector θ is updated by reparameterization gradients . Contrastive estimation Accurately predicting pixels in visually complex environments can be a challenging task. We can avoid reconstruction by instead predicting model states . While the observation marginal above was a constant, we now face the state marginal. Using the InfoNCE bound (Gutmann and Hyvärinen, 2010;) as described in Appendix C, where o q(s t | o) estimates the marginal by summing over observations o of the current sequence batch. Intuitively, q(s t | o t) makes the state predictable from the current image and ln o q(s t | o) keeps it diverse to prevent collapse. Instead of the observation model, the bound uses a state model, State model: We implement the state model as a CNN and again optimize the bound with respect to the combined parameter vector θ using reparameterization gradients. While avoiding pixel prediction, the amount of information this bound can extract efficiently is limited . We empirically compare reconstruction, contrastive, and reward objectives in our experiments in Figure 8. Visual control tasks We evaluate Dreamer on 20 continuous control tasks with image observations of the DeepMind Control Suite , illustrated in Figure 2. These tasks pose a variety of challenges, including partial observability, sparse rewards, contact dynamics, and 3D environments. We selected the tasks on which report non-zero performance from image inputs. Agent observations are images of shape 64 × 64 × 3, actions range from 1 to 12 dimensions, rewards are between 0 and 1, episodes contain 1000 steps, and initial states are randomized. Visualizations of our agent are available at https://dreamrl.github.io. Implementation All experiments used a single Nvidia V100 GPU and 10 CPU cores per training run. Our implementation uses TensorFlow Probability and will be open sourced. The training time for our implementation of Dreamer is 10 hours per 10 6 environment steps without parallelization, compared to 17 hours for online planning using PlaNet, and 24 hours for D4PG. We use the same hyper parameters across all tasks including a fixed action repeat of R = 2, as detailed in Appendix B. The world models are learned by reconstruction unless noted otherwise. Baseline methods We compare Dreamer to several baselines: The current best reported performance on the considered tasks is by D4PG , an improved variant of DDPG that uses distributed experience collection, distributional Q-learning, multi-step returns, and prioritized experience replay. PlaNet learns the same world model as Dreamer and selects actions using online planning instead of learning an action model. We include the numbers for D4PG from and re-run PlaNet with R = 2 for a fair comparison. Performance comparison To evaluate the performance of Dreamer, we compare with state-of-theart reinforcement learning agents. The are summarized in Figure 6. With an average score of 802 across tasks after 2 * 10 6 environment steps, Dreamer exceeds the performance of the strong model-free D4PG agent that achieves an average of 786 within 10 9 environment steps. At the same time, Dreamer inherits the data-efficiency of PlaNet, confirming that the learned world model can help to generalize from small amounts of experience. The empirical success of Dreamer shows that learning behaviors by latent imagination can outperform top methods based on experience replay. Long-horizon behavior To investigate its ability to learn long-horizon behaviors, we compare Dreamer to alternatives for deriving behaviors from the world model at various horizon lengths. For this, we learn an action model to maximize imagined rewards without value model and compare to online planning using PlaNet. Figure 4 shows the final performance for different imagination horizons, confirming that the value model makes Dreamer more robust to the horizon and in high performance even for short horizons. Performance curves for all 20 tasks with horizon of 20 are shown in Appendix D, where Dreamer outperforms the alternatives on 15 of 20 tasks and 3 ties. Representation learning Dreamer can be used with any dynamics model that predicts future rewards given actions and past observations. Since the representation learning objective is orthogonal to our algorithm, we compare three natural choices described in Section 4: pixel reconstruction, contrastive estimation, and pure reward prediction. Figure 8 shows clear differences in task performance for different representation learning approaches, with pixel reconstruction outperforming contrastive estimation on most tasks. This suggests that future improvements in representation learning are likely to transfer over to task performance with Dreamer. Reward prediction along was not sufficient to solve any of the tasks in our experiments. Further ablations are included in the appendix of the paper. Prior work learns latent dynamics for visual control by derivative-free policy learning or online planning, augments model-free agents with multi-step predictions, or uses analytic gradients of Qvalues or multi-step rewards, often for low-dimensional tasks. In comparison, Dreamer uses analytic gradients to efficiently learn long-horizon behaviors for visual control purely by latent imagination. Control with latent dynamics E2C and RCE embed images to predict forward in a compact space to solve simple tasks. World Models learn latent dynamics in a two-stage process to evolve linear controllers in imagination. PlaNet learns them jointly and solves visual locomotion tasks by latent online planning. Similarly, SOLAR latent space. I2A hands imagined trajectories to a model-free policy, while and learn belief representations to accelerate model-free agents. Imagined multi-step returns VPN , MVE , and STEVE learn dynamics for multi-step Q-learning from a replay buffer. AlphaGo combines predictions of actions and state values with planning, assuming access to the true dynamics. Also assuming access to the dynamics, POLO plans to explore by learning a value ensemble. PETS , VisualMPC , and PlaNet plan online using derivative-free optimization, and POPLIN improves online planning by self-imitation. Planning with neural network gradients was shown on small problems but has been challenging to scale . Analytic value gradients DPG , DDPG, and SAC leverage gradients of learned immediate action values to learn a policy by experience replay. SVG reduces the variance of model-free on-policy algorithms by analytic value gradients of one-step model predictions. ME-TRPO accelerates learning of a model-free agent via gradients of predicted rewards for proprioceptive inputs. DistGBP directly uses model gradients for online planning in simple tasks. We present Dreamer, an agent that learns long-horizon behaviors purely by latent imagination. For this, we propose a novel actor critic method that optimizes a parametric policy by propagating analytic gradients of multi-step values back through latent neural network dynamics. Dreamer outperforms previous approaches in data-efficiency, computation time, and final performance on a variety of challenging continuous control tasks from image inputs. While our approach compares favourably on these tasks, future research on learning representations is likely needed to scale latent imagination to visually more complex environments. A DETAILED ALGORITHM Update θ to predict rewards using representation learning. Imagine trajectories {(s τ, a τ)} t+H τ =t from each s t using action model. Predict rewards E q θ (r τ | s τ) and values v ξ (s τ). Compute value estimates V λ (s τ) via Equation 6. Update φ according to maximize t+H τ =t V λ (s τ) by gradient ascent. Update ξ to minimize Compute action a t ∼ q φ (a t | s t) with the action model. Add exploration noise to action. Model components We use the convolutional encoder and decoder networks from , the RSSM of , and implement all other functions as three dense layers of size 300 with ELU activations . Distributions in latent space are 30-dimensional diagonal Gaussians. The action model outputs an unconstrained mean and softplus standard deviation for the Normal distribution that is then transformed using tanh. Learning updates We draw batches of 50 sequences of length 50 to train the world model, value model, and action model models using Adam with learning rates 10 −3, 3 * 10 −4, 3 * 10 −4, respectively. We do not scale the KL regularizers (β = 1) but clip them below 3 free nats as in PlaNet. The imagination horizon is H = 20 and the same trajectories are used to update both action and value models. We use a slow moving value network that is updated every 100 gradient steps to compute the V λ value estimates with γ = 0.99 and λ = 0.95. Environment interaction The dataset is initialized with C = 5 episodes collected using random actions. We iterate between 100 training steps and collecting 1 episode by executing the predicted mode action with Normal(0, 0.3) exploration noise. Instead of manually selecting the action repeat for each environment as in and, we fix it to 2 for all environments. See Figure 11 for an assessment of the robustness to different action repeat values. The information bottleneck objective defined in Equation 9 for latent dynamics models is, For the generative objective, we lower bound the first term using the non-negativity of the KL divergence and drop the marginal data probability as it does not depend on the representation model, For the contrastive objective, we subtract the constant marginal probability of the data under the variational encoder, apply Bayes rule, and use the InfoNCE mini-batch bound , E ln q(o t | s t) + ln q(r t | s t) + = E ln q(o t | s t) − ln q(o t) + ln q(r t | s t) = E ln q(s t | o t) − ln q(s t) + ln q(r t | s t) ≥ E ln q(s t | o t) − ln o q(s t | o) + ln q(r t | s t). For the second term, we use the non-negativity of the KL divergence to obtain an upper bound, I(s 1:T ; i 1:T | a 1:T) = E p(o 1:T,r 1:T,s 1:T,a 1:T,i 1:T) t ln p(s t | s t−1, a t−1, i t) − ln p(s t | s t−1, a t−1) = E t ln p(s t | s t−1, a t−1, o t) − ln p(s t | s t−1, a t−1) ≤ E t ln p(s t | s t−1, a t−1, o t) − ln q(s t | s t−1, a t−1) = E t KL p(s t | s t−1, a t−1, o t) q(s t | s t−1, a t−1). This lower bounds the objective. Comparison of action selection schemes on the continuous control tasks of the DeepMind Control Suite from pixel inputs. The lines show mean scores over environment steps and the shaded areas show the standard deviation across 3 seeds. We compare Dreamer that learns both actions and values in imagination, to only learning actions in imagination, and to online planning using CEM without policy learning. The baselines include the top model-free algorithm D4PG, the common A3C agent, and the hybrid SLAC agent.

Represent the TLDR for finding the corresponding scientific article!

We present Dreamer, an agent that learns long-horizon behaviors purely by latent imagination using analytic value gradients.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.


Question: Represent the article for finding a single-sentence TLDR!

Question: Transfer reinforcement learning (RL) aims at improving learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under unknown diverse dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. We envision a future scenario where a variety of robotic systems, which are each trained or manually engineered to solve a similar task, provide their policies for a new robot to learn a relevant task quickly. For example, imagine various pick-and-place robots working in factories all over the world. Depending on the manufacturer, these robots will differ in their kinematics (e.g., link length, joint orientations) and dynamics (e.g., link mass, joint damping, friction, inertia). They could provide their policies to a new robot , even though their dynamics factors, on which the policies are implicitly conditioned, are not typically available . Moreover, we cannot rely on a history of their individual experiences, as they may be unavailable due to a lack of communication between factories or prohibitively large dataset sizes. In such scenarios, we argue that a key technique to develop is the ability to transfer knowledge from a collection of robots to a new robot quickly only by exploiting their policies while being agnostic to their different kinematics and dynamics, rather than collecting a vast amount of samples to train the new robot from scratch. The scenario illustrated above poses a new challenge in the transfer learning for reinforcement learning (RL) domains. Formally, consider multiple instances of a single environment that differ in their state transition dynamics, e.g., independent ant robots with different leg designs in Figure 1, which reach different locations by executing the same walking actions. These source agents interacting with one of the environment instances provide their deterministic policy to a new target agent in another environment instance. Then, our problem is: can we efficiently learn the policy of a target agent given only the collection of source policies? Note that information about source environmental dynamics, such as the exact state transition distribu- Figure 2: Overview of MULTIPOLAR. We formulate a target policy π target with the sum of 1) the adaptive aggregation F agg of deterministic actions from source policies L and 2) the auxiliary network F aux for predicting residuals around F agg. tions and the history of environmental states, will not be visible to the target agent as mentioned above. Also, the source policies are neither trained nor hand-engineered for the target environment instance, and therefore not guaranteed to work optimally and may even fail . These conditions prevent us from adopting existing work on transfer RL between different environmental dynamics, as they require access to source environment instances or their dynamics for training a target policy (e.g., ; ; ;). Similarly, meta-learning approaches (; ;) cannot be used here because they typically train an agent on a diverse set of tasks (i.e., environment instances). Also, existing techniques that utilize a collection of source policies, e.g., policy reuse frameworks (Fernández & ; ;) and option frameworks (; ;), are not a promising solution because, to our knowledge, they assume source policies have the same environmental dynamics but have different goals. As a solution to the problem, we propose a new transfer RL approach named MULTI-source POLicy AggRegation (MULTIPOLAR). As shown in Figure 2, our key idea is twofold; 1) In a target policy, we adaptively aggregate the deterministic actions produced by a collection of source policies. By learning aggregation parameters to maximize the expected return at a target environment instance, we can better adapt the aggregated actions to unseen environmental dynamics of the target instance without knowing source environmental dynamics nor source policy performances. 2) We also train an auxiliary network that predicts a residual around the aggregated actions, which is crucial for ensuring the expressiveness of the target policy even when some source policies are not useful. As another notable advantage, the proposed MULTIPOLAR can be used for both continuous and discrete action spaces with few modifications while allowing a target policy to be trained in a principled fashion. Similar to;;;; , our method assumes that the environment structure (state/action space) is identical between the source and target environments, while dynamics/kinematics parameters are different. This assumption holds in many real-world applications such as in sim-to-real tasks , industrial insertion tasks (different dynamics comes from the differences in parts), and wearable robots (with users as dynamics). We evaluate MULTIPOLAR in a variety of environments ranging from classic control problems to challenging robotics simulations. Our experimental demonstrate the significant improvement of sample efficiency with the proposed approach, compared to baselines that trained a target policy from scratch or from a single source policy. We also conducted a detailed analysis of our approach and found it works well even when some of the source policies performed poorly in their original environment instance. Main contributions: a new transfer RL problem that leverages multiple source policies collected under diverse environmental dynamics to train a target policy in another dynamics, and MULTIPOLAR, a simple yet principled and effective solution verified in our extensive experiments. Reinforcement Learning We formulate our problem under the standard RL framework , where an agent interacts with its environment modeled by a Markov decision process (MDP). An MDP is represented by the tuple M = (ρ 0, γ, S, A, R, T) where ρ 0 is the initial state distribution and γ is a discount factor. At each timestep t, given the current state s t ∈ S, the agent executes an action a t ∈ A based on its policy π(a t | s t ; θ) that is parameterized by θ. The environment returns a reward R(s t, a t) ∈ R and transitions to the next state s t+1 based on the state transition distribution T (s t+1 | s t, a t). In this framework, RL aims to maximize the expected return with respect to the policy parameters θ. In this work, we consider K instances of the same environment that differ only in their state transition dynamics. We model each environment instance by an indexed MDP: where no two state transition distributions T i, T j; i = j are identical. We also assume that each T i is unknown when training a target policy, i.e., agents cannot access the exact form of T i nor a collection of states sampled from T i. Source Policies For each of the K environment instances, we are given a deterministic source policy µ i: S → A that only maps states to actions. Each source policy µ i can be either parameterized (e.g., learned from interacting with the environment modeled by M i) or non-parameterized (e.g., heuristically designed by humans). Either way, we assume no prior knowledge about µ i is available for a target agent, such as their representations or original performances, except that they were acquired in M i with an unknown T i. Problem Statement Given the set of source policies L = {µ 1, . . ., µ K}, our goal is to train a new target agent's policy π target (a t | s t ; L, θ) in a sample efficient fashion, where the target agent interacts with another environment instance M target = (ρ 0, S, A, R, T target) and T target is not necessarily identical to T i (i = 1 . . ., K). As shown in Figure 2, with the Multi-Source Policy Aggregation (MULTIPOLAR), we formulate a target policy π target using a) the adaptive aggregation of deterministic actions from the set of source policies L, and b) the auxiliary network predicting residuals around the aggregated actions. We first present our method for the continuous action space, and then extend it to the discrete space. Adaptive Aggregation of Source Policies Let us denote by a (i) t = µ i (s t) the action predicted deterministically by source policy µ i given the current state s t. For the continuous action space, a D is a D-dimensional real-valued vector representing D actions performed jointly in each timestep. For the collection of source policies L, we derive the matrix of their deterministic actions: The key idea of this work is to aggregate A t adaptively in an RL loop, i.e., to maximize the expected return. This adaptive aggregation gives us a "baseline" action that could introduce a strong inductive bias in the training of a target policy, without knowing source environmental dynamics T i. More specifically, we define the adaptive aggregation function F agg: S → A that produces the baseline action based on the current state s t as follows: where θ agg ∈ R K×D is a matrix of trainable parameters, is the element-wise multiplication, and 1 K is the all-ones vector of length K. θ agg is neither normalized nor regularized, and can scale each action of each policy independently. This means that we do not merely adaptively interpolate action spaces, but more flexibly emphasize informative source actions while suppressing irrelevant ones. Predicting Residuals around Aggregated Actions Moreover, we learn auxiliary network F aux: S → A jointly with F agg, to predict residuals around the aggregated actions. F aux is used to improve the target policy training in two ways. 1) If the aggregated actions from F agg are already useful in the target environment instance, F aux will correct them for a higher expected return. 2) Otherwise, F aux learns the target task while leveraging F agg as a prior to have a guided exploration process. Any network could be used for F aux as long as it is parameterized and fully differentiable. Finally, the MULTIPOLAR function is formulated as: where θ aux denotes a set of trainable parameters for F aux. Note that the idea of predicting residuals for a source policy has also been presented by;;. The main difference here is that, while these works just add raw action outputs provided from a single hand-engineered source policy, we adaptively aggregate actions from multiple source policies in order to obtain a more flexible and canonical representation. indicates that the j-th action is to be executed. Following Eqs. and, the output of F (s t ; L, θ agg, θ aux) can be viewed as D-dimensional un-normalized action scores, from which we can sample a discrete action after normalizing it by the softmax function. We aim to empirically demonstrate the sample efficiency of a target policy trained with MULTIPO-LAR (denoted by "MULTIPOLAR policy"). To complete the experiments in a reasonable amount of time, we set the number of source policies to be K = 4 unless mentioned otherwise. Moreover, we investigate the factors that affect the performance of MULTIPOLAR. To ensure fair comparisons and reproducibility of experiments, we followed the guidelines introduced by and François- for conducting and evaluating all of our experiments. Baseline Methods To show the benefits of leveraging source policies, we compared our MULTI-POLAR policy to the standard multi-layer perceptron (MLP) trained from scratch, which is typically used in RL literature (; François-). As another baseline, we also used MULTIPOLAR with K = 1, which is an extension of residual policy learning (; ;) (denoted by "RPL") with adaptive residuals as well as the ability to deal with both continuous and discrete action spaces. We stress here that the existing transfer RL or meta RL approaches that train a universal policy network agnostic to the environmental dynamics, such as; , cannot be used as a baseline since they require a policy to be trained on a distribution of environment instances, which is not possible in our problem setting. Also, other techniques using multiple source policies, such as policy reuse frameworks, are not applicable because their source policies should be collected under the target environmental dynamics. Environments To show the general effectiveness of the MULTIPOLAR policy, we conducted comparative evaluations of MULTIPOLAR on the following six OpenAI Gym environments: Roboschool Hopper, Roboschool Ant, Roboschool InvertedPendulumSwingUp, Acrobot, CartPole, and LunarLander. We chose these six environments because 1) the parameterization of their dynamics and kinematics is flexible enough, 2) they cover discrete action space (Acrobot and CartPole) as well as continuous action space, and 3) they are samples of three distinct categories of OpenAI Gym environments, namely Box2d, Classic Control, and Roboschool. Experimental Procedure For each of the six environments, we first created 100 environment instances by randomly sampling the dynamics and kinematics parameters from a specific range. For example, these parameters in the Hopper environment were link lengths, damping, friction, armature, and link mass 1 Then, for each environment instance, we trained an MLP policy. The trained MLP policies were used in two ways: a) the baseline MLP policy for each environment instance, and b) a pool of 100 source policy candidates from which we sample K of them to train MULTIPOLAR policies and one of them to train RPL policies 2. Specifically, for each environment instance, we trained three MULTIPOLAR and three RPL policies with distinct sets of source policies selected randomly from the candidate pool. The learning procedure explained above was done three times with fixed different random seeds to reduce variance in due to stochasticity. As a , for each of the six environments, we had 100 environment instances × 3 random seeds = 300 experiments for MLP and 100 environment instances × 3 choices of source policies × 3 random seeds = 900 experiments for RPL and MULTIPOLAR. The aim of this large number of experiments is to obtain correct insights into the distribution of performances . Due to the large number of experiments for all the environments, our detailed analysis and ablation study of MULTIPOLAR components were conducted with only Hopper, as its sophisticated second-order dynamics plays a crucial role in agent performance . Implementation Details All the experiments were done using the Stable Baselines implementation of learning algorithms as well as its default hyperparameters and MLP network architecture for each environment (see Appendix A.1 for more details). Based on the performance of learning algorithms reported in the , all the policies were trained with Soft Actor-Critic in the LunarLander environment and with Proximal Policy Optimization in the rest of the environments. For fair comparisons, in all experiments, auxiliary network F aux had an identical architecture to that of the MLP. Therefore, the only difference between MLP and MULTIPOLAR was the aggregation part F agg, which made it possible to evaluate the contribution of transfer learning based on adaptive aggregation of source policies. Also, we avoided any random seed optimization since it has been shown to alter the policies' performance . Evaluation Metric Following the guidelines of , to measure sampling efficiency of training policies, i.e., how quick the training progresses, we used the average episodic reward over a various number of training samples. Also, to ensure that higher average episodic reward is representative of better performance and to estimate the variation of it, we used the sample bootstrap method to estimate statistically relevant 95% confidence bounds of the of our experiments. Across all the experiments, we used 10K bootstrap iterations and the pivotal method. Further details on evaluation method can be found in Appendix A.3. Sample Efficiency of MULTIPOLAR Figure 3 and Table 1 clearly show that on average, in all the environments, MULTIPOLAR outperformed baseline policies in terms of sample efficiency and sometimes the final episodic reward 3. For example, in Hopper over 2M training samples, MULTI-POLAR with K = 4 achieved a mean of average episodic reward about three times higher than MLP (i.e., training from scratch) and about twice higher than RPL (i.e., using only a single source policy). It is also noteworthy that MULTIPOLAR with K = 4 had on par or better performance than RPL, which indicates the effectiveness of leveraging multiple source policies 4. Figure 7 in Appendix, shows the individual average learning curve for each of the instances of Roboschool environments. Ablation Study To demonstrate the importance of each component of MULTIPOLAR, we evaluated the following degraded versions: θ agg fixed to 1, which just averages the deterministic actions from the source policies without adaptive weights (similar to the residual policy learning methods that used raw action outputs of a source policy), and F aux learned independent of s t, which replaces the state-dependent MLP with an adaptive "placeholder" parameter vector making actions just a linear combination of source policy outputs. As shown in Table 2, the full version of MULTIPOLAR significantly outperformed both of the degraded versions, suggesting that the adaptive aggregation and predicting residuals are both critical. Figure 4 illustrates an example of the histogram of final episodic reward (average rewards of the last 100 training episodes) for the source policy candidates obtained in the Hopper environment. As shown in the figure, the source policies were diverse in terms of the performance on their original environment instances 5. In this setup, we investigate the effect of source policies performances on MULTIPOLAR sample efficiency. We created two separate pools of source policies, where one contained only high-performing and the other only lowperforming source policies 6. Table 3 summarizes the of sampling source policies from these pools (4 high, 2 high & 2 low, and 4 low performances) and compares them to the original MULTIPOLAR (shown as 'Random') also reported in Table 1. Not surprisingly, MULTIPOLAR performed the best when all the source policies were sampled from the highperformance pool. However, we emphasize that such highquality policies are not always available in practice, due to the variability of how they are learned or hand-crafted under their own environment instance. Figure 6 in Appendix B.1 illustrates that MULTIPOLAR can successfully learn to suppress the useless low-performing source policies. Effect of Number of Source Policies Finally, we show how the number of source policies contributes to MULTIPOLAR's sample efficiency in Table 4. Specifically, we trained MULTIPOLAR policies up to K = 16 to study how the mean of average episodic rewards changes. The monotonic performance improvement over K (for K ≤ 16), is achieved at the cost of increased training and inference time. In practice, we suggest balancing this speed-performance trade-off by using as many source policies as possible before reaching the inference time limit required by the application. Our work is broadly categorized as an instance of transfer RL , in which a policy for a target task is trained using information collected from source tasks. In this section, we highlight how our work is different from the existing approaches and also discuss the current limitations as well as future directions. Transfer between Different Dynamics There has been very limited work on transferring knowledge between agents in different environmental dynamics. As introduced briefly in Section 1, some methods require training samples collected from source tasks. These sampled experiences are then used for measuring the similarity between environment instances (; ;) or for conditioning a target policy to predict actions . Alternative means to quantify the similarity is to use a full specification of MDPs or environmental dynamics. In contrast, the proposed MULTI-POLAR allows the knowledge transfer only through the policies acquired from source environment instances, which is beneficial when source and target environments are not always connected to exchange information about their environmental dynamics and training samples. Leveraging Multiple Policies The idea of utilizing multiple source policies can be found in the literature of policy reuse frameworks (Fernández & ; ; ; ;). The basic motivation behind these works is to provide "nearly-optimal solutions" for short-duration tasks by reusing one of the source policies, where each source would perform well on environment instances with different rewards (e.g., different goals in maze tasks). In our problem setting, where environmental dynamics behind each source policy are different, reusing a single policy without an adaptation is not the right approach, as described in and also demonstrated in our experiment. Another relevant idea is hierarchical RL (; ;) that involves a hierarchy of policies (or action-value functions) to enable temporal abstraction. In particular, option frameworks (; ;) make use of a collection of policies as a part of "options". However, they assumed all the policies in the hierarchy to be learned in a single environment instance. Another relevant work along this line of research is , which meta-learns a hierarchy of multiple sub-policies by training a master policy over the distribution of tasks. Nevertheless, hierarchical RL approaches are not useful for leveraging multiple source policies each acquired under diverse environmental dynamics. Learning Residuals in RL Finally, some recent works adopt residual learning to mitigate the limited performance of hand-engineered policies (; ;). We are interested in a more extended scenario where various source policies with unknown performances are provided instead of a single sub-optimal policy. Also, these approaches focus only on RL problems for robotic tasks in the continuous action space, while our approach could work on both of continuous and discrete action spaces in a broad range of environments. Limitations and Future Directions Currently, our work has several limitations. First, MULTI-POLAR may not be scalable to a large number of source policies, as its training and testing times will increase almost linearly with the number of source policies. One possible solution for this issue would be pre-screening source policies before starting to train a target agent, for example, by testing each source on the target task and taking them into account in the training phase only when they are found useful. Moreover, our work assumes source and target environment instances to be different only in their state transition distribution. An interesting direction for future work is to involve other types of environmental differences, such as dissimilar rewards and state/action spaces. We presented a new problem setting of transfer RL that aimed to train a policy efficiently using a collection of source policies acquired under diverse environmental dynamics. We demonstrated that the proposed MULTIPOLAR is, despite its simplicity, a principled approach with high training sample efficiency on a variety of environments. Our transfer RL approach is advantageous when one does not have access to a distribution of diverse environmental dynamics. Future work will seek to adapt our approach to more challenging domains such as a real-world robotics task. In this section, we present all the experimental details for the six environments we used. Note that we did not do any hyperparameter-tuning but followed the default parameters of. We used the Roboschool implementation of Hopper, Ant, and InvertedPendulumSwingup since they are based on an open-source engine, which makes it possible for every researcher to reproduce our experiments. To run our experiments in parallel, we used GNU Parallel tool . A.1 HYPERPARAMETERS Tables 5 and 6 summarize all the hyperparameters used for experiments on each environment. As done by , to have a successful training, rewards and input observations are normalized using their running average and standard deviation for all the environments except CartPole and LunarLander. Also, in all of the experiments, θ agg is initialized to be the all-ones matrix. Tables 7, 8, 9, 10, 11 and 12. We defined these sampling ranges such that the ing environments are stable enough for successfully training an MLP policy. To do so, we trained MLP policies across wide ranges of environmental parameters and chose the ranges in which the policy converged. In this section, we explain how we calculated the mean of average episodic rewards in Tables 1, 2, 3, and 4, over a specific number of training samples (the numbers at the header of the tables e.g., 25K, 50K, 75K, and 100K for the CartPole) which we denote by T in what follows. For each experiment in an environment instance, we computed the average episodic reward by taking the average of the rewards over all the episodes the agent played from the beginning of the training until collecting T number of training samples. Then we collected the computed average episodic rewards of all the experiments, i.e., all the combinations of three random seeds, three random sets of source policies (for RPL and MULTIPOLAR), and 100 target environment instances. Finally, we used the sample bootstrap method to estimate the mean and the 95% confidence bounds of the collected average episodic rewards. We used the Facebook Boostrapped implementation: https://github.com/facebookincubator/bootstrapped. To generate environment instances, we uniformly sampled the dynamics and kinematics parameters from the ranges defined in Section A.2. Figure 5 illustrates the histograms of the final episodic rewards of source policies on the original environment instances in which they were acquired. Figure 6 visualizes an example of how the aggregation parameters θ agg for the four policies and their three actions were learned during the 2M timestep training of MULTIPOLAR (K = 4) policy in the Hopper environment. In this example, the source policies in the first and second rows were sampled from low-performance pools whereas those in the third and fourth rows were sampled from high-performance pools (see Section 4.2 for more details). It illustrates that MULTIPOLAR can successfully suppress the two useless low-performing policies as the training progresses. To further study how having low-performing source policies affects MULTIPOLAR sample efficiency, we evaluated MULTIPOLAR (K=4) in the Hopper environment, where the sources are randomly initialized policies, i.e., policies that predict actions randomly. Following our experimental procedure explained in Section 4.1, Table 13 reports the bootstrap mean and 95% confidence bounds of average episodic rewards over various training samples for this experiment and compares it with MULTIPOLAR with four low-performing sources. This suggests that the sample efficiency of MULTIPOLAR (K = 4) with low-performing source policies (i.e., source policies which had low final episodic rewards in their own environments) is almost the same as with randomly initialized source policies.

We propose MULTIPOLAR, a transfer RL method that leverages a set of source policies collected under unknown diverse environmental dynamics to efficiently learn a target policy in another dynamics.

Represent the TLDR for finding the corresponding scientific article

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.