Question: Represent the scientific article to find a short TLDR

Question: Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this to the well-known convex duality of Shannon entropy and the softmax function. Such a is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning. • Policy gradients (V.), looks to maximize the expected reward by improving policies to favor high-reward actions. In general, the target loss function is regularized by the addition of an entropic functional for the policy. This makes policies more diffuse and less likely to yield degenerate . A critical step in the theoretical understanding of the field has been a smooth relaxation of the greedy max operation involved in selecting actions, turned into a Boltzmann softmax O. BID8. This new context has lead to a breakthrough this year J. BID7 with the proof of the equivalence of both methods of Q-learning and policy gradients. While that is extremely impressive in its unification, we argue that it is critical to look additionally at the fundamental reasons as to why it occurs. We believe that the convexity of the entropy functional used for policy regularization is at the root of the phenomenon, and that (Lagrangian) duality can be exploited as well, either yielding faster proofs, or further understanding. The contributions of our paper are as follows:1. We show how convex duality expedites the proof of the equivalence between soft Qlearning and softmax entropic policy gradients -heuristically in the general case, rigorously in the bandit case.2. We introduce a transportation inequality that relates the expected optimality gap of any policy with its Kullback-Leibler divergence to the optimal policy. We describe our notations here. Abusing notation heavily by identifying measures with their densities as in dπ(a|s) = π(a|s)da, if we note as either r(s, a) or r(a, s) the reward obtained by taking action a in state s, the expected reward expands as: DISPLAYFORM0 K r is a linear functional of π. Adding Shannon entropic regularization 1 improves numerical stability of the algorithm, and prevents early convergence to degenerate solutions. Noting regularization strength β, the objective becomes a free energy functional, named by analogy with a similar quantity in statistical mechanics: DISPLAYFORM1 Crucially, viewed as a functional of π, J is convex and is the sum of two parts DISPLAYFORM2 2 THE GIBBS VARIATIONAL PRINCIPLE FOR POLICY EVALUATION Here we are interested in the optimal value of the policy functional J, achieved for an optimal policy π *. We hence look for J * = J(π *) = sup π∈P J(π). In the one step-one state bandit setting we are in, this is in fact almost the same as deriving the state-value function. The principles of convex duality BID3 BID16 DISPLAYFORM0 with H the entropy functional defined above, we recover exactly the definition of the LegendreFenchel transformation, or convex conjugate, of β · H. The word convex applies to the entropy functional, and doesn't make any assumptions on the rewards r(s, a), other that they be well-behaved enough to be integrable in a. The Legendre transform inverts derivatives. A simple calculation shows that the formal convex conjugate of f: t → t log t is f *: p → e (p−1) -this because their respective derivatives log and exp are reciprocal. We can apply this to f (π(a|s)) = π(a|s) log π(a|s), and then this relationship can also be integrated in a. Hence the dual Legendre representation of the entropy functional H is known. The Gibbs variational principle states that, taking β = 1/λ as the inverse temperature parameter, and for each Borelian (measurable) test function Φ ∈ C b (A): DISPLAYFORM1 or in shorter notation, for each real random variable X with exponential moments, DISPLAYFORM2 We can prove a stronger . If µ is a reference measure (or policy), and we now consider the relative entropy (or Kullback-Leibler divergence) with respect to µ, H µ (·), instead of the entropy H(·), then the Gibbs variational principle still holds BID15, chapter 22). This regarding dual representation formulas for entropy is important and in fact found in several areas of science:• as above, in thermodynamics, where it is named the Gibbs variational principle;• in large deviations, this also known as the Donsker-Varadhan variational formula BID4;• in statistics, it is the well-known duality between maximum entropy and maximum likelihood estimation BID0; • finally, the theory of information geometry BID1 groups all three views and posits that there exists a general, dually flat Riemannian information manifold. The general form of the is as follows. For each Φ representing a rewards function r(s, a) or an estimator of it: DISPLAYFORM3 and the supremum is reached for the measure π * ∈ P defined by its Radon-Nikodym derivative equal to the Gibbs-Boltzmann measure yielding an energy policy: DISPLAYFORM4 In the special case where µ is the Lebesgue measure on a bounded domain (that is, the uniform policy), we find back the 5 above, up to a constant irrelevant for maximization. In the general case, the mathematically inclined reader will also see this as a rephrasing of the fact the Bregman divergence associated with Shannon entropy is the Kullback-Leibler divergence. For completeness' sake, we provide here its full proof: Proposition 1. Donsker-Varadhan variational formula. Let G be a bounded measurable function on A and π,π be probability measures on A, with π absolutely continuous w.r.t.π. Then DISPLAYFORM5 where π * is a probability measure defined by the Radon-Nikodym derivative: DISPLAYFORM6 Proposition 2. Corollary: DISPLAYFORM7 and the maximum is attained uniquely by π *.Proof. DISPLAYFORM8 Under review as a conference paper at ICLR 2018The link with reinforcement learning is made by picking Φ = r(s, a), π = π(a|s), λ = 1/β, and by recalling the implicit dependency of the right member on s but not on π at optimality, so that we can write DISPLAYFORM9 which is the definition of the one-step soft Bellman operator at optimum R. Fox & Tishby. FORMULA0; O. Nachum & Schuurmans. (2017b.); T. BID12. Note that here V * (s) depends on the reference measure µ which is used to pick actions frequency -we can be off-policy, in which case V * is only a pseudo state-value function. In this simplified one-step setting, this provides a short and direct proof that in expectation, and trained to optimality, soft Q-learning and policy gradients ascent yield the same J. BID7. Standard Q-learning is the special case β → 0, λ → ∞ where by the Laplace principle we recover V (s) → max A r(s, a); that is, the zero-temperature limit, with no entropy regularization. For simplicity of exposition, we have restricted so far to the proof in the bandit setting; now we extend it to the general case. First by inserting V * (s) = sup π V π (s) in the representation formulas above applied to r(s, a) + γV * (s), so that DISPLAYFORM0 The proof in the general case will then be finished if we assume that we could apply the Bellman optimality principle not to the hard-max, but to the soft-max operator. This requires proving that the soft-Bellman operator admits a unique fixed point, which is the above. By the Brouwer fixed point theorem, it is enough to prove that it is a contraction, or at least non-expansive (we assume that the discount factor γ < 1 to that end). We do so below, noting that this has been shown many times in the literature, for instance in O. BID8. Refining the soft-Bellman operator just like above, but in the multi-step case, by the expression DISPLAYFORM1 we get the:Proposition 3. Nonexpansiveness of the soft-Bellman operator for the supremum norm f ∞. DISPLAYFORM2 Proof. Let us consider two state-value functions V (s) and V (s) along with the associated action-value functions Q (s, a) and Q (s, a). Besides, denote MDP transition probability by p(s |s, a). Then: DISPLAYFORM3 ∞ by Hölder's inequality DISPLAYFORM4 In summary, the program of the proof was as below:1. Write down the entropy-regularised policy gradient functional, and apply the DonskerVaradhan formula to it. 2. Write down the ing softmax Bellman operator as a solution to the sup maximization -this obviously also proves existence. 3. Show that the softmax operator, just like the hard max, is still a contraction for the max norm, hence prove uniqueness of the solution by fixed point theorem. The above also shows formally that, should we discretize the action space A to replace integration over actions by finite sums, any strong estimatorr(s, a) of r(s, a), applied to the partition function of rewards 1 λ log a e λr(s,a), could be used for Q-learning-like iterations. This is because strong convergence would imply weak convergence (especially convergence of the characteristic function, via Levy's continuity theorem), and hence convergence towards the log-sum-exp cumulant generative function above. Different estimatorsr(s, a) lead to different algorithms. When the MDP and the rewards function r are not known, the parameterised critic choicer(s, a) ≈ Q w (s, a) recovers Nachum's Path Consistency Learning O. BID8 BID15. O'Donoghue's PGQ method B. O'Donoghue & Mnih. FORMULA0 can be seen as a control variate balancing of the two terms in 7. In theory, the rewards distribution could be also recovered simply by varying λ (or β), for instance by inverse Laplace transform. In this section, we propose an inequality that relates the optimality gap of a policy -by how much that policy is sub-optimal on average -to the Kullback-Leibler divergence between the current policy and the optimum. The proof draws on ideas of convex analysis and Legendre transormation exposed earlier in the context of soft Q-learning. Let us assume that X is a real-valued bounded random variable. We denote sup |X| ≤ M with M constant. Furthermore we assume that X is centered, that is, E[X] = 0. This can always be achieved just by picking DISPLAYFORM0 Then, by the Hoeffding inequality: DISPLAYFORM1 with K a positive real constant, i.e., the variable X is sub-Gaussian, so that its cumulant generating function grows less than quadratically. By taking a Legendre transformation and inverting it, we get that for any pair of measures P and Q that are mutually absolutely continuous, one has DISPLAYFORM2 which by specializing Q to be the measure associated to P * the optimal policy, P θ the current parameterized policy, and X an advantage return r: DISPLAYFORM3 By the same logic, any upper bound on log E e βX can give us information about E Q X − E P X. This enables us to relate the size of Kullback-Leibler trust regions to the amount by which our policy could be improved. In fact by combining the entropy duality formula with the Legendre transformation, one easily proves the below: Proposition 4. Let X a real-valued integrable random variable, and f a convex and differentiable function such that f = f = 0. Then with f *: x → f * (x) = sup(βx − f (β)) the Legendre transformation of f, f * −1 its reciprocal, and P and Q any two mutually absolutely continuous measures, one has the equivalence: DISPLAYFORM4 Proof. By Donsker-Varadhan formula, one has that the equivalence is proven if and only if DISPLAYFORM5 but this right term is easily proven to be nothing but DISPLAYFORM6 the inverse of the Legendre transformation of f applied to D KL (Q||P).This also opens up the possibility of using various softmax temperatures β i in practical algorithms in order to estimate f. Finally, note that if P θ is a parameterized softmax policy associated with action-value functions Q θ (a, s) and temperature β, then because P * is proportional to e −r(a,s)/β, one readily has DISPLAYFORM7 which can easily be inserted in the inequality above for the special case Q = P *. Entropic reinforcement learning has appeared early in the literature with two different motivations. The view of exploration with a self-information intrinsic reward was pioneered by Tishby, and developed in Ziebart's PhD. thesis BID16. It was rediscovered recently that within the asynchronous actor-critic framework, entropic regularization is crucial to ensure convergence in practice V.. Furthermore, the idea of taking steepest KL divergence steps as a practical reinforcement learning method per se was adopted by Schulman J. Schulman & Abbeel. The key common development in these works has been to make entropic regularization recursively follow the Bellman equation, rather than naively regularizing one-step policies G. BID5. Schulman thereafter proposed a general proof of the equivalence, in the limit, of policy gradient and soft Q-learning methods J. BID7, but the proof does not explicitly make the connection with convex duality and the expeditive justification it yields in the one-step case. Applying the Gibbs/Donsker-Varadhan variational formula to entropy in a machine learning context is, however, not new; see for instance BID0. Some of the convex optimization they invoke, including proximal stepping, can be found in the complete treatment by BID3. In the context of neural networks, convex analysis and partial differential equation methods are covered by BID10. Using dual formulas for the entropy functional in reinforcement learning has vast potential ramifications. One avenue of research will be to interpret our findings in a large deviations framework -the log-sum-exp cumulant generative function being an example of rate function governing fluctuations of the tail of empirical n-step returns. Smart drift change techniques could lead to significant variance reduction for Monte-Carlo rollout estimators. We also hope to exploit further concentration inequalities in order to provide more bounds for the state value function. Finally, a complete theory of the one-to-one correspondence between convex approximation algorithms and reinforcement learning methods is still lacking to date. We hope to be able to contribute in this direction through further work.

Represent the TLDR for finding the corresponding scientific article

A short proof of the equivalence of soft Q-learning and policy gradients.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Computer simulation provides an automatic and safe way for training robotic control policies to achieve complex tasks such as locomotion. However, a policy trained in simulation usually does not transfer directly to the real hardware due to the differences between the two environments. Transfer learning using domain randomization is a promising approach, but it usually assumes that the target environment is close to the distribution of the training environments, thus relying heavily on accurate system identification. In this paper, we present a different approach that leverages domain randomization for transferring control policies to unknown environments. The key idea that, instead of learning a single policy in the simulation, we simultaneously learn a family of policies that exhibit different behaviors. When tested in the target environment, we directly search for the best policy in the family based on the task performance, without the need to identify the dynamic parameters. We evaluate our method on five simulated robotic control problems with different discrepancies in the training and testing environment and demonstrate that our method can overcome larger modeling errors compared to training a robust policy or an adaptive policy. Recent developments in Deep Reinforcement Learning (DRL) have shown the potential to learn complex robotic controllers in an automatic way with minimal human intervention. However, due to the high sample complexity of DRL algorithms, directly training control policies on the hardware still remains largely impractical for agile tasks such as locomotion. A promising direction to address this issue is to use the idea of transfer learning which learns a model in a source environment and transfers it to a target environment of interest. In the context of learning robotic control policies, we can consider the real world the target environment and the computer simulation the source environment. Learning in simulated environment provides a safe and efficient way to explore large variety of different situations that a real robot might encounter. However, due to the model discrepancy between physics simulation and the real-world environment, also known as the Reality Gap BID2 BID18, the trained policy usually fails in the target environment. Efforts have been made to analyze the cause of the Reality Gap BID20 and to develop more accurate computer simulation to improve the ability of a policy when transferred it to real hardware. Orthogonal to improving the fidelity of the physics simulation, researchers have also attempted to cross the reality gap by training more capable policies that succeed in a large variety of simulated environments. Our method falls into the second category. To develop a policy capable of performing in various environments with different governing dynamics, one can consider to train a robust policy or to train an adaptive policy. In both cases, the policy is trained in environments with randomized dynamics. A robust policy is trained under a range of dynamics without identifying the specific dynamic parameters. Such a policy can only perform well if the simulation is a good approximation of the real world dynamics. In addition, for more agile motor skills, robust policies may appear over-conservative due to the uncertainty in the training environments. On the other hand, when an adaptive policy is used, it learns to first identify, implicitly or explicitly, the dynamics of its environment, and then selects the best action according to the identified dynamics. Being able to act differently according to the dynamics allows the adaptive policy to achieve higher performance on a larger range of dynamic systems. However, when the target dynamics is notably different from the training dynamics, it may still produce sub-optimal for two reasons. First, when a sequence of novel observations is presented, the learned identification model in an adaptive policy may produce inaccurate estimations. Second, even when the identification model is perfect, the corresponding action may not be optimal for the new situation. In this work, we introduce a new method that enjoys the versatility of an adaptive policy, while avoiding the challenges of system identification. Instead of relating the observations in the target environment to the similar experiences in the training environment, our method searches for the best policy directly based on the task performance in the target environment. Our algorithm can be divided to two stages. The first stage trains a family of policies, each optimized for a particular vector of dynamic parameters. The family of policies can be parameterized by the dynamic parameters in a continuous representation. Each member of the family, referred to as a strategy, is a policy associated with particular dynamic parameters. Using a locomotion controller as an example, a strategy associated with low friction coefficient may exhibit cautious walking motion, while a strategy associated with high friction coefficient may in more aggressive running motion. In the second stage we perform a search over the strategies in the target environment to find the one that achieves the highest task performance. We evaluate our method on three examples that demonstrate transfer of a policy learned in one simulator DART, to another simulator MuJoCo. Due to the differences in the constraint solvers, these simulators can produce notably different simulation . A more detailed description of the differences between DART and MuJoCo is provided in Appendix A. We also add latency to the MuJoCo environment to mimic a real world scenario, which further increases the difficulty of the transfer. In addition, we use a quadruped robot simulated in Bullet to demonstrate that our method can overcome actuator modeling errors. Latency and actuator modeling have been found to be important for Sim-to-Real transfer of locomotion policies BID20. Finally, we transfer a policy learned for a robot composed of rigid bodies to a robot whose end-effector is deformable, demonstrating the possiblity of using our method to transfer to problems that are challenging to model faithfully. While DRL has demonstrated its ability to learn control policies for complex and dynamic motor skills in simulation BID28 BID23 BID15, very few learning algorithms have successfully transferred these policies to the real world. Researchers have proposed to address this issue by optimizing or learning a simulation model using data from the real-world BID9 BID13 BID1. The main drawback for these methods is that for highly agile and high dimensional control problems, fitting an accurate dynamic model can be challenging and data inefficient. Complementary to learning an accurate simulation model, a different line of research in sim-to-real transfer is to learn policies that can work under a large variety of simulated environments. One common approach is domain randomization. Training a robust policy with domain randomization has been shown to improve the ability to transfer a policy BID34 BID26 BID25. BID34 trained an object detector with randomized appearance and applied it in a real-world gripping task. showed that training a robust policy with randomized dynamic parameters is crucial for transferring quadruped locomotion to the real world. Designing the parameters and range of the domain to be randomized requires specific knowledge for different tasks. If the range is set too high, the policy may learn a conservative strategy or fail to learn the task, while a small range may not provide enough variation for the policy to transfer to real-world. A similar idea is to train an adaptive policy with the current and the past observations as input. Such an adaptive policy is able to identify the dynamic parameters online either implicitly BID21 BID24 or explicitly BID38 ) and apply actions appropriate for different system dynamics. Recently, adaptive policies have been used for sim-to-real transfer, such as in-hand manipulation tasks BID21 or non-prehensile manipulation tasks BID24. Instead of training one robust or adaptive policy, trained multiple policies for a set of randomized environments and learned to combine them linearly in a separate set of environments. The main advantage of these methods is that they can be trained entirely in simulation and deployed in real-world without further fine-tuning. However, policies trained in simulation may not generalize well when the discrepancy between the target environment and the simulation is too large. Our method also uses dynamic randomization to train policies that exhibit different strategies for different dynamics, however, instead of relying on the simulation to learn an identification model for selecting the strategy, we propose to directly optimize the strategy in the target environment. A few recent works have also proposed the idea of training policies in a source environment and fine-tune it in the target environment. For example, BID7 proposed MAP-Elite to learn a large set of controllers and applied Bayesian optimization for fast adaptation to hardware damages. Their approach searches for individual controllers for discrete points in a behavior space, instead of a parameterized family of policies as in our case, making it potentially challenging to be applied to higher dimensional behavior spaces. BID27 used progressive network to adapt the policy to new environments by designing a policy architecture that can effectively utilize previously learned representations. BID4 learned an implicit representation of the environment variations by optimizing a latent policy input for each discrete instance of the environment. They showed that fine-tuning on this learned policy achieved improved learning efficiency. In contrast to prior work in which the fine-tuning phase adjusts the neural network weights in the target environment, we optimize only the dynamics parameters input to the policy. This allows our policies to adapt to the target environments with less data and to use sparse reward signal. We formulate the motor skill learning problem as a Markov Decision Process (MDP), M = (S, A, r, P, p 0, γ), where S is the state space, A is the action space, r: S × A → R is the reward function, P: S × A → S is the transition function, p 0 is the initial state distribution and γ is the discount factor. The goal of reinforcement learning is to find a control policy π: S → A that maximizes the expected accumulated reward: J M (π) = E τ =(s0,a0,...,s T) T t=0 γ t r(s t, a t), where s 0 ∼ p 0, a t ∼ π(s t) and s t+1 = P(s t, a t). In practice, we usually only have access to an observation of the robot that contains a partial information of the robot's state. In this case, we will have a Partially-Observable Markov Decision Process (POMDP) and the policy would become π: O → A, where O is the observation space. In the context of transfer learning, we can define a source MDP M s and a target MDP M t and the goal would be to learn a policy π s for M s such that it also works well on M t. In this work, P is regarded as a parameterized space of transition functions, s t+1 = P µ (s t, a t), where µ is a vector of physical parameters defining the dynamic model (e.g. friction coefficient). The transfer learning in this context learns a policy under P s and transfers to P t, where P s = P t. We propose a new method for transferring a policy learned in simulated environment to a target environment with unknown dynamics. Our algorithm consists of two stages: learning a family of policies and optimizing strategy. The first stage of our method is to learn a family of policies, each for a particular dynamics P s µ (·). One can potentially train each policy individually and interpolate them to cover the space of µ BID30 BID8. However, as the dimension of µ increases, the number of policies required for interpolation grows exponentially. Since many of these policies are trained under similar dynamics, our method merges them into one neural network and trains the entire family of policies simultaneously. We follow the work by BID38, which trains a policy π: (o, µ) → a that takes as input not only the observation of the robot o, but also the physical parameters µ. At the beginning of each rollout during the training, we randomly pick a new set of physical parameters for the simulation and fix it throughout the rollout. After training the policy this way, we obtain a family of policies that is parameterized by the dynamics parameters µ. Given a particular µ, we define the corresponding policy as π µ: o → a. We will call such an instantiated policy a strategy. The second stage of our method is to search for the optimal strategy in the space of µ for the target environment. Previous work learns a mapping between the experiences under source dynamics P s µ and the corresponding µ. When new experiences are generated in the target environment, this mapping will identify a µ based on similar experiences previously generated in the source environment. While using experience similarity as a metric to identify µ transfers well to a target environment that has the same dynamic parameter space BID38, it does not generalize well when the dynamic parameter space is different. Since our goal is to find a strategy that works well in the target environment, a more direct approach is to use the performance of the task, i.e. the accumulated reward, in the target environment as the metric to search for the strategy: DISPLAYFORM0 Solving Equation 1 can be done efficiently because the search space in Equation 1 is the space of dynamic parameters µ, rather than the space of policies, which are represented as neural networks in our implementation. To further reduce the number of samples from the target environment needed for solving Equation 1, we investigated a number of algorithms, including Bayesian optimization, model-based methods and an evolutionary algorithm (CMA). A detailed description and comparison of these methods are provided in Appendix C.We chose Covariance Matrix Adaptation (CMA) BID14, because it reliably outperforms other methods in terms of sample-efficiency. At each iteration of CMA, a set of samples are drawn from a Gaussian distribution over the space of µ. For each sample, we instantiate a strategy π µ and use it to generate rollouts in the target environment. The fitness of the sample is determined by evaluating the rollouts using J M t. Based on the fitness values of the samples in the current iteration, the mean and the covariance matrix of the Gaussian distribution are updated for the next iteration. To evaluate the ability of our method to overcome the reality gap, we train policies for four locomotion control tasks (hopper, walker2d, half cheetah, quadruped robot) and transfer each policy to environments with different dynamics. To mimic the reality gap seen in the real-world, we use target environments that are different from the source environments in their contact modeling, latency or actuator modeling. In addition, we also test the ability of our method to generalize to discrepancies in body mass, terrain slope and end-effector materials. FIG0 shows the source and target environments for all the tasks and summarizes the modeled reality gap in each task. During training, we choose different combinations of dynamic parameters to randomize and make sure they do not overlap with the variations in the testing environments. For clarity of exposition, we denote the dimension of the dynamic parameters that are randomized during training as dim(µ). For all examples, we use the Proximal Policy Optimization (PPO) to optimize the control policy. A more detailed description of the experiment setup as well as the simulated reality gaps are provided in Appendix B. For each example presented, we run three trials with different random seeds and report the mean and one standard deviation for the total reward. We compare our method, Strategy Optimization with CMA-ES (SO-CMA) to three baseline methods: training a robust policy (Robust), training an adaptive policy (Hist) and training a Universal Policy with Online System Identification (UPOSI) BID38. The robust policy is represented as a feed forward neural network, which takes as input the most recent observation from the robot, i.e. π robust: o → a. The policy needs to learn actions that work for all the training environments, but the dynamic parameters cannot be identified from its input. In contrast, an adaptive policy is given a history of observations as input, i.e. π adapt: (o t−h, . . ., o t) → a t. This allows the policy to potentially identify the environment being tested and adaptively choose the actions based on the identified environment. There are many possible ways to train an adaptive policy, for example, one can use an LSTM network to represent the policy or use a history of observations as input to a feed-forward network. We find that for the tasks we demonstrate, directly training an LSTM policy using PPO is much less efficient and reaches lower end performance than training a feed-forward network with history input. Therefore, in our experiments we use a feed-forward network with a history of 10 observations to represent the adaptive policy π adapt. We also compare our method to UPOSI, which decouples the learning of an adaptive policy into training a universal policy via reinforcement learning and a system identification model via supervised learning. In theory UPOSI and Hist should achieve similar performance, while in practice we expect UPOSI to learn more efficiently due to the decoupling. We adopt the same training procedure as done by BID38, and use a history of 10 observations as input to the online system identification model. For fair comparison, we continue to train the baseline methods after transferring to the target environment, using the same amount of samples SO-CMA consumes in the target environment. We refer this additional training step as'fine-tuning'. In addition to the baseline methods, we also compare our method to the performance of policies trained directly in the target environments, which serves as an'Oracle' benchmark. The Oracle policies for Hopper, Walke2d, HalfCheetah and Hopper Soft was trained for 1, 000, 000 samples in the target environment as in. For the quadruped example, we run PPO for 5, 000, 000 samples, similar to. We detail the process of'fine-tuning' in Appendix B.4 In the first example, we build a single-legged robot in DART similar to the Hopper environment simulated by MuJoCo in OpenAI Gym BID3. We investigate two questions in this example: 1) does SO-CMA work better than alternative methods in transferring to unknown environments? and 2) how does the choice of dim(µ) affect the performance of policy transfer? To this end, we perform experiments with dim(µ) = 2, 5 and 10. For the experiment with dim(µ) = 2, we randomize the mass of the robot's foot and the restitution coefficient between the foot and the ground. For dim(µ) = 5, we in addition randomize the friction coefficient, the mass of the robot's torso and the joint strength of the robot. We further include the mass of the rest two body parts and the joint damping to construct the randomized dynamic parameters for dim(µ) = 10. The specific ranges of randomization are described in Appendix B.4.We first evaluate how the performance of different methods varies with the number of samples in the target environment. As shown in FIG1, when dim(µ) is low, none of the four methods were able to transfer to the MuJoCo Hopper successfully. This is possibly due to there not being enough variation in the dynamics to learn diverse strategies. When dim(µ) = 5, SO-CMA can successfully transfer the policy to MuJoCo Hopper with good performance, while the baseline methods were not able to adapt to the new environment using the same sample budget. We further increase dim(µ) to 10 as shown in FIG1 (c) and find that SO-CMA achieved similar end performance to dim(µ) = 5, while the baselines do not transfer well to the target environment. We further investigate whether SO-CMA can generalize to differences in joint limits in addition to the discrepancies between DART and MuJoCo. Specifically, we vary the magnitude of the ankle joint limit in [0.5, 1.0] radians (default is 0.785) for the MuJoCo Hopper, and run all the methods with 30, 000 samples. The can be found in FIG2. We can see a similar trend that with low dim(µ) the transfer is challenging, and with higher value of dim(µ) SO-CMA is able to achieve notably better transfer performance than the baseline methods. In this example, we use the lower body of a biped robot constrained to a 2D plane, according to the Walker2d environment in OpenAI Gym. We find that with different initializations of the policy network, training could lead to drastically different gaits, e.g. hopping with both legs, running with one legs dragging the other, normal running, etc. Some of these gaits are more robust to environment changes than others, which makes analyzing the performance of transfer learning algorithms challenging. To make sure the policies are more comparable, we use the symmetry loss from, which leads to all policies learning a symmetric running gait. To mimic modeling error seen on real robots, we add a latency of 8ms to the MuJoCo simulator. We train policies with dim(µ) = 8, for which we randomize the friction coefficient, restitution coefficient and the joint damping of the six joints during training. FIG3 (a) shows the transfer performance of different method with respect to the sample numbers in the target environment. We further vary the mass of the robot's right foot inkg in the MuJoCo Walker2d environment and compare the transfer performance of SO-CMA to the baselines. The default foot mass is 2.9 kg. We use in total 30, 000 samples in the target environment for all methods being compared and the can be found in FIG3 (b). In both cases, our method achieves notably better performance than Hist and UPOSI, while being comparable to Robust. In the third example, we train policies for the HalfCheetah environment from OpenAI Gym. We again test the performance of transfer from DART to MuJoCo for this example. In addition, we add a latency of 50ms to the target environment. We randomize 11 dynamic parameters in the source environment consisting of the mass of all body parts, the friction coefficient and the restitution coefficient during training, i.e. dim(µ) = 11. The of the performance with respect to sample numbers in target environment can be found in FIG4 (a). We in addition evaluate transfer to environments where the slope of the ground varies, as shown in FIG4 (b). We can see that SO-CMA outperforms Robust and Hist, while achieves similar performance as UPOSI. As demonstrated by, when a robust policy is used, having an accurate actuator model is important to the successful transfer of policy from simulation to real-world for a quadruped robot, Minitaur FIG0 ). Specifically, they found that when a linear torque-current relation is assumed in the actuator dynamics in the simulation, the policy learned in simulation transfers poorly to the real hardware. When the actuator dynamics is modeled more accurately, in their case using a non-linear torque-current relation, the transfer performance were notably improved. In our experiment, we investigate whether SO-CMA is able to overcome the error in actuator models. We use the same simulation environment from, which is simulated in Bullet (. During the training of the policy, we use a linear torque-current relation for the actuator model, and we transfer the learned policy to an environment with the more accurate non-linear torque-current relation. We use the same 25 dynamic parameters and corresponding ranges used by for dynamics randomization during training. When applying the robust policy to the accurate actuator model, we observe that the quadruped tends to sink to the ground, similar to what was observed by . SO-CMA, on the other hand, can successfully transfer a policy trained with a crude actuator model to an environment with more realistic actuators( FIG5 (a)). Applying deep reinforcement learning to environments with deformable objects can be computationally inefficient BID5. Being able to transfer a policy trained in a purely rigid-body environment to an environment containing deformable objects can greatly improve the efficiency of learning. In our last example, we transfer a policy trained for the Hopper example with rigid objects only to a Hopper model with a deformable foot FIG0 ). The soft foot is modeled using the soft shape in DART, which uses an approximate but relatively efficient way of modeling deformable objects BID17. We train policies in the rigid Hopper environment and randomize the same set of dynamic parameters as in the in the DART-to-MuJoCo transfer example with dim(µ) = 5. We then transfer the learned policy to the soft Hopper environment where the Hopper's foot is deformable. The can be found in FIG5 (b). SO-CMA is able to successfully control the robot to move forward without falling, while the baseline methods fail to do so. We have demonstrated that our method, SO-CMA, can successfully transfer policies trained in one environment to a notably different one with a relatively low amount of samples. One advantage of SO-CMA, compared to the baselines, is that it works consistently well across different examples, while none of the baseline methods achieve successful transfer for all the examples. We hypothesize that the large variance in the performance of the baseline methods is due to their sensitivity to the type of task being tested. For example, if there exists a robust controller that works for a large range of different dynamic parameters µ in the task, such as a bipedal running motion in the Walker2d example, training a Robust policy may achieve good performance in transfer. However, when the optimal controller is more sensitive to µ, Robust policies may learn to use overly-conservative strategies, leading to sub-optimal performance (e.g. in HalfCheetah) or fail to perform the task (e.g. in Hopper). On the other hand, if the target environment is not significantly different from the training environments, UPOSI may achieve good performance, as in HalfCheetah. However, as the reality gap becomes larger, the system identification model in UPOSI may fail to produce good estimates and in non-optimal actions. Furthermore, Hist did not achieve successful transfer in any of the examples, possibly due to two reasons: 1) it shares similar limitation to UPOSI when the reality gap is large and 2) it is in general more difficult to train Hist due to the larger input space, so that with a limited sample budget it is challenging to fine-tune Hist effectively. We also note that although in some examples certain baseline method may achieve successful transfer, the fine-tuning process of these methods relies on having a dense reward signal. In practice, one may only have access to a sparse reward signal in the target environment, e.g. distance traveled before falling to the ground. Our method, using an evolutionary algorithm (CMA), naturally handles sparse rewards and thus the performance gap between our method (SO-CMA) and the baseline methods will likely be large if a sparse reward is used. We have proposed a policy transfer algorithm where we first learn a family of policies simultaneously in a source environment that exhibits different behaviors and then search directly for a policy in the family that performs the best in the target environment. We show that our proposed method can overcome large modeling errors, including those commonly seen on real robotic platforms with relatively low amount of samples in the target environment. These suggest that our method has the potential to transfer policies trained in simulation to real hardware. There are a few interesting directions that merit further investigations. First, it would be interesting to explore other approaches for learning a family of policies that exhibit different behaviors. One such example is the method proposed by BID11, where an agent learns diverse skills without a reward function in an unsupervised manner. Another example is the HCP-I policy proposed by BID4, which learns a latent representation of the environment variations implicitly. Equipping our policy with memories is another interesting direction to investigate. The addition of memory will extend our method to target environments that vary over time. We have investigated in a few options for strategy optimization and found that CMA-ES works well for our examples. However, it would be desired if we can find a way to further reduce the sample required in the target environment. One possible direction is to warm-start the optimization using models learned in simulation, such as the calibration model in or the online system identification model in BID38.A DIFFERENCES BETWEEN DART AND MUJOCO DART BID19 and MuJoCo BID35 are both physically-based simulators that computes how the state of virtual character or robot evolves over time and interacts with other objects in a physical way. Both of them have been demonstrated for transferring controllers learned for a simulated robot to a real hardware, and there has been work trying to transfer policies between DART and MuJoCo BID36. The two simulators are similar in many aspects, for example both of them uses generalized coordinates for representing the state of a robot. Despite the many similarities between DART and MuJoCo, there are a few important differences between them that makes transferring a policy trained in one simulator to the other challenging. For the examples of DART-to-MuJoCo transfer presented in this paper, there are three major differences as described below:1. Contact Handling Contact modeling is important for robotic control applications, especially for locomotion tasks, where robots heavily rely on manipulating contacts between end-effector and the ground to move forward. In DART, contacts are handled by solving a linear complementarity problem (LCP) (Tan et al.), which ensures that in the next timestep, the objects will not penetrate with each other, while satisfying the laws of physics. In MuJoCo, the contact dynamics is modeled using a complementarity-free formulation, which means the objects might penetrate with each other. The ing impulse will increase with the penetration depth and separate the penetrating objects eventually. Similar to the contact solver, DART tries to solve the joint limit constraints exactly so that the joint limit is not violated in the next timestep, while MuJoCo uses a soft constraint formulation, which means the character may violate the joint limit constraint. In MuJoCo, a diagonal matrix σI n is added to the joint space inertia matrix that can help stabilize the simulation, where σ ∈ R is a scalar named Armature in MuJoCo and I n is the n × n identity matrix. This is not modeled in DART.To illustrate how much difference these simulator characteristics can lead to, we compare the Hopper example in DART and MuJoCo by simulating both using the same sequence of randomly generated actions from an identical state. We plot the linear position and velocity of the torso and foot of the robot, which is shown in FIG6. We can see that due to the differences in the dynamics, the two simulators would control the robot to reach notably different states even though the initial state and control signals are identical. B EXPERIMENT DETAILS We use Proximal Policy Optimization (PPO) implemented in OpenAI Baselines for training all the policies in our experiments. For simulation in DART, we use DartEnv (Yu BID38, which implements the continuous control benchmarks in OpenAI Gym using PyDart BID12 . For all of our examples, we represent the policy as a feed-forward neural network with three hidden layers, each consists of 64 hidden nodes. The observation space, action space and the reward function used in all of our examples can be found in TAB0 . For the Walker2d environment, we found that with the original environment settings in OpenAI Gym, the robot sometimes learn to hop forward, possibly due to the ankle being too strong. Therefore, we reduce the torque limit of the ankle joint in both DART and MuJoCo environment for the Walker2d problem from [−100, 100] to [−20, 20]. We found that with this modification, we can reliably learn locomotion gaits that are closer to a human running gait. Below we list the dynamic randomization settings used in our experiments. TAB1 and TAB3 shows the range of the randomization for different dynamic parameters in different environments. For the quadruped example, we used the same settings as in. To evaluate the ability of our method to overcome the modeling error, we designed six types of modeling errors. Each example shown in our experiments contains one or more modeling errors listed below. For the Hopper, Walker2d and HalfCheetah example, we trained policies that transfers from DART environment to MuJoCo environment. As discussed in Appendix A, the major differences between DART and MuJoCo are contacts, joint limits and armature. The second type of modeling error we tested is latency in the signals. Specifically, we model the latency between when an observation o is sent out from the robot, and when the action corresponding to this observation a = π(o) is executed on the robot. When a policy is trained without any delay, it is usually very challenging to transfer it to problems with delay added. The value of delay is usually below 50ms and we use 8ms and 50ms in our examples. As noted by, error in actuator modeling is an important factor that contributes to the reality gap. They solved it by identifying a more accurate actuator model by fitting a piece-wise linear function for the torque-current relation. We use their identified actuator model as the ground-truth target environment in our experiments and used the ideal linear torque-current relation in the source environments. In the example of Walker2d, we vary the mass of the right foot on the robot to create a family of target environments for testing. The range of the torso mass varies inkg. In the example of HalfCheetah, we vary the slope of the ground to create a family of target environments for testing. This is implemented as rotating the gravity direction by the same angle. The angle varies in the range [−0.18, 0.0] radians. The last type of modeling error we test is that a deformable object in the target environment is modeled as a rigid object in the source environment. The deformable object is modeled using the soft shape object in DART. In our example, we created a deformable box of size 0.5m × 0.19m × 0.13m around the foot of the Hopper. We set the stiffness of the deformable object to be 10, 000 and the damping to be 1.0. We refer readers to BID17 for more details of the softbody simulation. For training policies in the source environment, we run PPO for 500 iterations. In each iteration, we sample 40, 000 steps from the source environment to update the policy. For the rest of the hyperparameters, we use the default value from OpenAI Baselines. We use a large batch size in our experiments as the policy needs to be trained to work on different dynamic parameters µ.For fine-tuning of the Robust and Adaptive policy in the target environment, we sample 2, 000 steps from the target environment at each iteration of PPO, which is the default value used in OpenAI Baselines. Here we use a smaller batch size for two reasons: 1) since the policy is trained to work on only one dynamics, we do not need as many samples to optimize the policy in general and 2) the fine-tuning process has a limited sample budget and thus we want to use a smaller batch size so that the policy can be improved more. In the case where we use a maximum of 50, 000 samples for fine-tuning, this amounts to 50 iterations of PPO updates. Furthermore, we use a maximum rollout length of 1, 000, while the actual length of the rollout collected during training is general shorter due to the early termination, e.g. when the robot falls to the ground. Therefore, with 50, 000 samples in total, the fine-tuning process usually consists of 100 ∼ 300 rollouts, depending on the task. We use the CMA-ES implementation in python by (PyC). At each iteration of CMA-ES, we generate 4 + 3 * log(N) samples from the latest Gaussian distribution, where N is the dimension of the dynamic parameters. During evaluation of each sample µ i, we run the policy π µi in the target environment for three trials and average the returns to obtain the fitness of this sample. In addition to CMA-ES, we have also experimented with a few other options for finding the best µ such that π µ works well in the target environment. Here we show some experiment for Strategy Optimization with Bayesian Optimization (SO-BO) and Model-based Optimization (SO-MB). Bayesian Optimization is a gradient-free optimization method that is known to work well for low dimensional continuous problems where evaluating the quality of each sample can be expensive. The main idea in Bayesian optimization is to incrementally build a Gaussian process (GP) model that estimates the loss of a given search parameter. At each iteration, a new sample is drawn by optimizing an acquisition function on the GP model. The acquisition function takes into account the exploration (search where the GP has low uncertainty) and exploitation (search where the GP predicts low loss). The new sample is then evaluated and added to the training dataset for GP.We test Bayesian Optimization on the Hopper and Quadruped example, as shown in FIG7. We can see that Bayesian Optimization can achieve comparable performance as CMA-ES and thus is a viable choice to our problem. However, SO-BA appears in general noisier than CMA-ES and is in general less computationally efficient due to the re-fitting of GP models. Another possible way to perform strategy optimization is to use a model-based method. In a modelbased method, we learn the dynamics of the target environment using generic models such as neural networks, Gaussian process, linear functions, etc. After we have learned a dynamics model, we can use it as an approximation of the target environment to optimize µ.We first tried using feed-forward neural networks to learn the dynamics and optimize µ. However, this method was not able to reliably find µ that lead to good performance. This is possibly due to that any error in the prediction of the states would quickly accumulate over time and lead to inaccurate predictions. In addition, this method would not be able to handle problems where latency is involved. In the experiments presented here, we learn the dynamics of the target environment with a Long Short Term Memory (LSTM) network BID16. Given a target environment, we first sample µ uniformly and collect experience using π µ until we have 5, 000 samples. We use these samples to fit an initial LSTM dynamic model. We then alternate between finding the best dynamic parametersμ such that πμ achieves the best performance under the latest LSTM dynamic model and update the LSTM dynamic model using data generated from πμ. This is repeated until we have reached the sample budget. We found that LSTM notably outperformed feed-forward networks when applied to strategy optimization. One for Hopper DART-to-MuJoCo can be found in FIG8. It can be seen that Model-based method with LSTM is able to achieve similar performance as CMA-ES. Model-based method provides more flexibility over CMA-ES and Bayesian optimization. For example, if the target environment changes over time, it may be desired to have µ also be time-varying. However, this would lead to a high dimensional search space, which might require significantly more samples for CMA-ES or Bayesian Optimization to solve the problem. If we can learn an accurate enough model from the data, we can use it to generate synthetic data for solving the problem. However, there are two major drawbacks for Model-based method. The first is that to learn the dynamics model, we need to have access to the full state of the robot, which can be challenging or troublesome in the real-world. In contrast, CMA-ES and Bayesian optimization only require the final return of a rollout. Second, the Model-based method is significantly slower to run than the other methods due to the frequent training of the LSTM network.

represent the short summary for finding the corresponding scientific article (long)

We propose a policy transfer algorithm that can overcome large and challenging discrepancies in the system dynamics such as latency, actuator modeling error, etc.

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition. Learning discrete representations of speech has gathered much recent interest . A popular approach to discover discrete units is via autoencoding (; ;) sometimes coupled with an autoregressive model. Another line of research is to learn continuous speech representations in a self-supervised way via predicting context information (; van den ;). In this paper, we combine these two lines of research by learning discrete representations of speech via a context prediction task instead of reconstructing the input. This enables us to directly apply well performing NLP algorithms to speech data (Figure 1a). The vq-wav2vec encoder maps raw audio (X) to a dense representation (Z) which is quantized (q) toẐ and aggregated into context representations (C); training requires future time step prediction. (b) Acoustic models are trained by quantizing the raw audio with vq-wav2vec, then applying BERT to the discretized sequence and feeding the ing representations into the acoustic model to output transcriptions. Our new discretization algorithm, vq-wav2vec, learns discrete representations of fixed length segments of audio signal by utilizing the wav2vec loss and architecture (; §2). To choose the discrete variables, we consider a Gumbel-Softmax approach as well as online k-means clustering, similar to VQ-VAE (; ; §3). We then train a Deep Bidirectional Transformer (BERT; ; on the discretized unlabeled speech data and input these representations to a standard acoustic model (Figure 1b; §4). Our experiments show that BERT representations perform better than log-mel filterbank inputs as well as dense wav2vec representations on both TIMIT and WSJ benchmarks. Discretization of audio enables the direct application of a whole host of algorithms from the NLP literature to speech data. For example, we show that a standard sequence to sequence model from the NLP literature can be used to perform speech recognition over discrete audio tokens (§5, §6). 2.1 WAV2VEC wav2vec learns representations of audio data by solving a self-supervised context-prediction task with the same loss function as word2vec (; van den). The model is based on two convolutional neural networks where the the encoder produces a representation z i for each time step i at a rate of 100 Hz and the aggregator combines multiple encoder time steps into a new representation c i for each time step i. Given an aggregated representation c i, the model is trained to distinguish a sample z i+k that is k steps in the future from distractor samplesz drawn from a distribution p n, by minimizing the contrastive loss for steps k = 1,..., K: where T is the sequence length, σ(x) = 1/(1 + exp(−x)), and where σ(z i+k h k (c i)) is the probability of z i+k being the true sample. We consider a step-specific affine transformation et al., 2018). We optimize the loss L = K k=1 L k, summing over different step sizes. After training, the representations produced by the context network c i are input to the acoustic model instead of log-mel filterbank features. BERT is a pre-training approach for NLP tasks, which uses a transformer encoder model to build a representation of text. Transformers uses self-attention to encode the input sequence as well as an optional source sequence . The original BERT model combined two tasks for training: first, masked language modeling randomly removes some of the input tokens and the model has to predict those missing tokens. Second, next sentence prediction splices two different text passages together into a single example and the model needs to predict whether the passages are from the same document. Our approach, vq-wav2vec, learns vector quantized (VQ) representations of audio data using a future time-step prediction task. We follow the same architectual choices as wav2vec (§2.1) with two convolutional networks f: X → Z and g:Ẑ → C for feature extraction and aggregation, as well as a new quantization module q: Z →Ẑ to build discrete representations (Figure 1a). We first map 30ms segments of raw speech to a dense feature representation z at a stride of 10ms using the encoder network f. Next, the quantizer (q) turns these dense representations into discrete indices which are mapped to a reconstructionẑ of the original representation z. We feedẑ into the aggregator g and optimize the same context prediction task as wav2vec outlined in §2.1. The quantization module replaces the original representation z byẑ = e i from a fixed size codebook e ∈ R V ×d which contains V representations of size d. We consider the Gumbel-Softmax which is a differentiable approximation of the argmax for computing one-hot representations (§3.1; Figure 2a) as well as online k-means clustering, similar to the vector quantized variational autoencoder (VQ-VAE; ; §3.2; Figure 2b). Finally, we perform multiple vector quantizations over different parts of z to mitigate mode collapse (§3.3). The Gumbel-Softmax (; ;) enables selecting discrete codebook variables in a fully differentiable way and we use the straight-through estimator of. Given the dense representation z, we apply a linear layer, followed by a ReLU and another linear which outputs l ∈ R V logits for the Gumbel-Softmax. At inference, we simply pick the largest index in l. At training, the output probabilities for choosing the j-th variable are where v = − log(− log(u)) and u are uniform samples from U. During the forward pass, i = argmax j p j and in the backward pass, the true gradient of the Gumbel-Softmax outputs is used. The vector quantization approach of van den is an alternative to making the index selection procedure fully differentiable. Different to their setup, we optimize a future time step prediction loss instead of the reconstruction loss of an autoencoder. We choose the codebook variable representation by finding the closest variable to the input features z in terms of the Euclidean distance, yielding i = argmin j z − e j 2 2. During the forward pass, we selectẑ = e i by choosing the corresponding variable from the codebook. We obtain gradients for the encoder network by back-propagating dL wav2vec /dẑ (van den). The final loss has two additional terms: where sg(x) ≡ x, d dx sg(x) ≡ 0 is the stop gradient operator and γ is a hyper-parameter. The first term is the future prediction task and gradients do not change the codebook because of the straightthrough gradient estimation of mapping z toẑ. The second term sg(z) −ẑ 2 moves the codebook vectors closer to the encoder output, and the third term z − sg(ẑ) 2 makes sure that the encoder outputs are close to a centroid (codeword). So far, we considered replacing the encoder feature vector z by a single entry e i in the codebook. This is prone to mode collapse where only some of the codewords are actually used. Previously, this problem has been mitigated by workarounds such as re-initializing codewords or applying additional regularizers to the loss function . In the following, we describe another strategy where we independently quantize partitions of z, similar to product quantization . This in larger dictionaries and increased downstream performance (Appendix A). The dense feature vector z ∈ R d is first organized into multiple groups G into the matrix form z ∈ R G×(d/G). We then represent each row by an integer index, and hence can represent the full feature vector by the indices i ∈ [V] G, where V again denotes the possible number of variables for this particular group and each element i j corresponds to a fixed codebook vector. For each of the G groups, we apply either one of the two VQ approaches (§3.1 and §3.2). The codebook itself can be initialized in two possible ways: Codebook variables can be shared across groups, i.e., a particular index in group j would reference the same vector as the same index in group j. This yields a codebook e ∈ R V ×(G/d). In contrast, not sharing the codebook variables yields a codebook of size e ∈ R V ×G×(G/d). In practise, we observe that sharing the codebook variables generally yields competitive to a non-shared representation. Once we trained a vq-wav2vec model we can discretize audio data and make it applicable to algorithms that require discrete inputs. One possibility is to use the discretized training data and apply BERT pre-training where the task is to predict masked input tokens based on an encoding of the surrounding context . Once the BERT model is trained, we can use it to build representations and feed them into an acoustic model to improve speech recognition. We follow recent advances in BERT training which only use the masked input token prediction. Since each of the discretized tokens represents around 10 ms of audio it is likely too easy to predict a single masked input token. We therefore change BERT training by masking spans of consecutive discretized speech tokens, similar to. To mask the input sequence, we randomly sample p = 0.05 of all tokens to be a starting index, without replacement, and mask M = 10 consecutive tokens from every sampled index; spans may overlap. This makes the masked token prediction harder and we show later that it improves accuracy over masking individual tokens (§6.5). We generally pre-train vq-wav2vec and BERT on the full 960h of Librispeech and after vq-wav2vec training it is discretized to 345m tokens. Where indicated we perform ablations on a clean 100h subset which is discretized to 36M tokens. We evaluate models on two benchmarks: TIMIT (b) is a 5h dataset with phoneme labels and Wall Street Journal (WSJ; Garofolo et al. 1993a) is a 81h dataset for speech recognition. For TIMIT, we apply the standard evaluation protocol and consider 39 different phonemes. For WSJ, we train acoustic models directly on 31 graphemes, including the English alphabet, the apostrophe, the silence token and tokens for repeating characters. We adapt the fairseq implmentation of wav2vec (; and use vqwav2vec/wav2vec models with 34 × 10 6 parameters. The encoder has 8 layers with 512 channels each, kernel sizes and strides, yielding a total stride of 160. Each layer contains a convolution, followed by dropout, group normalization with a single group and a ReLU non-linearity. The aggregator is composed of 12 layers, with 512 channels, stride 1, and kernel sizes starting at 2 and increasing by 1 for every subsequent layer. The block structure is the same as for the encoder network, except we introduce skip connections between each subsequent block. We train with the wav2vec context prediction loss (Equation 1) for 400k updates, predicting K = 8 steps into the future and sample 10 negatives from the same audio example. Training is warmed up for 500 steps where the learning rate is increased from 1 × 10 −7 to 5 × 10 −3, and then annealed to 1e-06 using a cosine schedule . The batch size is 10, and we crop a random section of 150,000 frames for each example (approximately 9.3 seconds for 16kHz sampling rate). All models are trained on 8 GPUs. For ablations and experiments on the 100h Librispeech subset, we use a smaller model with kernels and strides in the encoder and seven convolutional layers with stride one and kernel size three in the aggregator. This model is trained for 40k updates. Gumbel-Softmax Models. We use G = 2 groups and V = 320 latents per group and the linear layer projects the features produced by the encoder into G · V = 640 logits. The Gumbel-Softmax produces a one-hot vector for each group G. The temperature τ is linearly annealed from 2 to 0.5 over the first 70% of updates and then kept constant at 0.5. This enables the model to learn which latents work best for each input before committing to a single latent. After training this model on 960h of Librispeech and quantizing the training dataset, we are left with 13.5k unique codewords combinations (out of V G = 102k possible codewords). k-means Models. We use G = 2 groups and V = 320 variables per group. vq-wav2vec on full Librispeech yields 23k unique codewords. Following van den , we found γ = 0.25 to be a robust choice for balancing the VQ auxiliary loss. BERT base models have 12 layers, model dimension 768, inner dimension (FFN) 3072 and 12 attention heads . The learning rate is warmed up over the first 10,000 updates to a peak value of 1 × 10 −5, and then linearly decayed over a total of 250k updates. We train on 128 GPUs with a batch size of 3072 tokens per GPU giving a total batch size of 393k tokens . Each token represents 10ms of audio data. BERT small. For ablations we use a smaller setup with model dimension 512, FFN size 2048, 8 attention heads and dropout 0.05. Models are trained for 250k updates with a batch size of 2 examples per GPU. We use wav2letter as accoustic model (; and train for 1k epochs on 8 GPUs for both TIMIT and WSJ using the auto segmentation criterion. For decoding the emissions from the acoustic model on WSJ we use a lexicon as well as a separate language model trained on the WSJ language modeling data only. We consider a 4-gram KenLM language model and a character based convolutional language model and tune the models with the same protocol as. We first evaluate on the WSJ speech recognition benchmark. We train a vq-wav2vec model on the unlabeled version of Librispeech, then discretize the same data with the ing model to estimate a BERT model. Finally, we train a wav2letter acoustic model on WSJ by inputting either the BERT or vq-wav2vec representations instead of log-mel filterbanks. 2 We compare to various from the literature, including wav2vec and we consider three setups: performance without any language model (No LM), with an n-gram LM (4-gram LM) and with a character convolutional LM (Char ConvLM). We report the accuracy of wav2letter with log-mel filterbanks as input (Baseline) and wav2vec. For vq-wav2vec we first experiment with the Gumbel-Softmax, with and without a BERT base model (§5.3). Table 1 shows that vq-wav2vec together with BERT training can achieve a new state of the art of 2.34 WER on nov92. Gains are largest when no language model is used which is the fastest setting. vq-wav2vec with Gumbel-Softmax uses only 13.5k distinct codewords to represent the audio signal and this limited set of codewords is not sufficient to outperform the baseline. However, it does enable training BERT models which require a relatively small vocabulary. Next, we compare Gumbel-Softmax to k-means for vector quantization. For this experiment we use the faster to train BERT small configuration (§5.3). We also train a vq-wav2vec k-means model with a very large number of codewords (39m) to test whether a more expressive model can close dev PER test PER CNN + TD-filterbanks 15.6 18.0 Li-GRU + fMLLR -14.9 wav2vec 12.9 14.7 Baseline (log-mel) 16.9 17.6 vq-wav2vec, gumbel 15.34 17.78 + BERT small 9.64 11.64 vq-wav2vec, k-means 15.65 18.73 + BERT small 9.80 11.40 Table 3: TIMIT phoneme recognition in terms of phoneme error rate (PER). All our models use the CNN-8L-PReLU-do0.7 architecture (Table 4 : Librispeech for a standard sequence to sequence model trained on discretized audio without BERT pre-training and from the literature. All are without a language model. the gap to wav2vec. Table 2 shows that Gumbel-Softmax and k-means clustering perform relatively comparably: in the no language model setup without BERT, Gumbel-Softmax is more accurate than k-means but these differences disappear with BERT. For 4-gram LM setup, k-means is better but those differences disappear again after BERT training. Finally, the large codeword model can substantially reduce the gap to the original wav2vec model. Next, we experiment on the much smaller TIMIT phoneme recognition task where we also pre-train vq-wav2vec on the full Librispeech corpus. Table 3 shows that vq-wav2vec and BERT achieve a new state of the art of 11.67 PER which corresponds to a 21% reduction in error over the previous best of wav2vec. So far we used vq-wav2vec to train BERT on discretized speech. However, once the audio is discretized we can also train a standard sequence to sequence model to perform speech recognition. In preliminary experiments, we trained an off-the-shelf Big Transformer (; on the vq-wav2vec Gumbel-Softmax discretized Librispeech corpus and evaluated on the Librispeech dev/test sets; we use a 4k BPE output vocabulary . Table 4 shows that are promising, even though they are not as good as the state of the art which depends on data augmentation that we do not use. Next, we investigate how well vq-wav2vec can compress the audio data. Specifically, we train models with different numbers of groups G and variables V to vary the size of the possible codebook size V G and measure accuracy on TIMIT phoneme recognition without BERT training. We measure compression with the bitrate r · G log 2 V at sampling rate r = 100Hz and report the tradeoff between bitrate and accuracy on our phoneme recognition task. We experiment with vq-wav2vec k-means and train models with 1,2,4,8,16 and 32 groups, using 40,80,160,...,1280 variables, spanning a bitrate range from 0.53 kbit/s (G = 1, V = 40) to 33.03 kbit/s (G = 32, V = 1280). We place the quantization module after the aggregator module and train all models in the small vq-wav2vec setup (§5.2) on the 100h clean Librispeech subset. As baselines, we consider various lossy compression algorithms applied to the TIMIT audio data and train wav2vec models on the ing audio: Codec2 3 as a low bitrate codec, Opus as a medium bitrate codec and MP3 and Ogg Vorbis as high bitrate codecs. We use the whole spectrum of both variable and constant bitrate settings of the codecs; we encode and decode with ffmpeg (ffmpeg developers, 2016). Figure 3 shows the trade-off between the bitrate and TIMIT accuracy. Acoustic models on vq-wav2vec achieve the best across most bitrate settings. Table 5a shows that masking entire spans of tokens performs significantly better than individual tokens (M = 1). Furthermore, BERT training on discretized audio data is fairly robust to masking large parts of the input (Table 5b). vq-wav2vec is a self-supervised algorithm that quantizes unlabeled audio data which makes it amenable to algorithms requiring discrete data. This approach improves the state of the art on the WSJ and TIMIT benchmarks by leveraging BERT pre-training. In future work, we plan to apply other algorithms requiring discrete inputs to audio data and to explore self-supervised pre-training algorithms which mask part of the continuous audio input. Another future work avenue is to finetune the pre-trained model to output transcriptions instead of feeding the pre-trained features to a custom ASR model. We investigate the relationship between number of variables V and groups G. Table 6 shows that multiple groups are beneficial compared to a single group with a large number of variables. Table 7 shows that with a single group and many variables, only a small number of codewords survive. Table 6: PER on TIMIT dev set for vq-wav2vec models trained on Libri100. Results are based on three random seeds.

Represent the short summary for finding the corresponding scientific article (long).

Learn how to quantize speech signal and apply algorithms requiring discrete inputs to audio data such as BERT.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.