Question: Represent the scientific article to find a short TLDR

Question: Intelligent creatures can explore their environments and learn useful skills without supervision. In this paper, we propose ``Diversity is All You Need''(DIAYN), a method for learning useful skills without a reward function. Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy. On a variety of simulated robotic tasks, we show that this simple objective in the unsupervised emergence of diverse skills, such as walking and jumping. In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward. We show how pretrained skills can provide a good parameter initialization for downstream tasks, and can be composed hierarchically to solve complex, sparse reward tasks. Our suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efficiency in reinforcement learning. Deep reinforcement learning (RL) has been demonstrated to effectively learn a wide range of rewarddriven skills, including playing games , controlling robots (; b), and navigating complex environments . However, intelligent creatures can explore their environments and learn useful skills even without supervision, so that when they are later faced with specific goals, they can use those skills to satisfy the new goals quickly and efficiently. Learning skills without reward has several practical applications. Environments with sparse rewards effectively have no reward until the agent randomly reaches a goal state. Learning useful skills without supervision may help address challenges in exploration in these environments. For long horizon tasks, skills discovered without reward can serve as primitives for hierarchical RL, effectively shortening the episode length. In many practical settings, interacting with the environment is essentially free, but evaluating the reward requires human feedback BID7. Unsupervised learning of skills may reduce the amount of supervision necessary to learn a task. While we can take the human out of the loop by designing a reward function, it is challenging to design a reward function that elicits the desired behaviors from the agent . Finally, when given an unfamiliar environment, it is challenging to determine what tasks an agent should be able to learn. Unsupervised skill discovery partially answers this question. 1 Autonomous acquisition of useful skills without any reward signal is an exceedingly challenging problem. A skill is a latent-conditioned policy that alters the state of the environment in a consistent way. We consider the setting where the reward function is unknown, so we want to learn a set of skills by maximizing the utility of this set. Making progress on this problem requires specifying a learning objective that ensures that each skill individually is distinct and that the skills collectively explore large parts of the state space. In this paper, we show how a simple objective based on mutual information can enable RL agents to autonomously discover such skills. These skills are useful for a number of applications, including hierarchical reinforcement learning and imitation learning. We propose a method for learning diverse skills with deep RL in the absence of any rewards. We hypothesize that in order to acquire skills that are useful, we must train the skills so that they maximize coverage over the set of possible behaviors. While one skill might perform a useless behavior like random dithering, other skills should perform behaviors that are distinguishable from random dithering, and therefore more useful. A key idea in our work is to use discriminability between skills as an objective. Further, skills that are distinguishable are not necessarily maximally diverse -a slight difference in states makes two skills distinguishable, but not necessarily diverse in a semantically meaningful way. To combat this problem, we want to learn skills that not only are distinguishable, but also are as diverse as possible. By learning distinguishable skills that are as random as possible, we can "push" the skills away from each other, making each skill robust to perturbations and effectively exploring the environment. By maximizing this objective, we can learn skills that run forward, do backflips, skip backwards, and perform face flops (see Figure 3). Our paper makes five contributions. First, we propose a method for learning useful skills without any rewards. We formalize our discriminability goal as maximizing an information theoretic objective with a maximum entropy policy. Second, we show that this simple exploration objective in the unsupervised emergence of diverse skills, such as running and jumping, on several simulated robotic tasks. In a number of RL benchmark environments, our method is able to solve the benchmark task despite never receiving the true task reward. In these environments, some of the learned skills correspond to solving the task, and each skill that solves the task does so in a distinct manner. Third, we propose a simple method for using learned skills for hierarchical RL and find this methods solves challenging tasks. Four, we demonstrate how skills discovered can be quickly adapted to solve a new task. Finally, we show how skills discovered can be used for imitation learning. Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., BID2 ; ; ; ;) ). One problem with joint training (also noted by) is that the meta-policy does not select "bad" options, so these options do not receive any reward signal to improve. Our work prevents this degeneracy by using a random meta-policy during unsupervised skill-learning, such that neither the skills nor the meta-policy are aiming to solve any single task. A second importance difference is that our approach learns skills with no reward. Eschewing a reward function not only avoids the difficult problem of reward design, but also allows our method to learn task-agnostic. Related work has also examined connections between RL and information theory (; ; ;) and developed maximum entropy algorithms with these ideas Haarnoja et al. (2018; . Recent work has also applied tools from information theory to skill discovery. and use the mutual information between states and actions as a notion of empowerment for an intrinsically motivated agent. Our method maximizes the mutual information between states and skills, which can be interpreted as maximizing the empowerment of a hierarchical agent whose action space is the set of skills. Hausman et al., Florensa et al., and Gregor et al. (2016 showed that a discriminability objective is equivalent to maximizing the mutual information between the latent skill z and some aspect of the corresponding trajectory. considered the setting with many tasks and reward functions and considered the setting with a single task reward. Three important distinctions allow us to apply our method to tasks significantly more complex than the gridworlds in . First, we use maximum entropy policies to force our skills to be diverse. Our theoretical analysis shows that including entropy maximization in the RL objective in the mixture of skills being maximum entropy in aggregate. Second, we fix the prior distribution over skills, rather than learning it. Doing so prevents our method from collapsing to sampling only a handful of skills. Third, while the discriminator in while not converged do Sample skill z ∼ p(z) and initial state s0 ∼ p0(s) for t ← 1 to steps_per_episode do Sample action at ∼ π θ (at | st, z) from skill. Step environment: st+1 ∼ p(st+1 | st, at). Compute q φ (z | st+1) with discriminator. Set skill reward rt = log q φ (z | st+1) − log p(z) Update policy (θ) to maximize rt with SAC. Update discriminator (φ) with SGD.Figure 1: DIAYN Algorithm: We update the discriminator to better predict the skill, and update the skill to visit diverse states that make it more discriminable.the final state, our discriminator looks at every state, which provides additional reward signal. These three crucial differences help explain how our method learns useful skills in complex environments. Prior work in neuroevolution and evolutionary algorithms has studied how complex behaviors can be learned by directly maximizing diversity (a; b; ; ; ;). While this prior work uses diversity maximization to obtain better solutions, we aim to acquire complex skills with minimal supervision to improve efficiency (i.e., reduce the number of objective function queries) and as a stepping stone for imitation learning and hierarchical RL. We focus on deriving a general, information-theoretic objective that does not require manual design of distance metrics and can be applied to any RL task without additional engineering. Previous work has studied intrinsic motivation in humans and learned agents. BID3. While these previous works use an intrinsic motivation objective to learn a single policy, we propose an objective for learning many, diverse policies. Concurrent work BID0 draws ties between learning discriminable skills and variational autoencoders. We show that our method scales to more complex tasks, likely because of algorithmic design choices, such as our use of an off-policy RL algorithm and conditioning the discriminator on individual states. We consider an unsupervised RL paradigm in this work, where the agent is allowed an unsupervised "exploration" stage followed by a supervised stage. In our work, the aim of the unsupervised stage is to learn skills that eventually will make it easier to maximize the task reward in the supervised stage. Conveniently, because skills are learned without a priori knowledge of the task, the learned skills can be used for many different tasks. Our method for unsupervised skill discovery, DIAYN ("Diversity is All You Need"), builds off of three ideas. First, for skills to be useful, we want the skill to dictate the states that the agent visits. Different skills should visit different states, and hence be distinguishable. Second, we want to use states, not actions, to distinguish skills, because actions that do not affect the environment are not visible to an outside observer. For example, an outside observer cannot tell how much force a robotic arm applies when grasping a cup if the cup does not move. Finally, we encourage exploration and incentivize the skills to be as diverse as possible by learning skills that act as randomly as possible. Skills with high entropy that remain discriminable must explore a part of the state space far away from other skills, lest the randomness in its actions lead it to states where it cannot be distinguished. We construct our objective using notation from information theory: S and A are random variables for states and actions, respectively; Z ∼ p(z) is a latent variable, on which we condition our policy; we refer to a the policy conditioned on a fixed Z as a "skill"; I(·; ·) and H[·] refer to mutual information and Shannon entropy, both computed with base e. In our objective, we maximize the mutual information between skills and states, I(S; Z), to encode the idea that the skill should control which states the agent visits. Conveniently, this mutual information dictates that we can infer the skill from the states visited. To ensure that states, not actions, are used to distinguish skills, we minimize the mutual information between skills and actions given the state, I(A; Z | S). Viewing all skills together with p(z) as a mixture of policies, we maximize the entropy H[A | S] of this mixture policy. In summary, we maximize the following objective with respect to our policy parameters, θ: DISPLAYFORM0 We rearranged our objective in Equation 2 to give intuition on how we optimize it. 2 The first term encourages our prior distribution over p(z) to have high entropy. We fix p(z) to be uniform in our approach, guaranteeing that it has maximum entropy. The second term suggests that it should be easy to infer the skill z from the current state. The third term suggests that each skill should act as randomly as possible, which we achieve by using a maximum entropy policy to represent each skill. As we cannot integrate over all states and skills to compute p(z | s) exactly, we approximate this posterior with a learned discriminator q φ (z | s). Jensen's Inequality tells us that replacing p(z | s) with q φ (z | s) gives us a variational lower bound G(θ, φ) on our objective F(θ) (see BID1 for a detailed derivation): DISPLAYFORM1 We implement DIAYN with soft actor critic (SAC) , learning a policy π θ (a | s, z) that is conditioned on the latent variable z. Soft actor critic maximizes the policy's entropy over actions, which takes care of the entropy term in our objective G. Following Haarnoja et al. FORMULA0, we scale the entropy regularizer H[a | s, z] by α. We found empirically that an α = 0.1 provided a good trade-off between exploration and discriminability. We maximize the expectation in G by replacing the task reward with the following pseudo-reward: DISPLAYFORM0 We use a categorical distribution for p(z). During unsupervised learning, we sample a skill z ∼ p(z) at the start of each episode, and act according to that skill throughout the episode. The agent is rewarded for visiting states that are easy to discriminate, while the discriminator is updated to better infer the skill z from states visited. Entropy regularization occurs as part of the SAC update. Unlike prior adversarial unsupervised RL methods (e.g., Sukhbaatar et al. FORMULA0), DIAYN forms a cooperative game, which avoids many of the instabilities of adversarial saddle-point formulations. On gridworlds, we can compute analytically that the unique optimum to the DIAYN optimization problem is to evenly partition the states between skills, with each skill assuming a uniform stationary distribution over its partition (proof in Appendix B). In the continuous and approximate setting, convergence guarantees would be desirable, but this is a very tall order: even standard RL methods with function approximation (e.g., DQN) lack convergence guarantees, yet such techniques are still useful. Empirically, we find DIAYN to be robust to random seed; varying the random seed does not noticeably affect the skills learned, and has little effect on downstream tasks (see Fig.s 4, 6, and 13). In this section, we evaluate DIAYN and compare to prior work. First, we analyze the skills themselves, providing intuition for the types of skills learned, the training dynamics, and how we avoid problematic behavior in previous work. In the second half, we show how the skills can be used for downstream tasks, via policy initialization, hierarchy, imitation, outperforming competitive baselines on most tasks. We encourage readers to view videos 3 and code 4 for our experiments. (We study the skills learned by DIAYN on tasks of increasing complexity, ranging from point navigation (2 dimensions) to ant locomotion (111 dimensions). We first applied DIAYN to a simple 2D navigation environment. The agent starts in the center of the box, and can take actions to directly move its (x, y) position. FIG1 illustrates how the 6 skills learned for this task move away from each other to remain distinguishable. Next, we applied DIAYN to two classic control tasks, inverted pendulum and mountain car. Not only does our approach learn skills that solve the task without rewards, it learns multiple distinct skills for solving the task. (See Appendix D for further analysis.)Finally, we applied DIAYN to three continuous control tasks BID6: half cheetah, hopper, and ant. As shown in Figure 3, we learn a diverse set of primitive behaviors for all tasks. For half cheetah, we learn skills for running forwards and backwards at various speeds, as well as skills for doing flips and falling over; ant learns skills for jumping and walking in many types of curved trajectories (though none walk in a straight line); hopper learns skills for balancing, hopping forward and backwards, and diving. See Appendix D.4 for a comparison with VIME. While DIAYN learns skills without a reward function, as an outside observer, can we evaluate the skills throughout training to understand the training dynamics. FIG1 shows how the skills for inverted pendulum and mountain car become increasingly diverse throughout training (FIG12 repeats this experiment for 5 random seeds, and shows that are robust to initialization). Recall that our skills are learned with no reward, so it is natural that some skills correspond to small task reward while others correspond to large task reward. Question 3. Does discriminating on single states restrict DIAYN to learn skills that visit disjoint sets of states?Our discriminator operates at the level of states, not trajectories. While DIAYN favors skills that do not overlap, our method is not limited to learning skills that visit entirely disjoint sets of states. FIG1 shows a simple experiment illustrating this. The agent starts in a hallway (green star), and can move more freely once exiting the end of the hallway into a large room. Because RL agents are incentivized to maximize their cumulative reward, they may take actions that initially give no reward to reach states that eventually give high reward. In this environment, DIAYN learns skills that exit the hallway to make them mutually distinguishable. The key difference from the most similar prior work on unsupervised skill discovery, VIC, is our decision to not learn the prior p(z). We found that VIC suffers from the "Matthew Effect": VIC's learned prior p(z) will sample the more diverse skills more frequently, and hence only those skills will receive training signal to improve. To study this, we evaluated DIAYN and VIC on the half-cheetah environment, and plotting the effective number of skills (measured as exp(H[Z])) throughout training (details and more figures in Appendix E.2). The figure to the right shows how VIC quickly converges to a setting where it only samples a handful of skills. In contrast, DIAYN fixes the distribution over skills, which allows us to discover more diverse skills. The perhaps surprising finding that we can discover diverse skills without a reward function creates a building block for many problems in RL. For example, to find a policy that achieves a high reward on a task, it is often sufficient to simply choose the skill with largest reward. Three less obvious applications are adapting skills to maximize a reward, hierarchical RL, and imitation learning. After DIAYN learns task-agnostic skills without supervision, we can quickly adapt the skills to solve a desired task. Akin to the use of pre-trained models in computer vision, we propose that DIAYN can serve as unsupervised pre-training for more sample-efficient finetuning of task-specific policies. Question 5. Can we use learned skills to directly maximize the task reward?We take the skill with highest reward for each benchmark task and further finetune this skill using the task-specific reward function. We compare to a "random initialization" baseline that is initialized from scratch. Our approach differs from this baseline only in how weights are initialized. We initialize both the policy and value networks with weights learned during unsupervised pretraining. Although the critic networks learned during pretraining corresponds to the pseudo-reward from the discriminator (Eq. 3) and not the true task reward, we found empirically that the pseudo-reward was close to the true task reward for the best skill, and initializing the critic in addition to the actor further sped up learning. FIG3 shows both methods applied to half cheetah, hopper, and ant. We assume that the unsupervised pretraining is free (e.g., only the reward function is expensive to compute) or can be amortized across many tasks, so we omit pretraining steps from this plot. On all tasks, unsupervised pretraining enables the agent to learn the benchmark task more quickly. In theory, hierarchical RL should decompose a complex task into motion primitives, which may be reused for multiple tasks. In practice, algorithms for hierarchical RL can encounter many problems: each motion primitive reduces to a single action BID2, the hierarchical policy only samples a single motion primitive , or all motion primitives attempt to do the entire task. In contrast, DIAYN discovers diverse, task-agnostic skills, which hold the promise of acting as a building block for hierarchical RL.Question 6. Are skills discovered by DIAYN useful for hierarchical RL?We propose a simple extension to DIAYN for hierarchical RL, and find that simple algorithm outperforms competitive baselines on two challenging tasks. To use the discovered skills for hierarchical RL, we learn a meta-controller whose actions are to choose which skill to execute for the next k steps (100 for ant navigation, 10 for cheetah hurdle). The meta-controller has the same observation space as the skills. As an initial test, we applied the hierarchical RL algorithm to a simple 2D point navigation task (details in Appendix C.2). FIG4 illustrates how the reward on this task increases with the number of skills; error bars show the standard deviation across 5 random seeds. To ensure that our goals were not cherry picked, we sampled 25 goals evenly from the state space, and evaluated each random seed on all goals. We also compared to Variational Information Maximizing Exploration (VIME) . Note that even the best random seed from VIME significantly under-performs DIAYN. This is not surprising: whereas DIAYN learns a set of skills that effectively partition the state space, VIME attempts to learn a single policy that visits many states. Next, we applied the hierarchical algorithm to two challenging simulated robotics environment. On the cheetah hurdle task, the agent is rewarded for bounding up and over hurdles, while in the ant navigation task, the agent must walk to a set of 5 waypoints in a specific order, receiving only a sparse reward upon reaching each waypoint. The sparse reward and obstacles in these environments make them exceedingly difficult for non-hierarchical RL algorithms. Indeed, state of the art RL algorithms that do not use hierarchies perform poorly on these tasks. FIG6 shows how DIAYN outperforms state of the art on-policy RL (TRPO (a)), off-policy RL (SAC ), and exploration bonuses (VIME). This experiment suggests that unsupervised skill learning provides an effective mechanism for combating challenges of exploration and sparse rewards in RL. If the number of possible skills grows exponentially with the dimension of the task observation, one might imagine that DIAYN would fail to learn skills necessary to solve some tasks. While we found that DIAYN does scale to tasks with more than 100 dimensions (ant has 111), we can also use a simple modification to bias DIAYN towards discovering particular types of skills. We can condition the discriminator on only a subset of the observation space, or any other function of the observations. In this case, the discriminator maximizes E[log q φ (z | f (s))]. For example, in the ant navigation task, f (s) could compute the agent's center of mass, and DIAYN would learn skills that correspond to changing the center of mass. The "DIAYN+prior" in FIG6 (right) shows how incorporating this prior knowledge can aid DIAYN in discovering useful skills and boost performance on the hierarchical task. (No other experiments or figures in this paper used this prior.) The key takeaway is that while DIAYN is primarily an unsupervised RL algorithm, there is a simple mechanism for incorporating supervision when it is available. Unsurprisingly, we perform better on hierarchical tasks when incorporating more supervision. Expert trajectories DIAYN imitations Figure 9: Imitating an expert: DIAYN imitates an expert standing upright, flipping, and faceplanting, but fails to imitate a handstand. Question 8. Can we use learned skills to imitate an expert?Aside from maximizing reward with finetuning and hierarchical RL, we can also use learned skills to follow expert demonstrations. One use-case is where a human manually controls the agent to complete a task that we would like to automate. Simply replaying the human's actions fails in stochastic environments, cases where closed-loop control is necessary. A second use-case involves an existing agent with a hard coded, manually designed policy. Imitation learning replaces the existing policy with a similar yet differentiable policy, which might be easier to update in response to new constraints or objectives. We consider the setting where we are given an expert trajectory consisting of states, without actions, defined as τ * = {(s i)} 1≤i≤N. Our goal is to obtain a feedback controller that will reach the same states. Given the expert trajectory, we use our learned discriminator to estimate which skill was most likely to have generated the trajectory. This optimization problem, which we solve for categorical z by enumeration, is equivalent to an M-projection : DISPLAYFORM0 We qualitatively evaluate this approach to imitation learning on half cheetah. Figure 9 (left) shows four imitation tasks, three of which our method successfully imitates. We quantitatively evaluate this imitation method on classic control tasks in Appendix G. In this paper, we present DIAYN, a method for learning skills without reward functions. We show that DIAYN learns diverse skills for complex tasks, often solving benchmark tasks with one of the learned skills without actually receiving any task reward. We further proposed methods for using the learned skills to quickly adapt to a new task, to solve complex tasks via hierarchical RL, and to imitate an expert. As a rule of thumb, DIAYN may make learning a task easier by replacing the task's complex action space with a set of useful skills. DIAYN could be combined with methods for augmenting the observation space and reward function. Using the common language of information theory, a joint objective can likely be derived. DIAYN may also more efficiently learn from human preferences by having humans select among learned skills. Finally, the skills produced by DIAYN might be used by game designers to allow players to control complex robots and by artists to animate characters. The log p(z) term in Equation 3 is a baseline that does not depend on the policy parameters θ, so one might be tempted to remove it from the objective. We provide a two justifications for keeping it. First, assume that episodes never terminate, but all skills eventually converge to some absorbing state (e.g., with all sensors broken). At this state, the discriminator cannot distinguish the skills, so its estimate is log q(z | s) = log(1/N), where N is the number of skills. For practical reasons, we want to restart the episode after the agent reaches the absorbing state. Subtracting log(z) from the pseudo-reward at every time step in our finite length episodes is equivalent to pretending that episodes never terminate and the agent gets reward log(z) after our "artificial" termination. Second, assuming our discriminator q φ is better than chance, we see that q φ (z | s) ≥ p(z). Thus, subtracting the log p(z) baseline ensures our reward function is always non-negative, encouraging the agent to stay alive. Without this baseline, an optimal agent would end the episode as soon as possible. For simple environments, we can compute an analytic solution to the DIAYN objective. For example, consider a N × N gridworld, where actions are to move up/down/left/right. Any action can be taken in any state, but the agent will stay in place if it attempts to move out of the gridworld. We use (x, y) to refer to states, where x, y ∈ {1, 2, · · ·, N}.For simplicity, we assume that, for every skill, the distribution of states visited exactly equals that skill's stationary distribution over states. To clarify, we will use π z to refer to the policy for skill z. We use ρ πz to indicate skill z's stationary distribution over states, andρ πz as the empirical distribution over states within a single episode. Our assumption is equivalent to saying ρ πz (s) =ρ πz (s) ∀s ∈ S One way to ensure this is to assume infinite-length episodes. We want to show that a set of skills that evenly partitions the state space is the optimum of the DIAYN objective for this task. While we will show this only for the 2-skill case, the 4 skill case is analogous. The optimum policies for a set of two skills are those which evenly partition the state space. We will show that a top/bottom partition is one such (global) optima. The left/right case is analogous. Lemma B.1. A pair of skills with state distributions given below (and shown in FIG8) are an optimum for the DIAYN objective with no entropy regularization (α = 0). DISPLAYFORM0 Before proving Lemma B.1, we note that there exist policies that achieve these stationary distributions. FIG8 shows one such policy, were each arrow indicates a transition with probability 1 4. Note that when the agent is in the bottom row of yellow states, it does not transition to the green states, and instead stays in place with probability Proof. Recall that the DIAYN objective with no entropy regularization is: DISPLAYFORM1 Because the skills partition the states, we can always infer the skill from the state, so H[Z | S] = 0. By construction, the prior distribution over H[Z] is uniform, so H[Z] = log is maximized. Thus, a set of two skills that partition the state space maximizes the un-regularized DIAYN objective. Next, we consider the regularized objective. In this case, we will show that while an even partition is not perfectly optimal, it is "close" to optimal, and its "distance" from optimal goes to zero as the gridworld grows in size. This analysis will give us additional insight into the skills preferred by the DIAYN objective. Lemma B.2. A pair of skills with state distributions given given in Equation 4 achieve an DIAYN objective within a factor of O(1/N) of the optimum, where N is the gridworld size. Proof. Recall that the DIAYN objective with no entropy regularization is: DISPLAYFORM2 We have already computed the second two terms in the previous proof: H[Z | S] = 0 and H[Z] = log. For computing the first term, it is helpful to define the set of "border states" for a particular skill as those that do not neighbor another skill. For skill 1 defined in FIG8 Figure 11: The DIAYN objective prefers skills that (Left) partition states into sets with short borders and (Right) which correspond to bottleneck states. at states along the border between two skills. Everything else being equal, this conflict encourages DIAYN to produce skills that have small borders, as shown in Figure 11. For example, in a gridworld with dimensions N < M, a pair of skills that split along the first dimension (producing partitions of size (N, M/2)) would achieve a larger (better) objective than skills that split along the second dimension. This same intuition that DIAYN seeks to minimize the border length between skills in DIAYN preferring partitions that correspond to bottleneck states (see Figure 11b). In our experiments, we use the same hyperparameters as those in , with one notable exception. For the Q function, value function, and policy, we use neural networks with 300 hidden units instead of 128 units. We found that increasing the model capacity was necessary to learn many diverse skills. When comparing the "skill initialization" to the "random initialization" in Section 4.2, we use the same model architecture for both methods. To pass skill z to the Q function, value function, and policy, we simply concatenate z to the current state s t. As in , epochs are 1000 episodes long. For all environments, episodes are at most 1000 steps long, but may be shorter. For example, the standard benchmark hopper environment terminates the episode once it falls over. FIG1 show up to 1000 epochs, which corresponds to at most 1 million steps. We found that learning was most stable when we scaled the maximum entropy objective (H[A | S, Z] in Eq. 1) by α = 0.1. We use this scaling for all experiments. The cheetah hurdle environment is a modification of HalfCheetah-v1, where we added boxes with shape H = 0.25m, W = 0.1m, D = 1.0m, where the width dimension is along the same axis as the cheetah's forward movement. We placed the boxes ever 3 meters, start at x = −1m. The ant navigation environment is a modification of Ant-v1. To improve stability, we follow Pong et al. FORMULA0 and lower the gear ratio of all joints to 30. The goals are the corners of a square, centered at the origin, with side length of 4 meters: [, (2, −2), (−2, −2), (−2, 2),]. The ant starts at the origin, and receives a reward of +1 when its center of mass is within 0.5 meters of the correct next goal. Each reward can only be received once, so the maximum possible reward is +5. For the 2D navigation experiment shown in FIG4, we first learned a set of skills on the point environment. Next, we introduced a reward function r g (s) = − s − g 2 2 penalizing the distance from the agent's state to some goal, and applied the hierarchical algorithm above. In this task, the DIAYN skills provided sufficient coverage of the state space that the hierarchical policy only needed to take a single action (i.e., choose a single skill) to complete the task. FIG1: Objectives: We plot the two terms from our objective (Eq. 1) throughout training. While the entropy regularizer (blue) quickly plateaus, the discriminability term (orange) term continues to increase, indicating that our skills become increasingly diverse without collapsing to deterministic policies. This plot shows the mean and standard deviation across 5 seeds for learning 20 skills in half cheetah environment. Note that log 2 (1/20) ≈ −3, setting a lower bound for log q φ (z | s). To provide further intuition into our approach, FIG1 plots the two terms in our objective throughout training. Our skills become increasingly diverse throughout training without converging to deterministic policies. To illustrate the stability of DIAYN to random seed, we repeated the experiment in FIG1 for 5 random seeds. FIG12 illustrates that the random seed has little effect on the training dynamics. Question 9. Does entropy regularization lead to more diverse skills? DISPLAYFORM0 To answer this question, we apply our method to a 2D point mass. The agent controls the orientation and forward velocity of the point, with is confined within a 2D box. We vary the entropy regularization α, with larger values of α corresponding to policies with more stochastic actions. With small α, we learn skills that move large distances in different directions but fail to explore large parts of the state space. Increasing α makes the skills visit a more diverse set of states, which may help with exploration in complex state spaces. It is difficult to discriminate skills when α is further increased. DISPLAYFORM1 Figure 15: Task reward of skills learned without reward: While our skills are learned without the task reward function, we evaluate each with the task reward function for analysis. The wide range of rewards shows the diversity of the learned skills. In the hopper and half cheetah tasks, many skills achieve large task reward, despite not observing the task reward during training. As discussed in prior work , standard model-free algorithms trained directly on the task reward converge to scores of 1000 -3000 on hopper, 1000 -5000 on cheetah, and 700 -2000 on ant. In FIG3, we take the skills learned without any rewards, and evaluate each of them on the standard benchmark reward function. We compare to random (untrained) skills. The wide distribution over rewards is evidence that the skills learned are diverse. For hopper, some skills hop or stand for the entire episode, receiving a reward of at least 1000. Other skills aggressively hop forwards or dive backwards, and receive rewards between 100 and 1000. Other skills fall over immediately and receive rewards of less than 100. The benchmark half cheetah reward includes a control penalty for taking actions. Unlike random skills, learned skills rarely have task reward near zero, indicating that all take actions to become distinguishable. Skills that run in place, flop on their nose, or do backflips receive reward of -100. Skills that receive substantially smaller reward correspond to running quickly backwards, while skills that receive substantially larger reward correspond to running forward. Similarly, the benchmark ant task reward includes both a control penalty and a survival bonus, so random skills that do nothing receive a task reward near 1000. While no single learned skill learns to run directly forward and obtain a task reward greater than 1000, our learned skills run in different patterns to become discriminable, ing in a lower task reward. Question 10. Does DIAYN explore effectively in complex environments?We apply DIAYN to three standard RL benchmark environments: half-cheetah, hopper, and ant. In all environments, we learn diverse locomotion primitives, as shown in Figure 3. Despite never receiving any reward, the half cheetah and hopper learn skills that move forward and achieve large task reward on the corresponding RL benchmarks, which all require them to move forward at a fast pace. Half cheetah and hopper also learn skills that move backwards, corresponding to receiving a task reward much smaller than what a random policy would receive. Unlike hopper and half cheetah, the ant is free to move in the XY plane. While it learns skills that move in different directions, most skills move in arcs rather than straight lines, meaning that we rarely learn a single skill that achieves large task reward on the typical task of running forward. In the appendix, we visualize the objective throughout training. In FIG4, we evaluate all skills on three reward functions: running (maximize X coordinate), jumping (maximize Z coordinate) and moving (maximize L2 distance from origin). For each skill, DIAYN learns some skills that achieve high reward. We compare to single policy trained with a pure exploration objective (VIME ). Whereas previous work (e.g., Pathak et al. FORMULA0 ; BID4) finds a single policy that explores well, DIAYN optimizes a collection of policies, which enables more diverse exploration. FIG4: Exploration: We take DIAYN skills learned without a reward function, and evaluate on three natural reward functions: running, jumping, and moving away from the origin. For all tasks, DIAYN learns some skills that perform well. In contrast, a single policy that maximizes an exploration bonus (VIME) performs poorly on all tasks. We used our method as a starting point when comparing to VIC in Section 4.2. While p(z) is fixed in our method, we implement VIC by learning p(z). In this section, we describe how we learned p(z), and show the effect of learning p(z) rather than leaving it fixed. We choose p(z) to optimize the following objective, where p z (s) is the distribution over states induced by skill s: DISPLAYFORM0 For clarity, we define p t z (s) as the distribution over states induced by skill z at epoch t, and define t (z) as an approximation of E[log p(z | s)] using the policy and discriminator from epoch t: DISPLAYFORM1 Noting that p(z) is constrained to sum to 1, we can optimize this objective using the method of Lagrange multipliers. The corresponding Lagrangian is DISPLAYFORM2 Setting the derivative equal to zero, we get DISPLAYFORM3 and finally arrive at DISPLAYFORM4 Figure 17: Effect of learning p(z): We plot the effective number of skills that are sampled from the skill distribution p(z) throughout training. Note how learning p(z) greatly reduces the effective number on inverted pendulum and mountain car. We show from 3 random seeds for each environment. Figure 18: Learning p(z) with varying number of skills: We repeat the experiment in Figure 4 for varying sizes of z. Regardless of the size of z, learning p(z) causes the effective number of skills to drop to less than 10. The two subplots show the same data (Left) on a linear scale and (Right) logarithmic scale. We plot the mean and standard deviation across 3 random seeds. In this section, we briefly discuss the effect of learning p(z) rather than leaving it fixed. To study the effect of learning p(z), we compared the entropy of p(z) throughout training. When p(z) is fixed, the entropy is a constant (log ≈ 3.9). To convert nats to a more interpretable quantity, we compute the effective number of skills by exponentiation the entropy: effective num. skills e Figure 17 shows the effective number of skills for half cheetah, inverted pendulum, and mountain car. Note how the effective number of skills drops by a factor of 10x when we learn p(z). This observation supports our claim that learning p(z) in learning fewer diverse skills. FIG6 is a repeat of the experiment in FIG5, where we varying the dimension of z. Note that the dimension of z equals the maximum number of skills that the agent could learn. We observe that the effective number of skills plummets throughout training, even when using a high-dimensional vector for z. In this section, we visualize the skills learned for inverted pendulum and mountain car without a reward. Not only does our approach learn skills that solve the task without rewards, it learns multiple distinct skills for solving the task. FIG13 shows the X position of the agent across time, within one episode. For inverted pendulum FIG13, we plot only skills that solve the task. Horizontal lines with different X coordinates correspond to skills balancing the pendulum at different positions along the track. The periodic lines correspond to skills that oscillate back and forth while balancing the pendulum. Note that skills that oscillate have different X positions, amplitudes, and periods. For For every skill, we collect one trajectory and plot the agent's X coordinate across time. For inverted pendulum (top), we only plot skills that balance the pendulum. Note that among balancing skills, there is a wide diversity of balancing positions, control frequencies, and control magnitudes. For mountain car (bottom), we show skills that achieve larger reward (complete the task), skills with near-zero reward, and skills with very negative reward. Note that skills that solve the task (green) employ varying strategies. mountain car FIG13, skills that climb the mountain employ a variety of strategies for to do so. Most start moving backwards to gather enough speed to summit the mountain, while others start forwards, then go backwards, and then turn around to summit the mountain. Additionally, note that skills differ in when the turn around and in their velocity (slope of the green lines). Figures 20, 21, and 22 show more skills learned without reward. Given the expert trajectory, we use our learned discriminator to estimate which skill was most likely to have generated the trajectory:ẑ = arg max z Π st∈τ * q φ (z | s t)As motivation for this optimization problem, note that each skill induces a distribution over states, p z p(s | z). We use p * to denote the distribution over states for the expert policy. With a fixed prior distribution p(z) and a perfect discriminator q φ (z | s) = p(z | s), we have p(s | z) ∝ q φ (z | s) as a function of z. Thus, Equation G is an M-projection of the expert distribution over states onto the family of distributions over states, P = {p z}:arg min DISPLAYFORM0 For clarity, we omit a constant that depends only on p *. Note that the use of an M-projection, rather than an I-projection, helps guarantee that the retrieved skill will visit all states that the expert visits BID5. In our experiments, we solve Equation 5 by simply iterating over skills. The "expert" trajectories are actually generated synthetically in these experiments, by running a different random seed of our algorithm. A different seed is used to ensure that the trajectories are not actually produced by any of the currently available skills. Of course, in practice, the expert trajectories might be provided by any other means, including a human. For each expert trajectory, we retrieve the closest DIAYN skillẑ using Equation 4.2.3. Evaluating q φ (ẑ | τ *) gives us an estimate of the probability that the imitation will match the expert (e.g., for a safety critical setting). This quantity is useful for predicting how accurately our method will imitate an expert before executing the imitation policy. In a safety critical setting, a user may avoid attempting tasks where this score is low. We compare our method to three baselines. The "low entropy" baseline is a variant on our method with lower entropy regularization. The "learned p(z)" baseline learns the distribution over skills. Note that Variational Intrinsic Control is a combination of the "low entropy" baseline and the "learned p(z)" baseline. Finally, the "few skills" baseline learns only 5 skills, whereas all other methods learn 50. FIG1 shows the aggregated across 600 imitation tasks. The X-axis shows the discriminator score, our estimate for how well the imitation policy will match the expert. The Y-axis shows the true distance between the trajectories, as measured by L2 distance in state space. For all methods, the distance between the expert and the imitation decreases as the discriminator's score increases, indicating that the discriminator's score is a good predictor of task performance. Our method consistently achieves the lowest trajectory distance among all methods. The "low entropy" baseline is slightly worse, motivating our decision to learn maximum entropy skills. When imitating tasks using the "few skills" baseline, the imitation trajectories are even further from the expert trajectory. This is expected -by learning more skills, we obtain a better "coverage" over the space of skills. A "learn p(z)" baseline that learns the distribution over skills also performs poorly. Recalling that is a combination of the "low entropy" baseline and the "learn p(z)" baseline, this plot provides evidence that using maximum entropy policies and fixing the distribution for p(z) are two factors that enabled our method to scale to more complex tasks.

Represent the short summary for finding the corresponding scientific article (long)

We propose an algorithm for learning useful skills without a reward function, and show how these skills can be used to solve downstream tasks.

Represent the TLDR for finding the corresponding scientific article

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Lexical ambiguity, i.e., the presence of two or more meanings for a single word, is an inherent and challenging problem for machine translation systems. Even though the use of recurrent neural networks and attention mechanisms are expected to solve this problem, machine translation systems are not always able to correctly translate lexically ambiguous sentences. In this work, I attempt to resolve the problem of lexical ambiguity in English--Japanese neural machine translation systems by combining a pretrained Bidirectional Encoder Representations from Transformer (BERT) language model that can produce contextualized word embeddings and a Transformer translation model, which is a state-of-the-art architecture for the machine translation task. These two proposed architectures have been shown to be more effective in translating ambiguous sentences than a vanilla Transformer model and the Google Translate system. Furthermore, one of the proposed models, the Transformer_BERT-WE, achieves a higher BLEU score compared to the vanilla Transformer model in terms of general translation, which is concrete proof that the use of contextualized word embeddings from BERT can not only solve the problem of lexical ambiguity, but also boost the translation quality in general. Lexical ambiguity, i.e., the presence of two or more meanings for a single word, is an inherent and challenging problem for machine translation systems. Even though the use of recurrent neural networks (RNN) and attention mechanisms are expected to solve this problem, machine translation systems are not always able to correctly translate lexically ambiguous sentences. In this work, we attempt to resolve the problem of lexical ambiguity in English-Japanese neural machine translation systems by combining a pretrained Bidirectional Encoder Representations from Transformer (BERT) language model that can produce contextualized word embeddings and a Transformer translation model, which is a state-of-the-art architecture for the machine translation task. These two proposed architectures have been shown to be more effective in translating ambiguous sentences than a vanilla Transformer model and the Google Translate system. Furthermore, one of the proposed models, the Transformer BERT−WE, achieves a higher BLEU score compared to the vanilla Transformer model in terms of general translation, which is concrete proof that the use of contextualized word embeddings from BERT can not only solve the problem of lexical ambiguity, but also boosts the translation quality in general. Machine translation is one of the most important tasks in the field of natural language processing. In 2014, Sutskever and his fellow researchers at Google introduced the sequence-to-sequence (seq2seq) model , marking the advent of neural machine translation (NMT) in a breakthrough in the field of machine translation. Since then, seq2seq models have been growing rapidly, evolving from a purely recurrent neural network (RNN)-based encoder-decoder model to recurrence-free models that rely on convolution or attention mechanisms . The Transformer architecture , which is based on attention mechanism, is currently the standard model for machine translation tasks because of its effectiveness and efficiency. It also provides a foundation for the advent of state-of-the-art language models, such as Bidirectional Encoder Representations from Transformer (BERT) and GPT-2 . Section 2 shows how seq2seq models transformed from a purely RNN-based encoder-decoder model to a transformer model that relies entirely on attention mechanism. Although many significant improvements have been made in the NMT field, lexical ambiguity is still a problem that causes difficulty for machine translation models. show that the performance of RNNbased seq2seq model decreases as the number of senses for each word increases. Section 3 demonstrates that even modern translation models, such as Google Translate, cannot translate some lexically ambiguous sentences and forms hypotheses concerning some causes of this problem. Section 4 describes the BERT language model and explains why BERT vector representations can help resolve the problem of lexical ambiguity. Subsequently, two context-aware machine translation architectures that integrate pretrained BERT and Transformer models are proposed in section 5. For comparison purposes, a vanilla Transformer was built with the same set of hyperparameters and trained with the same settings as the proposed models. Finally, the three models were evaluated based on two criteria: i.e., the capability to produce good translations in general and the ability to translate lexically ambiguous sentences. The evaluation and sample translations are shown in section 6.3. 2 Neural machine translation 2.1 Sequence-to-sequence model NMT is an approach to machine translation, where a large neural network model learns to predict the likelihood of a sequence of words given a source sentence in an end-to-end fashion. The neural network model used for machine translation is called a seq2seq model, which is composed of an encoder and a decoder. RNN and its variants such as long short-term memory (LSTM) and gated recurrent unit (GRU) have been a common choice to build a seq2seq model. The encoder, which is a multilayered RNN cell, encodes the input sequence x into a fixed-sized vector v, which is essentially the last hidden state of the encoder's RNN. The decoder, which is another RNN, maps this context vector to the target sequence y. In other words, a seq2seq model learns to maximize the conditional probability: where T and S are the lengths of the input sentence of the source language and the output sentence of the target language, respectively. The attention mechanism proposed by , is a significant improvement to seq2seq models. By using the attention mechanism, each position in the decoder can selectively focus on all positions in the encoder instead of relying entirely on the last hidden state of the encoder, which consequently boosts the model's capability to learn long-term dependencies. Basically, the attention mechanism is a mapping of a query and a set of key-value pairs to an output vector. Each query vector represents a decoder's hidden state h t, while the key and values vectors represent all the encoder's hidden states h s. The output vector c t is a weighted sum of the value vectors, where the weight corresponding to each value vector is computed by an alignment function. where the value of the similarity function describes to what extent an input at position s and an output at position t match. The two most commonly used similarity functions are additive attention and multiplicative attention, which were proposed by and respectively, as shown in Eq. 3. 2.3 Transformer The Transformer model was first introduced by , which removes all recurrence and relies entirely on selfattention. In a self-attention layer, all queries, keys, and values come from the same place. As a , each position can attend to all other positions in the same sequence, which dispenses with the need for recurrence. This architecture does not only outperform RNN-based seq2seq models in terms of performance, but also it is parallelizable and requires less time to train. Thus, it has replaced the RNN-based seq2seq model as the de facto standard in neural machine translation. Like other seq2seq models, a Transformer consists of an encoder and a decoder, as shown in Figure 1. The encoder is a stack of N = 6 identical layers, each of which is composed of two linked sublayers: a multihead attention mechanism and a fully connected feed-forward network. The decoder stack also consists of N = 6 identical layers. Unlike the encoder, each decoder layer is composed of three consecutive sublayers: a masked multihead attention, a multihead attention mechanism, and a fully connected feed-forward network. The first attention sublayer in each decoder layer performs self-attention, while the second one pays attention to the output of the encoder stack. In both the encoder and decoder, each sublayer's input and output are added using a residual connection and normalized using a layer normalization method . All sublayers in the model and the embedding layers produce outputs of dimension d model = 512. The Transformer model uses a multihead attention mechanism, i.e., many attention functions are performed simultaneously in the same attention layer. Specifically, all queries, keys, and values of dimension d model are projected h = 8 times with different learned linear projections to. Each attention head produces output vectors of dimension d v, which are concatenated and linearly projected one more time to produce the final output vectors of dimension d model. The alignment function used in the Transformer model is called scaled dot-product attention: Each fully connected feed-forward network in the model consists of a hidden layer with ReLU activation and an output layer, producing outputs of dimensions d f f = 2048 and d model = 512, respectively. The self-attention mechanism uses symmetrical matrix multiplication; therefore, it cannot capture information about the sequence order. Thus, in addition to word embedding layers, it is necessary to add some information about the relative or absolute positions of the tokens in the sequence . The positional encodings are added to the outputs of embedding layers before being fed to the encoder and decoder stacks. For the model to attend to relative positions, the sine and cosine functions of different frequencies are used to generate positional encodings: where pos is the position and i is the dimension. Lexical ambiguity, which is also called semantic ambiguity, can be defined as the presence of more than one meaning for a single word. Words that possess two or more possible meanings are called homographs. Translating homographs is not a trivial task because their meanings vary based on their contexts. For example, given a sentence "The fisherman went to the bank.", the word "bank" may refer to "a financial institution" or "the side of a river." In this case, it is acceptable to interpret this sentence in two ways. However, given another sentence "A fisherman is sitting on the bank.", it is unreasonable to interpret the word "bank" as "a financial institution" in this case. However, this sentence is challenging for machine translation systems to produce a correct translation, as shown in the later part of the paper. Even though many advancements have been made in the field of neural machine translation to date, contemporary translation systems are still struggling to deal with semantic ambiguity. For instance, although the sentence "He left a book on the table and left the room." contains two words "left" of different meanings, Google Translate is able to correctly translate it into " On the other hand, Google Translate misinterprets the word "bank" in the sentence "A fisherman is sitting on the bank." and thus translates it into "漁 師が銀 銀 銀行 行 行に座っています。". The cause of this problem can be hypothesized that machine translation models use only one embedding vector to represent a homograph, even though the senses of a single homograph can be completely unrelated. Consequently, if a machine translation model fails to understand the meaning of a homograph from the context, it will tend to choose the dominant meaning. Furthermore, another hypothesis is that the parallel corpus used to train Google Translate system does not provide enough information for the system to understand the context of the latter example. This problem of semantic ambiguity can be addressed if both following conditions are satisfied: 1. Different senses of a homograph are represented by different embedding vectors. To achieve this, the model needs to understand the meanings of homographs from the context. 2. The training set must be exceptionally large so that the model can properly understand the context of unseen sentences. Although it is possible to obtain a large monolingual data set, it becomes difficult when it comes to finding a parallel corpus. An extensive pretrained language representation model, BERT supports transfer learning and finetuning on a wide range of tasks, such as question answering and language inference . BERT is composed of multiple layers of bidirectional Transformer encoders, with 12 layers for BERT BASE and 24 layers for BERT LARGE model . As a , BERT can learn bidirectional word representations by conditioning on both left and right contexts, which outperforms unidirectional language models, such as ELMo and OpenAI-GPT . BERT is simultaneously pretrained on two different tasks: masked language modeling and next sentence prediction. As for the masked language modeling task, 15% of all tokens in each sequence are selected at random to be predicted. If a token is chosen, it is replaced with a [MASK] token 80% of the time, with a random token 10% of the time or with the same token 10% of the time. BERT learns to predict the masked tokens instead of regenerating the entire input. As for Figure 2: The BERT architecture the next sentence prediction task, BERT is trained with a collection of concatenated sentence pairs and tries to predict whether the two sentences in each pair are contiguous. Consequently, the pretrained BERT is capable of understanding the relationship between two sentences and can be finetuned on downstream tasks, such as question answering and natural language inference . The corpora used for pretraining BERT are the BooksCorpus (800M words) and English Wikipedia (2,500M words). BERT uses WordPiece embeddings with a vocabulary of 30,000 tokens. Each sequence input to BERT starts with a [CLS] token followed by two concatenated sentences, which are separated by a special [SEP] token. Since the pretrained BERT generates the vector representation for a word by considering all other words in the same sequence, this vector can change dynamically with the context where it is used. In other words, BERT can produce different embeddings for different senses of the same word. To examine this feature of BERT, a pretrained BERT was used to generate the vector representations of words in different examples: 1. "There is an old man fishing on the bank." 2. "It's on the north bank of the Thames." 3. "a house on the banks of the River Severn" 4. " He jumped in and swam to the opposite bank." 5. "Many of these banks issue both credit and debit cards." 6. " My salary is paid directly into my bank." 7. "A group of ten international banks is to underwrite and sell the bonds." The words bank and banks in the first four examples mean "the edge of a river," while the ones in the next three examples mean "a financial institution." After word embeddings of dimension 768 are extracted from the pretrained BERT, t-SNE algorithm is used to extract the two most significant features of each word and the reduced word embeddings are visualized as shown in Figure 3. It can be clearly seen that the points representing the words "bank" and "banks" are clustered in two separate groups based on their meaning. Furthermore, another interesting point is that the words "bank" and "banks," which mean "the edge of a river" are located near related words such as "river," "fishing," and "swam," while the ones meaning "a financial institution" are near to some monetary terms such as "credit" and "mortgage." In the original Transformer architecture, each word in the predefined vocabulary list is represented by only one embedding vector. These word embeddings are trained as the model's parameters, therefore depend greatly on the limited training set. Apparently, the original Transformer does not satisfy the conditions mentioned in section 3. Consequently, it is unable to correctly translate semantically ambiguous sentences, as demonstrated in section 6.3 The BERT BASE model was integrated into a Transformer translation model to address lexical ambiguity in neural machine translation. Specifically, two architectures are proposed: Transformer BERT−WE using the pretrained BERT as input word embedding layer and Transformer BERT−Encoder replacing the encoder stack of a Transformer with the pretrained BERT model. The outputs of the last ten layers of the BERT were extracted and averaged. All the parameters of the pretrained BERT were kept unchanged during the training phase of both models. In this work, we implement and compare the performance of three models: a baseline Transformer, a Transformer BERT−WE, and a Transformer BERT−Encoder, which share the same hyperparameters for comparison purposes. We denote the number of layers in both the encoder and decoder stacks as N, the dimension of embedding layers and all sublayers' output as d model, the dimension of the inner layer in every fully connected feed-forward layer as d f f, and the number of attention heads as h. Due to the lack of memory capacity, N is set to 3, as opposed to N = 6 in the original Transformer paper. In addition, to match the hidden size h = 768 of the pretrained BERT model, d model and d f f are set to 768 and 3072 respectively. Dropout is applied to the output of each sublayer and the sum of the embeddings and positional encodings in both the encoder and decoder with a rate of 0.1. The models are trained with Japanese-English Subtitle Corpus (JESC) , consisting of over 3.2 million sentence pairs. This data set includes casual language, colloquialisms, expository writing, and narrative discourse. The train/val/testsplits are of size 3,237,374 / 2000 / 2001. Rakuten MA (morphological analyzer) is used to tokenize Japanese sentences and tokens that appear at least 10 times are shortlisted. Likewise, as for the baseline Transformer model, English sentences are tokenized by using nltk library and tokens are shortlisted in the same manner. By contrast, for BERT Transformer models, the BERT's internal word tokenizer and vocabulary are used. All out-of-vocabulary words are replaced with a special [UNK] token. For the pretrained BERT to effectively generate vector representations, special tokens [CLS] and [SEP] are added to the beginning and the end of the input English sentences, respectively. Over 6,000 sentences that contain homographs are extracted from the IWSLT 2017 EnglishJapanese data set to evaluate the performance of the models on ambiguous sentences. The Bilingual Evaluation Understudy (BLEU) score is used as the evaluation metric to assess the models' performance. The three models were trained with the same training settings. The batch size was set to 32 and AdamOptimizer with decayed learning rate used. The initial learning rate is set to 0.0001. The models are evaluated on a validation set after each training epoch. When validation loss increases for the first time, the learning rate starts decreasing with the rate of 0.33 per epoch. The training process is stopped when validation loss increases again. It takes from 1.5 to 2 days to finish training a model on a GTX 1080Ti. The models were evaluated based on two criteria: general translation and homograph translation. The JESC test set was used to evaluate the models' capability of translating English sentences in general, while the IWSLT 2017 data set was used to evaluate the models' ability to correctly translate semantically ambiguous sentences. The of BLEU score evaluation are shown in Table 1. It can be clearly seen that the Transformer BERT−WE model outperforms the other two models in both evaluations, achieving a BLEU score of 20.31 on JESC test set and 8.67 on IWSLT 2017 data set. Transformer BERT−Encoder model's performance is slightly worse than the vanilla Transformer in terms of general translation; however, it outperforms the vanilla Transformer when it comes to translating homographs. As shown in Table 2, Google Translate and the vanilla Transformer wrongly translate the word "bank" in the two given English sentences. By contrast, the two models Transformer BERT−WE and Transformer BERT−Encoder can correctly translate the word "bank" into "土手" or "岸", which means "the edge of a river" in Japanese. Another approach to solving lexical ambiguity was proposed by . The authors proved that the performance of translation models degrades as the number of senses for each word increases. They concatenated embedding vectors from a word sense disambiguation (WSD) system to a translation model's word embeddings and applied gating functions to generate contextualized word embeddings. The contextualized word embeddings are fed into the encoder of an RNN-based seq2seq model to generate translations. Their model was trained on three different language pairs: English-German, English-French, and English-Chinese. Using pretrained word embeddings was empirically proved to increase the BLEU score for the machine translation task. compared the performance of different translation systems that used either random initialization or pretraining on both source and target languages. The word embeddings used in their experiments were trained by using the Common Bag of Words (CBOW) algorithm . According to their , using pretrained word embeddings, especially on the source language side, considerably boosts the performance of machine translation systems . In this work, we demonstrate that lexical ambiguity is an inherent problem that contemporary machine translation systems cannot completely address, hypothesize two causes of the problem, and prove that this issue can be addressed by using contextualized word embeddings that dynamically change based on the context of given words. In addition, the BERT language model is demonstrated to be effective at generating contextualized word representations and two machine translation architectures that integrate pretrained BERT and Transformer translation models are proposed. The two architectures are shown to be able to translate semantically ambiguous sentences effectively. Furthermore, the Transformer BERT−WE model outperforms the vanilla Transformer model, proving that our approach can not only resolve the problem of lexical ambiguity, but also increases the translation quality in general.

Represent the TLDR for finding the corresponding scientific article

The paper solves a lexical ambiguity problem caused from homonym in neural translation by BERT.

LSTMs learn long-range dependencies compositionally by building them from shorter constituents over the course of training.


Question: Represent the article for finding a single-sentence TLDR:

Question: This paper focuses on the synthetic generation of human mobility data in urban areas. We present a novel and scalable application of Generative Adversarial Networks (GANs) for modeling and generating human mobility data. We leverage actual ride requests from ride sharing/hailing services from four major cities in the US to train our GANs model. Our model captures the spatial and temporal variability of the ride-request patterns observed for all four cities on any typical day and over any typical week. Previous works have succinctly characterized the spatial and temporal properties of human mobility data sets using the fractal dimensionality and the densification power law, respectively, which we utilize to validate our GANs-generated synthetic data sets. Such synthetic data sets can avoid privacy concerns and be extremely useful for researchers and policy makers on urban mobility and intelligent transportation. Ride sharing or hailing services have disrupted urban transportation in hundreds of cities around the globe (; BID2 . In United States, it has been estimated that between 24% to 43% of the population have used ride-sharing services in 2018 BID21 . Uber alone operates in more than 600 cities around the globe BID22 . Ride sharing services have turned urban transportation into a convenient utility (available any place at any time), and become an important part of the economy in large urban areas BID8.Ride request data from ride sharing services can potentially be of great value. Data gathered from ride sharing services could be used to provide insights about traffic and human mobility patterns which are essential for intelligent transportation systems. Ride requests in major cities with high penetration by such services exhibit spatial and temporal variability. Modeling of such variability is a challenging problem for researchers. Moreover, there are still unresolved challenges, such as: optimal algorithms for dynamic pooling of ride requests BID1, real-time preplacement of vehicles BID12, and city scale traffic congestion prediction BID17 and avoidance. Access to large amount of actual ride request data is essential to understanding and addressing these challenges. Data from ride sharing services have been used for real-time sensing and analytics to yield insights on human mobility patterns BID25 BID11. Each city exhibits a different pattern of urban mobility -there could be cultural or economical factors governing these patterns. If ride sharing services constitute a significant percentage of the riders in a city, can we build models from ride request data to model urban mobility for the whole city and provide societal benefit without compromising personal privacy? This question motivates us to explore the potential of using Generative Adversarial Networks (GANs) to generate synthetic ride request data sets that exhibit very similar attributes as the actual ride request data sets. This work proposes a novel approach of generating synthetic ride request data sets using GANs. This approach involves viewing ride requests as a (temporal) sequence of (spatial) images of ride request locations. The approach uses GANs to match the properties of the synthetic data sets with that of real ride request data sets. Many recent works using neural networks have looked at demand prediction BID29 BID30 and traffic prediction at intersections BID28. In our work, we are looking at generating actual ride requests for both spatially and temporally granular intervals. Also, we compare and validate the spatial and temporal variations of the DISPLAYFORM0 Figure 1: Ride requests for a small region of downtown San Francisco for a typical week day. Each figure shows the aggregated ride-locations (red dots) over a period of an hour. Each red dot may represent one or more ride-locations. Ride density varies spatially and temporally.synthetic data sets with the real data sets. In dealing with large amount of data for many cities and long training times for GANs, we develop effective ways to parallelize and scale our GANs training runs using large CPU clusters on AWS. We present our GANs scaling approach and experimental , and show that significant reduction in training times can be achieved. In this section, we introduce the actual (real) ride request data sets used for our GANs training and evaluation. We use the real data sets to compare with and validate the GANs generated synthetic data sets. Our real ride request data sets consist of all the ride requests for an entire week for the four cities. There is a strong repeating pattern from week to week as shown in FIG0. Hence the week-long data should be quite representative. For all four cities, the ride sharing services have significant penetration. Hence we believe the ride request data sets also reflect the overall urban mobility patterns for these cities. Our real data sets are actual ride requests for four cities over one week period from ride sharing services operating in the United States. Each ride request in the data set includes: request time and pickup location (latitude & longitude), and drop-off time and location (latitude & longitude). For this work we focus on ride request time and pickup location for generating pickup locations; and ride request time and drop-off location to generate drop-off locations. After training independent GANs models for pickup and drop-off locations, we generate synthetic locations using GANs and leverage graph generator approach BID4 BID11 to pair all pickup and drop-off locations to obtain synthetic ride requests. The trajectory or optimal route for a ride is not within the scope of this work. For the rest of the paper, we will use the term ride-locations to refer to both pickup and drop-off locations wherever they can be used interchangeably. We do temporal and spatial quantization of the raw ride request data from ride sharing services. We partition the entire week into 2016 time intervals of 5 minutes each, and lump together all the ride requests within each interval. We partition spatially the area of the entire city into small squares with side length,, of 50 meters, and lump together all the ride-locations occurring within the same square area. Each square area is then represented by a single pixel in a 2-D image with the gray scale intensity of the pixel reflecting the number of ride-locations in that square area (in a time interval). Occurrence of no ride-locations in an area is denoted by zero pixel intensity; positive integers (1, 2, 3, . . .) as pixel intensity denote the number of ride-locations in the square area. Combining the temporal and spatial quantizations, the real ride request data set for each city becomes a time sequence of images with each image spatially capturing all the ride requests occurring in a particular 5-min interval. The actual ride requests in every city exhibit distinct patterns of variability in both the spatial dimension (over geographical area of the city) and the temporal dimension (over each day and over each week). In Figure 1, this variability is illustrated. The ride request density is at its highest at 6pm, and continually decreases over time till 3am. Spatially there are dense patches of ride requests and these dense patches can shift with time, reflecting shifting concentrations of commuters in different areas at different times of day. We observe similar repeating patterns of temporal and spatial variability for all four cities. Previous works have been able to characterize these temporal and spatial variability patterns BID11. A graph can be used to model the ride requests within a 5-min interval, with nodes 1 representing pickup and drop off locations and a directed edge connecting the pickup node and the drop-off node. It was shown in BID11 ) that the size and density of this Ride Request Graph (RRG) evolves in time in response to the fluctuation of ride requests during each day and through out each week. It was observed that these ride request graphs exhibit and obey the Densification Power Law (DPL) property, similar to other graphs modeling human behaviors such as social networking graphs and publication citation graphs BID16. It was further observed that the ride request graphs for each city exhibit a distinct degree or exponent of the DPL, and that this DPL Exponent (α) can be viewed as a very succinct quantitative characterization of the temporal variability of the ride request patterns for that city. For any time snapshot t: DISPLAYFORM0 where e(t) and n(t) are the number of edges and number of nodes respectively, formed by all ride requests occurring in the time interval t. Edge weight denote the number of requests from the same source (pickup) to destination (drop-off) nodes in time snapshot t. The number of edges grows according to a specific exponential power (α) of the number of nodes. There is also a comparable quantitative characterization of the spatial variability of the ride request patterns for each city. The actual geographical locations of the nodes of the ride request graphs is not explicitly represented and therefore another characterization is needed. Correlation Fractal Dimension BID24 BID0 ) provides a succinct description of a kdimensional point-set to provide statistics about the distribution of points; it provides a quantitative measure of self-similarity. The spatial distribution of ride requests in each time interval can be viewed as a point-set image. We can measure the Correlation Fractal Dimension (D 2) as described in BID12. Values for correlation fractal dimension computed for each time snapshot t fall within a range for each city indicating the degree of self-similarity, and the consistent weekly pattern. For our 2-dimenional space, we impose a 2D-grid with square of side 2. For the i-th square, let C,i be the count of requests in each square. The correlation fractal dimension is defined as: DISPLAYFORM1 For self-similar data sets, we expect the derivative to be constant for a certain range of BID27. We observe that this range varies for our four cities, and each city exhibits a distinct value range for its correlation fractal dimension (D 2).We use the Densification Power Law Exponent (α) and the Correlation Fractal Dimension (D 2) to capture and characterize the temporal and spatial variability, respectively, for the ride request patterns for each city. RRG created for every time snapshot captures ridership fluctuations over time; nodes in a RRG do not encode any spatial information. Therefore, we compute Correlation Fractal Dimension for each time snapshot to capture the spatial distribution of both pickup and dropoff locations. The temporal evolution, and spatial distribution at any give time snapshot capture the dynamics of ride requests. We use these two parameters independently to confirm the similarity between the real data sets and the GANs generated synthetic data sets. We can claim strong similarity if the values of these two parameters (α and D 2) of the synthetic data sets match closely the values of the same two parameters of the real data sets. Generative Adversarial Networks learn to generate high quality samples i.e. sample from the data distribution p(x). Previous works by BID3 BID14 ) synthesized images of a higher quality using GANs which were hard for humans to distinguish from real images. Conditional GANs are an extension of GANs to sample from a conditional distribution given each image has an associated label which is true for our case of ride requests. In our framework, we would apply conditional GANs using ride request data in the form of images; similar to as shown in Figure 1 but without the base map shown in color. GANs learn a mapping from a random noise vector z to output image x. Conditional GANs learn a mapping from noise vector z and a label y to x BID18 BID5. The additional variable in the model allows to generate and discriminate samples conditioned on y. The generator accepts noise data z along with y to produce an image. The discriminator accepts an image x and condition y to predict the probability under condition y that x came from the empirical data distribution rather than from the generative model. The objective function can be expressed as: DISPLAYFORM0 where G tries to minimize to this objective function against an adversarial D that tries to maximize it. Every image is assigned a label from the set {0, 1, 2, ..., 23} representing the hour of a day. All twelve instances of five minute snapshots within an hour are assigned the same hour label 3. To accelerate our training using multiple machines, we exploit spatial parallelism by dividing the entire geographical region of a city into an array of blocks. FIG1 illustrates the division of San Francisco downtown into nine blocks. Keeping our image size similar to MNIST BID23, each block is set to represent an image of size 24×24 pixels, with each pixel representing one 50m×50m square area. Hence, each block covers an area of 1200m × 1200m. Each block, representing a grey scale image of 24 × 24 pixels, depicts all the ride-locations in that block. Separate images are formed for pickup and drop-off locations; models trained are also separate for pickup and drop-off locations. Each image of a block is labeled with a time interval (for our experiments, the hour in a day) which is similar for both images created from pickup and drop-off locations. The synthetically generated images from an array of blocks with the same time interval label are combined by stitching together all the processed blocks of a city. The generator network takes an input of a 100-dimensional Gaussian noise sample as well as a onehot vector encoding of the time snapshot to be generated. It has a single, fully-connected hidden layer without any convolution BID6 consisting of 128 ReLU-activated neurons which then passes to a sigmoid activated output layer with the same number of output neurons as the total number of pixels in each block. The discriminator network has a single hidden layer of 128 ReLU-activated neurons with a single sigmoid activated output neuron. We find that small networks are appropriate for the training data and allow for a quick and stable convergence to be achieved between the discriminator and the generator. Using relatively simple network architectures makes it possible to ensure that the discriminator and generator are evenly matched such that the loss for either network does not saturate early in the training process. In addition to the standard GANs architecture of generator and discriminator, an additional network is introduced which is referred to as the classifier BID15; it is pre-trained on the training data with the five minute label of the data serving as the classification target. In this way the time information that is encoded into the synthetic data by the generator network is then decoded by the classifier network. The generator is then trained on a weighted sum of the loss from both the classifier and discriminator networks as shown in the following equation: DISPLAYFORM0 where β is a tune-able hyper-parameter. This allows for more explicit loss attribution such that the generator receives two different error signals; one indicating the realism of the synthetic data and the other indicating accuracy relative to the conditioning values. By experiments using MNIST data and BID15, we found adding a classifier increases the efficiency of the training process and in higher quality synthetic data while incurring considerably less training time than other conditional GANs architectures we have experimented. In this section, we present the cloud infrastructure used for running our experiments. We also present performance on scaling our GANs workloads on the cloud infrastructure. All experiments are conducted on Amazon Web Services (AWS) using c5.18x instances with each instance containing an Intel Xeon Scalable Processor with 72 virtual cores (vCores) running at 3.0GHz and 144 GB of RAM. In this work we set the block size for each of the four cities to be 1200 × 1200 meters; each block is trained separately. Enlarging the block size will increase the computational time for training; and the complexity of the model can potentially impact scalability. The total number of blocks for each city are shown in TAB0. The number of blocks are mainly determined by the size of the greater metropolitan area of each city. To help enhance the scalability of our GANs workload across multiple nodes we make use of Ray BID19 from Berkeley, a distributed framework for AI Applications, to efficiently parallelize our workload across cluster of CPU nodes on AWS. Ray provides a convenient API in Python to scale deep learning workloads for numerous libraries, and support for heterogeneous resources like CPUs and GPUs. We also make use of Intel's Math Kernel Library Intel (2018b) (MKL) which provides machine learning libraries for supporting operations like activation (ReLU), inner product, and other useful functions BID9. Using Ray we scale our training runs by using from 2 to 8 c5.18x instances (containing from 144 cores to 576 cores) on AWS. The scalability are shown in Figure 4. As can be seen increasing the number of c5.18X Xeon CPU instances can significantly reduce the GANs training time up to 8 c5.18x instances. For the city of Los Angeles, the training time can be reduced from over one hour to less than 20 minutes. For New York City the training time can be reduced to just minutes. Running times for sampling ride requests from the trained models and stitching the images of all the blocks together are significantly less than the training times, and are not included in these . We also conduct our GANs scaling experiments using GPU instances on AWS. In our initial experiments we observe no real performance improvements using GPUs. Training time using GPUs on AWS was observed to be 5.93 hours on a p3.8xlarge instance using NVIDIA's Multi-Process Service (MPS) . With MPS, the GPU utilization is close to maximum by running multiple of our small GANs training jobs in parallel on a single GPU. Although, the number of jobs which could be executed in parallel on a GPU are not that many in comparison to Xeons. Scaling on GPUs requires more investigation. In this work, we show that it is possible to achieve very nice scalability of our GANs workload using only CPU cores supported by Intel's MKL library and Berkeley's Ray framework. The correlation fractal dimension (D 2) gives a bound on the number of ride requests within a geographical region. This is an essential characteristic to match for the data set we are generating using GANs. In TAB3, we provide the fractal range for each city within which the fractal dimension remains constant. It is important to note that the fractal range for each city differs. The fractal range provides the range for which the data exhibits statistical self-similarity BID0. The variation in the fractal ranges for the different cities can be attributed to the geographical shape of the city for which the ride requests are generated. We hypothesize that due to Los Angeles's sprawling nature, a larger is needed to observe self-similar patterns in comparison to the other three cities, which have a more corridor-like geographical region. Table 2: Summary of measured correlation fractal dimensions (D 2) for four cities; computed over a day for every hour using pickup locations of real and synthetic data sets. One may also interpret D 2 as a way to measure the fidelity of generated images to that from real data. Comparison of the ranges of values of D 2, in terms of min, max, and mean values, for the real and the synthetic data sets are fairly close although not identical. In most instances the mean value for D 2 is lower for the synthetic data sets in comparison to the real data sets. We believe this discrepancy in the values of D 2 require further investigation. Recent works to improve capture learning of highresolution details of an image BID13 can potentially benefit the learning for our ride request images. DPL provides a characterization of the temporal evolution of ride requests. In the top row of Figure 5 we observe the plot of the DPL exponents α (slop of the line) based on the temporal patterns of the real data sets. For the ride request graph to obey DPL properties, we use graph generator proposed by BID11 to connect source and destination locations. In the bottom row of Figure 5 we see the same based on the synthetic data sets. We can see that the DPL exponent values α correlated quite nicely with that from the real data sets for New York, Chicago, and San Francisco. Figure 5: DPL plots from real data (top row) and synthetic data (bottom row) for four cities. The red line is the least square fit of the form y = Cx α, where y and x are number of edges and nodes respectively. R 2 ≈ 1.00 for all of them. For Los Angeles, the synthetic exponent is higher than the real observed value; the geographical region for LA is much larger and due to many prominent regions of high request density, the model may likely suffer from bias towards generating more requests in prominent regions leading to a faster increase of the number of edges connecting nodes present in high density regions. Another validation of our GANs approach is provided in Figure 6. Here we observe temporal variation of ride requests in terms of the volume of ride requests generated for each hour of a typical weekday. We see that for all four cities, the temporal variation of the synthetic data sets match quite well the temporal variation exhibited by the actual data set. The emergence of ride sharing services and the availability of extensive data sets from such services are creating unprecedented opportunities for: 1) doing city-scale data analytics on urban transportation for supporting Intelligent Transportation Systems (ITS); 2) improving the efficiency of ride sharing services; 3) facilitating real-time traffic congestion prediction; and 4) providing new public services for societal benefit. Moreover, the power of neural networks for machine learning has allowed the creation of useful models which can capture human behavior and dynamic real-world scenarios. The key contributions of this paper include:• We map the ride requests of ride sharing services into a time sequence of images that capture both the temporal and spatial attributes of ride request patterns for a city.• Based on extensive real world ride request data, we introduce a GANs based workflow for modeling and generating synthetic and realistic ride request data sets for a city.• We further show that our GANs workload can be effectively scaled using Xeon CPU clusters on AWS, in reducing training times from hours to minutes for each city.• Using previous work on modelling urban mobility patterns, we validate our GANs generated data sets for ride requests for four major US cities, by comparing the spatial and temporal properties of the GANs generated data sets against that of the real data sets. There are other promising avenues for further research. Some open research topics include: Figure 6: Plots for four cities highlighting the temporal variability of ride requests visible in both real and our model (predicted) for ride request generation. The pattern is representative of any typical day of week.• Using the GANs generated data sets for experiments on new algorithms for dynamic ride pooling, real-time pre-placement of vehicles, and real-time traffic congestion prediction. • Using the GANs generated data sets for conducting experiments on what-if scenarios related to traffic congestion prediction and mitigation, and planning for future development of transportation infrastructures. We are currently pursuing these research topics. As our GANs generated data sets are used in our follow up research, we plan to further validate the synthetic data sets by comparing our research with from using the real data sets. We plan to continue to tune our GANs models and generate improved synthetic data sets that can be made available for other researchers.

This paper focuses on the synthetic generation of human mobility data in urban areas using GANs.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.