Question:
represent the text to find a very short summary (1-2 sentences max)
Given Reinforcement learning typically requires carefully designed reward functions in order to learn the desired behavior. We present a novel reward estimation method that is based on a finite sample of optimal state trajectories from expert demon- strations and can be used for guiding an agent to mimic the expert behavior. The optimal state trajectories are used to learn a generative or predictive model of the “good” states distribution. The reward signal is computed by a function of the difference between the actual next state acquired by the agent and the predicted next state given by the learned generative or predictive model. With this inferred reward function, we perform standard reinforcement learning in the inner loop to guide the agent to learn the given task. Experimental evaluations across a range of tasks demonstrate that the proposed method produces superior performance compared to standard reinforcement learning with both complete or sparse hand engineered rewards. Furthermore, we show that our method successfully enables an agent to learn good actions directly from expert player video of games such as the Super Mario Bros and Flappy Bird. Reinforcement learning (RL) deals with learning the desired behavior of an agent to accomplish a given task. Typically, a scalar reward signal is used to guide the agent's behavior and the agent learns a control policy that maximizes the cumulative reward over a trajectory, based on observations. This type of learning is referred to as "model-free" RL since the agent does not know apriori or learn the dynamics of the environment. Although the ideas of RL have been around for a long time BID24 ), great achievements were obtained recently by successfully incorporating deep models into them with the recent success of deep reinforcement learning. Some notable breakthroughs amongst many recent works are, the work from BID12 who approximated a Q-value function using as a deep neural network and trained agents to play Atari games with discrete control; who successfully applied deep RL for continuous control agents achieving state of the art; and BID22 who formulated a method for optimizing control policies with guaranteed monotonic improvement. In most RL methods, it is very critical to choose a well-designed reward function to successfully learn a good action policy for performing the task. However, there are cases where the reward function required for RL algorithms is not well-defined or is not available. Even for a task for which a reward function initially seems to be easily defined, it is often the case that painful hand-tuning of the reward function has to be done to make the agent converge on an optimal behavior. This problem of RL defeats the benefits of automated learning. In contrast, humans often can imitate instructor's behaviors, at least to some extent, when accomplishing a certain task in the real world, and can guess what actions or states are good for the eventual accomplishment, without being provided with the detailed reward at each step. For example, children can learn how to write letters by imitating demonstrations provided by their teachers or other adults (experts). Taking inspiration from such scenarios, various methods collectively referred to as imitation learning or learning from experts' demonstrations have been proposed BID21 ) as a relevant technical branch of RL. Using these methods, expert demonstrations can be given as input to the learning algorithm. Inverse reinforcement learning BID15; BID1; BID28 ), behavior cloning BID20 ), imitation learning BID6; BID5 ), and curiosity-based exploration ) are examples of research in this direction. While most of the prior work using expert demonstrations assumes that the demonstration trajectories contain both the state and action information (τ = {(s t)}) to solve the imitation learning problem, we, however, believe that there are many cases among real world environments where action information is not readily available. For example, a human teacher cannot tell the student what amount of force to put on each of the fingers when writing a letter. As such, in this work, we propose a reward estimation method that can estimate the underlying reward based only on the expert demonstrations of state trajectories for accomplishing a given task. The estimated reward function can be used in RL algorithms in order to learn a suitable policy for the task. The proposed method has the advantage of training agents based only on visual observations of experts performing the task. For this purpose, it uses a model of the distribution of the expert state trajectories and defines the reward function in a way that it penalizes the agent's behavior for actions that cause it to deviate from the modeled distribution. We present two methods with this motivation; a generative model and a temporal sequence prediction model. The latter defines the reward function by the similarity between the state predicted by the temporal sequence model trained based on the expert's demonstrations and the currently observed state. We present experimental of the methods on multiple environments and with varied settings of input and output. The primary contribution of this paper is in the estimation of the reward function based on state similarity to expert demonstrations, that can be measured even from raw video input. Model-free Reinforcement Learning (RL) methods learn a policy π(a t |s t) that produces an action from the current observation. BID12 showed that a q-value function q(s t, a t) can be approximated with a deep neural network, which is trained using hand-engineered scalar reward signals given to the agent based on its behavior. Similarly, actor-critic networks in Deep Deterministic Policy Gradients (DDPG) can enable state of the art continuous control, e.g. in robotic manipulation by minimizing the distance between the end effector and the target position. Since the success with DDPG, other methods such as Trust Region Policy Optimization (TRPO) BID22 ) and Proximal Policy Optimization (PPO) BID23 ) have been proposed as further improvements for model-free RL in continuous control problems. Although RL enables an agent to learn an optimal policy without supervised training data, in the standard case, it requires a difficult task of hand-tuning good reward functions for each environment. This has been pointed out previously in the literature BID1 ). Several kinds of approaches have been proposed to workaround or tackle this problem. An approach that does not require reward hand-tuning is behavior cloning based on supervised learning instead of RL. It learns the conditional distribution of actions given states in a supervised manner. Although it has an advantage of fast convergence BID5 ) (as behavior cloning learns a single action from states in each step), it typically in compounding of errors in the future states. An alternate approach is Inverse Reinforcement Learning (IRL) proposed in the seminal work by BID15. In this work, the authors try to recover the optimal reward function as a best description behind the given expert demonstrations from humans or other expert agents, using linear programming methods. It is based on the assumption that expert demonstrations are solutions to a Markov Decision Process (MDP) defined with a hidden reward function BID15 ). It demonstrated successful estimation of the reward function in case of relatively simple environments such as the grid world and the mountain car problem. Another use of the expert demonstrations is initializing the value function; this was described by BID27. Extending the work by BID15, entropy-based methods that compute the suitable reward function by maximizing the entropy of the expert demonstrations have been proposed by BID29. In the work by BID1, a method was proposed for recovering the cost function based on expected feature matching between observed policy and the agent behavior. Furthermore, they showed this to be the necessary and sufficient condition for the agent to imitate the expert behavior. More recently, there was some work that extended this framework using deep neural networks as non-linear function approximator for both policy and the reward functions BID28 ). In other relevant work by BID6, the imitation learning problem was formulated as a two-players competitive game where a discriminator network tries to distinguish between expert trajectories and agent-generated trajectories. The discriminator is used as a surrogate cost function which guides the agent's behavior in each step to imitate the expert behavior by updating policy parameters based on Trust Region Policy Optimization (TRPO) BID22 ). Related recent works also include model-based imitation learning BID2 ) and robust imitation learning BID26 ) using generative adversarial networks. All the above-mentioned methods, however, rely on both state and action information provided by expert demonstrations. Contrarily, we learn only from expert state trajectories in this work. A recent line of work aims at learning useful policies for agents even in the absence of expert demonstrations. In this regard, trained an agent with a combination of reward inferred with intrinsic curiosity and a hand-engineered, complete or even very sparse scalar reward signal. The curiosity-based reward is designed to have a high value when the agent encounters unseen states and a low value when it is in a state similar to the previously explored states. The work reported successful navigation in games like Mario and Doom without any expert demonstrations. In this paper, we compare our proposed methods with the curiosity-based approach and show the advantage over it in terms of the learned behaviors. However, our methods assumed state demonstrations are available as expert data while the curiosity-based method did not use any demonstration data. We consider an incomplete Markov Decision Process (MDP), consisting of states S and action space A, where the reward signal r: S × A → R, is unknown. An agent can act in an environment defined by this MDP following a policy π(a t |s t). Here, we assume that we have knowledge of a finite set of optimal or expert state trajectories DISPLAYFORM0., n}. These trajectories can represent joints angles, raw images or any other information depicting the state of the environment. Since the reward signal is unknown, our primary goal is to find a reward signal that enables the agent to learn a policy π, that can maximize the likelihood of these set of expert trajectories τ. In this paper, we assume that the reward signal can be inferred entirely based on the current state and next state information, r: S × S → R. More formally, we would like to find a reward function that maximizes the following objective: DISPLAYFORM1 where r(s t+1 |s t) is the reward function of the next state given the current state and p(s t+1 |s t) is the transition probability. We assume the performing to maximize the likelihood of next step prediction in equation 1 will be leading the maximizing the future reward when the task is deterministic. Because this likelihood is based on similarity with demonstrations which are obtained while an expert agent is performing by maximizing the future reward. Therefore we assume the agent will be maximizing future reward when it takes the action that gets the similar next step to expert demonstration trajectory data τ. Let, τ = {s i t} i=1:M,t=1:N be the optimal states visited by the expert agent, where M is the number of demonstration episodes, and N is the number of steps within each episode. We estimate an appropriate reward signal based on the expert state trajectories τ, which in turn is used to guide a reinforcement learning algorithm and learn a suitable policy. We evaluate two approaches to implement this idea. A straightforward approach is to first train a generative model using the expert trajectories τ. Rewards can then be estimated based on similarity measures between a reconstructed state value and the actual currently experienced state value of the agent. This method constrains exploration to the states that have been demonstrated by an expert and enables learning a policy that closely matches the expert. However, in this approach, the temporal order of states are ignored or not readily accounted for. This temporal order of the next state in the sequence is important for estimating the state transition probability function. As such, the next approach we take is to consider a temporal sequence prediction model that can be trained to predict the next state value given current state, based on the expert trajectories. Once again the reward value can be estimated as a function of the similarity measure between the predicted next state and the one actually visited by the agent. The following sub-sections describes both these approaches in detail. We train a deep generative model (three-layered fully connected auto-encoder) using the state values s i t for each step number t, sampled from the expert agent trajectories τ. The generative model is trained to minimize the following reconstruction loss (maximize the likelihood of the training data): DISPLAYFORM0 where θ * g represents the optimum parameters of the generative model. Following typical settings, we assume p(s i t ; θ g) to be a Gaussian distribution, such that equation FORMULA2 DISPLAYFORM1 The reward value is estimated as a function of the difference between the actual state value s t+1 and the generated output g(s DISPLAYFORM2 where s t is the current state value, and ψ can be a linear or nonlinear function, typically hyperbolic tangent or gaussian function. In this formulation, if the current state is similar to the reconstructed state value, i.e. g(s t ; θ g), the estimated reward value will be higher. However, if the current state is not similar to the generated state, the reward value will be estimated to be low. Moreover, as a reward value is estimated at each time step, this approach can be used even in problems which originally had a highly sparse engineered reward structure. In this approach, we learn a temporal sequence prediction model (the specific networks used are mentioned in the corresponding experiments sections) such that we can maximize the likelihood of the next state given the current state. As such the network is trained using the following objective function, DISPLAYFORM0 where θ * h represents the optimal parameters of the prediction model. We also assume the probability of the next state given the previous state value, p(s i t+1 |s i t ; θ h) to be a Gaussian distribution. As such the objective function can be seen to be minimizing the mean square error, DISPLAYFORM1 DISPLAYFORM2 The estimated reward here, can also be interpreted akin to the generative model case. Here, if the agent's policy takes an action that changes the environment towards states far away from the expert trajectories, the corresponding estimated reward value is low. If the actions of agent bring it close to the expert demonstrated trajectories, thereby making the predicted next state match with the actual visited state value, the reward is estimated to be high. This process of reward shaping or guidance can enable the agent to learn a policy that is optimized based on the expert demonstration trajectories. Algorithm 1 explains the step by step flow of the proposed methods. Given trajectories τ from expert agent 3: DISPLAYFORM3 end for 6: end procedure 7: procedure REINFORCEMENT LEARNING 8: DISPLAYFORM4 Observe state s t Select/execute action a t, and observe state s t+1 11: DISPLAYFORM0 Update the deep reinforcement learning network using the tuple (s t, a t, r t, s t+1) end for 14: end procedure In order to evaluate our reward estimation methods, we conducted experiments across a range of environments. We consider five different tasks, namely: robot arm reaching task (reacher) to a fixed target position, robot arm reaching task to a random target position, controlling a point agent for reaching a target while avoiding an obstacle, learning an agent for longest duration of flight in the Flappy Bird video game, and learning an agent for maximizing the traveling distance in Super Mario Bros video game. We consider a 2-DoF robot arm in the x-y plane that has to learn to reach with the end-effector a point target. The first arm of the robot is a rigidly linked point, with the second arm linked to its edge. It has two joint values θ = (θ 1, θ 2), θ 1 ∈ (−∞, +∞), θ 2 ∈ [−π, +π] and the lengths of arms are 0.1 and 0.11 units, respectively. The robot arm was initialized by random joint values at the initial step for each episode. In the following experiments, we have two settings: fixed point target, and a random target. The applied continuous action values a t is used to control the joint angles, such that,θ = θ t − θ t−1 = 0.05 a t. Each action value has been clipped the range [−1, 1]. The reacher task is enabled using the physics engine within the roboschool environment BID3 BID16 ). Figure 1 describes the roboshool environment. The robot arms are in blue, the blue-green point is the end-effector, and the pink dot is the desired target location. In this experiment, the target point p tgt is always fixed at (0.1, 0.1). The state vector s t consists of the following values: absolute end position of first arm (p 2), joint value of elbow (θ 2), velocities of the joints (θ 1,θ 2), absolute target position (p tgt), and the relative end-effector position from target (p ee − p tgt). We used DDPG ) for this task, with the number of steps for each episode being 500 in this experiment 1. The reward functions used in this task were as follows:Dense reward: DISPLAYFORM0 Sparse reward: r t = − tanh(α p ee − p tgt 2) + r env t, DISPLAYFORM1 DISPLAYFORM2 GM with r env t: rGM (1000 episodes): DISPLAYFORM3 GM with action: DISPLAYFORM4 where r env t is the environment specific reward, which is calculated based on the cost for current action, − a t 2. This regularization is required for finding the shortest path to reach the target. As this cost is critical for fast convergence, we use this in all cases. The dense reward is a distance between end-effector and the target, and the sparse reward is based on a bonus for reaching. The generative model parameters θ 2k is trained by τ 2k trajectories that contains only states of 2000 episodes from an agent trained during 1k episodes with dense reward. The generative network has 400, 300 and 400 units fully-connected layers, respectively. They also have ReLU activation function, with the batch size being 16, and number of epochs being 50. θ 1k is trained from τ 1k trajectories that is randomly picked 1000 episodes from τ 2k. The GM with action uses a generative model θ 2k,+a that is trained pairs of state and action for 2000 episodes for same agents as τ 2k. We use a tanh nonlinear function for the estimated reward in order to keep a bounded value. The α, β change sensitiveness of distance or reward, were both set to 100 2. Here, we also compare our with behavior cloning (BC) method BID20 where the trained actor networks directly use obtained pairs of states and actions. DISPLAYFORM5 Figure 1: The environment of reacher task. The reacher has two arms, and objective of agent is reaching endeffector (green) to target point (red). FIG3 shows the difference in performance by using the different reward functions 3. All methods are evaluated by a score (y-axis), which was normalized to a minimum and a maximum value. The proposed method, especially "GM with r env t ", manages to achieve a score nearing that of the dense reward, with the performance being much better as compared to the sparse reward setting. Moreover, the learning curves based on the rewards estimated with the generative model show a faster convergence rate. However, the without environment specific reward, i.e. with the additional action regularization term, takes a longer time to converge. This is primarily because of the fact that GM reward is reflective of the distance between target and end-effector, and cannot directly account for the action regularization. The GM reward based on τ 1k underperforms as compared with GM reward based on τ 2k because of the lack of demonstration data. FIG4 shows the reward value of each end-effector point. GM estimated reward using τ 2k has better reward map as compared to GM estimated reward using τ 1k. However, these demonstrations data FIG11 ) are biased by the robot trajectories. Thus, a method of generating demonstration data that normalizes or avoids such bias will further improve the reward structure. If the demonstrations contain the action information in addition to state information, behavior cloning achieves good performance. Surprisingly, however, when using both state and action information in the generative model, "GM [state, action]", the performance of the agent is comparatively poor. In this experiment, the target point p tgt is initialized by a random uniform distribution of [−0.27, +0.27], that includes points outside of the reaching range of the robot arms. Furthermore, we removed the relative position of the target from the input state information. This makes the task more difficult. The state vector s t has the following values: p 2, θ 2,θ 1,θ 2, p tgt. In the previous experiment, the distribution of states in expert trajectories is expected to be similar to the reward structure due to a fixed target location. However, when the target position changes randomly, this distribution is not fixed. We, therefore, evaluate with the temporal sequence prediction model h(s t ; θ h) in this experiment. The RL setting is same as the previous experiment, however we changed the total number of steps within each episode to 400. The reward functions used in this experiment were calculated as follows: DISPLAYFORM0 Sparse reward: r t = tanh(−α p ee − p tgt 2) + r env t, DISPLAYFORM1 LSTM reward: r t = tanh(−γ s t+1 − h(s t:t-n ; θ lstm) 2 ) + r DISPLAYFORM2 The expert demonstrations τ were obtained using the states of 2000 episodes running a trained agent with dense hand-engineered reward. The GM estimated reward uses the same setting as in the previous experiment. NS is a model that predicts the next state given current state, and was trained using demonstration data τ 4. The LSTM model uses Long short-term memory BID7 ) as a temporal sequence prediction model. The state in reacher task does not contain time sequence data, hence we use a finite state history as input. LSTM model has three layers 5 and one fully-connected layer with 40 ReLU activation units. The forward model based reward estimation is based on predicting the next state given both the current state and action 6. Here, we also compared with the baseline behavior cloning method. In this experiment, α is 100, β is 1, and γ is 10. FIG3 shows the performance of the trained agent using the different reward functions. In all cases using estimated rewards performances significantly better than the sparse reward case. The LSTM based prediction method gives the best , reaching close to the performance obtained with dense hand engineered reward function. As expected, the GM based reward estimation fails to work well in this relatively complex experimental setting. The NS model estimated reward, which predicts next state given only the current state information, has comparable performance with LSTM based prediction model during the initial episodes. The FM based reward function also performs poorly in this experiment. Comparatively, the direct BC works relatively well. This indicates that it is better to use behavior cloning than reward prediction when both state and action information are available from demonstration data. Starting with this experiment, we evaluate using only the temporal sequence prediction method. As such, here we use a finite history of the state values in order to predict the next state value. We assume that predicting a part of the state that is related to a given action allows the model to make a better estimate of the reward function. Former work by predicts a function of the next state, φ(s t+1) rather than predicting the raw value s t+1, as in this paper. In this experiment, we also changed the non-linear function ψ in the proposed prediction method to a Gaussian function (as compared to the hyperbolic tangent function used in previous experiments). This allows us to compare the robustness of our proposed method for reward estimation to different non-linear functions. We develop a new environment that adds an obstacle to the reaching task. This reacher is a twodimensional point (x, y) that uses position control. In FIG5 we show the modified environment setup. The agent's goal is to reach the target while avoiding the obstacle in this case. The initial position of agent, the target position, and an obstacle position were initialized randomly. The state value contains the agent absolute position (p t), current velocity of the agent (ṗ t), a target absolute position (p tgt), an obstacle absolute position (p obs), and the relative location of target and obstacle with respect to the agent (p t −p tgt, p t −p obs). Once again the RL algorithm used in this experiment was DDPG ) for continuous control 7. The number of steps for each episode set to 500. Here, we used the following reward functions: DISPLAYFORM0 LSTM reward: DISPLAYFORM1 DISPLAYFORM2 where h (s t:t-n ; θ lstm) is a network that predicts a selected part of state values given a finite history of state information. The dense reward is composed of both, the target distance cost and the obstacle distance bonus. The optimal state trajectories τ contains 800 "human guided" demonstration data. In this case, the LSTM network consisted of two layers, each with 256 units with ReLU activations. In this experiment, σ 1 is 0.005, and σ 2 is 0.002 8. FIG6 shows the performance with the different estimated or hand-engineered reward settings. The LSTM based prediction method learns to reach the target faster than the dense reward, while LSTM (s) has the best overall performance by learning with human-guided demonstration data. The agent (green) will move to reach the target (red), while avoiding the obstacle (pink). DISPLAYFORM3 We use a re-implementation BID10 ) of Android game, "Flappy Bird", in python (pygame). The objective of this game is to pass through the maximum number of pipes without collision. The control is a single discrete command of whether to flap the bird wings or not. The state has four consecutive gray frames (4 x 80 x 80). The RL is trained by DQN BID12 ) 9, and the update frequency of deep network is 100 steps. The used rewards are, DISPLAYFORM0 if pass through a pipe; −1 if collide to a pipe. LSTM reward: DISPLAYFORM1 which s t+1 is an absolute position of the bird, which can be given from simulator or it could be processed by pattern matching or CNN from raw images, h (s t ; θ lstm) is a predicted the absolute position. Hence, LSTM is trained for predicting absolute position of bird location given images. The τ of this experiment is 10 episodes data from a trained agent in the repository by BID10. Also, we also compared with the baseline behavior cloning method. In this experiment, σ is 0.02. FIG8 shows the of LSTM reward is better than normal "hand-crafted" reward. The reason for this situation is, the normal dense reward just describes the traveling distance, but our LSTM reward will teach which absolute transition of bird is good. Also, LSTM has better convergence than BC ; the reason is the number of demonstrations is not enough for behavior cloning method. In the final task, we consider a more complex environment in order to evaluate our proposed reward estimation method using only state information. Here we use the Super Mario Bros classic Nintendo video game environment . Our proposed method estimates reward values based on expert gameplay video data (using only the state information in the form of image frames).In this experiment, we also benchmarked against the recently proposed curiosity-based method ) using the implementation provided by the same authors BID17 ). This was used as the baseline reinforcement learning technique. Unlike in the actual game, here we always initialize Mario to the starting position rather than a previously saved checkpoint. This is a discrete control setup, where, Mario can make 14 types of actions 10. The state information consists of sequential input of four 42 x 42 gray image frames 11. Here we used the A3C RL algorithm BID13 ). We used gameplay of stage "1-1" for this experiment, with the objective of the agent being to travel as far as possible and achieve as high a score as possible. The rewards functions used in this experiment were as follows: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 Curiosity DISPLAYFORM3 DISPLAYFORM4 where position t is the current position of Mario at time t, score t is the current score value at time t, and s t are screen images from the Mario game at time t. The position and score information are obtained using the Mario game emulator. In this experiment, we use a three-dimensional convolutional neural network BID8 ) (3D-CNN) for our temporal sequence prediction method. In order to capture expert demonstration data, we took 15 game playing videos by five different people 12. In total, the demonstration data consisted of 25000 frames. The length of skipped frames in input to the temporal sequence prediction model was 36, as humans cannot play as fast as an RL agent; however, we did not change the skip frame rate for the RL agent. The 3D-CNN consisted of 4 layers 13 and a final layer to reconstruct the image. The agent was trained using 50 epochs with a batch size of 8. We implemented two prediction methods for reward estimation. In the naïve method the Mario agent will end up getting positive rewards if it sits in a fixed place without moving. This is because it can avoid dying by just not moving. However, this is clearly a trivial suboptimal policy. Hence, we implemented the alternate reward function based on the same temporal sequence prediction model, but we apply a threshold value that prevents the agent from converging onto such a trivial solution. Here, the value of ζ is 0.025, which was calculated based on the reward value obtained by just staying fixed at the initial position. FIG10 shows the performance with the different reward functions. Here, the graphs directly show the average over multiple trials. As observed, the agent was unable to reach large distances even while using "hand-crafted" dense rewards and did not converge to the goal every time 14; this behavior was also observed by for their reward case. As observed from the average curves of FIG10, the proposed 3D-CNN method learns relatively faster as compared to the curiosity-based agent ). As expected the 3D-CNN (naïve) method converged to a solution of remaining fixed at the initial state. As future work, we hope to improve the performance in this game setting using deeper RL networks, as well as large input image sizes. Overall estimating reward from φ(s t) without the need of action data, allows an agent to learn suitable policy directly from raw video data. The abundance of visual data creates ample opportunity for this type of reward estimation method to be explored further in different video game settings.10 A single action is repeated for six consecutive frames. Please refer to ) for details. 11 Every next six frames were skipped. 12 All videos consisted of games where the player succeeded in clearing the stage. 13 Two layers with (2 x 5 x 5), two layers with (2 x 3 x 3) kernels, all have 32 filters, and every two layers with stride.14 By our experiment, even if it trained long steps, such as 3.0M; it just reached around 600 -700 averagely. In this paper, we proposed two variations of a reward estimation method via state prediction by using state-only trajectories of the expert; one based on an autoencoder-based generative model and one based on temporal sequence prediction using LSTM. Both the models were for calculating similarities between actual states and predicted states. We compared the methods with conventional reinforcement learning methods in five various environments. As overall trends, we found that the proposed method converged faster than using hand-crafted reward in many cases, especially when the expert trajectories were given by humans, and also that the temporal sequence prediction model had better than the generative model. It was also shown that the method could be applied to the case where the demonstration was given by videos. However, detailed trends were different for the different environments depending on the complexity of the tasks. Neither model of our proposed method was versatile enough to be applicable to every environment without any changes of the reward definition. As we saw in the necessity of the energy term of the reward for the reacher task and in the necessity of special handling of the initial position of Mario, the proposed method has a room of improvements especially in modeling global temporal characteristics of trajectories. We would like to tackle these problems as future work. The DDPG's actor network has 400 and 300 unites fully-connected (fc) layers, the critic network has also 400 and 300 fully-connected layers, and each layer has a ReLU (Nair & Hinton FORMULA1) activation function. We put the tanh activation function at the final layer of actor network. Without this modification, the normal RL takes a long time to converge. Also, initial weights will be set from a uniform distribution U (−0.003, 0.003). The exploration policy is Ornstein-Uhlenbeck process BID25 ) (θ = 0.15, µ = 0, σ = 0.01), size of reply memory is 1M, and optimizer is Adam (Kingma & Ba FORMULA1). We implemented these experiments by Keras-rl , Keras BID4 ), and Tensorflow BID0 ) libraries.: These are scatter-plots of end-effector positions (blue) for each state of captured demonstration τ 500, τ 1k, τ 2k, each point is drawn by α is 0.01. And the fixed target position is also plotted (red). Notes that this is just plotting end-effector position, there is more variation in other state values. For example, even if the end-effector position were same, arms' pose (joint values) might be different. Note that τ 500 is not used in the experiment. The DDPG's actor network has 64 and 64 unites fully-connected layers, the critic network has also 64 and 64 fully-connected layers, and each layer has a ReLU activation function. Initial weights will be set from a uniform distribution U (−0.003, 0.003). The exploration policy is Ornstein-Uhlenbeck process BID25 ) (θ = 0.15, µ = 0, σ = 0.01), size of reply memory is 500k, and optimizer is Adam., a positive would be Reward Estimation from Game Videos & a negative would be The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.

Question: Transferring representations from large-scale supervised tasks to downstream tasks have shown outstanding in Machine Learning in both Computer Vision and natural language processing (NLP). One particular example can be sequence-to-sequence models for Machine Translation (Neural Machine Translation - NMT). It is because, once trained in a multilingual setup, NMT systems can translate between multiple languages and are also capable of performing zero-shot translation between unseen source-target pairs at test time. In this paper, we first investigate if we can extend the zero-shot transfer capability of multilingual NMT systems to cross-lingual NLP tasks (tasks other than MT, e.g. sentiment classification and natural language inference). We demonstrate a simple framework by reusing the encoder from a multilingual NMT system, a multilingual Encoder-Classifier, achieves remarkable zero-shot cross-lingual classification performance, almost out-of-the-box on three downstream benchmark tasks - Amazon Reviews, Stanford sentiment treebank (SST) and Stanford natural language inference (SNLI). In order to understand the underlying factors contributing to this finding, we conducted a series of analyses on the effect of the shared vocabulary, the training data type for NMT models, classifier complexity, encoder representation power, and model generalization on zero-shot performance. Our provide strong evidence that the representations learned from multilingual NMT systems are widely applicable across languages and tasks, and the high, out-of-the-box classification performance is correlated with the generalization capability of such systems. Here, we first describe the model and training details of the base multilingual NMT model whose 135 encoder is reused in all other tasks. Then we provide details about the task-specific classifiers. For 136 each task, we provide the specifics of f pre, f pool and f post nets that build the task-specific classifier. All the models in our experiments are trained using the Adam optimizer with label smoothing. Unless otherwise stated below, layer normalization is applied to all LSTM gates and 139 feed-forward layer inputs. We apply L2 regularization to the model weights and dropout to layer 140 activations and sub-word embeddings. Hyper-parameters, such as mixing ratio λ of L2 regularization, 141 dropout rates, label smoothing uncertainty, batch sizes, learning rate of optimizers and initialization 142 ranges of weights are tuned on the development sets provided for each task separately. NMT Models. Our multilingual NMT model consists of a shared multilingual encoder and two 144 decoders, one for English and the other for French. The multilingual encoder uses one bi-directional respectively. We used max-pooling operator for the f pool network to pool activation over time. Multilingual SNLI. We extended the proposed multilingual Encoder-Classifier model to a multi-160 source model since SNLI is an inference task of relations between two input sentences, "premise" and "hypothesis". For the two sources, we use two separate encoders, which are initialized with 162 the same pre-trained multilingual NMT encoder, to obtain their representations. Following our 163 notation, the encoder outputs are processed using f pre, f pool and f post nets, again with two separate 164 network blocks. Specifically, f pre consists of a co-attention layer [62.60 BiCVM BID3 59.03 RANDOM BID5 63.21 RATIO BID5 58.64The Amazon Reviews and SNLI tasks have a French test set available, and we evaluate the perfor- In this section, we try to analyze why our simple multilingual Encoder-Classifier system is effective 208 at zero-shot classification. We perform a series of experiments to better understand this phenomenon. In particular, we study the effect of shared sub-word vocabulary, the amount of multilingual and French, the out-of-vocabulary (OOV) rate for the German test set using our vocabulary is just 219 0.078%. We design this experiment as a control to understand the effect of having a shared sub-word 220 for the proposed system to model a language agnostic representation (interlingua) which enables it to 252 perform better zero-shot classification. Moreover, it should be noted that best zero-shot performance 253 is obtained by using the complex classifier and up to layer 3 of the encoder. Although this gap is not 254 big enough to be significant, we hypothesize that top layer of the encoder could be very specific to 255 the MT task and hence might not be best suited for zero-shot classification. Effect of Early vs Late Phases of the Training. Figure 1 shows that as the number of training 257 steps increases, the test accuracy goes up whereas the test loss on the SNLI task increases slightly, hinting at over-fitting on the English task. As expected, choosing checkpoints which are before the 259 onset of the over-fitting seems to benefit zero-shot performance on the French SNLI test set. This suggests that over-training on the English task might hurt the ability of the model to generalize to a 261 new language and also motivated us to conduct the next set of analysis. which aims to smooth point estimates of the learned parameters by averaging n steps from the training 264 run and using it for inference. This is aimed at improving generalization and being less susceptible to 265 the effects of over-fitting at inference. We hypothesize that a system with enhanced generalization 266 might be better suited for zero-shot classification since it is a measure of the ability of the model to 267 generalize to a new task. learning multilingual representations from a set of bilingual lexical data. Here we combined the best of both worlds by learning contextualized representations which are 309 multilingual in nature and explored its performance in the zero-shot classification tasks. We demon-310 strated that using the encoder from a multilingual NMT system as a pre-trained component in other TAB10 summarizes the accuracy of our proposed system for these three different approaches and French tasks, freezing the encoder after initialization significantly improves the performance further. We hypothesize that since the Amazon dataset is a document level classification task, the long input 498 sequences are very different from the short sequences consumed by the NMT system, and hence 499 freezing the encoder seems to have a positive effect. This hypothesis is also supported by the SNLI 500 and SST , which contain sentence-level input sequences, where we did not find any significant 501 difference between freezing and not freezing the encoder.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Zero-shot cross-lingual transfer by using multilingual neural machine translation

A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.


Question: Represent the article for finding a single-sentence TLDR

Question: This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Discovering state-of-the-art neural network architectures requires substantial effort of human experts. Recently, there has been a growing interest in developing algorithmic solutions to automate the manual process of architecture design. The automatically searched architectures have achieved highly competitive performance in tasks such as image classification BID35 BID36 BID13 a; BID26 and object detection BID36.The best existing architecture search algorithms are computationally demanding despite their remarkable performance. For example, obtaining a state-of-the-art architecture for CIFAR-10 and ImageNet required 2000 GPU days of reinforcement learning (RL) BID36 or 3150 GPU days of evolution BID26. Several approaches for speeding up have been proposed, such as imposing a particular structure of the search space BID13 a), weights or performance prediction for each individual architecture and weight sharing/inheritance across multiple architectures BID0 BID24; ), but the fundamental challenge of scalability remains. An inherent cause of inefficiency for the dominant approaches, e.g. based on RL, evolution, MCTS BID20, SMBO BID12 or Bayesian optimization BID9, is the fact that architecture search is treated as a black-box optimization problem over a discrete domain, which leads to a large number of architecture evaluations required. In this work, we approach the problem from a different angle, and propose a method for efficient architecture search called DARTS (Differentiable ARchiTecture Search). Instead of searching over a discrete set of candidate architectures, we relax the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. The data efficiency of gradient-based optimization, as opposed to inefficient black-box search, allows DARTS to achieve competitive performance with the state of the art using orders of magnitude less computation resources. It also outperforms another recent efficient architecture search method, ENAS BID24. Notably, DARTS is simpler than many existing approaches as it does not involve controllers BID35; BID36 BID24 BID33, hypernetworks or performance predictors BID12 ), yet it is generic enough handle both convolutional and recurrent architectures. The idea of searching architectures within a continuous domain is not new BID27; BID30 BID28, but there are several major distinctions. While prior works seek to fine-tune a specific aspect of an architecture, such as filter shapes or branching patterns in a convolutional network, DARTS is able to learn high-performance architecture building blocks with complex graph topologies within a rich search space. Moreover, DARTS is not restricted to any specific architecture family, and is applicable to both convolutional and recurrent networks. In our experiments (Sect. 3) we show that DARTS is able to design a convolutional cell that achieves 2.76 ± 0.09% test error on CIFAR-10 for image classification using 3.3M parameters, which is competitive with the state-of-the-art by regularized evolution BID26 obtained using three orders of magnitude more computation resources. The same convolutional cell also achieves 26.7% top-1 error when transferred to ImageNet (mobile setting), which is comparable to the best RL method BID36. On the language modeling task, DARTS efficiently discovers a recurrent cell that achieves 55.7 test perplexity on Penn Treebank (PTB), outperforming both extensively tuned LSTM BID17 and all the existing automatically searched cells based on NAS BID35 and ENAS BID24.Our contributions can be summarized as follows:• We introduce a novel algorithm for differentiable network architecture search based on bilevel optimization, which is applicable to both convolutional and recurrent architectures.• Through extensive experiments on image classification and language modeling tasks we show that gradient-based architecture search achieves highly competitive on CIFAR-10 and outperforms the state of the art on PTB. This is a very interesting , considering that so far the best architecture search methods used non-differentiable search techniques, e.g. based on RL BID36 or evolution BID26 BID13 ).• We achieve remarkable efficiency improvement (reducing the cost of architecture discovery to a few GPU days), which we attribute to the use of gradient-based optimization as opposed to non-differentiable search techniques.• We show that the architectures learned by DARTS on CIFAR-10 and PTB are transferable to ImageNet and WikiText-2, respectively. The implementation of DARTS is available at https://github.com/quark0/darts 2 DIFFERENTIABLE ARCHITECTURE SEARCH We describe our search space in general form in Sect. 2.1, where the computation procedure for an architecture (or a cell in it) is represented as a directed acyclic graph. We then introduce a simple continuous relaxation scheme for our search space which leads to a differentiable learning objective for the joint optimization of the architecture and its weights (Sect. 2.2). Finally, we propose an approximation technique to make the algorithm computationally feasible and efficient (Sect. 2.3). Following BID36; BID26; BID12 BID25, we search for a computation cell as the building block of the final architecture. The learned cell could either be stacked to form a convolutional network or recursively connected to form a recurrent network. A cell is a directed acyclic graph consisting of an ordered sequence of N nodes. Each node x (i) is a latent representation (e.g. a feature map in convolutional networks) and each directed edge (i, j) is associated with some operation o (i,j) that transforms x (i). We assume the cell to have two input nodes and a single output node. For convolutional cells, the input nodes are defined as the cell outputs in the previous two layers BID36. For recurrent cells, these are defined as the input at the current step and the state carried from the previous step. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. Each intermediate node is computed based on all of its predecessors: A special zero operation is also included to indicate a lack of connection between two nodes. The task of learning the cell therefore reduces to learning the operations on its edges. DISPLAYFORM0 Let O be a set of candidate operations (e.g., convolution, max pooling, zero) where each operation represents some function o(·) to be applied to x (i). To make the search space continuous, we relax the categorical choice of a particular operation to a softmax over all possible operations: DISPLAYFORM0 where the operation mixing weights for a pair of nodes (i, j) are parameterized by a vector α (i,j) of dimension |O|. The task of architecture search then reduces to learning a set of continuous variables α = α (i,j), as illustrated in FIG0. At the end of search, a discrete architecture can be obtained by replacing each mixed operationō (i,j) with the most likely operation, i.e., o (i,j) = argmax o∈O α (i,j) o. In the following, we refer to α as the (encoding of the) architecture. After relaxation, our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters). Analogous to architecture search using RL BID35 BID36 BID24 or evolution BID13 BID26 where the validation set performance is treated as the reward or fitness, DARTS aims to optimize the validation loss, but using gradient descent. Denote by L train and L val the training and the validation loss, respectively. Both losses are determined not only by the architecture α, but also the weights w in the network. The goal for architecture search is to find α * that minimizes the validation loss L val (w *, α *), where the weights w * associated with the architecture are obtained by minimizing the training loss w * = argmin w L train (w, α *).This implies a bilevel optimization problem with α as the upper-level variable and w as the lower-level variable: DISPLAYFORM1 DISPLAYFORM2 The nested formulation also arises in gradient-based hyperparameter optimization BID16 BID22 BID2, which is related in a sense that the architecture α could be viewed as a special type of hyperparameter, although its dimension is substantially higher than scalar-valued hyperparameters such as the learning rate, and it is harder to optimize. Create a mixed operationō DISPLAYFORM0 (ξ = 0 if using first-order approximation) 2. Update weights w by descending ∇ w L train (w, α) Derive the final architecture based on the learned α. Evaluating the architecture gradient exactly can be prohibitive due to the expensive inner optimization. We therefore propose a simple approximation scheme as follows: DISPLAYFORM0 where w denotes the current weights maintained by the algorithm, and ξ is the learning rate for a step of inner optimization. The idea is to approximate w * (α) by adapting w using only a single training step, without solving the inner optimization (equation 4) completely by training until convergence. Related techniques have been used in meta-learning for model transfer BID1, gradientbased hyperparameter tuning BID15 and unrolled generative adversarial networks BID19. Note equation 6 will reduce to ∇ α L val (w, α) if w is already a local optimum for the inner optimization and thus ∇ w L train (w, α) = 0.The iterative procedure is outlined in Alg. 1. While we are not currently aware of the convergence guarantees for our optimization algorithm, in practice it is able to reach a fixed point with a suitable choice of ξ 1. We also note that when momentum is enabled for weight optimisation, the one-step unrolled learning objective in equation 6 is modified accordingly and all of our analysis still applies. Applying chain rule to the approximate architecture gradient (equation 6) yields DISPLAYFORM1 where w = w − ξ∇ w L train (w, α) denotes the weights for a one-step forward model. The expression above contains an expensive matrix-vector product in its second term. Fortunately, the complexity can be substantially reduced using the finite difference approximation. Let be a small scalar 2 and DISPLAYFORM2 Evaluating the finite difference requires only two forward passes for the weights and two backward passes for α, and the complexity is reduced from O(|α||w|) to O(|α| + |w|).First-order Approximation When ξ = 0, the second-order derivative in equation 7 will disappear. In this case, the architecture gradient is given by ∇ α L val (w, α), corresponding to the simple heuristic of optimizing the validation loss by assuming the current w is the same as w * (α). This leads to some speed-up but empirically worse performance, according to our experimental in TAB0. In the following, we refer to the case of ξ = 0 as the first-order approximation, and refer to the gradient formulation with ξ > 0 as the second-order approximation. To form each node in the discrete architecture, we retain the top-k strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes. The strength of an operation is defined as DISPLAYFORM0. To make our derived architecture comparable with 1 A simple working strategy is to set ξ equal to the learning rate for w's optimizer. 2 We found = 0.01/ ∇ w L val (w, α) 2 to be sufficiently accurate in all of our experiments. DISPLAYFORM1. The analytical solution for the corresponding bilevel optimization problem is (α *, w *) =, which is highlighted in the red circle. The dashed red line indicates the feasible set where constraint equation 4 is satisfied exactly (namely, weights in w are optimal for the given architecture α). The example shows that a suitable choice of ξ helps to converge to a better local optimum. those in the existing works, we use k = 2 for convolutional cells BID36 BID12 BID26 and k = 1 for recurrent cells BID24.The zero operations are excluded in the above for two reasons. First, we need exactly k non-zero incoming edges per node for fair comparison with the existing models. Second, the strength of the zero operations is underdetermined, as increasing the logits of zero operations only affects the scale of the ing node representations, and does not affect the final classification outcome due to the presence of batch normalization BID8. Our experiments on CIFAR-10 and PTB consist of two stages, architecture search (Sect. 3.1) and architecture evaluation (Sect. 3.2). In the first stage, we search for the cell architectures using DARTS, and determine the best cells based on their validation performance. In the second stage, we use these cells to construct larger architectures, which we train from scratch and report their performance on the test set. We also investigate the transferability of the best cells learned on CIFAR-10 and PTB by evaluating them on ImageNet and WikiText-2 (WT2) respectively. We include the following operations in O: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, 3 × 3 average pooling, identity, and zero. All operations are of stride one (if applicable) and the convolved feature maps are padded to preserve their spatial resolution. We use the ReLU-Conv-BN order for convolutional operations, and each separable convolution is always applied twice BID36 BID26 BID12.Our convolutional cell consists of N = 7 nodes, among which the output node is defined as the depthwise concatenation of all the intermediate nodes (input nodes excluded). The rest of the setup follows BID36; BID12 BID26, where a network is then formed by stacking multiple cells together. The first and second nodes of cell k are set equal to the outputs of cell k − 2 and cell k − 1, respectively, and 1 × 1 convolutions are inserted as necessary. Cells located at the 1/3 and 2/3 of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two. The architecture encoding therefore is (α normal, α reduce), where α normal is shared by all the normal cells and α reduce is shared by all the reduction cells. Detailed experimental setup for this section can be found in Sect. A.1.1. Our set of available operations includes linear transformations followed by one of tanh, relu, sigmoid activations, as well as the identity mapping and the zero operation. The choice of these candidate operations follows BID35; BID24. Each architecture snapshot is re-trained from scratch using the training set (for 100 epochs on CIFAR-10 and for 300 epochs on PTB) and then evaluated on the validation set. For each task, we repeat the experiments for 4 times with different random seeds, and report the median and the best (per run) validation performance of the architectures over time. As references, we also report the (under the same evaluation setup; with comparable number of parameters) of the best existing cells discovered using RL or evolution, including NASNet-A (2000 GPU days), AmoebaNet-A (3150 GPU days) BID26 and ENAS (0.5 GPU day) BID24.function, as done in the ENAS cell BID24. The rest of the cell is learned. Other settings are similar to ENAS, where each operation is enhanced with a highway bypass BID34 and the cell output is defined as the average of all the intermediate nodes. As in ENAS, we enable batch normalization in each node to prevent gradient explosion during architecture search, and disable it during architecture evaluation. Our recurrent network consists of only a single cell, i.e. we do not assume any repetitive patterns within the recurrent architecture. Detailed experimental setup for this section can be found in Sect. A.1.2. To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for a short period (100 epochs on CIFAR-10 and 300 epochs on PTB). This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive FIG2.To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set. We note the test set is never used for architecture search or architecture selection. Detailed experimental setup for architecture evaluation on CIFAR-10 and PTB can be found in Sect. A.2.1 and Sect. A.2.2, respectively. Besides CIFAR-10 and PTB, we further investigated the transferability of our best convolutional cell (searched on CIFAR-10) and recurrent cell (searched on PTB) by evaluating them on ImageNet (mobile setting) and WikiText-2, respectively. More details of the transfer learning experiments can be found in Sect. A.2.3 and Sect. A.2.4. BID33 3.54 39.8 96 8 RL AmoebaNet-A BID26 3.34 ± 0.06 3.2 3150 19 evolution AmoebaNet-A + cutout BID26 ) † 3.12 3.1 3150 19 evolution AmoebaNet-B + cutout BID26 2.55 ± 0.05 2.8 3150 19 evolution Hierarchical evolution BID13 3.75 ± 0.12 15.7 300 6 evolution PNAS BID12 3.41 ± 0.09 3.2 225 8 SMBO ENAS + cutout BID24 2.89 4.6 0.5 6 RL ENAS + cutout BID24 3.00 ± 0.14 3.3 1.5 7 gradient-based DARTS (second order) + cutout 2.76 ± 0.09 3.3 4 7 gradient-based * Obtained by repeating ENAS for 8 times using the code publicly released by the authors. The cell for final evaluation is chosen according to the same selection protocol as for DARTS. † Obtained by training the corresponding architectures using our setup. ‡ Best architecture among 24 samples according to the validation error after 100 training epochs. Table 2: Comparison with state-of-the-art language models on PTB (lower perplexity is better). Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (3 GPU days). BID18 60.7 58.8 24 --manual LSTM + skip connections BID17 60.9 58.3 24 --manual LSTM + 15 softmax experts BID31 BID23 publicly released by the authors. † Obtained by training the corresponding architecture using our setup. ‡ Best architecture among 8 samples according to the validation perplexity after 300 training epochs. BID36 27.2 8.7 5.3 488 2000 RL NASNet-C BID36 27.5 9.0 4.9 558 2000 RL AmoebaNet-A BID26 25.5 8.0 5.1 555 3150 evolution AmoebaNet-B BID26 26.0 8.5 5.3 555 3150 evolution AmoebaNet-C BID26 24.3 7.6 6.4 570 3150 evolution PNAS BID12 25. The CIFAR-10 for convolutional architectures are presented in TAB0. Notably, DARTS achieved comparable with the state of the art BID36 BID26 while using three orders of magnitude less computation resources (i.e. 1.5 or 4 GPU days vs 2000 GPU days for NASNet and 3150 GPU days for AmoebaNet). Moreover, with slightly longer search time, DARTS outperformed ENAS BID24 ) by discovering cells with comparable error rates but less parameters. The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is less important for convolutional cells however, because the performance of discovered architectures does not strongly depend on initialization FIG2.Alternative Optimization Strategies To better understand the necessity of bilevel optimization, we investigated a simplistic search strategy, where α and w are jointly optimized over the union of the training and validation sets using coordinate descent. The ing best convolutional cell (out of 4 runs) yielded 4.16 ± 0.16% test error using 3.1M parameters, which is worse than random search. In the second experiment, we optimized α simultaneously with w (without alteration) using SGD, again over all the data available (training + validation). The ing best cell yielded 3.56 ± 0.10% test error using 3.0M parameters. We hypothesize that these heuristics would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization. Note that α is not directly optimized on the training set in DARTS. Table 2 presents the for recurrent architectures on PTB, where a cell discovered by DARTS achieved the test perplexity of 55.7. This is on par with the state-of-the-art model enhanced by a mixture of softmaxes BID31, and better than all the rest of the architectures that are either manually or automatically discovered. Note that our automatically searched cell outperforms the extensively tuned LSTM BID17, demonstrating the importance of architecture search in addition to hyperparameter search. In terms of efficiency, the overall cost (4 runs in total) is within 1 GPU day, which is comparable to ENAS and significantly faster than NAS BID35.It is also interesting to note that random search is competitive for both convolutional and recurrent models, which reflects the importance of the search space design. Nevertheless, with comparable or less search cost, DARTS is able to significantly improve upon random search in both cases (2.76 ± 0.09 vs 3.29 ± 0.15 on CIFAR-10; 55.7 vs 59.4 on PTB). TAB4 show that the cell learned on CIFAR-10 is indeed transferable to ImageNet. It is worth noticing that DARTS achieves competitive performance with the state-of-the-art RL method BID36 while using three orders of magnitude less computation resources. TAB6 shows that the cell identified by DARTS transfers to WT2 better than ENAS, although the overall are less strong than those presented in Table 2 for PTB. The weaker transferability between PTB and WT2 (as compared to that between CIFAR-10 and ImageNet) could be explained by the relatively small size of the source dataset (PTB) for architecture search. The issue of transferability could potentially be circumvented by directly optimizing the architecture on the task of interest. BID4 -68.9 --manual LSTM BID18 69.1 66.0 33 -manual LSTM + skip connections BID17 69.1 65.9 24 -manual LSTM + 15 softmax experts BID31 66 We presented DARTS, a simple yet efficient architecture search algorithm for both convolutional and recurrent networks. By searching in a continuous space, DARTS is able to match or outperform the state-of-the-art non-differentiable architecture search methods on image classification and language modeling tasks with remarkable efficiency improvement by several orders of magnitude. There are many interesting directions to improve DARTS further. For example, the current method may suffer from discrepancies between the continuous architecture encoding and the derived discrete architecture. This could be alleviated, e.g., by annealing the softmax temperature (with a suitable schedule) to enforce one-hot selection. It would also be interesting to explore performance-aware architecture derivation schemes based on the one-shot model learned during the search process. A EXPERIMENTAL DETAILS A.1 ARCHITECTURE SEARCH A.1.1 CIFAR-10Since the architecture will be varying throughout the search process, we always use batch-specific statistics for batch normalization rather than the global moving average. Learnable affine parameters in all batch normalizations are disabled during the search process to avoid rescaling the outputs of the candidate operations. To carry out architecture search, we hold out half of the CIFAR-10 training data as the validation set. A small network of 8 cells is trained using DARTS for 50 epochs, with batch size 64 (for both the training and validation sets) and the initial number of channels 16. The numbers were chosen to ensure the network can fit into a single GPU. We use momentum SGD to optimize the weights w, with initial learning rate η w = 0.025 (annealed down to zero following a cosine schedule without restart BID14), momentum 0.9, and weight decay 3 × 10 −4. We use zero initialization for architecture variables (the α's in both the normal and reduction cells), which implies equal amount of attention (after taking the softmax) over all possible ops. At the early stage this ensures weights in every candidate op to receive sufficient learning signal (more exploration). We use Adam BID10 as the optimizer for α, with initial learning rate η α = 3 × 10 −4, momentum β = (0.5, 0.999) and weight decay 10 −3. The search takes one day on a single GPU 3.A.1.2 PTB For architecture search, both the embedding and the hidden sizes are set to 300. The linear transformation parameters across all incoming operations connected to the same node are shared (their shapes are all 300 × 300), as the algorithm always has the option to focus on one of the predecessors and mask away the others. Tying the weights leads to memory savings and faster computation, allowing us to train the continuous architecture using a single GPU. Learnable affine parameters in batch normalizations are disabled, as we did for convolutional cells. The network is then trained for 50 epochs using SGD without momentum, with learning rate η w = 20, batch size 256, BPTT length 35, and weight decay 5 × 10 −7. We apply variational dropout BID3 of 0.2 to word embeddings, 0.75 to the cell input, and 0.25 to all the hidden nodes. A dropout of 0.75 is also applied to the output layer. Other training settings are identical to those in BID18; BID31. Similarly to the convolutional architectures, we use Adam for the optimization of α (initialized as zeros), with initial learning rate η α = 3 × 10 −3, momentum β = (0.9, 0.999) and weight decay 10 −3. The search takes 6 hours on a single GPU. A large network of 20 cells is trained for 600 epochs with batch size 96. The initial number of channels is increased from 16 to 36 to ensure our model size is comparable with other baselines in the literature (around 3M). Other hyperparameters remain the same as the ones used for architecture search. Following existing works BID24 BID36 BID12 BID26, additional enhancements include cutout , path dropout of probability 0.2 and auxiliary towers with weight 0.4. The training takes 1.5 days on a single GPU with our implementation in PyTorch BID21. Since the CIFAR are subject to high variance even with exactly the same setup BID13, we report the mean and standard deviation of 10 independent runs for our full model. To avoid any discrepancy between different implementations or training settings (e.g. the batch sizes), we incorporated the NASNet-A cell BID36 and the AmoebaNet-A cell BID26 into our training framework and reported their under the same settings as our cells.

We propose a differentiable architecture search algorithm for both convolutional and recurrent networks, achieving competitive performance with the state of the art using orders of magnitude less computation resources.

Represent the short summary for finding the corresponding scientific article (long):

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the article for finding a single-sentence TLDR

Question: Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops. To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass. To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our show that maximization is an inappropriate decoding objective for open-ended text generation, the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and Nucleus Sampling is the best decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text. On February 14th 2019, OpenAI surprised the scientific community with an impressively highquality article about Ovid's Unicorn, written by GPT-2. 1 Notably, the top-quality generations obtained from the model rely on randomness in the decoding method, in particular through top-k sampling that samples the next word from the top k most probable choices (; ;), instead of aiming to decode text that maximizes likelihood. In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2-117M, as shown in Figure 1. This may seem counter-intuitive, as one would expect that good models would assign higher probability to more human-like, grammatical text. Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward. Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling -sampling directly from the probabilities predicted by the model - in text that is incoherent and almost unrelated to the context. Why is text produced by pure sampling so degenerate? In this work we show that the "unreliable tail" is to blame. This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate. To overcome these shortcomings we introduce Nucleus Sampling (§3.1). The key intuition of Nucleus Sampling is that the vast majority of probability mass at each time step is concentrated in the nucleus, a small subset of the vocabulary that tends to range between one and a thousand candidates. Instead of relying on a fixed top-k, or using a temperature parameter to control the shape of the distribution without sufficiently suppressing the unreliable tail distribution, we propose sampling from the top-p portion of the probability mass, expanding and contracting the candidate pool dynamically. In order to compare current methods to Nucleus Sampling, we compare various distributional properties of generated text to the reference distribution, such as the likelihood of veering into repetition and the perplexity of generated text. The latter reveals that text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution. On the other hand, pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality. Vocabulary usage and Self-BLEU statistics reveal that high values of k are needed to make top-k sampling match human statistics. Yet, generations based on high values of k are also found to have incredibly high variance in likelihood, hinting at qualitatively observable incoherency issues. Nucleus Sampling can easily match reference perplexity through a proper value of p, without facing the ing incoherence caused by setting k high enough to match distributional statistics. Finally, we perform Human Unified with Statistical Evaluation (HUSE;) to jointly assess the overall quality and diversity of the decoding strategies, which cannot be captured using either human or automatics evaluation alone. The HUSE evaluation demonstrates that Nucleus sampling is the best overall decoding strategy. We include generated examples for qualitative analysis -see Figure 9 for a representative example, and further examples in the appendix. A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (; ;). Generative Adversarial Networks (GANs) have been a prominent research direction , but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (; ;). Work on neural dialog systems have proposed methods for diverse beam search, using a task-specific diversity scoring function or constraining beam hypotheses to be sufficiently different (a; ; ;). While such utility functions encourage desirable properties in generations, they do not remove the need to choose an appropriate decoding strategy, and we believe that Nucleus Sampling will have complementary advantages in such approaches. Many text generation tasks are defined through (input, output) pairs, such that the output is a constrained transformation of the input. Example applications include machine translation , data-to-text generation , and summarization (Nallapati et al., dsfsadfd An unprecedented number of mostly young whales have become stranded on the West Australian coast since 2008. Poor nutrition has led to a rise in the number of stranded humpback whales on the West Australian coast, veterinary researchers have said. Carly Holyoake, from Murdoch University, at the Australian Veterinary Association's annual conference in Perth on Wednesday, said an unprecedented number of mostly young whales had become stranded on the coast since 2008. There has been an unprecedented number of calves caught in the nets of whaling stations that operate in WA. Pilot whales continue to migrate to feeding grounds to feed their calves. They are now vulnerable due to the decline of wild populations; they are restricted to one breeding site each year. Image copyright Yoon Bo Kim But, with sharp decline in wild populations the size of the Petrels are shrinking and dwindling population means there will only be room for a few new fowl. Pumping Station #3 shut down due to construction damage Find more at: www.abc.net.au/environment/species-worry/ in-the-top-10-killer-whale-catastrophes-in-history.html "In the top 10 killer whale catastrophes in history: 1) 1986: Up to 12 orcas struck by lightning; many drowned and many more badly injured. Last week's intense storms and a series of powerful cyclones have been officially blamed for the deaths of at least nine large fin whales near Whitsundays -the largest loss of any species globally. The fin whales: packed in the belly of one killer whale thrashing madly in fear as another tries to bring it to safety. When the colossal animal breached the waters of Whitsundays, he'd been seen tagged for a decade. The Australian Food Safety Authority has warned Australia's beaches may be revitalised this year because healthy seabirds and seals have been on the move. More than 50,000 seabirds, sea mammals and seahorses have been swept into the sea by the Holden CS118 and Adelaide Airport CS300 from 2013. A major white-bat and umidauda migration across Australia is under way in Australia for the first time, with numbers reaching an estimated 50,000. The number of stranded whales has increased by more than 50 per cent in the past year, with the number of stranded whales on the West Australian coast increasing by more than 50 per cent in the past year. The number of whales stranded on the West Australian coast has increased by more than 50 per cent in the past year, with the number of stranded whales on the West Australian coast increasing by more than 50 per cent in the past year. Sampling, t=0.9 Pure Sampling The whale's fate was confirmed late last week when the animal was found by fishermen off the coast of Bundaberg. Experts believe the whale was struck by a fishing vessel off the coast of Bundaberg, and died after being sucked into the ocean. The whale's fate was confirmed late last week when the animal was found by fishermen off the coast of Bundaberg. Figure 2: Example generations from an initial sentence. Maximization and top-k truncation methods lead to copious repetition (highlighted in blue), while sampling with and without temperature tends to lead to incoherence (highlighted in red). Nucleus Sampling largely avoids both issues. All generations for all hyperparameters will be made publicly available, along with our codebase. 2016). We refer to these tasks as directed generation. Typically encoder-decoder architectures are used, often with an attention mechanisms or using attention-based architectures such as the Transformer . Generation is usually performed using beam search; since output is tightly scoped by the input, repetition and genericness are not as problematic. Still, similar issues have been reported when using large beam sizes , a counter-intuitive observation since large beam sizes help maximize probability. Open-ended generation, which includes conditional story generation and contextual text continuation (as in Figure 1), has recently become a promising research direction due to significant advances in neural language models (; ; ; ;). While the input context restricts the space of acceptable output generations, there is a considerable degree of freedom in what can plausibly come next, unlike in directed generation settings. Our work addresses the challenges faced by neural text generation with this increased level of freedom, but we note that some tasks, such as goal-oriented dialog, may fall somewhere in between open-ended and directed generation. Figure 3: The probability of a repeated phrase increases with each repetition, creating a positive feedback loop. We found this effect to hold for the vast majority of phrases we tested, regardless of phrase length or if the phrases were sampled randomly rather than taken from human text. Given an input text passage as context, the task of open-ended generation is to generate text that forms a coherent continuation from the given context. More formally, given a sequence of m tokens x 1... x m as context, the task is to generate the next n continuation tokens to obtain the completed sequence x 1... x m+n. We assume that models compute P (x 1:m+n) using the common left-to-right decomposition of the text probability, which is used to generate the generation token-by-token using a particular decoding strategy. Maximization-based decoding The most commonly used decoding objective, in particular for directed generation, is maximization-based decoding. Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest likelihood. Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable , common practice is to use beam search (b; ;). However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text . We propose a new stochastic decoding method: Nucleus Sampling. The key idea is to use the shape of the probability distribution to determine the set of tokens to be sampled from. Given a distribution P (x|x 1:i−1), we define its top-p vocabulary V (p) ⊂ V as the smallest set such that Let p = x∈V (p) P (x|x 1:i−1). The original distribution is re-scaled to a new distribution, from which the next word is sampled: In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold p. The size of the sampling set will adjust dynamically based on the shape of the probability distribution at each time step. For high values of p, it is a small subset of vocabulary that takes up most of the probability mass -the nucleus. Figure 4: The probability mass assigned to partial human sentences. Flat distributions lead to many moderately probable tokens, while peaked distributions concentrate most probability mass into just a few tokens. The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k problematic. Top-k Sampling has recently become a popular alternative sampling procedure (; ;). Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate. Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone. At each time step, the top k possible next tokens are sampled from according to their relative probabilities. Formally, given a distribution P (x|x 1:i−1), we define its top-k vocabulary V (k) ⊂ V as the set of size k which maximizes x∈V (k) P (x|x 1:i−1). Let p = x∈V (k) P (x|x 1:i−1). The distribution is then re-scaled as in equation 3, and sampling is performed based on that distribution. Note that the scaling factor p will be different at each time-step, in contrast to Nucleus Sampling. Difficulty in choosing a suitable value of k While top-k sampling leads to considerably higher quality text than either beam search or sampling from the full distribution, the use of a constant k is sub-optimal across varying contexts. As illustrated on the left of Figure 4, in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g. nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure. Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased through the truncation. Under Nucleus Sampling, the number of candidates considered rises and falls dynamically, corresponding to the changes in the model's confidence region over the vocabulary which top-k sampling fails to capture for any one choice of k. Another common approach to sampling-based generation is to shape a probability distribution through temperature . Temperature sampling has been applied widely to text generation (; ;). Given the logits u 1:|V | and temperature t, the softmax is re-estimated as Setting t ∈ shapes the distribution to be more skewed towards high probability events, which implicitly lowers the mass in the tail distribution. Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling . However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (; has been the most successful in the extremely large-scale training setups in recent literature (; . In this study we use the Generative Pre-trained Transformer, version 2 , which was trained on WebText, a 40GB collection of text scraped from the web. We perform experiments using the Large model (762M parameters). Our analysis is based on generating 5,000 text passages, which end upon reaching an end-of-document token or a maximum length of 200 tokens. We consider a conditional generation setting, in which texts are conditioned on the initial paragraph (restricted to 1-40 tokens) of documents in the validation portion of WebText. Our first evaluation is to compute the perplexity of generated text using various decoding strategies, according to the model that is being generated from. We compare these perplexities against that of the gold text (Figure 5). Importantly, we argue that the optimal generation strategy should produce text which has a perplexity close to that of the gold text: Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to be low-diverstiy and get stuck in repetition loops, as shown in §5 and illustrated in Figure 3. We see that perplexity of text obtained from pure sampling (sampling with t = 1) is actually worse than the perplexity of the gold. This indicates that the model is actually confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as happens in Figure 1. Yet setting the temperature lower creates diversity repetition issues, as we shall see in §5. Interestingly, even with the relatively fine-grained parameter sweep conducted, Nucleus Sampling gets closer perplexity in both the conditional and under conditional case, than any other method, as shown in Table 1. One might wonder if the issue with maximization search error is a, i.e., there are higher quality sentences to which the model assigns higher probability than to the decoded ones, beam search has just failed to find them. Yet Figure 5 shows that the per-token probability of natural text is, on average, much lower than text generated by beam search. Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but Note that beam search has unnaturally low perplexities. A similar effect is seen using a temperature of 0.7 with top-k as in both and?. Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but we shall see the first two face coherency issues when their parameters are set this high. more informative tokens. Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 3. Why is naturally existing human text not the most probable text? We conjecture that this is an intrinsic property of human language. Language models that assign probabilities one word at a time without context for the underlying document will have trouble capturing this effect. Grice's Maxims of Communication show that people optimize against stating the obvious. Thus, it is natural that making every word as predictable as possible will be disfavored. This makes solving the problem simply by training larger models or improving neural architectures using standard per-word learning objectives unlikely: models that are blindly predicting the future will be forced to predict the lowest common denominator, rather than information rich language. In order to compare generations to the reference text, we begin by analyzing their use of vocabulary. Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text. The Zipfian coefficient s can be used to compare the distribution in a given text to a theoretically perfect exponential curve, where s = 1 . Figure 6 shows the vocabulary distributions along with estimated Zipf coefficients for selected parameters of different decoding methods. As expected, pure sampling is the closest to the human distribution, followed by Nucleus Sampling. The visualization of the distribution shows that pure sampling slightly overestimates the use of rare words, likely one reason why pure sampling also has higher perplexity than human text. Furthermore, lower temperature sampling avoids sampling these rare words from the tail, which is why it has been used in some recent work . We follow previous work and compute Self-BLEU as a metric of diversity. Self-BLEU is calculated by computing the BLEU score of each generated document using all other generations in the evaluation set as references. Thus a lower Self-BLEU score implies higher diversity. Figure 7 shows that Self-BLEU largely follow that of the Zipfian distribution analysis as a diversity measure. It is worth noting that very high values of k and t are needed to get close to the reference distribution, though these in unnaturally high perplexity (§4). One attribute of text quality that we can quantify is repetition. Figure 8 shows that Nucleus Sampling and top-k sampling have the least repetition for reasonable parameter ranges. Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity). Further, all stochastic methods face repetition issues when their tuning parameters are set too low, which tends to overtruncate, mimicking greedy search. Therefore we conclude that only Nucleus Sampling satisfies all the distributional criteria for desirable generations. 6 HUMAN EVALUATION Statistical evaluations are unable to measure the coherence of generated text properly. While the metrics in previous sections gave us vital insights into the different decoding methods we compare, human evaluation is still required to get a full measure of the quality of the generated text. However, pure human evaluation does not take into account the diversity of the generated text; therefore we use HUSE to combine human and statistical evaluation. HUSE is computed by training a discriminator to distinguish between text drawn from the human and model distributions, Self-BLEU of Generations Over 5000 Documents Self-BLEU5 Figure 7: Self-BLEU calculated on the unconditional generations produced by stochastic decoding methods; lower Self-BLEU scores imply higher diversity. Horizontal blue and orange lines represent human self-BLEU scores. Note how common values of t ∈ [0.5, 1] and k ∈ in high self-similarity, whereas "normal" values of p ∈ [0.9, 1) closely match the human distribution of text. tokens. Concretely, a phrase (sequence of words with minimum length 2) must repeat at least three times at the end of the generation to be considered a repetition. Since all the methods described used different parameter ranges, we label points with the appropriate parameter value except for t and p. Since values of k greater than 100 are rarely used in practice and values of p are usually in [0.9, 1), it becomes clear that Nucleus sampling is far closer to the human distribution in its usual parameter range. Finally, temperatures lower than 0.9 severely effect repetition, making it problematic for use in long-form generation. based on only two features: The probability assigned by the language model, and human judgements of typicality of generations. Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements. The in Table 1 shows that Nucleus Sampling obtains the highest HUSE score, with Top-k sampling performing next best. 6.2 QUALITATIVE ANALYSIS Figure 9 shows representative example generations. Unsurprisingly, Beam Search gets stuck in a repetition loop it cannot escape. Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word "umidauda", apparently a species of bird. The generation produced by Nucleus Sampling isn't perfect -the model appears to get get confused that it was talking about whales, and begins talking about birds instead. Yet, Top-k Sampling immediately veers off into an unrelated event. When top-k sampling is combine with a temperature of 0.7, as is commonly done , the output devolves into repetition. This paper provided a deep analysis into the properties of the most common decoding methods for open-ended language generation. We have shown that likelihood maximizing decoding causes repetition and overly generic language usage, while sampling methods without truncation risk sampling from the low-confidence tail of a model's predicted distribution. Further, we proposed Nucleus Samplingas a solution that captures the region of confidence of language models effectively. In future work, we wish to dynamically characterize this region of confidence and include a more semantic utility function to guide the decoding process. We include another set of examples for further qualitative comparison. What's the best way to read this one? The blog post from the authors discusses the differences between how the marketing mix plays out in terms of visibility, trustworthiness, and engagement, among other things. In my experience, this difference isn't necessarily obvious; it's hard to explain to a layperson, but in all honesty it's probably the most important point to consider when reviewing social media. Where can I purchase my prototypes and printed books in the USA? Currently we have a sign for purchase available in the guide. Since we are based in the States we will be shipping to you by FedEx. Do you make books in America? We do! We love our folks in the US. We haven't had a chance to make a documentary so we love to share our love for our culture. How does the book publisher pay for the book and art? Why are your caustics superior? Purest Grades Available. All US Manufactured. No Asian Imports. Quality Control on all incoming and outgoing shipments. Most suppliers only supply tech grade product. What is difference between Food and Tech Grade Lye? Food grade has a lower parts per million of heavy metal impurities. 80% of our soap makers prefer our food grade product. Figure 9: More example generations from an initial tag line. All generations for all hyperparameters will be made publicly available, along with our codebase.

Current language generation systems either aim for high likelihood and devolve into generic repetition or miscalibrate their stochasticity—we provide evidence of both and propose a solution: Nucleus Sampling.

Artificial neural networks evolved the same structures present in the olfactory systems of flies and mice after being trained to classify odors