Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use parameter perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods allows to combine the best of both worlds. We demonstrate that both off- and on-policy methods benefit from this approach through experimental comparison of DQN, DDPG, and TRPO on high-dimensional discrete action environments as well as continuous control tasks. Exploration remains a key challenge in contemporary deep reinforcement learning (RL). Its main purpose is to ensure that the agent's behavior does not converge prematurely to a local optimum. Enabling efficient and effective exploration is, however, not trivial since it is not directed by the reward function of the underlying Markov decision process (MDP). Although a plethora of methods have been proposed to tackle this challenge in high-dimensional and/or continuous-action MDPs, they often rely on complex additional structures such as counting tables BID6, density modeling of the state space BID22, learned dynamics models BID0 BID29, or self-supervised curiosity BID23 ).An orthogonal way of increasing the exploratory nature of these algorithms is through the addition of temporally-correlated noise, for example as done in bootstrapped DQN BID20. Along the same lines, it was shown that the addition of parameter noise leads to better exploration by obtaining a policy that exhibits a larger variety of behaviors BID32 ). We discuss these related approaches in greater detail in Section 5. Their main limitation, however, is that they are either only proposed and evaluated for the on-policy setting with relatively small and shallow function approximators (Rückstieß et al., 2008) or disregard all temporal structure and gradient information (; BID17 BID28 . This paper investigates how parameter space noise can be effectively combined with off-the-shelf deep RL algorithms such as DQN BID19, DDPG BID18, and TRPO (b) to improve their exploratory behavior. Experiments show that this form of exploration is applicable to both high-dimensional discrete environments and continuous control tasks, using on-and off-policy methods. Our indicate that parameter noise outperforms traditional action space noise-based baselines, especially in tasks where the reward signal is extremely sparse. We consider the standard RL framework consisting of an agent interacting with an environment. To simplify the exposition we assume that the environment is fully observable. An environment is modeled as a Markov decision process (MDP) and is defined by a set of states S, a set of actions A, a distribution over initial states p(s 0), a reward function r: S × A → R, transition probabilities p(s t+1 |s t, a t), a time horizon T, and a discount factor γ ∈. We denote by π θ a policy parametrized by θ, which can be either deterministic, π: S → A, or stochastic, π: S → P(A). The agent's goal is to maximize the expected discounted return η(π θ) = E τ [T t=0 γ t r(s t, a t)], where τ = (s 0, a 0, . . ., s T) denotes a trajectory with s 0 ∼ p(s 0), a t ∼ π θ (a t |s t), and s t+1 ∼ p(s t+1 |s t, a t). Experimental evaluation is based on the undiscounted return E τ [T t=0 r(s t, a t)]. Off-policy RL methods allow learning based on data captured by arbitrary policies. This paper considers two popular off-policy algorithms, namely Deep Q-Networks (DQN, BID19) and Deep Deterministic Policy Gradients (DDPG, BID18).Deep Q-Networks (DQN) DQN uses a deep neural network as a function approximator to estimate the optimal Q-value function, which conforms to the Bellman optimality equation:Q(s t, a t) = r(s t, a t) + γ max a ∈A Q(s t+1, a).The policy is implicitly defined by Q as π(s t) = argmax a ∈A Q(s t, a). Typically, a stochasticgreedy or Boltzmann policy is derived from the Q-value function to encourage exploration, which relies on sampling noise in the action space. The Q-network predicts a Q-value for each action and is updated using off-policy data from a replay buffer. DDPG is an actor-critic algorithm, applicable to continuous action spaces. Similar to DQN, the critic estimates the Q-value function using off-policy data and the recursive Bellman equation: DISPLAYFORM0 where π θ is the actor or policy. The actor is trained to maximize the critic's estimated Q-values by back-propagating through both networks. For exploration, DDPG uses a stochastic policy of the form π θ (s t) = π θ (s t) + w, where w is either w ∼ N (0, σ 2 I) (uncorrelated) or w ∼ OU(0, σ 2) (correlated).2 Again, exploration is realized through action space noise. In contrast to off-policy algorithms, on-policy methods require updating function approximators according to the currently followed policy. In particular, we will consider Trust Region Policy Optimization (TRPO, Schulman et al. (2015a) ), an extension of traditional policy gradient methods (b) using the natural gradient direction BID24 BID13.Trust Region Policy Optimization (TRPO) TRPO improves upon REINFORCE (b) by computing an ascent direction that ensures a small change in the policy distribution. More specifically, TRPO solves the following constrained optimization problem: DISPLAYFORM0 where ρ θ = ρ π θ is the discounted state-visitation frequencies induced by π θ, A(s, a) denotes the advantage function estimated by the empirical return minus the baseline, and δ KL is a step size parameter which controls how much the policy is allowed to change per iteration. This work considers policies that are realized as parameterized functions, which we denote as π θ, with θ being the parameter vector. We represent policies as neural networks but our technique can be applied to arbitrary parametric models. To achieve structured exploration, we sample from a set of policies by applying additive Gaussian noise to the parameter vector of the current policy: θ = θ + N (0, σ 2 I). Importantly, the perturbed policy is sampled at the beginning of each episode and kept fixed for the entire rollout. For convenience and readability, we denote this perturbed policy as π:= π θ and analogously define π:= π θ.State-dependent exploration As pointed out by Rückstieß et al., there is a crucial difference between action space noise and parameter space noise. Consider the continuous action space case. When using Gaussian action noise, actions are sampled according to some stochastic policy, generating a t = π(s t) + N (0, σ 2 I). Therefore, even for a fixed state s, we will almost certainly obtain a different action whenever that state is sampled again in the rollout, since action space noise is completely independent of the current state s t (notice that this is equally true for correlated action space noise). In contrast, if the parameters of the policy are perturbed at the beginning of each episode, we get a t = π(s t). In this case, the same action will be taken every time the same state s t is sampled in the rollout. This ensures consistency in actions, and directly introduces a dependence between the state and the exploratory action taken. Perturbing deep neural networks It is not immediately obvious that deep neural networks, with potentially millions of parameters and complicated nonlinear interactions, can be perturbed in meaningful ways by applying spherical Gaussian noise. However, as recently shown by , a simple reparameterization of the network achieves exactly this. More concretely, we use layer normalization BID2 between perturbed layers.3 Due to this normalizing across activations within a layer, the same perturbation scale can be used across all layers, even though different layers may exhibit different sensitivities to noise. Adaptive noise scaling Parameter space noise requires us to pick a suitable scale σ. This can be problematic since the scale will strongly depend on the specific network architecture, and is likely to vary over time as parameters become more sensitive to noise as learning progresses. Additionally, while it is easy to intuitively grasp the scale of action space noise, it is far harder to understand the scale in parameter space. We propose a simple solution that resolves all aforementioned limitations in an easy and straightforward way. This is achieved by adapting the scale of the parameter space noise over time and relating it to the variance in action space that it induces. More concretely, we can define a distance measure between perturbed and non-perturbed policy in action space and adaptively increase or decrease the parameter space noise depending on whether it is below or above a certain threshold: DISPLAYFORM0 where α ∈ R >0 is a scaling factor and δ ∈ R >0 a threshold value. The concrete realization of d(·, ·) depends on the algorithm at hand and we describe appropriate distance measures for DQN, DDPG, and TRPO in Appendix C.Parameter space noise for off-policy methods In the off-policy case, parameter space noise can be applied straightforwardly since, by definition, data that was collected off-policy can be used. More concretely, we only perturb the policy for exploration and train the non-perturbed network on this data by replaying it. Parameter space noise for on-policy methods Parameter noise can be incorporated in an onpolicy setting, using an adapted policy gradient, as set forth by Rückstieß et al.. Policy gradient methods optimize DISPLAYFORM1 Given a stochastic policy π θ (a|s) with θ ∼ N (φ, Σ), the expected return can be expanded using likelihood ratios and the re-parametrization trick BID16 as DISPLAYFORM2 for N samples i ∼ N (0, I) and τ i ∼ (π DISPLAYFORM3, p) (see Appendix B for a full derivation). Rather than updating Σ according to the previously derived policy gradient, we fix its value to σ 2 I and scale it adaptively as described in Appendix C. This section answers the following questions:(i) Do existing state-of-the-art RL algorithms benefit from incorporating parameter space noise?(ii) Does parameter space noise aid in exploring sparse reward environments more effectively?(iii) How does parameter space noise exploration compare against evolution strategies for deep policies with respect to sample efficiency?Reference implementations of DQN and DDPG with adaptive parameter space noise are available online. The added value of parameter space noise over action space noise is measured on both highdimensional discrete-action environments and continuous control tasks. For the discrete environments, comparisons are made using DQN, while DDPG and TRPO are used on the continuous control tasks. Discrete-action environments For discrete-action environments, we use the Arcade Learning Environment (ALE, BID3) benchmark along with a standard DQN implementation. We compare a baseline DQN agent with -greedy action noise against a version of DQN with parameter noise. We linearly anneal from 1.0 to 0.1 over the first 1 million timesteps. For parameter noise, we adapt the scale using a simple heuristic that increases the scale if the KL divergence between perturbed and non-perturbed policy is less than the KL divergence between greedy and -greedy policy and decreases it otherwise (see Section C.1 for details). By using this approach, we achieve a fair comparison between action space noise and parameter space noise since the magnitude of the noise is similar and also avoid the introduction of an additional hyperparameter. For parameter perturbation, we found it useful to reparametrize the network in terms of an explicit policy that represents the greedy policy π implied by the Q-values, rather than perturbing the Qfunction directly. To represent the policy π(a|s), we add a single fully connected layer after the convolutional part of the network, followed by a softmax output layer. Thus, π predicts a discrete probability distribution over actions, given a state. We find that perturbing π instead of Q in more meaningful changes since we now define an explicit behavioral policy. In this setting, the Q-network is trained according to standard DQN practices. The policy π is trained by maximizing the probability of outputting the greedy action accordingly to the current Q-network. Essentially, the policy is trained to exhibit the same behavior as running greedy DQN. To rule out this double-headed version of DQN alone exhibits significantly different behavior, we always compare our parameter space noise approach against two baselines, regular DQN and two-headed DQN, both with -greedy exploration. We furthermore randomly sample actions for the first 50 thousand timesteps in all cases to fill the replay buffer before starting training. Moreover, we found that parameter space noise performs better if it is combined with a bit of action space noise (we use a -greedy behavioral policy with = 0.01 for the parameter space noise experiments). Full experimental details are described in Section A.1.We chose 21 games of varying complexity, according to the taxonomy presented by BID4. The learning curves are shown in FIG2 for a selection of games (see Appendix D for full ). Each agent is trained for 40 M frames. The overall performance is estimated by running each configuration with three different random seeds, and we plot the median return (line) as well as the interquartile range (shaded area). Note that performance is evaluated on the exploratory policy since we are interested in its behavior especially. Overall, our show that parameter space noise often outperforms action space noise, especially on games that require consistency (e.g. Enduro, Freeway) and performs comparably on the remaining ones. Additionally, learning progress usually starts much sooner when using parameter space noise. Finally, we also compare against a double-headed version of DQN with -greedy exploration to ensure that this change in architecture is not responsible for improved exploration, which our confirm. Full are available in Appendix D.That being said, parameter space noise is unable to sufficiently explore in extremely challenging games like Montezuma's Revenge. More sophisticated exploration methods like BID4 are likely necessary to successfully learn these games. However, such methods often rely on some form of "inner" exploration method, which is usually traditional action space noise. It would be interesting to evaluate the effect of parameter space noise when combined with exploration methods. On a final note, proposed improvements to DQN like double DQN , prioritized experience replay , and dueling networks are orthogonal to our improvements and would therefore likely improve further. We leave the experimental validation of this theory to future work. We now compare parameter noise with action noise on the continuous control environments implemented in OpenAI Gym BID6. We use DDPG BID18 as the RL algorithm for all environments with similar hyperparameters as outlined in the original paper except for the fact that layer normalization BID2 ) is applied after each layer before the nonlinearity, which we found to be useful in either case and especially important for parameter space noise. We compare the performance of the following configurations: (a) no noise at all, (b) uncorrelated additive Gaussian action space noise (σ = 0.2), (c) correlated additive Gaussian action space noise (Ornstein-Uhlenbeck process with σ = 0.2), and (d) adaptive parameter space noise. In the case of parameter space noise, we adapt the scale so that the ing change in action space is comparable to our baselines with uncorrelated Gaussian action space noise (see Section C.2 for full details).We evaluate the performance on several continuous control tasks. FIG3 depicts the for three exemplary environments. Each agent is trained for 1 M timesteps, where 1 epoch consists of 10 thousand timesteps. In order to make comparable between configurations, we evaluate the performance of the agent every 10 thousand steps by using no noise for 20 episodes. On HalfCheetah, parameter space noise achieves significantly higher returns than all other configurations. We find that, in this environment, all other exploration schemes quickly converge to a local optimum (in which the agent learns to flip on its back and then "wiggles" its way forward). Parameter space noise behaves similarly initially but still explores other options and quickly learns to break out of this sub-optimal behavior. Also notice that parameter space noise vastly outperforms correlated action space noise on this environment, clearly indicating that there is a significant difference between the two. On the remaining two environments, parameter space noise performs on par with other exploration strategies. Notice, however, that even if no noise is present, DDPG is capable of learning good policies. We find that this is representative for the remaining environments (see Appendix E for full ), which indicates that these environments do not require a lot of exploration to begin with due to their well-shaped reward function. The for TRPO are depicted in FIG4. Interestingly, in the Walker2D environment, we see that adding parameter noise decreases the performance variance between seeds. This indicates that parameter noise aids in escaping local optima. The environments in the previous section required relatively little exploration. In this section, we evaluate whether parameter noise enables existing RL algorithms to learn on environments with very sparse rewards, where uncorrelated action noise generally fails BID20 BID0.A scalable toy example We first evaluate parameter noise on a well-known toy problem, following the setup described by BID20 as closely as possible. The environment consists of a chain of N states and the agent always starts in state s 2, from where it can either move left or right. In state s 1, the agent receives a small reward of r = 0.001 and a larger reward r = 1 in state s N. Obviously, it is much easier to discover the small reward in s 1 than the large reward in s N, with increasing difficulty as N grows. The environment is described in greater detail in Section A.3.We compare adaptive parameter space noise DQN, bootstrapped DQN, and -greedy DQN. The chain length N is varied and for each N three different seeds are trained and evaluated. After each episode, we evaluate the performance of the current policy by performing a rollout with all noise disabled (in the case of bootstrapped DQN, we perform majority voting over all heads). The problem is considered solved if one hundred subsequent rollouts achieve the optimal return. We plot the median number of episodes before the problem is considered solved (we abort if the problem is still unsolved after 2 thousand episodes). Full experimental details are available in Section A.3. Green indicates that the problem was solved whereas blue indicates that no solution was found within 2 K episodes. Note that less number of episodes before solved is better. FIG5 shows that parameter space noise clearly outperforms action space noise (which completely fails for moderately large N) and even outperforms the more computational expensive bootstrapped DQN. However, it is important to note that this environment is extremely simple in the sense that the optimal strategy is to always go right. In a case where the agent needs to select a different optimal action depending on the current state, parameter space noise would likely work less well since weight randomization of the policy is less likely to yield this behavior. Our thus only highlight the difference in exploration behavior compared to action space noise in this specific case. In the general case, parameter space noise does not guarantee optimal exploration. Continuous control with sparse rewards We now make the continuous control environments more challenging for exploration. Instead of providing a reward at every timestep, we use environments that only yield a non-zero reward after significant progress towards a goal. More concretely, we consider the following environments from rllab 5, modified according to: (a) SparseCartpoleSwingup, which only yields a reward if the paddle is raised above a given threshold, (b) SparseDoublePendulum, which only yields a reward if the agent reaches the upright position, and (c) SparseHalfCheetah, which only yields a reward if the agent crosses a target distance, (d) SparseMountainCar, which only yields a reward if the agent drives up the hill, (e) SwimmerGather, yields a positive or negative reward upon reaching targets. For all tasks, we use a time horizon of T = 500 steps before resetting. We consider both DDPG and TRPO to solve these environments (the exact experimental setup is described in Section A.2). FIG6 shows the performance of DDPG, while the for TRPO have been moved to Appendix F. The overall performance is estimated by running each configuration with five different random seeds, after which we plot the median return (line) as well as the interquartile range (shaded area).For DDPG, SparseDoublePendulum seems to be easy to solve in general, with even no noise finding a successful policy relatively quickly. The for SparseCartpoleSwingup and SparseMountainCar are more interesting: Here, only parameter space noise is capable of learning successful policies since all other forms of noise, including correlated action space noise, never find states with nonzero rewards. For SparseHalfCheetah, DDPG at least finds the non-zero reward but never learns a successful policy from that signal. On the challenging SwimmerGather task, all configurations of DDPG fail. Our clearly show that parameter space noise can be used to improve the exploration behavior of these off-the-shelf algorithms. However, it is important to note that improvements in exploration are not guaranteed for the general case. It is therefore necessary to evaluate the potential benefit of parameter space noise on a case-by-case basis. Evolution strategies (ES) are closely related to our approach since both explore by introducing noise in the parameter space, which can lead to improved exploration behavior . 6 However, ES disregards temporal information and uses black-box optimization to train the neural network. By combining parameter space noise with traditional RL algorithms, we can include temporal information as well rely on gradients computed by back-propagation for optimization while still benefiting from improved exploratory behavior. We now compare ES and traditional RL with parameter space noise directly. We compare performance on the 21 ALE games that were used in Section 4.1. The performance is estimated by running 10 episodes for each seed using the final policy with exploration disabled and computing the median returns. For ES, we use the obtained by , which were obtained after training on 1 000 M frames. For DQN, we use the same parameter space noise for exploration that was previously described and train on 40 M frames. Even though DQN with parameter space noise has been exposed to 25 times less data, it outperforms ES on 15 out of 21 Atari games (full are available in Appendix D). Combined with the previously described , this demonstrates that parameter space noise combines the desirable exploration properties of ES with the sample efficiency of traditional RL. The problem of exploration in reinforcement has been studied extensively. A range of algorithms BID14 BID5 BID1 have been proposed that guarantee near-optimal solutions after a number of steps that are polynomial in the number of states, number of actions, and the horizon time. However, in many real-world reinforcements learning problems both the state and action space are continuous and high dimensional so that, even with discretization, these algorithms become impractical. In the context of deep reinforcement learning, a large variety of techniques have been proposed to improve exploration BID29 BID6 BID20 BID22 BID30 BID21. However, all are non-trivial to implement and are often computational expensive. The idea of perturbing the parameters of a policy has been proposed by Rückstieß et al. for policy gradient methods. The authors show that this form of perturbation generally outperforms random exploration and evaluate their exploration strategy with the REINFORCE (a) and Natural Actor-Critic BID24 algorithms. However, their policies are relatively lowdimensional compared to modern deep architectures, they use environments with low-dimensional state spaces, and their contribution is strictly limited to the policy gradient case. In contrast, our method is applied and evaluated for both on and off-policy setting, we use high-dimensional policies, and environments with large state spaces. Our work is also closely related to evolution strategies (ES, BID26 BID27 FORMULA2). In the context of policy optimization, our work is closely related to BID17 and BID28. More recently, showed that ES can work for high-dimensional environments like Atari and OpenAI Gym continuous control problems. However, ES generally disregards any temporal structure that may be present in trajectories and typically suffers from sample inefficiency. Bootstrapped DQN BID20 has been proposed to aid with more directed and consistent exploration by using a network with multiple heads, where one specific head is selected at the beginning of each episode. In contrast, our approach perturbs the parameters of the network directly, thus achieving similar yet simpler (and as shown in Section 4.2, sometimes superior) exploration behavior. Concurrently to our work, BID8 have proposed a similar approach that utilizes parameter perturbations for more efficient exploration. In this work, we propose parameter space noise as a conceptually simple yet effective replacement for traditional action space noise like -greedy and additive Gaussian noise. This work shows that parameter perturbations can successfully be combined with contemporary on-and off-policy deep RL algorithms such as DQN, DDPG, and TRPO and often in improved performance compared to action noise. Experimental further demonstrate that using parameter noise allows solving environments with very sparse rewards, in which action noise is unlikely to succeed. Our indicate that parameter space noise is a viable and interesting alternative to action space noise, which is still the de facto standard in most reinforcement learning applications. A EXPERIMENTAL SETUP For ALE BID3, the network architecture as described in BID19 is used. This consists of 3 convolutional layers (32 filters of size 8 × 8 and stride 4, 64 filters of size 4 × 4 and stride 2, 64 filters of size 3 × 3 and stride 1) followed by 1 hidden layer with 512 units followed by a linear output layer with one unit for each action. ReLUs are used in each layer, while layer normalization BID2 ) is used in the fully connected part of the network. For parameter space noise, we also include a second head after the convolutional stack of layers. This head determines a policy network with the same architecture as the Q-value network, except for a softmax output layer. The target networks are updated every 10 K timesteps. The Q-value network is trained using the Adam optimizer BID15 with a learning rate of 10 −4 and a batch size of 32. The replay buffer can hold 1 M state transitions. For the -greedy baseline, we linearly anneal from 1 to 0.1 over the first 1 M timesteps. For parameter space noise, we adaptively scale the noise to have a similar effect in action space (see Section C.1 for details), effectively ensuring that the maximum KL divergence between perturbed and non-perturbed π is softly enforced. The policy is perturbed at the beginning of each episode and the standard deviation is adapted as described in Appendix C every 50 timesteps. Notice that we only perturb the policy head after the convolutional part of the network (i.e. the fully connected part, which is also why we only include layer normalization in this part of the network). To avoid getting stuck (which can potentially happen for a perturbed policy), we also use -greedy action selection with = 0.01. In all cases, we perform 50 K random actions to collect initial data for the replay buffer before training starts. We set γ = 0.99, clip rewards to be in [−1, 1], and clip gradients for the output layer of Q to be within [−1, 1]. For observations, each frame is down-sampled to 84 × 84 pixels, after which it is converted to grayscale. The actual observation to the network consists of a concatenation of 4 subsequent frames. Additionally, we use up to 30 noop actions at the beginning of the episode. This setup is identical to what is described by BID19. For DDPG, we use a similar network architecture as described by BID18: both the actor and critic use 2 hidden layers with 64 ReLU units each. For the critic, actions are not included until the second hidden layer. Layer normalization BID2 ) is applied to all layers. The target networks are soft-updated with τ = 0.001. The critic is trained with a learning rate of 10 −3 while the actor uses a learning rate of 10 −4. Both actor and critic are updated using the Adam optimizer BID15 with batch sizes of 128. The critic is regularized using an L2 penalty with 10 −2. The replay buffer holds 100 K state transitions and γ = 0.99 is used. Each observation dimension is normalized by an online estimate of the mean and variance. For parameter space noise with DDPG, we adaptively scale the noise to be comparable to the respective action space noise (see Section C.2). For dense environments, we use action space noise with σ = 0.2 (and a comparable adaptive noise scale). Sparse environments use an action space noise with σ = 0.6 (and a comparable adaptive noise scale).TRPO uses a step size of δ KL = 0.01, a policy network of 2 hidden layers with 32 tanh units for the nonlocomotion tasks, and 2 hidden layers of 64 tanh units for the locomotion tasks. The Hessian calculation is subsampled with a factor of 0.1, γ = 0.99, and the batch size per epoch is set to 5 K timesteps. The baseline is a learned linear transformation of the observations. The following environments from OpenAI Gym 7 BID6 are used: DISPLAYFORM0 • Swimmer (S ⊂ R 8, A ⊂ R 2), and DISPLAYFORM1 For the sparse tasks, we use the following environments from rllab 8, modified as described by: DISPLAYFORM2, which only yields a reward if the paddle is raised above a given threshold, DISPLAYFORM3, which only yields a reward if the agent crosses a distance threshold, DISPLAYFORM4, which only yields a reward if the agent drives up the hill,• SparseDoublePendulum (S ⊂ R 6, A ⊂ R), which only yields a reward if the agent reaches the upright position, and DISPLAYFORM5, which yields a positive or negative reward upon reaching targets. We follow the state encoding proposed by BID20 and use φ(s t) = (1{x ≤ s t}) as the observation, where 1 denotes the indicator function. DQN is used with a very simple network to approximate the Q-value function that consists of 2 hidden layers with 16 ReLU units. Layer normalization BID2 ) is used for all hidden layers before applying the nonlinearity. Each agent is then trained for up to 2 K episodes. The chain length N is varied and for each N three different seeds are trained and evaluated. After each episode, the performance of the current policy is evaluated by sampling a trajectory with noise disabled (in the case of bootstrapped DQN, majority voting over all heads is performed). The problem is considered solved if one hundred subsequent trajectories achieve the optimal episode return. Figure 6 depicts the environment. DISPLAYFORM0 Figure 6: Simple and scalable environment to test for exploratory behavior BID20.We compare adaptive parameter space noise DQN, bootstrapped DQN BID20 ) (with K = 20 heads and Bernoulli masking with p = 0.5), and -greedy DQN (with linearly annealed from 1.0 to 0.1 over the first one hundred episodes). For adaptive parameter space noise, we only use a single head and perturb Q directly, which works well in this setting. Parameter space noise is adaptively scaled so that δ ≈ 0.05. In all cases, γ = 0.999, the replay buffer holds 100 K state transitions, learning starts after 5 initial episodes, the target network is updated every 100 timesteps, and the network is trained using the Adam optimizer BID15 with a learning rate of 10 −3 and a batch size of 32. DISPLAYFORM1 Given a stochastic policy π θ (a|s) with θ ∼ N (φ, Σ), the expected return can be expanded using likelihood ratios and the reparametrization trick BID16 as DISPLAYFORM2 DISPLAYFORM3 for N samples i ∼ N (0, I) and τ i ∼ (π DISPLAYFORM4 This also allows us to subtract a variance-reducing baseline b i t, leading to DISPLAYFORM5 In our case, we set Σ := σ 2 I and use our proposed adaption method to re-scale as appropriate. Parameter space noise requires us to pick a suitable scale σ. This can be problematic since the scale will highly depend on the specific network architecture, and is likely to vary over time as parameters become more sensitive as learning progresses. Additionally, while it is easy to intuitively grasp the scale of action space noise, it is far harder to understand the scale in parameter space. We propose a simple solution that resolves all aforementioned limitations in an easy and straightforward way. This is achieved by adapting the scale of the parameter space noise over time, thus using a time-varying scale σ k . Furthermore, σ k is related to the action space variance that it induces, and updated accordingly. Concretely, we use the following simple heuristic to update σ k every K timesteps: DISPLAYFORM0 where d(·, ·) denotes some distance between the non-perturbed and perturbed policy (thus measuring in action space), α ∈ R >0 is used to rescale σ k, and δ ∈ R >0 denotes some threshold value. This idea is based on the Levenberg-Marquardt heuristic BID25. The concrete distance measure and appropriate choice of δ depends on the policy representation. In the following sections, we outline our choice of d(·, ·) for methods that do (DDPG and TRPO) and do not (DQN) use behavioral policies. In our experiments, we always use α = 1.01. For DQN, the policy is defined implicitly by the Q-value function. Unfortunately, this means that a naïve distance measure between Q and Q has pitfalls. For example, assume that the perturbed policy has only changed the bias of the final layer, thus adding a constant value to each action's Q-value. In this case, a naïve distance measure like the norm Q − Q 2 would be nonzero, although the policies π and π (implied by Q and Q, respectively) are exactly equal. This equally applies to the case where DQN as two heads, one for Q and one for π. We therefore use a probabilistic formulation 9 for both the non-perturbed and perturbed policies: π, π: S × A → by applying the softmax function over predicted Q values: DISPLAYFORM0, where Q i (·) denotes the Q-value of the i-th action. π is defined analogously but uses the perturbed Q instead (or the perturbed head for π). Using this probabilistic formulation of the policies, we can now measure the distance in action space: DISPLAYFORM1 where D KL (· ·) denotes the Kullback-Leibler (KL) divergence. This formulation effectively normalizes the Q-values and therefore does not suffer from the problem previously outlined. We can further relate this distance measure to -greedy action space noise, which allows us to fairly compare the two approaches and also avoids the need to pick an additional hyperparameter δ. More concretely, the KL divergence between a greedy policy π(s, a) = 1 for a = argmax a Q(s, a) and π(s, a) = 0 otherwise and an -greedy policy π(s, a) = 1 − + |A| for a = argmax a Q(s, a) and π(s, a) = |A| otherwise is D KL (π π) = − log (1 − + |A|), where |A| denotes the number of actions (this follows immediately from the definition of the KL divergence for discrete probability distributions). We can use this distance measure to relate action space noise and parameter space noise to have similar distances, by adaptively scaling σ so that it matches the KL divergence between greedy and -greedy policy, thus setting δ:= − log (1 − + |A|). For DDPG, we relate noise induced by parameter space perturbations to noise induced by additive Gaussian noise. To do so, we use the following distance measure between the non-perturbed and perturbed policy: DISPLAYFORM0 where E s [·] is estimated from a batch of states from the replay buffer and N denotes the dimension of the action space (i.e. A ⊂ R N). It is easy to show that d(π, π + N (0, σ 2 I)) = σ. Setting δ:= σ as the adaptive parameter space threshold thus in effective action space noise that has the same standard deviation as regular Gaussian action space noise. In order to scale the noise for TRPO, we adapt the sampled noise vectors σ by computing a natural step H −1 σ. We essentially compute a trust region around the noise direction to ensure that the perturbed policy π remains sufficiently close to the non-perturbed version via DISPLAYFORM0 Concretely, this is computed through the conjugate gradient algorithm, combined with a line search along the noise direction to ensure constraint conformation, as described in Appendix C of Schulman et al. (2015b). FIG8 provide the learning curves for all 21 Atari games. TAB2 compares the final performance of ES after 1 000 M frames to the final performance of DQN with -greedy exploration and parameter space noise exploration after 40 M frames. In all cases, the performance is estimated by running 10 episodes with exploration disabled. We use the numbers reported by for ES and report the median return across three seeds for DQN. FIG9. The for InvertedPendulum and InvertedDoublePendulum are very noisy due to the fact that a small change in policy can easily degrade performance significantly, and thus hard to read. Interestingly, adaptive parameter space noise achieves the most stable performance on InvertedDoublePendulum. Overall, performance is comparable to other exploration approaches. Again, no noise in either the action nor the parameter space achieves comparable , indicating that these environments combined with DDPG are not well-suited to test for exploration. The performance of TRPO with noise scaled according to the parameter curvature, as defined in Section C.3 is shown in FIG10. The TRPO baseline uses only action noise by using a policy network that outputs the mean of a Gaussian distribution, while the variance is learned. These show that adding parameter space noise aids in either learning much more consistently on these challenging sparse environments.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.

represent the tldr for finding the corresponding scientific article
Examples:
Provided: "We introduced a novel, simple, and efficient data augmentation method that boosts the performances of existing GANs when training data is limited and diverse.  " Match: "The need for large amounts of training image data with clearly defined features is a major obstacle to applying generative adversarial networks(GAN) on image generation where training data is limited but diverse, since insufficient latent feature representation in the already scarce data often leads to instability and mode collapse during GAN training. To overcome the hurdle of limited data when applying GAN to limited datasets, we propose in this paper the strategy of \textit{parallel recurrent data augmentation}, where the GAN model progressively enriches its training set with sample images constructed from GANs trained in parallel at consecutive training epochs. Experiments on a variety of small yet diverse datasets demonstrate that our method, with little model-specific considerations, produces images of better quality as compared to the images generated without such strategy. The source code and generated images of this paper will be made public after review. Generative Adversarial Networks(GAN) BID5 ) are powerful unsupervised learning models that have recently achieved great success in learning high-dimensional distributions in various types of problems and on different datasets. In the context of image generation, the basic framework of a GAN model consists of two parts: a generator G that generates images by translating random input z into an image, and a discriminator D which determines the authenticity of a generated image x as compared to the real data. These two components are alternatively optimized against each other during the training process, with the goal of minimizing the difference between the distribution of generated image data and target distribution of real image data. A notable challenge in GAN training, however, lies in the need for large amounts of clearly labeled data to capture the diversity features across various types of images into the model. Such requirement makes it difficult or even impossible to utilize GAN in applications where the amount of available training data is small but diverse. Moreover, recent deep learning models BID6 ) have demonstrated tendencies of misrepresentation in classification tasks when influenced by adversarial noise. Such vulnerability may also translate to unsatisfactory image generation as most generative models are implemented with deep networks. Thus, given these considerations, we propose in this paper the strategy of parallel recurrent sample augmentation agnostic to specific model details. Our contributions can be summarized as follows:• We proposed a general black-box method using recurrent image addition to diversify training data and enhance its quality over a large class of GANs without model specifications.• We also includes in our model a novel K-fold parallel framework, which better augments training data by stabilizing model output and preventing overfitting.• Experiments across various datasets and GAN objectives demonstrate the effectiveness of our method using authenticity measures such as Inception Score and Frechet Inception Distance. Building reliable deep generative adversarial models on limited amounts of training data has been a persistent challenge within the research community. Previous efforts to address the issue of labeled data scarcity generally fall into two groups: optimizing the structures of GANs to allow for better feature representation of data, and augmenting the training data through techniques. Along the first line of research, prior research optimized the GAN in BID5 by considering stronger mathematical objectives for more powerful latent space representation in general BID14,, BID7, BID13 ). In addition, recent research on GANs BID8, BID14 ) reparametrized the input noise using variational inference by assuming that the latent space could be modeled by a tractable prior, but noise reparametrization has severe mathematical limitation that prevents applicability to more general models. Furthermore, distributed multi-discriminator models BID10, ) also enhance the performances, with great potential room for further optimization. For the second line of research, data augmentation has already enjoyed considerable success in the problem of image classification BID12 ). Traditional data augmentation methods such as crop, mirror, rotation and distortion BID12, BID17 ) generally require domain-specific expert knowledge and manual operations, and only produce limited variation in augmented images. Recent developments centered on automatic augmentation of training data using controlled RNNs after transforming the problem into policy search in reinforcement learning , but the space complexity the search algorithm requires still does not apply to quick augmentation with limited data. In this section we describe the details of Parallel Recurrent Data Augmentation(PRDA), which consists of recurrent image data construction via noise addition on images and parallel generation with fold division. Recent research suggests that adversarial noise to deep learning models greatly perturbs the performances of deep neural networks. In CNNs for classification, for instance, the neural network may assign irrelevant labels to semantically unambiguous images after the addition of noise BID6 ). This phenomeon is due to the omission of possible latent features in new data generation caused by over-dependency on our limited training data. Since virtually all GANs are implemented with deep networks, a similar deficiency in representation of latent feature space may translate to lower qualities of the generated images. To counter this effect, we consider the strategy of recurrent data augmentation, which constructs varied images given the limited training set by repeatedly generating and modifying these samples on training set for subsequent training sample generation. Running the original generative model for a fixed number of times, we extract sampled images using standard procedures of sample image generation as described in BID15, BID7. Random noise is then added to these samples to produce new images, which are then used for subsequent training. This procedure is repeated for a fixed number of times or until convergence. FIG0 is a flow-chart of our procedure. Notice that the addition of high dimensional normal random noise allows the additional images to retain the original latent features to be learned by GAN. Compared with traditional methods such as rotation, cropping and mirroring BID17, BID12 ) which may lead to information loss, random noise addition doesn't reduce the information about the latent features in training set while making the model more robust, because the expectation of noise is invariant at 0. Additionally, noise addition is agnostic to the type of generative model, since the procedure is independent from the specific choice of neural network or objective functions. Additionally, we introduce a parallel data generation strategy inspired by K-fold cross validation in machine learning BID2 ). Dividing the training data into K folds at the beginning, we run in parallel K independent generators on K data groups, each consisting of K − 1 folds of the training set. When data is generated in each generator, the sample images produced by each generator at the given epochs are then added with random noise. These noised images, in turn, are fed back into the respective training data sets. To allow for maximal usage of each generated image, we insert the images such that the image generated by one generator goes to the augmented training set of all other K − 1 generators. This is to insure that the different generators in parallel have access to as many varied data pieces as possible in subsequent steps of training, so as to prevent overfitting and bolster the robustness of our model. Figure 2 demonstrates the mechanism of our algorithm. Notice that our K-fold division goes hand in hand with the recurrent data generation with no need for model specific considerations. As demonstrated by our experiments Section 4, training different GANs in parallel from different folds of data substantially boosts the quality of the training set and that of the generated images. For a comparative analysis, we have conducted experiments on previous GAN architectures over various datasets with/without data augmentation. The GANs we have tested on include DC-GAN BID15 ), BEGAN BID1 ), and WGAN-GP BID7 ). Additionally, to simulate limited data, we randomly select 5000 images from the datasets CIFAR-10, CelebA and Places to create their corresponding reduced datasets named reduced-CIFAR, reducedCelebA, and reduced-Places, and conduct our experiments on these limited datasets. All of our experiments are conducted with CPU Intel(R) Core(R) CPU 8700-K (3.7GHz) and GPU GTX 1080. In our experiments, we augment the training set with 8 noised images every 100 training epoches, and repeat the procedure 3-5 times. By comparison, the unaugmented GAN is run over the same initial training data, with the number of epochs the same as the product of 100 and augmentation times. FIG1,4,5 are some sample images that our method produces with the state-of-the-art GAN WGAN-GP as compared to the ones produced by GAN without data augmentation. We observe that GANs with parallel recurrent image augmentation produce semantically coherent and visually diverse images earlier than the unaugmented GANs, while able to avoid fluctuations seen in unaugmented GANs during training. To evaluate the quality of the images generated by our augmentation method as compared with those generated without augmentation, we use the Inception Score(IS) BID16 ) and Frechet BID9 ). IS measures the entropy of generated images, with higher scores indicating greater diversity. On the other hand, FID measures the distance between the generated data and real data with two respective means and variances. Thus, the larger the IS and the smaller the FID, the better the performances of the model. Table 2: IS/FID scores of GANs on Reduced-CIFAR with/without Augmentation, given K = 5 Table 1 lists the combinations of GAN and dataset we tested our strategy on, as well as the Inception Score and Frechet Inception Distance of the images that are generated with and without our method using the state-of-the-art GAN WGAN-GP. Table 2 lists IS and FID of different GAN models on the Reduced-CIFAR with/without data augmentation. Clearly, on a variety of GAN structures and Datasets, recurrent sample augmentation produces better images as measured quantitatively. In sum, our paper shows that parallel recurrent sample augmentation can significantly improve the quality of synthetic images for a large class of GAN models. Our strategy is not only simple to implement, but also agnostic to the specific type of GAN to be improved on. As a further step, we are investigating the relationship between our proposed approach and other established methods. One possible pathway, for instance, lies in reinforcement learning as described in BID3 that gives more control to image generation via reward designation. We also hope to apply our idea to other generative models such as the VAE BID11 ) and further optimize our strategy using recent theoretical advances."

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet have recently achieved state-of-the-art performance on a variety of language understanding tasks. However, their size makes them impractical for a number of scenarios, especially on mobile and edge devices. In particular, the input word embedding matrix accounts for a significant proportion of the model's memory footprint, due to the large input vocabulary and embedding dimensions. Knowledge distillation techniques have had success at compressing large neural network models, but they are ineffective at yielding student models with vocabularies different from the original teacher models. We introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocabulary as well as lower embedding and hidden state dimensions. Specifically, we employ a dual-training mechanism that trains the teacher and student models simultaneously to obtain optimal word embeddings for the student vocabulary. We combine this approach with learning shared projection matrices that transfer layer-wise knowledge from the teacher model to the student model. Our method is able to compress the BERT-BASE model by more than 60x, with only a minor drop in downstream task metrics, ing in a language model with a footprint of under 7MB. Experimental also demonstrate higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques. Recently, contextual-aware language models such as ELMo , GPT , BERT and XLNet have shown to greatly outperform traditional word embedding models including Word2Vec and GloVe in a variety of NLP tasks. These pre-trained language models, when finetuned on downstream language understanding tasks such as sentiment classification , natural language inference and reading comprehension , have achieved state-of-the-art performance. However, the large number of parameters in these models, often above hundreds of millions, makes it impossible to host them on resource-constrained tasks such as doing real-time inference on mobile and edge devices. Besides utilizing model quantization techniques which aim to reduce the floating-point accuracy of the parameters, significant recent research has focused on knowledge distillation techniques. Here, the goal is to train a small-footprint student model by borrowing knowledge, such as through a soft predicted label distribution, from a larger pre-trained teacher model. However, a significant bottleneck that has been overlooked by previous efforts is the input vocabulary size and its corresponding word embedding matrix, often accounting for a significant proportion of all model parameters. For instance, the embedding table of the BERT BASE model, comprising over 30K WordPiece tokens (b), accounts for over 21% of the model size. While there has been existing work on reducing NLP model vocabulary sizes , distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space. This profoundly limits their potential to further reduce model sizes. We present two novel ideas to improve the effectiveness of knowledge distillation, in particular for BERT, with the focus on bringing down model sizes to as much as a few mega-bytes. Our model is among the first to propose to use a significantly smaller vocabulary for the student model learned during distillation. In addition, instead of distilling solely on the teacher model's final-layer outputs, our model leverages layer-wise teacher model parameters to directly optimize the parameters of the corresponding layers in the student model. Specifically, our contributions are: • Dual Training: Our teacher and student models have different vocabularies and incompatible tokenizations for the same sequence. To address this during distillation, we feed the teacher model a mix of teacher vocabulary-tokenized and student vocabulary-tokenized words within a single sequence. Coupled with the masked language modeling task, this encourages an implicit alignment of the teacher and student WordPiece embeddings, since the student vocabulary embedding may be used as context to predict a word tokenized by the teacher vocabulary and vice versa. • Shared Variable Projections: To minimize the loss of information from reducing the hidden state dimension, we introduce a separate loss to align the teacher and student models' trainable variables. This allows for more direct layer-wise transfer of knowledge to the student model. Using the combination of dual training and shared variable projections, we train a 12-layer highlycompressed student BERT model, achieving a maximum compression ratio of ∼61.94x (with 48 dimension size) compared to the teacher BERT BASE model. We conduct experiments for measuring both generalized language modeling perspective and for downstream tasks, demonstrating competitive performance with high compression ratios for both families of tasks. Research in neural network model compression has been concomitant with the rise in popularity of neural networks themselves, since these models have often been memory-intensive for the hardware of their time. Work in model compression for NLP applications falls broadly into four categories: matrix approximation, parameter pruning/sharing, weight quantization and knowledge distillation. A family of approaches seeks to compress the matrix parameters of the models by low-rank approximation i.e. the full-rank matrix parameter is approximated using multiple low-rank matrices, thereby reducing the effective number of model parameters. Another line of work explores parameter pruning and sharing-based methods (; ; ;), which explore the redundancy in model parameters and try to remove redundant weights as well as neurons, for a variety of neural network architectures. Model weight quantization techniques (; a;) focus on mapping model weights to lower-precision integers and floating-point numbers. These can be especially effective with hardware supporting efficient low-precision calculations. More recently, apply quantization to BERT-based transformer models. Knowledge distillation differs from the other discussed approaches: the smaller student model may be parametrized differently from the bigger teacher model, affording more modeling freedom. Teaching a student model to match the soft output label distributions from a larger model alongside the hard ground-truth distribution works well for many tasks, such as machine translation and language modeling . Not limited to the teacher model outputs, some approaches perform knowledge distillation via attention transfer , or via feature maps or intermediate model outputs (; ;). More relevant to current work, and employ variants of these techniques to BERT model compression by reducing the number of transformer layers. However, as explained before, these approaches are not immediately applicable to our setting due to incompatible teacher and student model vocabularies, and do not focus sufficiently on the embedding matrix size . Our knowledge distillation approach is centered around reducing the number of WordPiece tokens in the model vocabulary. In this section, we first discuss the rationale behind this reduction and the challenges it introduces, followed by our techniques, namely dual training and shared projection. (Right) A student BERT model trained from scratch with smaller vocab (5K) and hidden state dimension (e.g., 48). During distillation, the teacher model randomly selects a vocabulary to segment each input word. The red and green square nexts to the transformer layers indicate trainable parameters for both the student and teacher models -note that our student models have smaller model dimensions. The projection matrices U and V, shown as having representative shapes, are shared across all layers for model parameters that have the same dimensions. We follow the general knowledge distillation paradigm of training a small student model from a large teacher model. Our teacher model is a 12-layer uncased BERT BASE, trained with 30522 WordPiece tokens and 768-dimensional embeddings and hidden states. We denote the teacher model parameters by θ t. Our student model consists of an equal number of transformer layers with parameters denoted by θ s, but with a smaller vocabulary as well as embedding/hidden dimensions, illustrated in Figure 1. Using the same WordPiece algorithm and training corpus as BERT, we obtain a vocabulary of 4928 WordPieces, which we use for the student model. WordPiece tokens (b) are sub-word units obtained by applying a greedy segmentation algorithm to the training corpus: a desired number (say, D) of WordPieces are chosen such that the segmented corpus is minimal in the number of WordPieces used. A cursory look at both vocabularies reveals that 93.9% of the WordPieces in the student vocabulary also exist in the teacher vocabulary, suggesting room for a reduction in the WordPiece vocabulary size from 30K tokens. Since we seek to train a general-purpose student language model, we elect to reuse the teacher model's original training objective to optimize the student model, i.e., masked language modeling and next sentence prediction, before any fine-tuning. In the former task, words in context are randomly masked, and the language model needs to predict those words given the masked context. In the latter task, given a pair of sentences, the language model predicts whether the pair is consistent. However, since the student vocabulary is not a complete subset of the teacher vocabulary, the two vocabularies may tokenize the same words differently. As a , the outputs of the teacher and student model for the masked language modeling task may not align. Even with the high overlap between the two vocabularies, the need to train the student embedding from scratch, and the change in embedding dimension precludes existing knowledge distillation techniques, which rely on the alignment of both models' output spaces. As a , we explore two alternative approaches that enable implicit transfer of knowledge to the student model, which we describe below. During distillation, for a given training sequence input to the teacher model, we propose to mix the teacher and student vocabularies by randomly selecting (with a probability p DT, a hyperparameter) tokens from the sequence to segment using the student vocabulary, with the other tokens segmented using the teacher vocabulary. As illustrated in Figure 1, given the input context ['I', 'like', 'machine', 'learning'], the words'I' and'machine' are segmented using the teacher vocabulary (in green), while'like' and'learning' are segmented using the student vocabulary (in blue). Similar to cross-lingual training in , this encourages alignment of the representations for the same word as per the teacher and student vocabularies. This is effected through the masked language modeling task: the model now needs to learn to predict words from the student vocabulary using context words segmented using the teacher vocabulary, and vice versa. The expectation is that the student embeddings can be learned effectively this way from the teacher embeddings as well as model parameters θ t. Note that we perform dual training only for the teacher model inputs: the student model receives words segmented exclusively using the student vocabulary. Also, during masked language modeling, the model uses different softmax layers for the teacher and the student vocabularies depending on which one was used to segment the word in question. Relying solely on teacher model outputs to train the student model may not generalize well . Therefore, some approaches utilize and try to align the student model's intermediate predictions to those of the teacher . In our setting, however, since the student and teacher model output spaces are not identical, intermediate model outputs may prove hard to align. Instead, we seek to directly minimize the loss of information from the teacher model parameters θ t to the student parameters θ s with smaller dimensions. We achieve this by projecting the model parameters into the same space, to encourage alignment. More specifically, as in Figure 1, we project each trainable variable in θ t to the same shape as the corresponding variable in θ s. For example, for all the trainable variables θ t with shape 768×768, we learn two projection matrices U ∈ R d×768 and V ∈ R 768×d to project them into the corresponding space of the student model variable θ t, where d is the student model's hidden dimension. U and V are common to all BERT model parameters of that dimensionality; in addition, U and V are not needed for fine-tuning or inference after distillation. In order to align the student variable and the teacher variable's projection, we introduce a separate mean square error loss defined in Equation 1, where ↓ stands for down projection (since the projection is to a lower dimension). The above loss function aligns the trainable variables in the student space. Alternatively, we can project trainable variables in θ s to the same shape as in θ t. This way, the loss function in Equation 2, (↑ denotes up projection) can compare the trainable variables in the teacher space. 3.4 OPTIMIZATION OBJECTIVE Our final loss function includes, in addition to an optional projection loss, masked language modeling cross-entropy losses for the student as well as the teacher models, since the teacher model is trained with dual-vocabulary inputs and is not static. P (y i = c|θ s) and P (y i = c|θ t) denote the stu-dent and teacher model prediction probabilities for class c respectively, and 1 denotes an indicator function. Equations 3 and 4 below define the final loss L f inal, where is a hyperparameter. To evaluate our knowledge distillation approach, we design two classes of experiments. First, we evaluate the distilled student language models using the masked word prediction task on an unseen evaluation corpus, for an explicit evaluation of the language model. Second, we fine-tune the language model by adding a task-specific affine layer on top of the student language model outputs, on a suite of downstream sentence and sentence pair classification tasks. This is meant to be an implicit evaluation of the quality of the representations learned by the student language model. We describe these experiments, along with details on training, implementation and our baselines below. During the distillation of the teacher BERT model to train the student BERT language model, we utilize the same corpus as was used to train the teacher i.e. BooksCorpus and English Wikipedia, with whitespaces used tokenize the text into words. We only use the masked language modeling task to calculate the overall distillation loss from Section 3.3, since the next sentence prediction loss hurt performance slightly. Dual training is enabled for teacher model inputs, with p DT, the probability of segmenting a teacher model input word using the student vocabulary, set to 0.5. For experiments including shared projection, the projection matrices U and V utilized Xavier initialization . The loss weight coefficient is set to 1 after tuning. It is worth noting that in contrast to a number of existing approaches, we directly distill the teacher BERT language model, not yet fine-tuned on a downstream task, to obtain a student language model that is task-agnostic. For downstream tasks, we fine-tune this distilled student language model. Distillation is carried out on Cloud TPUs in a 4x4 pod configuration 1 (32 TPU cores overall). We optimized the loss using LAMB for 250K steps, with a learning rate of 0.00125 and batch size of 4096. Depending on the student model dimension, training took between 2-4 days. We evaluate three variants of our distilled student models: with only dual training of the teacher and student vocabularies (DualTrain) and with dual training along with down-projection (DualTrain + SharedProjDown) or up-projection (DualTrain + SharedProjUp) of the teacher model parameters. For each of these configurations, we train student models with embedding and hidden dimensions 48, 96 and 192, for 9 total variants, each using a compact 5K-WordPiece vocabulary. Table 1 presents some statistics on these models' sizes: our smallest model contains two orders of magnitude fewer parameters, and requires only 1% floating-point operations when compared to the BERT BASE model. For the language modeling evaluation, we also evaluate a baseline without knowledge distillation (termed NoKD), with a model parameterized identically to the distilled student models but trained directly on the teacher model objective from scratch. For downstream tasks, we compare with NoKD as well as Patient Knowledge Distillation (PKD) from , who distill the 12-layer BERT BASE model into 3 and 6-layer BERT models by using the teacher model's hidden states. Table 1: A summary of our student models' sizes compared to BERT BASE. #Params indicates the number of parameters in the student model, model size is measured in megabytes, and FLOPS ratio measures the relative ratio of floating point operations required for inference on the model. For explicit evaluation of the generalized language perspective of the distilled student language models, we use the Reddit dataset 2 to measure word mask prediction accuracy of the student models, since the language used on Reddit is different from that in the training corpora. The dataset is preprocessed similarly to the training corpora, except we do not need to tokenize it using the teacher vocabulary, since we only run and evaluate the student models. For implicit evaluation on downstream language understanding tasks, we fine-tune and evaluate the distilled student models on three tasks from the GLUE benchmark : • Stanford Sentiment Treebank (SST-2) , a two-way sentence sentiment classification task with 67K training instances, • Microsoft Research Paraphrase Corpus (MRPC) , a two-way sentence pair classification task to identify paraphrases, with 3.7K training instances, and • Multi-Genre Natural Language Inference (MNLI) , a three-way sentence pair classification task with 393K training instances, to identify premise-hypothesis relations. There are separate development and test sets for genre-matched and genre-mismatched premisehypothesis pairs; we tune our models solely on the genre-matched development set. For all downstream task evaluations, we fine-tune for 10 epochs using LAMB with a learning rate of 0.0002 and batch size of 32. Since our language models are trained with a maximum sequence length of 128 tokens, we do not evaluate on reading comprehension datasets such as SQuAD or RACE , which require models supporting longer sequences. Table 2: Masked language modeling task accuracy for the distilled student models and a fine-tunefrom-scratch baseline. We observe consistently better performance for our proposed approaches. Table 2 contains masked word prediction accuracy figures for the different models and the NoKD baseline. We observe that dual training significantly improves over the baseline for all model dimensions, and that both shared projection losses added to dual training further improve the word prediction accuracy. It is interesting to note that for all model dimensions, SharedProjUp projecting into the teacher space outperforms SharedProjDown, significantly so for dimension 48. Expectedly, there is a noticeable performance drop going from 192 to 96 to 48-dimensional hidden state models. Table 3: Results of the distilled models, the teacher model and baselines on the downstream language understanding task test sets, obtained from the GLUE server, along with the size parameters and compression ratios of the respective models compared to the teacher BERT BASE. MNLI-m and MNLI-mm refer to the genre-matched and genre-mismatched test sets for MNLI. Note that because of the differing teacher and student model vocabularies, masked word prediction accuracy for the teacher BERT BASE model is not directly comparable with the student models. Table 3 shows on the downstream language understanding tasks, as well as model sizes, for our approaches, the BERT BASE teacher model, and the PKD and NoKD baselines. We note that models trained with our proposed approaches perform strongly and consistently improve upon the identically parametrized NoKD baselines, indicating that the dual training and shared projection techniques are effective, without incurring significant losses against the BERT BASE teacher model. Comparing with the PKD baseline, our 192-dimensional models, achieving a higher compression rate than either of the PKD models, perform better than the 3-layer PKD baseline and are competitive with the larger 6-layer baseline on task accuracy while being nearly 5 times as small. Another observation we make is that the performance drop from 192-dimensional to 96-dimensional models is minimal (less than 2% for most tasks). For the MRPC task, in fact, the 96-dimensional model trained with dual training achieves an accuracy of 80.5%, which is higher than even the PKD 6-layer baseline with nearly 12 times as many parameters. Finally, our highly-compressed 48-dimensional models also perform respectably: the best 48-dimensional models are in a similar performance bracket as the 3-layer PKD model, a model 25 times larger by memory footprint. Shared projections and model performance: We see that for downstream task performance, dual training still consistently improves upon the direct fine-tuning approach for virtually all experiments. The effect of shared variable projection, however, is less pronounced, with consistent improvements visible only for MRPC and for the 48-dimensional models i.e. the smallest dataset and models respectively in our experiments. This aligns with our intuition for variable projection as a more direct way to provide a training signal from the teacher model internals, which can assume more importance for a low-data or small-model scenario. However, for larger models and more data, the linear projection of parameters may be reducing the degrees of freedom available to the model, since linear projection is a fairly simple function to align the teacher and student parameter spaces. A related comparison of interest is between up-projection and down-projection of the model variables: we note up-projection does visibly better on the language modeling task and slightly better on the downstream tasks. The parameters of a well-trained teacher model represent a high-quality local minimum in the teacher space, which may be easier to search for during up-projection. Vocabulary size tradeoffs: Issues with input vocabulary size are peculiar to problems in natural language processing: they do not always apply to other areas such as computer vision, where a small fixed number of symbols can encode most inputs. There has been some work on reducing input vocabulary sizes for NLP, but typically not targeting model compression. One concern with reducing the vocabularies of NLP models is it pushes the average tokenized sequence lengths up, making model training harder. In this work, however, we consider classification tasks on shorter texts, which are not as affected by input sequence lengths as, say, tasks such as machine translation are. Furthermore, many real-world applications revolve around short text inputs, which is why a better trade-off between vocabulary size and sequence lengths may be worthwhile for such applications. Order of distillation and fine-tuning: Most of the existing work on distilling language models such as BERT and reporting on downstream tasks, including some of the baselines in this work, first fine-tune a teacher model on the downstream tasks, and then distill this model. Our goal in this work, however, is to explore the limits to which BERT's language modeling capacity itself, and how much of it is driven by its large WordPiece vocabulary. We leave experiments on distilling fine-tuned teacher models, potentially yielding better on downstream tasks, to future work. We proposed two novel ideas to improve the effectiveness of knowledge distillation for BERT, focusing on using a significantly smaller vocabulary, as well as smaller embedding and hidden dimensions for the student BERT language models. Our dual training mechanism encourages implicit alignment of the teacher and student WordPiece embeddings, and shared variable projection allows for the faster and direct layer-wise transfer of knowledge to the student BERT model. Combining the two techniques, we trained a series of highly-compressed 12-layer student BERT models. Experiments on these models, to evaluate both generalized language perspective and four standardized downstream tasks, demonstrate the effectiveness of our proposed methods on both model accuracy and compression efficiency. One future direction of interest is to combine our approach with existing work to reduce the number of layers in the student models and explore other approaches such as low-rank matrix factorization to transfer model parameters from the teacher space to the student space. In addition, taking into account the frequency distribution of the WordPiece tokens while training embeddings may help optimize the model size further.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We present novel distillation techniques that enable training student models with different vocabularies and compress BERT by 60x with minor performance drop.

Represent the TLDR for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.