Question: represent the article for finding a single-sentence tldr

Question: We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline in cases this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL. Recent work in the machine learning community has shown how optimization procedures can create new building-blocks for the end-to-end machine learning pipeline (; ; ; ; ; ; ; ;). In this paper we focus on the setting of optimizing an unconstrained, non-convex, and continuous objective function f θ (x): R n × Θ → R asx = arg min x f θ (x), where f is parameterized by θ ∈ Θ and has inputs x ∈ R n. If it exists, some (sub-)derivative ∇ θx is useful in the machine learning setting to make the output of the optimization procedure end-to-end learnable. For example, θ could parameterize a predictive model that is generating potential outcomes conditional on x happening that you want to optimize over. End-to-end learning in these settings can be done by defining a loss function L on top ofx and taking gradient steps ∇ θ L. If f θ were convex this gradient is easy to analyze and compute when it exists and is unique (; ; . Unfortunately analyzing and computing a "derivative" through the non-convex arg min here is not as easy and is challenging in theory and practice. No such derivative may exist in theory, it might not be unique, and even if it uniquely exists, the numerical solver being used to compute the solution may not find a global or even local optimum of f . One promising direction to sidestep these issues is to approximate the arg min operation with an explicit optimization procedure that is interpreted as just another compute graph and unrolled through. This is most commonly done with gradient descent as in ; ; ; ; ; ; . This approximation adds significant definition and structure to an otherwise extremely ill-defined desiderata at the cost of biasing the gradients and enabling the learning procedure to over-fit to the hyper-parameters of the optimization algorithm, such as the number of gradient steps or the learning rate. In this paper we show that the Cross-Entropy Method (CEM) is a reasonable alternative to unrolling gradient descent for approximating the derivative through an unconstrained, non-convex, and continuous arg min. CEM for optimization is a zeroth-order optimizer and works by generating a sequence of samples from the objective function. We show a simple and computationally negligible way of making CEM differentiable that we call DCEM by using the smooth top-k operation from. This also brings CEM into the end-to-end learning process in cases where there is otherwise a disconnection between the objective that is being learned and the objective that is induced by deploying CEM on top of those models. We first quickly study DCEM in a simple non-convex energy-based learning setting for regression. We contrast using unrolled gradient descent and DCEM for optimizing over a SPEN . We show that unrolling through gradient descent in this setting over-fits to the number of gradient steps taken and that DCEM generates a more reasonable energy surface. Our main application focus is on using DCEM in the context of non-convex continuous control. This setting is especially interesting as vanilla CEM is the state-of-the-art method for solving the control optimization problem with neural network transition dynamics as in;. We show that DCEM is useful for embedding action sequences into a lower-dimensional space to make solving the control optimization process significantly less computationally and memory expensive. This gives us a controller that induces a differentiable policy class parameterized by the model-based components. We then use PPO to fine-tune the modelbased components, demonstrating that it is possible to use standard policy learning for model-based RL in addition to just doing maximum-likelihood fitting to observed trajectories. Optimization-based modeling is a way of integrating specialized operations and domain knowledge into end-to-end machine learning pipelines, typically in the form of a parameterized arg min operation. Convex, constrained, and continuous optimization problems, e.g. as in;;;, capture many standard layers as special cases and can be differentiated through by applying the implicit function theorem to a set of optimality conditions from convex optimization theory, such as the KKT conditions. Non-convex and continuous optimization problems, e.g. as in; , are more difficult to differentiate through. Differentiation is typically done by unrolling gradient descent or applying the implicit function theorem to some set of optimality conditions, sometimes forming a locally convex approximation to the larger non-convex problem. Unrolling gradient descent is the most common way and approximates the arg min operation with gradient descent for the forward pass and interprets the operations as just another compute graph for the backward pass that can all be differentiated through. In contrast to these works, we show how continuous and nonconvex arg min operations can also be approximated with the cross entropy method as an alternative to unrolling gradient descent. Oftentimes the solution space of high-dimensional optimization problems may have structural properties that an optimizer can exploit to find a better solution or to find the solution quicker than an otherwise naïve optimizer. This is done in the context of meta-learning in where gradient-descent is unrolled over a latent space. In the context of Bayesian optimization this has been explored with random feature embeddings, hand-coded embeddings, and auto-encoder-learned embeddings (; ; ; ; ; ;). We show that DCEM is another reasonable way of learning an embedded domain for exploiting the structure in and efficiently solving larger optimization problems, with the significant advantage of DCEM being that the latent space is directly learned to be optimized over as part of the end-to-end learning pipeline. High-dimensional non-convex optimization problems that have a lot of structure in the solution space naturally arise in the control setting where the controller seeks to optimize the same objective in the same controller dynamical system from different starting states. This has been investigated in, e.g., planning (; ; ; ; ; ;), and policy distillation . shows how to learn an action space for model-free learning and; embed action sequences with a VAE. There has also been a lot of work on learning reasonable latent state space representations (; ; ; Miladinović et al., 2019) that may have structure imposed to make it more controllable (; ; ; ;). In contrast to these works, we learn how to encode action sequences directly with DCEM instead of auto-encoding the sequences. This has the advantages of 1) never requiring the expensive expert's solution to the control optimization problem, 2) potentially being able to surpass the performance of an expert controller that uses the full action space, and 3) being end-to-end learnable through the controller for the purpose of finding a latent space of sequences that DCEM is good at searching over. Another direction the RL and control has been pursuing is on the combination of model-based and model-free methods (; ; ; ; ; ; ;). proposes differentiable MPC and only do imitation learning on the cartpole and pendulum tasks with known or lightly-parameterized dynamics -in contrast, we are able to 1) scale our differentiable controller up to the cheetah and walker tasks, 2) use neural network dynamics inside of our controller, and 3) backpropagate a policy loss through the output of our controller and into the internal components. We focus on uses of the Cross-Entropy Method (CEM) for optimization in this paper. In this setting, suppose we have a non-convex, deterministic, and continuous objective function f θ (x) parameterized by θ over a domain R n and we want to solve the optimization problem The original form of CEM is an iterative and zeroth-order algorithm to approximate the solution of eq. with a sequence of samples from a sequence of parametric sampling distributions g φ defined over the domain R n, such as Gaussians. We refer the reader to for more details and motivations for using CEM and briefly describe how it works here. Given a sampling distribution g φ, the hyper-parameters of CEM are the number of candidate points sampled in each iteration N, the number of elite candidates k to use to fit the new sampling distribution to, and the number of iterations T. The iterates of CEM are the parameters φ of the sampling distribution. CEM starts with an initial sampling distribution g φ1 (X) ∈ R n, and in each iteration t generates N samples from the domain, evaluates the function at those points v t,i = f θ (X t,i), and re-fits the sampling distribution to the top-k samples by solving the maximum-likelihood problem DCEM minimizes a parameterized objective function f θ and is differentiable w.r.t. θ. Each DCEM iteration samples from the distribution g φ, starting with φ 1. Evaluate the objective function at those points Compute the soft top-k projection of the values with eq. Update φ t+1 by solving the maximum weighted likelihood problem in eq. (5 Differentiating through CEM's output with respect to the objective function's parameters with ∇ θx is useful, e.g., to bring CEM into the end-to-end learning process in cases where there is otherwise a disconnection between the objective that is being learned and the objective that is induced by deploying CEM on top of those models. Unfortunately in the vanilla form presented above the top-k operation in eq. makesx non-differentiable with respect to θ. The function samples can usually be differentiated through with some estimator such as the reparameterization trick , which we use in all of our experiments. The top-k operation can be made differentiable by replacing it with a soft version as done in;; , or by using a stochastic oracle as in?. Here we use the Limited Multi-Label Projection (LML) layer , which projects points from R n onto the LML polytope defined by which is the set of points in the unit n-hypercube with coordinates that sum to k. Notationally, if n is implied by the context we will leave it out and write L k. We propose a temperature-scaled LML variant to project onto the interior of the LML polytope with where τ > 0 is the temperature parameter and H b (y) = − i y i log y i + (1 − y i) log(1 − y i) is the binary entropy function. Equation is a convex optimization layer and can be solved in a negligible amount of time with a GPU-amenable bracketing method on the univariate dual and quickly backpropagated through with implicit differentiation. We can use the LML layer to make a soft and differentiable version of eq. as This is now a maximum weighted likelihood estimation problem (; ;), which still admits an analytic closed-form solution in many cases, e.g. for the natural exponential family . Thus using the soft top-k operation with the reparameterization trick, e.g., on the samples from g in a differentiable variant of CEM that we call DCEM and summarize in alg. 1. We note that we usually also normalize the values in each iteration to help separate the scaling of the values from the temperature parameter. Proposition 2. The temperature-scaled LML layer Π L k (x /τ) approaches the hard top-k operation as τ → 0 + when all components of x are unique. We prove this in app. A by using the KKT conditions of eq.. Corollary 1. DCEM captures CEM as a special case as τ → 0 +. Proposition 3. With an isotropic Gaussian sampling distribution, the update in eq. becomes, where the soft top-k indexing set is This can be proved by differentiating eq., as discussed in, e.g., . As a corollary, this captures prop. 1 as τ → 0 +. Energy-based learning for regression and classification estimate the conditional probability P(y|x) of an output y ∈ Y given an input x ∈ X with a parameterized energy function Predictions are made by solving the optimization problemŷ = arg min Historically linear energy functions have been well-studied, e.g. in Taskar easier to solve and analyze. More recently non-convex energy functions that are parameterized by neural networks are being explored -a popular one being Structured Prediction Energy Networks (SPENs) which propose to model E θ with neural networks. suggests to do supervised learning of SPENs by approximating eq. with gradient descent that is then unrolled for T steps, i.e. by starting with some y 0, making gradient updates y t+1 = y t + γ∇ y E θ (y t |x) ing in an outputŷ = y T, defining a loss function L on top ofŷ, and doing learning with gradient updates ∇ θ L that go through the inner gradient steps. In this context we can alternatively use DCEM to approximate eq.. One potential consideration when training deep energy-based models with approximations to eq. is the impact and bias that the approximation is going to have on the energy surface. We note that for gradient descent, e.g., it may cause the energy surface to overfit to the number of gradient steps so that the output of the approximate inference procedure isn't even a local minimum of the energy surface. One potential advantage of DCEM is that the output is more likely to be near a local minimum of the energy surface so that, e.g., more test-time iterations can be used to refine the solution. We empirically illustrate the impact of the optimizer choice on a synthetic example in sect. 5.1. Our main application focus is in the continuous control setting where we show how to use DCEM to learn a latent control space that is easier to solve than the original problem and induces a differentiable policy class that allows parts of the controller to be fine-tuned with auxiliary policy or imitation losses. We are interested in controlling discrete-time dynamical systems with continuous state-action spaces. Let H be the horizon length of the controller and U H be the space of control sequences over this horizon length, e.g. U could be a multi-dimensional real space or box therein and U H could be the Cartesian product of those spaces representing the sequence of controls over H timesteps. We are interested in repeatedly solving the control optimization problem 2 u 1:H = arg min Algorithm 2 Learning an embedded control space with DCEM Fixed Inputs: Dynamics f trans, per-step state-action cost Update the decoder to improve the controller's cost end while where we are in an initial system state x init governed by deterministic system transition dynamics f trans, and wish to find the optimal sequence of actionsû 1:H such that we find a valid trajectory {x 1:H, u 1:H} that optimizes the cost C t (x t, u t). Typically these controllers are used for receding horizon control where only the first action u 1 is deployed on the real system, a new state is obtained from the system, and the eq. is solved again from the new initial state. In this case we can say the controller induces a policy π(x init) ≡û 1 3 that solves eq. and depends on the cost and transition dynamics, and potential parameters therein. In all of the cases we consider f trans is deterministic, but may be approximated by a stochastic model for learning. Some model-based reinforcement learning settings consider cases where f trans and C are parameterized and potentially used in conjunction with another policy class. For sufficiently complex dynamical systems, eq. is computationally expensive and numerically instable to solve and rife with sub-optimal local minima. The Cross-Entropy Method is the state-ofthe-art method for solving eq. with neural network transitions f trans . CEM in this context samples full action sequences and refines the samples towards ones that solve the control problem. uses CEM with 1000 samples in each iteration for 10 iterations with a horizon length of 12. This requires 1000 × 10 × 12 = 120, 000 evaluations of the transition dynamics to predict the control to be taken given a system stateand the transition dynamics may use a deep recurrent architecture as in or an ensemble of models as in. One comparison point here is a model-free neural network policy takes a single evaluation for this prediction, albeit sometimes with a larger neural network. The first application we show of DCEM in the continuous control setting is to learn a latent action space Z with a parameterized decoder f dec θ: Z → U H that maps back up to the space of optimal action sequences, which we illustrate in fig. 3. For simplicity starting out, assume that the dynamics and cost functions are known (and perhaps even the ground-truth) and that the only problem is to estimate the decoder in isolation, although we will show later that these assumptions can be relaxed. The motivation for having such a latent space and decoder is that the millions of times eq. is being solved for the same dynamic system with the same cost, the solution space of optimal action sequencesû 1:H ∈ U H has an extremely large amount of spatial (over U) and temporal (over time in U H) structure that is being ignored by CEM on the full space. The space of optimal action sequences only contains the knowledge of the trajectories that matter for solving the task at hand, such as different parts of an optimal gait, and not irrelevant control sequences. We argue that CEM over the full action space wastes a lot of computation considering irrelevant action sequences and show that these can be ignored by learning a latent space of more reasonable candidate solutions here that we search over instead. Given a decoder, the control optimization problem in eq. can then be transformed into an optimization problem over Z aŝ which is still a challenging non-convex optimization problem that searches over a decoder's input space to find the optimal control sequence. We illustrate what this looks like in fig. 3, and note the impact of the decoder initialization in app. C. We propose in alg. 2 to use DCEM to approximately solve eq. and then learn the decoder directly to optimize the performance of eq.. Every time we solve eq. with DCEM and obtain an optimal latent representationẑ along with the induced trajectory {x t, u t}, we can take a gradient step to push down the ing cost of that trajectory with ∇ θ C(ẑ), which goes through the DCEM process that uses the decoder to generate samples to obtainẑ. We note that the DCEM machinery behind this is not necessary if a reasonable local minima is consistently found as this is an instance of min-differentiation (, Theorem 10.13) but in practice this breaks down in non-convex cases when the minimum cannot be consistently found.; solve related problems in this space and we discuss them in sect. 2.3. We also note that to learn an action embedding we still need to differentiate through the transition dynamics and cost functions to compute ∇ θ C(ẑ), even if the ground-truth ones are being used, since the latent space needs to have the knowledge of how the control cost will change as the decoder's parameters change. DCEM in this setting also induces a differentiable policy class π(x init) ≡ u 1 = f dec (ẑ) 1. This enables a policy or imitation loss J to be defined on the policy that can fine-tune the parts of the controller (decoder, cost, and transition dynamics) gradient information from ∇ θ J. In theory the same approach could be used with CEM on the full optimization problem in eq.. For realistic problems without modification this is intractable and memory-intensive as it would require storing and backpropagating through every sampled trajectory, although as a future direction we note that it may be possible to delete some of the low-influence trajectories to help overcome this. We use PyTorch and will openly release our DCEM library, model-based control code, and the source, plotting, and analysis code for all of our experiments. In this section we briefly explore the impact of the inner optimizer on the energy surface of a SPEN as discussed in sect. 4.1. For illustrative purposes we consider a simple unidimensional regression task where the ground-truth data is generated from f (x) = x sin(x) for x ∈ [0, 2π]. We model P(y|x) ∝ exp{−E θ (y|x)} with a single neural network E θ and make predictionsŷ by solving the optimization problem eq.. Given the ground-truth output y, we use the loss L(ŷ, y) = ||ŷ − y || 2 2 and take gradient steps of this loss to shape the energy landscape. We consider approximating eq. with unrolled gradient descent and DCEM with Gaussian sampling distributions. Both of these are trained to take 10 optimizer steps and we use an inner learning rate of 0.1 for gradient descent and with DCEM we use 10 iterations with 100 samples per iteration and 10 elite candidates, with a temperature of 1. For both algorithms we start the initial iterate at y 0 = 0. We show in app. B that both of these models attain the same loss on the training dataset but, since this is a unidimensional regression task, we can visualize the entire energy surfaces over the joint input-output space in fig. 1. This shows that gradient descent has learned to adapt from the initial y 0 = 0 position to the final position by descending along the function's surface as we would expect, but there is no reason why the energy surface should be a local minimum around the last iterateŷ = y 10. The energy surface learned by CEM captures local minima around the regression target as the sequence of Gaussian iterates are able to capture a more global view of the function landscape and need to focus in on a minimum of it for regression. We show ablations in app. B from training for 10 inner iterations and then evaluating with a different number of iterations and show that gradient descent quickly steps away from making reasonable predictions. Discussion and limitations. We note that other tricks could be used to force the output to be at a local minimum with gradient descent, such as using multiple starting points or randomizing the number of gradient descent steps taken -our intention here is to highlight this behavior in the vanilla case. We also note that DCEM is susceptible to overfitting to the hyper-parameters behind it in similar, albeit less obvious ways. Eval Reward walker.walk Figure 4: We evaluated our final models by running 100 episodes each on the cheetah and walker tasks. CEM over the full action space uses 10,000 trajectories for control at each time step while embedded DCEM samples only 1000 trajectories. DCEM almost recovers the performance of CEM over the full action space and PPO fine-tuning of the model-based components helps bridge the gap. We first show that it is possible to learn an embedded control space as discussed in sect. 4.2 in an isolated setting. We use the standard cartpole dynamical system from with a continuous state-action space. We assume that the ground-truth dynamics and cost are known and use the differentiable ground-truth dynamics and cost implemented in PyTorch from. This isolates the learning problem to only learning the embedding so that we can study what this is doing without the additional complications that arise from exploration, estimating the dynamics, learning a policy, and other non-stationarities. We show experiments with these assumptions relaxed in sect. 5.2.2. We use DCEM and alg. 2 to learn a 2-dimensional latent space Z = 2 that maps back up to the full control space U H = H where we focus on horizons of length H = 20. For DCEM over the embedded space we use 10 iterations with 100 samples in each iteration and 10 elite candidates, again with a temperature of 1. We show the details in app. D that we are able to recover the performance of an expert CEM controller that uses an order-of-magnitude more samples fig. 2 shows a visualization of what the CEM and embedded DCEM iterates look like to solve the control optimization problem from the same initial system state. CEM spends a lot of evaluations on sequences in the control space that are unlikely to be optimal, such as the ones the bifurcate between the boundaries of the control space at every timestep, while our embedded space is able to learn more reasonable proposals. Next we show that we can relax the assumptions of having known transition dynamics and reward and show that we can learn a latent control space on top of a learned model on the cheetah.run and walker.walk tasks with frame skips of 4 and 2, respectively, from the DeepMind control suite using the MuJoCo physics engine . We then fine-tune the policy induced by the embedded controller with PPO , sending the policy loss directly back into the reward and latent embedding modules underlying the controller. We start with a state-of-the-art model-based RL approach by noting that the PlaNet restricted state space model (RSSM) is a reasonable architecture for proprioceptive-based control in addition to just pixel-based control. We show the graphical model we use in fig. 3, which maintains deterministic hidden states h t and stochastic (proprioceptive) system observations x t and rewards r t. We model transitions as h t+1 = f trans θ (h t, x t), observations with x t ∼ f odec θ (h t), rewards with r t = f rew θ (h t, x t), and map from the latent action space to action sequences with u 1:T = f dec (z). We follow the online training procedure of to initialize all of the models except for the action decoder f dec, using approximately 2M timesteps. We then use a variant of alg. 2 to learn f dec to embed the action space for control with DCEM, which we also do online while updating the models. We describe the full training process in app. E. Figure 3: Our RSSM with action sequence embeddings Our DCEM controller induces a differentiable policy class π θ (x init) where θ are the parameters of the models that impact the actions that the controller is selecting. We then use PPO to define a loss on top of this policy class and fine-tune the components (the decoder and reward module) so that they improve the episode reward rather than the maximumlikelihood solution of observed trajectories. We chose PPO because we thought it would be able to fine-tune the policy with just a few updates because the policy is starting at a reasonable point, but this did not turn out to be the case and in the future other policy optimizers can be explored. We implement this by making our DCEM controller the policy in the PyTorch PPO implementation by. We provide more details behind our training procedure in app. E. We evaluate our controllers on 100 test episodes and the rewards in fig. 4 show that DCEM is almost (but not exactly) able to recover the performance of doing CEM over the full action space while using an order-of-magnitude less trajectory samples (1,000 vs 10,0000). PPO fine-tuning helps bridge the gap between the performances. Videos of our trained models are available at: https://sites.google.com/view/diff-cross-entropy-method Discussion and limitations. DCEM in the control setting has many potential future directions to explore and help bring efficiency and policy-based fine-tuning to model-based reinforcement learning. Much more analysis and experimentation is necessary to achieve this as we faced many issues getting the model-based cheetah and walker tasks to work that did not arise in the ground-truth cartpole task. We discuss this more in app. E. We also did not focus on the sample complexity of our algorithms getting these proof-of-concept experiments working. We also note that other reasonable baselines on this task could involve distilling the controller into a model-free policy and then doing search on top of that policy, as done in POPLIN . We have laid the foundations for differentiating through the cross-entropy method and have brought CEM into the end-to-end learning pipeline. Beyond further explorations in the energy-based learning and control contexts we showed here, DCEM can be used anywhere gradient descent is unrolled. We find this especially promising for meta-learning, potentially building on LEO . Inspired by DCEM, other more powerful sampling-based optimizers could be made differentiable in the same way, potentially optimizers that leverage gradient-based information in the inner optimization steps (; ; ;) or by also learning the hyper-parameters of structured optimizers (; ;). We acknowledge the scientific Python community for developing the core set of tools that enabled this work, including PyTorch , Jupyter , Matplotlib (, seaborn , numpy , pandas (, and SciPy . A PROOF OF PROP. 2 Proof. We first note that a solution exists to the projection operation, and it is unique, which comes from the strict convexity of the objective . The Lagrangian of the temperature-scaled LML projection in eq. is Differentiating eq. gives and the first-order optimality condition ∇ y L(y, ν) = 0 gives y i = σ(τ −1 (x i + ν *)), where σ is the sigmoid function. Using lem. 1 as τ → 0 + gives Substituting this back into the constraint 1 y = k gives that π(x) k < −ν * < π(x) k+1, where π(x) sorts x ∈ R n in ascending order so that π(x) 1 ≤ π(x) 2 ≤... ≤ π(x) n. Thus we have that y i = 1{x i ≥ π(x) k }, which is 1 when x i is in the top-k components of x and 0 otherwise, and therefore the temperature-scaled LML layer approaches the hard top-k function as τ → 0 +. where σ(x /τ) = (1 + exp{ −x /τ}) −1 is the temperature-scaled sigmoid. We have found the decoder to be influenced by the activation function that's used with it and have found the ELU to perform the best. fig. 7 conveys some intuition behind this choice. We randomly initialize a neural network u = f θ (z) with no biases, where θ = {W i} i for every layer weight W i, and then scale the weights with αθ. We then sample z ∼ N (0, I), pass them through f αθ, and plot the outputs. The ReLU induces an extremely biased distribution which is seen more prevalently as α grows that is not as present when using the ELU or hyperbolic tangent since they are almost linear around zero. Despite the reasonable looking initializations for the hyperbolic tangent, we found that it does not perform as well in practice in our experiments. We found that the initial scale α of the decoder's parameters is also important for learning because of the network is not initially producing samples that cover the full output space as shown with α = 1, it seems hard for it to learn how to expand to cover the full output space. In this section we discuss some of the ablations we considered when learning the latent action space for the cartpole task. In all settings we use DCEM to unroll 10 inner iterations that samples 100 candidate points in each iteration and has an elite set of 10 candidates. For training, we randomly sample initial starting points of the cartpole and for validation we use a fixed set of initial points. Figure 8 shows the convergence of models as we vary the latent space dimension and temperature parameter, and fig. 9 shows that DCEM is able to fully recover the expert performance on the cartpole. Because we are operating in the ground-truth dynamics setting we measure the performance by comparing the controller costs. We use τ = 0 to indicate the case where we optimize over the latent space with vanilla CEM and then update the decoder with, where the gradient doesn't go back into the optimization process that producedẑ. This is non-convex min differentiation and is reasonable whenẑ is near-optimal, but otherwise is susceptible to making the decoder difficult to search over. These show a few interesting points that come up in this setting, which of course may be different in other settings. Firstly that with a two-dimensional latent space, all of the temperature values are able to find a reasonable latent space at some point during training. However after more updates, the lower-temperature experiments start updating the decoder in ways that make it more difficult to search over and start achieving worse performance than the τ = 1 case. For higher-dimensional latent spaces, the DCEM machinery is necessary to keep the decoder searchable. Furthermore we notice that just a 16-dimensional latent space for this task can be difficult for learning, one reason this could be is from DCEM having too many degrees of freedom in ways it can update the decoder to improve the performance of the optimizer.: Improvement factor on the ground-truth cartpole task from embedding the action space with DCEM compared to running CEM on the full action space, showing that DCEM is able to recover the full performance. We use the DCEM model that achieves the best validation loss. The error lines show the 95% confidence interval around three trials. Figure 10: Learned DCEM reward surfaces for the cartpole task. Each row shows a different initial state of the system. We can see that as the temperature decreases, the latent representation can still capture near-optimal values, but they are in much narrower regions of the latent space than when τ = 1. Algorithm 3 PlaNet variant that we use for proprioceptive control with optional DCEM embedding Models: a deterministic state model, a stochastic state model, a reward model, and (if using DCEM) an action sequence decoder. Initialize dataset D with S random seed episodes. Initialize the transition model's deterministic hidden state h 0 and initialize the environment, obtaining the initial state estimate x 0. CEM-Solve can use DCEM or full CEM for t = 1,..., T do u t ← CEM-solve(h t−1, x t−1) Add exploration noise ∼ p to the action u t. Obtain the hidden states of the {h τ,x τ} from the model. Compute the multi-step likelihood bound L(τ, h τ,x τ) (, eq 6 .) Optimize the likelihood bound if using DCEM then Update the decoder end if end if end for For the cheetah.run and walker.walk DeepMind control suite experiments we start with a modified PlaNet architecture that does not have a pixel decoder. We started with this over PETS to show that this RSSM is reasonable for proprioceptivebased control and not just pixel-based control. This model is graphically shown in fig. 3 and has 1) a deterministic state model h t = f (h t−1, x t−1, u t−1), 2) a stochastic state model x t ∼ p(x t, h t), and 3) a reward model: r t ∼ p(r t |h t, x t). In the proprioceptive setting, we posit that the deterministic state model is useful for multi-step training even in fully observable environments as it allows the model to "push forward" information about what is potentially going to happen in the future. For the modeling components, we follow the recommendations in and use a GRU with 200 units as the deterministic path in the dynamics model and implement all other functions as two fully-connected layers, also with 200 units with ReLU activations. Distributions over the state space are isotropic Gaussians with predicted mean and standard deviation. We train the model to optimize the variational bound on the multi-step likelihood as presented in on batches of size 50 with trajectory sequences of length 50. We start with 5 seed episodes with random actions and in contrast to , we have found that interleaving the model updates with the environment steps instead of separating the updates slightly improves the performance, even in the pixel-based case, which we do not report on here. For the optimizers we either use CEM over the full control space or DCEM over the latent control space and use a horizon length of 12 and 10 iterations here. For full CEM, we sample 1000 candidates in each iteration with 100 elite candidates. For DCEM we use 100 candidates in each iteration with 10 elite candidates. Our training procedure has the following three phases, which we set up to isolate the DCEM additions. We evaluate the models output from these training runs on 100 random episodes in fig. 4 in the main paper. Now that these ideas have been validated, promising directions of future work include trying to combine them all into a single training run and trying to reduce the sample complexity and number of timesteps needed to obtain the final model. Under review as a conference paper at ICLR 2020 Phase 1: Model initialization. We start in both environments by launching a single training run of fig. 11 to get initial system dynamics. fig. 11 shows that these starting points converge to near-stateof-the-art performance on these tasks. These models take slightly longer to converge than in , likely due to how often we update our models. We note that at this point, it would be ideal to use the policy loss to help fine-tune the components so that policy induced by CEM on top of the models can be guided, but this is not feasible to do by backpropagating through all of the CEM samples due to memory, so we instead next move on to initializing a differentiable controller that is feasible to backprop through. Phase 2: Embedded DCEM initialization. Our goal in this phase is to obtain a differentiable controller that is feasible to backprop through. Our first failed attempt to achieve this was to use offline training on the replay buffer, which would have been ideal as it would require no additional transitions to be collected from the environment. We tried using alg. 2, the same procedure we used in the ground-truth cartpole setting, to generate an embedded DCEM controller that achieves the same control cost on the replay buffer as the full CEM controller. However we found that when deploying this controller on the system, it quickly stepped off of the data manifold and failed to control it -this seemed to be from the controller finding holes in the model that causes the reward to be over-predicted. We then used an online data collection process identical to the one we used for phase 1 to jointly learn the embedded control space while updating the models so that the embedded controller doesn't find bad regions in them. We show where the DCEM updates fit into alg. 3. One alternative that we tried to updating the decoder to optimize the control cost on the samples from the replay buffer is that the decoder can also be immediately updated after planning at every step. This seemed nice since it didn't require any additional DCEM solves, but we found that the decoder became too biased during the episode as samples at consecutive timesteps have nearly identical information. For the hyper-parameters, we kept most of the DCEM hyper-parameters fixed throughout this phase to 100 samples, 10 elites, and a temperature τ = 1. We ablated across 1) the number of DCEM iterations taken to be {3, 5, 10}, 2) deleting the replay buffer from phase 1 or not, and 3) re-initializing the model or not from phase 1. We report the best runs that we use as the starting point for the next phase in fig. 12, which achieve reasonable performance but don't match the performance of doing CEM over the full action space. These runs all use 10 DCEM iterations and both keep the replay buffer from phase 1. The Cheetah run keeps the models from phase 1 and the Walker re-initializes the models. The cheetah curve around timestep 600k shows, the stability here can be improved as sometimes the decoder finds especially bad regions in the model that induce extremely high losses. Phase 3: Policy optimization into the controller. Finally now that we have a differentiable policy class induced by this differentiable controller we can do policy learning to fine-tune parts of it. We initially chose Proximal Policy Optimization (PPO) for this phase because we thought that it would be able to fine-tune the policy in a few iterations without requiring a good estimate of the value function, but this phase also ended up consuming many timesteps from the environment. Crucially in this phase, we do not do likelihood fitting at all, as our goal is to show that PPO can be used as another useful signal to update the parts of a controller -we did this to isolate the improvement from PPO but in practice we envision more unified algorithms that use both signals at the same time. Using the standard PPO hyper-parameters, we collect 10 episodes for each PPO training step and ablate across 1) the number of passes to make through these episodes {1, 2, 4}, 2) every combination of the reward, transition, and decoder being fine-tuned or frozen, 3) using a fixed variance of 0.1 around the output of the controller or learning this, 4) the learning rate of the fine-tuned model-based portions {10 −4, 10 −5}. Figure 13 shows the of the best runs from this search. We conclude by showing the PPO-fine-tuned DCEM iterates for solving a single control optimization problem from a random system state for the cheetah fig. 14. and walker fig. 15 tasks. PlaNet+DCEM+PPO walker.walk

represent this text to find a much much longer article about it with say hundreds-thousands of words

DCEM learns latent domains for optimization problems and helps bridge the gap between model-based and model-free RL --- we create a differentiable controller and fine-tune parts of it with PPO

Represent the short summary for finding the corresponding scientific article (long)

first posing and solving the sample efficiency optimization problem in the non-parameterized policy space, and then solving a supervised regression problem to find a parameterized policy that is near the optimal non-parameterized policy.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose a new framework for entity and event extraction based on generative adversarial imitation learning -- an inverse reinforcement learning method using generative adversarial network (GAN). We assume that instances and labels yield to various extents of difficulty and the gains and penalties (rewards) are expected to be diverse. We utilize discriminators to estimate proper rewards according to the difference between the labels committed by ground-truth (expert) and the extractor (agent). Experiments also demonstrate that the proposed framework outperforms state-of-the-art methods. Event extraction (EE) is a crucial information extraction (IE) task that focuses on extracting structured information (i.e., a structure of event trigger and arguments, "what is happening", and "who or what is involved ") from unstructured texts. In most recent five years, many event extraction approaches have brought forth encouraging by retrieving additional related text documents BID18, introducing rich features of multiple categories [BID26, incorporating relevant information within or beyond context BID23, BID24 BID7 and adopting neural network frameworks BID4, BID8, BID8, BID17, BID13 BID27 .There are still challenging cases: for example, in the following sentences: "Masih's alleged comments of blasphemy are punishable by death under Pakistan Penal Code" and "Scott is charged with first-degree homicide for the death of an infant. ", the word death can trigger an Execute event in the former sentence and a Die event in the latter one. With similar local information (word embeddings) or contextual features (both sentences include legal events), supervised models pursue the probability distribution which resembles that in the training set (in ACE2005 data, we have overwhelmingly more Die annotation on death than Execute), and will label both as Die event, causing error in the former instance. Such mistake is due to the lack of a mechanism that explicitly deals with wrong and confusing labels. Many multi-classification approaches utilize cross-entropy loss, which aims at boosting the probability of the correct labels. Many approaches -including AdaBoost which focuses weights on difficult cases -usually treat wrong labels equally and merely inhibits them indirectly. Models are trained to capture features and weights to pursue correct labels, but will become vulnerable and unable to avoid mistakes when facing ambiguous instances, where the probabilities of the confusing and wrong labels are not sufficiently "suppressed". Therefore, exploring information from wrong labels is a key to make the models robust. In this paper, we propose a dynamic mechanism -inverse reinforcement learning -to directly assess correct and wrong labels on instances in entity and event extraction. We assign explicit scores on cases -or rewards in terms of Reinforcement Learning (RL). We adopt discriminators from generative adversarial networks (GAN) to estimate the reward values. Discriminators ensures the highest reward for ground-truth (expert) and the extractor attempts to imitate the expert by pursuing highest rewards. For challenging cases, if the extractor continues selecting wrong labels, the GAN keeps expanding the margins between rewards for ground-truth labels and (wrong) extractor labels and eventually deviates the extractor from wrong labels. The main contributions of this paper can be summarized as follows: • We apply reinforcement learning framework to event extraction tasks, and the proposed framework is an end-to-end and pipelined approach that extracts entities and event triggers and determines the argument roles for detected entities.• With inverse reinforcement learning propelled by GAN, we demonstrate that a dynamic reward function ensures more optimal performance in a complicated RL task. We follow the schema of Automatic Content Extraction (ACE) 1 to detect the following elements from unstructured natural language text:• Entity: word or phrase that describes a real world object such as a person ("Masih" as PER in Figure 1). ACE schema defines 7 types of entities.• Event Trigger: the word that most clearly expresses an event (interaction or change of status). ACE schema defines 33 types of events such as Sentence ("punishable" in Figure 1) and Execute ("death").• Event argument: an entity that serves as a participant or attribute with a specific role in an event mention, e.g., a PER "Masih" serves as a Defendant in a Sentence event triggered by "punishable". For broader readers who might not be familiar with reinforcement learning, we briefly introduce by their counterparts or equivalent concepts in supervised models with the RL terms in the parentheses: our goal is to train an extractor (agent A) to label entities, event triggers and argument roles (actions a) in text (environment e); to commit correct labels, the extractor consumes features (state s) and follow the ground truth (expert E); a reward R will be issued to the extractor according to whether it is different from the ground truth P l a c e N / A P e r s o n A g e n t P l a c e N / A P e r s o n A g e n tFigure 1: Our framework includes a reward estimator based on GAN to issue dynamic rewards with regard to the labels (actions) committed by event extractor (agent). The reward estimator is trained upon the difference between the labels from ground truth (expert) and extractor (agent). If the extractor repeatedly misses Execute label for "death", the penalty (negative reward values) is strengthened; if the extractor make surprising mistakes: label "death" as Person or label Person "Masih" as Place role in Sentence event, the penalty is also strong. For cases where extractor is correct, simpler cases such as Sentence on "death" will take a smaller gain while difficult cases Execute on "death" will be awarded with larger reward values.and how serious the difference is -as shown in Figure 1, a repeated mistake is definitely more serious -and the extractor improves the extraction model (policy π) by pursuing maximized rewards. Our framework can be briefly described as follows: given a sentence, our extractor scans the sentence and determines the boundaries and types of entities and event triggers using Q-Learning (Section 3.1); meanwhile, the extractor determines the relations between triggers and entities -argument roles with policy gradient (Section 3.2). During the training epochs, GANs estimate rewards which stimulate the extractor to pursue the most optimal joint model (Section 4). The entity and trigger detection is often modeled as a sequence labeling problem, where longterm dependency is a core nature; and reinforcement learning is a well-suited method [BID15 are also good candidates for context embeddings. From RL perspective, our extractor (agent A) is exploring the environment, or unstructured natural language sentences when going through the sequences and committing labels (actions a) for the tokens. When the extractor arrives at tth token in the sentence, it observes information from the environment and its previous action a t−1 as its current state s t; the extractor commits a current action a t and moves to the next token, it has a new state s t+1. The information from the environment is token's context embedding v t, which is usually acquired from Bi-LSTM BID12 outputs; previous action a t−1 may impose some constraint for current action a t, e.g., I-ORG does not follow B-PER 2. With the aforementioned notations, we have DISPLAYFORM0 To determine the current action a t, we generate a series of Q-tables with DISPLAYFORM1 where f sl (·) denotes a function that determine the Q-values using the current state as well as previous states and actions. Then we achievê DISPLAYFORM2 Equation 2 and 3 suggest that an RNN-based framework which consumes current input and previous inputs and outputs can be adopted, and we use a unidirectional LSTM as BID1. We have a full pipeline as illustrated in Figure 2.2. In this work, we use BIO, e.g., "B-Meet" indicates the token is beginning of Meet trigger, "I-ORG" means that the token is inside an organization phrase, and "O" denotes null., with fixed rewards r = ±5 for correct/wrong labels and discount factor λ = 0.01. Score for wrong label is penalized while correct one is reinforced. For each label (action a t) with regard to s t, a reward r t = r(s t, a t) is assigned to the extractor (agent). We use Q-learning to pursue the most optimal sequence labeling model (policy π) by maximizing the expected value of the sum of future rewards E(R t), where R t represents the sum of discounted future rewards r t + γr t+1 + γ 2 r t+2 +... with a discount factor γ, which determines the influence between current and next states. We utilize Bellman Equation to update the Q-value with regard to the current assigned label to approximate an optimal model (policy π *). DISPLAYFORM3 As illustrated in FIG0, when the extractor assigns a wrong label on the "death" token because the Q-value of Die ranks first, Equation 4 will penalize the Q-value with regard to the wrong label; while in later epochs, if the extractor commits a correct label of Execute, the Q-value will be boosted and make the decision reinforced. We minimize the loss in terms of mean squared error between the original and updated Q-values notated as Q sl (s t, a t): DISPLAYFORM4 and apply back propagation to optimize the parameters in the neural network. After the extractor determines the entities and triggers, it takes pairs of one trigger and one entity (argument candidate) to determine whether the latter serves a role in the event triggered by the former. In this task, for each pair of trigger and argument candidate, our extractor observes the context embeddings of trigger and argument candidate -v ttr and v tar respectively, as well as the output of another Bi-LSTM consuming the sequence of context embeddings between trigger and argument candidates in the state; the state also includes a representation (onehot vector) of the entity type of the argument candidate a tar, and the event type of the trigger a tar also determine the available argument role labels, e.g., an Attack event never has Adjudicator arguments as Sentence events. With these notations we have: DISPLAYFORM0 where the footnote tr denotes the trigger, ar denotes argument candidate, and f ss denotes the sub-sentence Bi-LSTM for the context embeddings between trigger and argument. We have another ranking table for argument roles: DISPLAYFORM1 where f tr,ar represents a mapping function whose output sizes is determined by the trigger event type a ttr. e.g., Attack event has 5 -Attacker, Target, Instrument, Place and Not-a-role labels and the mapping function for Attack event contains a fully-connected layer with output size of 5. And we determine the role witĥ a tr,ar = arg max atr,ar Q tr,ar (s tr,ar, a tr,ar).We assign a reward r(s tr,ar, a tr,ar) to the extractor, and since there is one step in determining the argument role label, the expected values of R = r(s tr,ar, a tr,ar).We utilize another RL algorithm -Policy Gradient BID19 to pursue the most optimal argument role labeling performance. We have probability distribution of argument role labels that are from the softmax output of Q-values: P (a tr,ar |s tr,ar) = softmax(Q tr,ar (s tr,ar, a tr,ar)).To update the parameters, we minimize loss function DISPLAYFORM2 From Equation 10 and FIG1 we acknowledge that, when the extractor commits a correct label (Agent for the GPE entity "Pakistan"), the reward encourages P (a tr,ar |s tr,ar) to increase; and when the extractor is wrong (e.g., Place for "Pakistan"), the reward will be negative, leading to a decreased P (a tr,ar |s tr,ar). Here we have a brief clarification on different choices of RL algorithms in the two tasks. In the sequence labeling task, we do not take policy gradient approach due to high variance of E(R t), i.e., the sum of future rewards R t should be negative when the extractor chooses a wrong label, but an ill-set reward and discount factor γ assignment or estimation may give a positive R t (often with a small value) and still push up the probability of the wrong action, which is not desired. There are some variance reduction approaches to constrain the R t but they still need additional estimation and bad estimation will introduce new risk. Q-learning only requires rewards on current actions r t, which are relatively easy to constrain. In the argument role labeling task, determination on each trigger-entity pair consists of only one single step and R t is exactly the current reward r, then policy gradient approach performs correctly if we ensure negative rewards for wrong actions and positive for correct ones. However, this one-step property impacts the Q-learning approach: without new positive values from further steps, a small positive reward on current correct label may make the updated Q-value smaller than those wrong ones. So far in our paper, the reward values demonstrated in the examples are fixed, we have DISPLAYFORM0 and typically we have c 1 > c 2. This strategy makes RL-based approach no difference from classification approaches with cross-entropy in terms of "treating wrong labels equally" as discussed in introductory section. Moreover, recent RL approaches on relation extraction BID25 BID9 ] adopt a fixed setting of reward values with regard to different phases of entity and relation detection based on empirical tuning, which requires additional tuning work when switching to another data set or schema. In event extraction task, entity, event and argument role labels yield to a complex structure with variant difficulties. Errors should be evaluated case by case, and from epoch to epoch. In the earlier epochs, when parameters in the neural networks are slightly optimized, all errors are tolerable, e.g., in sequence labeling, extractor within the first 2 or 3 iterations usually labels most tokens with O labels. As the epoch number increases, the extractor is expected to output more correct labels, however, if the extractor makes repeated mistakes -e.g., the extractor persistently labels"death" as O in the example sentence "... are punishable by death..." during multiple epochs -or is stuck in difficult cases -e.g., whether FAC (facility) token "bridges" serves as a Place or Target role in an Attack event triggered by "bombed " in sentence "U.S. aircraft bombed Iraqi tanks holding bridges... "-a mechanism is required to assess these challenges and to correct them with salient and dynamic rewards. We describe the training approach as a process of extractor (agent A) imitating the ground-truth (expert E), and during the process, a mechanism ensures that the highest reward values are issued to correct labels (actions a), including the ones from both expert E and a. DISPLAYFORM1 This mechanism is Inverse Reinforcement Learning BID0, which estimates the reward first in an RL framework. Equation 12 reveals a scenario of adversary between ground truth and extractor and Generative Adversarial Imitation Learning (GAIL) BID11, which is based on GAN BID10, fits such adversarial nature. In the original GAN, a generator generates (fake) data and attempts to confuse a discriminator D which is trained to distinguish fake data from real data. In our proposed GAIL framework, the extractor (agent A) substitutes the generator and commits labels to the discriminator D; the discriminator D, now serves as reward estimator, aims to issue largest rewards to labels (actions) from the ground-truth (expert E) or identical ones from the extractor but provide lower rewards for other/wrong labels. Rewards R(s, a) and the output of D are now equivalent and we ensure: DISPLAYFORM2 where s, a E and a A are input of the discriminator. In the sequence labeling task, s consists of the context embedding of current token v t and a one-hot vector that represents the previous action a t−1 according to Equation 1, in the argument role labeling task, s comes from the representations of all elements mentioned in Equation 6; a E is a one-hot vector of ground-truth label (expert, or "real data") while a A denotes the counterpart from the extractor (agent, or "generator"). The concatenated s and a E is the input for "real data" channel while s and a A build the input for "generator" channel of the discriminator. In our framework, due to different dimensions in two tasks and event types, we have 34 discriminators (1 for sequence labeling, 33 for event argument role labeling with regard to 33 event types). Every discriminator consists of 2 fully-connected layers with a sigmoid output. The original output of D denotes a probability which is bounded in, and we use linear transformation to shift and expand it: Figure 5: An illustrative example of the GAN structure in sequence labeling scenario (argument role labeling scenario has the identical frameworks except vector dimensions). As introduced in Section 4, the "real data" in the original GAN is replaced by feature/state representation (Equation 1, or Equation 6 for argument role labeling scenario) and ground-truth labels (expert actions) in our framework, while the "generator data" consists of features and extractor's attempt labels (agent actions). The discriminator serves as the reward estimator and a linear transform is utilized to extend the D's original output of probability range. DISPLAYFORM3 e.g., in our experiments, we set α = 20 and β = 0.5 and make R(s, a) ∈ [−10, 10].To pursue Equation 13, we minimize the loss function and optimize the parameters in the neural network: DISPLAYFORM4 During the training process, after we feed neural network mentioned in Section 3.1 and 3.2 with a mini-batch of data, we collect the features (or states s), corresponding extractor labels (agent actions a A) and ground-truth (expert actions a E) to update the discriminators according to Equation 15; then we feed features and extractor labels into the discriminators to acquire reward values and train the extractor -or the generator from GAN's perspective. Since the discriminators are continuously optimized, if the extractor (generator) makes repeated mistakes or makes surprising ones (e.g., considering a PER as a Place), the margin of rewards between correct and wrong labels expands and outputs reward with larger absolute values. Hence, in sequence labeling task, the updated Q-values are updated with a more discriminative difference, and, similarly, in argument role labeling task, the P (a|s) also increases or decreases more significantly with a larger absolute reward values. Figure 5 illustrates how we utilize GAN for reward estimation. In case where discriminators are not sufficiently optimized (e.g., in early epochs) and may output undesired values -e.g., negative for correct actions, we impose a hard margiñ R(s, a) = max(0.1, R(s, a)) when a is correct, min(−0.1, R(s, a)) otherwise to ensure that correct actions will always take positive reward values and wrong ones take negative. In training phase, the extractor selects labels according to the rankings of Q-values in Equation 3 and 8 and GANs will issue rewards to update the Q-tables and policy probabilities; and we also adopt -greedy strategy: we set a probability threshold ∈ and uniformly sample a number ρ ∈ before the extractor commits a label for an instance: a = arg max a Q(s, a), if ρ ≥ Randomly pick up an action, if othersWith this strategy, the extractor is able to explore all possible labels (including correct and wrong ones), and acquires rewards with regard to all labels to update the neural networks with richer information. Moreover, after one step of -greedy exploration, we also force the extractor to commit ground-truth labels and issue it with expert (highest) rewards, and update the parameters accordingly. This additional step is inspired by BID14 , 2018], which combines cross-entropy loss from supervised models with RL loss functions 3. Such combination can simultaneously and explicitly encourage correct labels and penalize wrong labels and greatly improve the efficiency of pursuing optimal models. To evaluate the performance with our proposed approach, we utilize ACE2005 documents excluding informal documents from cts (Conversational Telephone Speech) and un (UseNet) and we have 5, 272 triggers and 9, 612 arguments. We follow training (529 documents with 14, 180 sentences), validation (30 documents with 863 sentences) and test (40 documents with 672 sentences) splits and adopt the same criteria of the evaluation to align with [BID17 :• An entity (named entities and nominals) is correct if its entity type and offsets find a match in the ground truth.• A trigger is correct if its event type and offsets find a match in the ground truth.• An argument is correctly labeled if its event type, offsets and role find a match in the ground truth. We use ELMo embeddings 4 BID15. Because ELMo is delivered with builtin Bi-LSTMs, we treat ELMo embedding as context embeddings in Figure 2 and 4. We use GAIL-ELMo in the tables to denote the setting. Moreover, in order to disentangle the contribution from ELMo embeddings, we also present the performance in a non-ELMo setting (denoted as GAIL-W2V) which utilizes the following embedding techniques to represent tokens in the input sentence.• Token surface embeddings: for each unique token in the training set, we have a lookup dictionary for embeddings which is randomly initialized and updated in the training phase.• Character-based embeddings: each character also has a randomly initialized embedding, and will be fed into a token-level Bi-LSTM network, the final output of this network will enrich the information of token.• POS embeddings: We apply Part-of-Speech (POS) tagging on the sentences using Stanford CoreNLP tool BID21. The POS tags of the tokens also have a trainable look-up dictionary (embeddings).• Pre-trained embeddings: We also acquire embeddings trained from a large and publicly available corpus. These embeddings preserve semantic information of the tokens and they are not updated in the training phase. We concatenate these embeddings and feed them into the Bi-LSTM networks as demonstrated in Figure 2 and 4. To relieve over-fitting issues, we utilize dropout strategy on the input data during the training phase. We intentionally set "UNK" (unknown) masks, which hold entries in the look-up dictionaries of tokens, POS tags and characters. We randomly mask known tokens, POS tags and characters in the training sentences with "UNK" mask. We also set an all-0 vector on Word2Vec embeddings of randomly selected tokens. We tune the parameters according to the F1 score of argument role labeling. For Qlearning, we set a discount factor γ = 0.01. For all RL tasks, we set exploration threshold = 0.1. We set all hidden layer sizes (including the ones on discriminators) and LSTM (for subsentence Bi-LSTM) cell memory sizes as 128. The dropout rate is 0.2. When optimizing the parameters in the neural networks, we use SGD with Momentum and the learning rates start from 0.02 (sequence labeling), 0.005 (argument labeling) and 0.001 (discriminators), then the learning rate will decay every 5 epochs with exponential of 0.9; all momentum values are set as 0.9.For the non-ELMo setting, we set 100 dimensions for token embeddings, 20 for PoS embeddings, and 20 for character embeddings. For pre-trained embeddings, we train a 100-dimension Word2Vec [] model from English Wikipedia articles (January 1st, 2017), with all tokens preserved and a context window of 5 from both left and right. We also implemented an RL framework with fixed rewards of ±5 as baseline with identical parameters as above. For sequence labeling (entity and event trigger detection task), we also set an additional reward value of −50 for B-I errors, namely an I-label does not follow B-label with the same tag name (e.g., I-GPE follows B-PER). We use RL-W2V and RL-ELMo to denote these fixed-reward settings. We compare the performance of entity extraction (including named entities and nominal mentions) with the following state-of-the-art and high-performing approaches:• JointIE []: a joint approach that extracts entities, relations, events and argument roles using structured prediction with rich local and global linguistic features.• JointEntityEvent BID23: an approach that simultaneously extracts entities and arguments with document context. • Tree-LSTM []: a Tree-LSTM based approach that extracts entities and relations. JointIE [] 85.2 76.9 80.8 JointEntityEvent BID23 83.5 80.2 81.8 Tree-LSTM [] 82.9 83.9 83.4 KBLSTM BID24 85 BID24 • KBLSTM BID24: an LSTM-CRF hybrid model that applies knowledge base information on sequence labeling. From Table 1 we can conclude that our proposed method outperforms the other approaches, especially with an impressively high performance of recall. CRF-based models are applied on sequence labeling tasks because CRF can consider the label on previous token to avoid mistakes such as appending an I-GPE to a B-PER, but it neglects the information from the later tokens. Our proposed approach avoids the aforementioned mistakes by issuing strong penalties (negative reward with large absolute value); and the Q-values in our sequence labeling sub-framework also considers rewards for the later tokens, which significantly enhances our prediction performance. For event extraction performance with system-predicted entities as argument candidates, besides [] and BID23 we compare 5 our performance with:• dbRNN BID17: an LSTM framework incorporating the dependency graph (dependency-bridge) information to detect event triggers and argument roles. TAB6 demonstrates that the performance of our proposed framework is better than state-of-the-art approaches except lower F1 score on argument identification against BID17. BID17 utilizes Stanford CoreNLP to detect the noun phrases and take the detected phrases as argument candidates, while our argument candidates come from system predicted entities and some entities may be missed. However, BID17's approach misses entity type information, which cause many errors in argument role labeling task, whereas our argument candidates hold entity types, and our final role labeling performance is better than BID17.Our framework is also flexible to consume ground-truth (gold) annotation of entities as argument candidates. And we demonstrate the performance comparison with the following state-of-the-art approaches on the same setting besides BID17:• JointIE-GT []: similar to [], the only difference is that this approach detects arguments based on ground-truth entities.5. Some high-performing event approaches such as [,] have no argument role detection, thus they are not included for the sake of fair comparison. ], an RNN-based approach which integrates local lexical features. For this setting, we keep the identical parameters (including both trained and preset ones) and network structures which we used to report our performance in Table 1 and 2, and we substitute system-predicted entity types and offsets with ground-truth counterparts. TAB7 demonstrates that, without any further deliberate tuning, our proposed approach can still provide better performance. We notice that some recent approaches [] consolidate argument role labels with same names from different event types (e.g., Adjudicator in Trial-Hearing, Charge-Indict, Sue, Convict, etc.), for argument role labeling they only deal with 37 categories while our setting consists of 143 categories (with a hierarchical routine of 33 event types and 3-7 roles for each type). The strategy of consolidation can boost the scores and our early exploration with similar strategy reaches an argument role labeling F1 score of 61.6 with gold entity annotation, however, the appropriateness with regard to ACE schema definition still concerns us. For example, the argument role Agent appear in Injure, Die, Transport, Start-Org, Nominate, Elect, Arrest-Jail and Release-Parole events, the definition of each Agent in these types includes criminals, business people, law enforcement officers and organizations which have little overlap and it is meaningless and ridiculous to consider these roles within one single label. Moreover, when analyzing errors from this setting, we encounter errors such as Attacker in Meet events or Destination in Mary events, which completely violate ACE schema. Hence, for the sake of solid comparison, we do not include this setting, though we still appreciate and honor any work and attempt to pursue higher performance. The statistical in Table 1, 2 and 3 demonstrate that dynamic rewards outperforms the settings with fixed rewards. As presented in Section 4, fixed reward setting resembles classification methods with cross-entropy loss, which treat errors equally and do not incorporate much information from errors, hence the performance is similar to some earlier approaches but does not outperform state-of-the-art. For instances with ambiguity, our dynamic reward function can provide more salient margins between correct and wrong labels: e.g., "... they sentenced him to death...", with the identical parameter set as aforementioned, reward for the wrong Die label is −5.74 while correct Execute label gains 6.53. For simpler cases, e.g., "... submitted his resignation...", we have flatter rewards as 2.74 for End-Position, −1.33 for None or −1.67 for Meet, which are sufficient to commit correct labels. Scores in Table 1, 2 and 3 prove that non-ELMo settings already outperform state-of-the-art, which confirms the advantage and contribution of our GAIL framework. Moreover, in spite of insignificant drop in fixed reward setting, we agree that ELMo is a good replacement 6 for a combination of word, character and PoS embeddings. The only shortcoming according to our empirical practice is that ELMo takes huge amount of GPU memory and the training procedure is slow (even we do not update the pre-trained parameters during our training phase). Losses of scores are mainly missed trigger words and arguments. For example, the Meet trigger "pow-wow " is missed because it is rarely used to describe a formal political meeting; and there is no token with similar surface form -which can be recovered using character embedding or character information in ELMo setting -in the training data. We observe some special erroneous cases due to fully biased annotation. In the sentence "Bombers have also hit targets...", the entity "bombers" is mistakenly classified as the Attacker argument of the Attack event triggered by the word "hit". Here the "bombers" refers to aircraft and is considered as a VEH (Vehicle) entity, and should be an Instrument in the Attack event, while "bombers" entities in the training data are annotated as Person (who detonates bombs), which are never Instrument. This is an ambiguous case, however, it does not compromise our claim on the merit of our proposed framework against ambiguous errors, because our proposed framework still requires a mixture of different labels to acknowledge ambiguity. One of the recent event extraction approaches mentioned in the introductory section BID13 utilizes GAN in event extraction. The GAN in the cited work outputs generated features to regulate the event model from features leading to errors, while our approach directly assess the mistakes to explore levels of difficulty in labels. Moreover, our approach also covers argument role labeling, while the cited paper does not. RL-based methods have been recently applied to a few information extraction tasks such as relation extraction; and both relation frameworks from BID9 BID25 apply RL on entity relation detection with a series of predefined rewards. We are aware that the term imitation learning is slightly different from inverse reinforcement learning. Techniques of imitation learning BID5 BID16 BID3 attempt to map the states to expert actions by following demonstration, which resembles supervised learning, while inverse reinforcement learning BID0 BID20 BID28 BID11 BID2 estimates the rewards first and apply the rewards to RL. BID22 is an imitation learning application on bio-medical event extraction, and there is no reward estimator used. We humbly recognize our work as inverse reinforcement learning approach although "GAIL" is named after imitation learning. In this paper, we propose an end-to-end entity and event extraction framework based on inverse reinforcement learning. Experiments have demonstrated that the performance benefits from dynamic reward values estimated from discriminators in GAN, and we also demonstrate the performance of recent embedding work in the experiments. In the future, besides releasing the source code, we also plan to further visualize the reward values and attempt to interpret these rewards so that researchers and event extraction system developers are able to better understand and explore the algorithm and remaining challenges. Our future work also includes using cutting edge approaches such as BERT BID6, and exploring joint model in order to alleviate impact from upstream errors in current pipelined framework.

Represent the short summary for finding the corresponding scientific article (long)

We use dynamic rewards to train event extractors.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.