Question: Represent the article for finding a single-sentence TLDR

Question: Reinforcement learning provides a powerful and general framework for decision making and control, but its application in practice is often hindered by the need for extensive feature and reward engineering. Deep reinforcement learning methods can remove the need for explicit engineering of policy or value features, but still require a manually specified reward function. Inverse reinforcement learning holds the promise of automatic reward acquisition, but has proven exceptionally difficult to apply to large, high-dimensional problems with unknown dynamics. In this work, we propose AIRL, a practical and scalable inverse reinforcement learning algorithm based on an adversarial reward learning formulation that is competitive with direct imitation learning algorithms. Additionally, we show that AIRL is able to recover portable reward functions that are robust to changes in dynamics, enabling us to learn policies even under significant variation in the environment seen during training. While reinforcement learning (RL) provides a powerful framework for automating decision making and control, significant engineering of elements such as features and reward functions has typically been required for good practical performance. In recent years, deep reinforcement learning has alleviated the need for feature engineering for policies and value functions, and has shown promising on a range of complex tasks, from vision-based robotic control BID12 to video games such as Atari BID13 and Minecraft BID16. However, reward engineering remains a significant barrier to applying reinforcement learning in practice. In some domains, this may be difficult to specify (for example, encouraging "socially acceptable" behavior), and in others, a naïvely specified reward function can produce unintended behavior BID2. Moreover, deep RL algorithms are often sensitive to factors such as reward sparsity and magnitude, making well performing reward functions particularly difficult to engineer. Inverse reinforcement learning (IRL) BID19 BID14 refers to the problem of inferring an expert's reward function from demonstrations, which is a potential method for solving the problem of reward engineering. However, inverse reinforcement learning methods have generally been less efficient than direct methods for learning from demonstration such as imitation learning BID10, and methods using powerful function approximators such as neural networks have required tricks such as domain-specific regularization and operate inefficiently over whole trajectories BID6. There are many scenarios where IRL may be preferred over direct imitation learning, such as re-optimizing a reward in novel environments BID7 or to infer an agent's intentions, but IRL methods have not been shown to scale to the same complexity of tasks as direct imitation learning. However, adversarial IRL methods BID6 a) hold promise for tackling difficult tasks due to the ability to adapt training samples to improve learning efficiency. Part of the challenge is that IRL is an ill-defined problem, since there are many optimal policies that can explain a set of demonstrations, and many rewards that can explain an optimal policy BID15. The maximum entropy (MaxEnt) IRL framework introduced by BID24 handles the former ambiguity, but the latter ambiguity means that IRL algorithms have difficulty distinguishing the true reward functions from those shaped by the environment dynamics. While shaped rewards can increase learning speed in the original training environment, when the reward is deployed at test-time on environments with varying dynamics, it may no longer produce optimal behavior, as we discuss in Sec. 5. To address this issue, we discuss how to modify IRL algorithms to learn rewards that are invariant to changing dynamics, which we refer to as disentangled rewards. In this paper, we propose adversarial inverse reinforcement learning (AIRL), an inverse reinforcement learning algorithm based on adversarial learning. Our algorithm provides for simultaneous learning of the reward function and value function, which enables us to both make use of the efficient adversarial formulation and recover a generalizable and portable reward function, in contrast to prior works that either do not recover a reward functions BID10, or operates at the level of entire trajectories, making it difficult to apply to more complex problem settings BID6 a). Our experimental evaluation demonstrates that AIRL outperforms prior IRL methods BID6 on continuous, high-dimensional tasks with unknown dynamics by a wide margin. When compared to GAIL BID10, which does not attempt to directly recover rewards, our method achieves comparable on tasks that do not require transfer. However, on tasks where there is considerable variability in the environment from the demonstration setting, GAIL and other IRL methods fail to generalize. In these settings, our approach, which can effectively disentangle the goals of the expert from the dynamics of the environment, achieves superior . Inverse reinforcement learning (IRL) is a form of imitation learning and learning from demonstration BID3. Imitation learning methods seek to learn policies from expert demonstrations, and IRL methods accomplish this by first inferring the expert's reward function. Previous IRL approaches have included maximum margin approaches BID0 BID18, and probabilistic approaches such as BID24; BID4. In this work, we work under the maximum causal IRL framework of BID23. Some advantages of this framework are that it removes ambiguity between demonstrations and the expert policy, and allows us to cast the reward learning problem as a maximum likelihood problem, connecting IRL to generative model training. Our proposed method most closely resembles the algorithms proposed by BID21; BID10 BID5. Generative adversarial imitation learning (GAIL) BID10 differs from our work in that it is not an IRL algorithm that seeks to recover reward functions. The critic or discriminator of GAIL is unsuitable as a reward since, at optimality, it outputs 0.5 uniformly across all states and actions. Instead, GAIL aims only to recover the expert's policy, which is a less portable representation for transfer. BID21 does not interleave policy optimization with reward learning within an adversarial framework. Improving a policy within an adversarial framework corresponds to training an amortized sampler for an energy-based model, and prior work has shown this is crucial for performance BID6. BID22 also consider learning cost functions with neural networks, but only evaluate on simple domains where analytically solving the problem with value iteration is tractable. Previous methods which aim to learn nonlinear cost functions have used boosting BID17 and Gaussian processes BID11, but still suffer from the feature engineering problem. Our IRL algorithm builds on the adversarial IRL framework proposed by BID5, with the discriminator corresponding to an odds ratio between the policy and exponentiated reward distribution. The discussion in BID5 is theoretical, and to our knowledge no prior work has reported a practical implementation of this method. Our experiments show that direct implementation of the proposed algorithm is ineffective, due to high variance from operating over entire trajectories. While it is straightforward to extend the algorithm to single state-action pairs, as we discuss in Section 4, a simple unrestricted form of the discriminator is susceptible to the reward ambiguity described in BID15, making learning the portable reward functions difficult. As illustrated in our experiments, this greatly limits the generalization capability of the method: the learned reward functions are not robust to environment changes, and it is difficult to use the algo-rithm for the purpose of inferring the intentions of agents. We discuss how to overcome this issue in Section 5. BID1 consider learning reward functions which generalize to new tasks given multiple training tasks. Our work instead focuses on how to achieve generalization within the standard IRL formulation. Our inverse reinforcement learning method builds on the maximum causal entropy IRL framework BID23, which considers an entropy-regularized Markov decision process (MDP), defined by the tuple (S, A, T, r, γ, ρ 0). S, A are the state and action spaces, respectively, γ ∈ is the discount factor. The dynamics or transition distribution T (s |a, s), the initial state distribution ρ 0 (s), and the reward function r(s, a) are unknown in the standard reinforcement learning setup and can only be queried through interaction with the MDP.The goal of (forward) reinforcement learning is to find the optimal policy π * that maximizes the expected entropy-regularized discounted reward, under π, T, and ρ 0: DISPLAYFORM0 where τ = (s 0, a 0, ...s T, a T) denotes a sequence of states and actions induced by the policy and dynamics. It can be shown that the trajectory distribution induced by the optimal policy π * (a|s) takes the form π * (a|s) ∝ exp{Q * soft (s t, a t)} BID23 BID9, where DISPLAYFORM1 Inverse reinforcement learning instead seeks infer the reward function r(s, a) given a set of demonstrations D = {τ 1, ..., τ N}. In IRL, we assume the demonstrations are drawn from an optimal policy π * (a|s). We can interpret the IRL problem as solving the maximum likelihood problem: DISPLAYFORM2 Where st,at) parametrizes the reward function r θ (s, a) but fixes the dynamics and initial state distribution to that of the MDP. Note that under deterministic dynamics, this simplifies to an energy-based model where for feasible trajectories, p θ (τ) ∝ e T t=0 γ t r θ (st,at) BID24. BID5 propose to cast optimization of Eqn. 1 as a GAN BID8 optimization problem. They operate in a trajectory-centric formulation, where the discriminator takes on a particular form (f θ (τ) is a learned function; π(τ) is precomputed and its value "filled in"): DISPLAYFORM3 DISPLAYFORM4 and the policy π is trained to maximize DISPLAYFORM5 Updating the discriminator can be viewed as updating the reward function, and updating the policy can be viewed as improving the sampling distribution used to estimate the partition function. If trained to optimality, it can be shown that an optimal reward function can be extracted from the optimal discriminator as f * (τ) = R * (τ)+const, and π recovers the optimal policy. We refer to this formulation as generative adversarial network guided cost learning (GAN-GCL) to discriminate it from guided cost learning (GCL) BID5. This formulation shares similarities with GAIL BID10, but GAIL does not place special structure on the discriminator, so the reward cannot be recovered. In practice, using full trajectories as proposed by GAN-GCL can in high variance estimates as compared to using single state, action pairs, and our experimental show that this in very poor learning. We could instead propose a straightforward conversion of Eqn. 2 into the single state and action case, where: DISPLAYFORM0.As in the trajectory-centric case, we can show that, at optimality, f * (s, a) = log π * (a|s) = A * (s, a), the advantage function of the optimal policy. We justify this, as well as a proof that this algorithm solves the IRL problem in Appendix A.This change in an efficient algorithm for imitation learning. However, it is less desirable for the purpose of reward learning. While the advantage is a valid optimal reward function, it is a heavily entangled reward, as it supervises each action based on the action of the optimal policy for the training MDP. Based on the analysis in the following Sec. 5, we cannot guarantee that this reward will be robust to changes in environment dynamics. In our experiments we demonstrate several cases where this reward simply encourages mimicking the expert policy π *, and fails to produce desirable behavior even when changes to the environment are made. We now discuss why IRL methods can fail to learn robust reward functions. First, we review the concept of reward shaping. BID15 describe a class of reward transformations that preserve the optimal policy. Their main theoretical is that under the following reward transformation, DISPLAYFORM0 the optimal policy remains unchanged, for any function Φ: S → R. Moreover, without prior knowledge of the dynamics, this is the only class of reward transformations that exhibits policy invariance. Because IRL methods only infer rewards from demonstrations given from an optimal agent, they cannot in general disambiguate between reward functions within this class of transformations, unless the class of learnable reward functions is restricted. We argue that shaped reward functions may not be robust to changes in dynamics. We formalize this notion by studying policy invariance in two MDPs M, M which share the same reward and differ only in the dynamics, denoted as T and T, respectively. Suppose an IRL algorithm recovers a shaped, policy invariant rewardr(s, a, s) under MDP M where Φ = 0. Then, there exists MDP pairs M, M where changing the transition model from T to T breaks policy invariance on MDP M. As a simple example, consider deterministic dynamics T (s, a) → s and state-action rewardsr(s, a) = r(s, a) + γΦ(T (s, a)) − Φ(s). It is easy to see that changing the dynamics T to T such that T (s, a) = T (s, a) means thatr(s, a) no longer lies in the equivalence class of Eqn. 3 for M. First, let the notation Q * r,T (s, a) denote the optimal Q-function with respect to a reward function r and dynamics T, and π * r,T (a|s) denote the same for policies. We first define our notion of a "disentangled" reward. Definition 5.1 (Disentangled Rewards). A reward function r (s, a, s) is (perfectly) disentangled with respect to a ground-truth reward r(s, a, s) and a set of dynamics T such that under all dynamics T ∈ T, the optimal policy is the same: π * r,T (a|s) = π * r,T (a|s)We could also expand this definition to include a notion of suboptimality. However, we leave this direction to future work. Under maximum causal entropy RL, the following condition is equivalent to two optimal policies being equal, since Q-functions and policies are equivalent representations (up to arbitrary functions of state f (s)): DISPLAYFORM0 To remove unwanted reward shaping with arbitrary reward function classes, the learned reward function can only depend on the current state s. We require that the dynamics satisfy a decomposability Collect trajectories τ i = (s 0, a 0, ..., s T, a T) by executing π. Train D θ,φ via binary logistic regression to classify expert data τ E i from samples τ i.6: DISPLAYFORM0 Update π with respect to r θ,φ using any policy optimization method. 8: end for condition where functions over current states f (s) and next states g(s) can be isolated from their sum f (s) + g(s). This can be satisfied for example by adding self transitions at each state to an ergodic MDP, or any of the environments used in our experiments. The exact definition of the condition, as well as proof of the following statements are included in Appendix B. Theorem 5.1. Let r(s) be a ground-truth reward, and T be a dynamics model satisfying the decomposability condition. Suppose IRL recovers a state-only reward r (s) such that it produces an optimal policy in T: DISPLAYFORM1 DISPLAYFORM2 Then r is only a function of state. In the traditional IRL setup, where we learn the reward in a single MDP, our analysis motivates learning reward functions that are solely functions of state. If the ground truth reward is also only a function of state, this allows us to recover the true reward up to a constant. In the method presented in Section 4, we cannot learn a state-only reward function, r θ (s), meaning that we cannot guarantee that learned rewards will not be shaped. In order to decouple the reward function from the advantage, we propose to modify the discriminator of Sec. 4 with the form: DISPLAYFORM0 where f θ,φ is restricted to a reward approximator g θ and a shaping term h φ as DISPLAYFORM1 The additional shaping term helps mitigate the effects of unwanted shaping on our reward approximator g θ (and as we will show, in some cases it can account for all shaping effects). The entire training procedure is detailed in Algorithm 1. Our algorithm resembles GAIL BID10 and GAN-GCL BID5, where we alternate between training a discriminator to classify expert data from policy samples, and update the policy to confuse the discriminator. The advantage of this approach is that we can now parametrize g θ (s) as solely a function of the state, allowing us to extract rewards that are disentangled from the dynamics of the environment in which they were trained. In fact, under this restricted case, we can show the following under deterministic environments with a state-only ground truth reward (proof in Appendix C): DISPLAYFORM2 where r * is the true reward function. Since f * must recover to the advantage as shown in Sec. 4, h recovers the optimal value function V *, which serves as the reward shaping term. To be consistent with Sec. 4, an alternative way to interpret the form of Eqn. 4 is to view f θ,φ as the advantage under deterministic dynamics DISPLAYFORM3 In stochastic environments, we can instead view f (s, a, s) as a single-sample estimate of A * (s, a). In our experiments, we aim to answer two questions:1. Can AIRL learn disentangled rewards that are robust to changes in environment dynamics?2. Is AIRL efficient and scalable to high-dimensional continuous control tasks?To answer 1, we evaluate AIRL in transfer learning scenarios, where a reward is learned in a training environment, and optimized in a test environment with significantly different dynamics. We show that rewards learned with our algorithm under the constraint presented in Section 5 still produce optimal or near-optimal behavior, while naïve methods that do not consider reward shaping fail. We also show that in small MDPs, we can recover the exact ground truth reward function. To answer 2, we compare AIRL as an imitation learning algorithm against GAIL BID10 ) and the GAN-based GCL algorithm proposed by BID5, which we refer to as GAN-GCL, on standard benchmark tasks that do not evaluate transfer. Note that BID5 does not implement or evaluate GAN-GCL and, to our knowledge, we present the first empirical evaluation of this algorithm. We find that AIRL performs on par with GAIL in a traditional imitation learning setup while vastly outperforming it in transfer learning setups, and outperforms GAN-GCL in both settings. It is worth noting that, except for BID6, our method is the only IRL algorithm that we are aware of that scales to high dimensional tasks with unknown dynamics, and although GAIL BID10 resembles an IRL algorithm in structure, it does not recover disentangled reward functions, making it unable to re-optimize the learned reward under changes in the environment, as we illustrate below. For our continuous control tasks, we use trust region policy optimization BID20 as our policy optimization algorithm across all evaluated methods, and in the tabular MDP task, we use soft value iteration. We obtain expert demonstrations by training an expert policy on the ground truth reward, but hide the ground truth reward from the IRL algorithm. In this way, we simulate a scenario where we wish to use RL to solve a task but wish to refrain from manual reward engineering and instead seek to learn a reward function from demonstrations. Our code and additional supplementary material including videos will be available at https://sites.google.com/view/ adversarial-irl, and hyper-parameter and architecture choices are detailed in Appendix D. We first consider MaxEnt IRL in a toy task with randomly generated MDPs. The MDPs have 16 states, 4 actions, randomly drawn transition matrices, and a reward function that always gives a reward of 1.0 when taking an action from state 0. The initial state is always state 1.The optimal reward, learned reward with a state-only reward function, and learned reward using a state-action reward function are shown in FIG1. We subtract a constant offset from all reward functions so that they share the same mean for visualization -this does not influence the optimal policy. AIRL with a state-only reward function is able to recover the ground truth reward, but AIRL with a state-action reward instead recovers a shaped advantage function. We also show that in the transfer learning setup, under a new transition matrix T, the optimal policy under the state-only reward achieves optimal performance (it is identical to the ground truth reward) whereas the state-action reward only improves marginally over uniform random policy. The learning curve for this experiment is shown in Learning curve for the transfer learning experiment on tabular MDPs. Value iteration steps are plotted on the x-axis, against returns for the policy on the y-axis. To evaluate whether our method can learn disentangled rewards in higher dimensional environments, we perform transfer learning experiments on continuous control tasks. In each task, a reward is learned via IRL on the training environment, and the reward is used to reoptimize a new policy on a test environment. We train two IRL algorithms, AIRL and GAN-GCL, with state-only and stateaction rewards. We also include for directly transferring the policy learned with GAIL, and an oracle that involves optimizing the ground truth reward function with TRPO. Numerical for these environment transfer experiments are given in TAB2 The first task involves a 2D point mass navigating to a goal position in a small maze when the position of the walls are changed between train and test time. At test time, the agent cannot simply mimic the actions learned during training, and instead must successfully infer that the goal in the maze is to reach the target. The task is shown in FIG3. Only AIRL trained with state-only rewards is able to consistently navigate to the goal when the maze is modified. Direct policy transfer and state-action IRL methods learn rewards which encourage the agent to take the same path taken in the training environment, which is blocked in the test environment. We plot the learned reward in FIG4 In our second task, we modify the agent itself. We train a quadrupedal "ant" agent to run forwards, and at test time we disable and shrink two of the front legs of the ant such that it must significantly change its gait. We find that AIRL is able to learn reward functions that encourage the ant to move forwards, acquiring a modified gait that involves orienting itself to face the forward direction and crawling with its two hind legs. Alternative methods, including transferring a policy learned by GAIL (which achieves near-optimal performance with the unmodified agent), fail to move forward at all. We show the qualitative difference in behavior in FIG5.We have demonstrated that AIRL can learn disentangled rewards that can accommodate significant domain shift even in high-dimensional environments where it is difficult to exactly extract the true reward. GAN-GCL can presumably learn disentangled rewards, but we find that the trajectorycentric formulation does not perform well even in learning rewards in the original task, let alone transferring to a new domain. GAIL learns successfully in the training domain, but does not acquire a representation that is suitable for transfer to test domains. Illustration of the shifting maze task, where the agent (blue) must reach the goal (green). During training the agent must go around the wall on the left side, but during test time it must go around on the right. Bottom row: Behavior acquired by optimizing a state-only reward learned with AIRL on the disabled ant environment. Note that the ant must orient itself before crawling forward, which is a qualitatively different behavior from the optimal policy in the original environment, which runs sideways. Finally, we evaluate AIRL as an imitation learning algorithm against the GAN-GCL and the stateof-the-art GAIL on several benchmark tasks. Each algorithm is presented with 50 expert demonstrations, collected from a policy trained with TRPO on the ground truth reward function. For AIRL, we use an unrestricted state-action reward function as we are not concerned with reward transfer. Numerical are presented in TAB3.These experiments do not test transfer, and in a sense can be regarded as "testing on the training set," but they match the settings reported in prior work BID10.We find that the performance difference between AIRL and GAIL is negligible, even though AIRL is a true IRL algorithm that recovers reward functions, while GAIL does not. Both methods achieve close to the best possible on each task, and there is little room for improvement. This goes against the belief that IRL algorithms are indirect, and less efficient that direct imitation learning algorithms BID10. The GAN-GCL method is ineffective on all but the simplest Pendulum task when trained with the same number of samples as AIRL and GAIL. We find that a discriminator trained over trajectories easily overfits and provides poor learning signal for the policy. Our illustrate that AIRL achieves the same performance as GAIL on benchmark imitation tasks that do not require any generalization. On tasks that require transfer and generalization, illustrated in the previous section, AIRL outperforms GAIL by a wide margin, since our method is able to recover disentangled rewards that transfer effectively in the presence of domain shift. We presented AIRL, a practical and scalable IRL algorithm that can learn disentangled rewards and greatly outperforms both prior imitation learning and IRL algorithms. We show that rewards learned with AIRL transfer effectively under variation in the underlying domain, in contrast to unmodified IRL methods which tend to recover brittle rewards that do not generalize well and GAIL, which does not recover reward functions at all. In small MDPs where the optimal policy and reward are unambiguous, we also show that we can exactly recover the ground-truth rewards up to a constant. In this section, we show that the objective of AIRL matches that of solving the maximum causal entropy IRL problem. We use a similar method as BID5, which shows the justification of GAN-GCL for the trajectory-centric formulation. For simplicity we derive everything in the undiscounted case. A.1 SETUP As mentioned in Section 3, the goal of IRL can be seen as training a generative model over trajectories as: max st,at). We can compute the gradient with respect to θ as follows: DISPLAYFORM0 DISPLAYFORM1 Let p θ,t (s t, a t) = s t =t,a t =t p θ (τ) denote the state-action marginal at time t. Rewriting the above equation, we have: DISPLAYFORM2 As it is difficult to draw samples from p θ, we instead train a separate importance sampling distribution µ(τ). For the choice of this distribution, we follow BID5 and use a mixture policy µ(a|s) = 1 2 π(a|s) + 1 2p (a|s), wherep(a|s) is a rough density estimate trained on the demonstrations. This is justified as reducing the variance of the importance sampling estimate when the policy π(a|s) has poor coverage over the demonstrations in the early stages of training. Thus, our new gradient is: DISPLAYFORM3 We additionally wish to adapt the importance sampler π to reduce variance, by min- DISPLAYFORM4 The policy trajectory distribution factorizes as π(τ) = p(s 0)T −1 t=0 p(s t+1 |s t, a t)π(a t |s t). The dynamics and initial state terms inside π(τ) and p θ (τ) cancel, leaving the entropy-regularized policy objective: DISPLAYFORM5 In AIRL, we replace the cost learning objective with training a discriminator of the following form: DISPLAYFORM6 The objective of the discriminator is to minimize cross-entropy loss between expert demonstrations and generated samples: DISPLAYFORM7 We replace the policy optimization objective with the following reward: DISPLAYFORM8 A.2 DISCRIMINATOR OBJECTIVE First, we show that training the gradient of the discriminator objective is the same as Eqn. 5. We write the negative loss to turn the minimization problem into maximization, and use µ to denote a mixture between the dataset and policy samples. DISPLAYFORM9 Taking the derivative w.r.t. θ, DISPLAYFORM10 Multiplying the top and bottom of the fraction in the second expectation by the state marginal π(s t) = a π t (s t, a t), and grouping terms we get: DISPLAYFORM11 Where we have writtenp θ,t (s t, a t) = exp{f θ (s t, a t)}π t (s t), andμ to denote a mixture between p θ (s, a) and policy samples. This expression matches Eqn. 5, with f θ (s, a) serving as the reward function, when π maximizes the policy objective so thatp θ (s, a) = p θ (s, a). Next, we show that the policy objective matches that of the sampler of Eqn. 6. The objective of the policy is to maximize with respect to the rewardr t (s, a). First, note that: DISPLAYFORM0 Thus, whenr(s, a) is summed over entire trajectories, we obtain the entropy-regularized policy objective DISPLAYFORM1 Where f θ serves as the reward function. DISPLAYFORM2 The global minimum of the discriminator objective is achieved when π = π E, where π denotes the learned policy (the "generator" of a GAN) and π E denotes the policy under which demonstrations were collected BID8. At this point, the output of the discriminator is 1 2 for all values of s, a, meaning we have exp{f θ (s, a)} = π E (a|s), or f * (s, a) = log π E (a|s) = A * (s, a). In this section we include proofs for Theorems 5.1 and 5.2, and the condition on the dynamics necessary for them to hold. Definition B.1 (Decomposability Condition). Two states s 1, s 2 are defined as "1-step linked" under a dynamics or transition distribution T (s |a, s) if there exists a state s that can reach s 1 and s 2 with positive probability in one time step. Also, we define that this relationship can transfer through transitivity: if s 1 and s 2 are linked, and s 2 and s 3 are linked, then we also consider s 1 and s 3 to be linked. A transition distribution T satisfies the decomposability condition if all states in the MDP are linked with all other states. The key reason for needing this condition is that it allows us to decompose the functions state dependent f (s) and next state dependent g(s) from their sum f (s) + g(s), as stated below: Lemma B.1. Suppose the dynamics for an MDP satisfy the decomposability condition. Then, for functions a(s), b(s), c(s), d(s), if for all s, s: DISPLAYFORM0 Then for for all s, a(s) = c(s) + const DISPLAYFORM1 Proof. Rearranging, we have: DISPLAYFORM2 for some function only dependent on s. In order for this to be representable, the term b(s) − d(s) must be equal for all successor states s from s. Under the decomposability condition, all successor states must therefore be equal in this manner through transitivity, meaning we have b(s) − d(s) must be constant with respect to s. Therefore, a(s) = c(s) + const. We can then substitute this expression back in to the original equation to derive b(s) = d(s) + const. We consider the case when the ground truth reward is state-only. We now show that if the learned reward is also state-only, then we guarantee learning disentangled rewards, and vice-versa (sufficiency and necessity). Theorem 5.1. Let r(s) be a ground-truth reward, and T be a dynamics model satisfying the decomposability condition. Suppose IRL recovers a state-only reward r (s) such that it produces an optimal policy in T: Q * r,T (s, a) = Q * r,T (s, a) − f (s) Then, r (s) is disentangled with respect to all dynamics. Proof. We show that r (s) must equal the ground-truth reward up to constants (modifying rewards by constants does not change the optimal policy). Let r (s) = r(s) + φ(s) for some arbitrary function of state φ(s). We have: Proof. We show the converse, namely that if r (s, a, s) can depend on a or s, then there exists a dynamics model T such that the optimal policy is changed, i.e. Q * r,T (s, a) = Q * r,T (s, a) + f (s) ∀s, a. Consider the following 3-state MDP with deterministic dynamics and starting state S: We denote the action with a small letter, i.e. taking the action a from S brings the agent to state A, receiving a reward of 0. For simplicity, assume the discount factor γ = 1. The optimal policy here takes the a action, returns to s, and repeat for infinite positive reward. An action-dependent reward which induces the same optimal policy would be to move the reward from the action returning to s to the action going to a or s: Optimizing r on this new MDP in a different policy than optimizing r, as the agent visits B, ing in infinite negative reward. In this section, we prove that AIRL can recover the ground truth reward up to constants if the ground truth is only a function of state r(s). For simplicity, we consider deterministic environments, so that s is uniquely defined by s, a, and we restrict AIRL's reward estimator g to only be a function of state.

represent the short summary for finding the corresponding scientific article (long)

We propose an adversarial inverse reinforcement learning algorithm capable of learning reward functions which can transfer to new, unseen environments.

Represent the short summary for finding the corresponding scientific article (long)

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the scientific article to find a short TLDR

Question: We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? Our work responds to \citet{zhang2016understanding}, who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. We show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy. We propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the ``noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$ the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, $B_{opt} \propto \epsilon N$. We verify these predictions empirically. This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. BID27 trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these contradict learning theory, although this claim is disputed BID18 BID7. Nonetheless, their beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile BID19 observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking shows improving our estimate of the full-batch gradient can harm performance. BID11 observed a linear scaling rule between batch size and learning rate in a deep ResNet, while BID15 proposed a square root rule on theoretical grounds. Many authors have suggested "broad minima" whose curvature is small may generalize better than "sharp minima" whose curvature is large BID4 BID14. Indeed, BID7 argued the of BID27 can be understood using "nonvacuous" PAC-Bayes generalization bounds which penalize sharp minima, while BID19 showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However BID6 challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization. In this work we show:• The of BID27 are not unique to deep learning; we observe the same phenomenon in a small "over-parameterized" linear model. We demonstrate that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.• SGD integrates a stochastic differential equation whose "noise scale" g ≈ N/B, where is the learning rate, N training set size and B batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is proportional to the learning rate and training set size 1.We describe Bayesian model comparison in section 2. In section 3 we replicate the observations of BID27 in a linear model, and show they are explained by the Bayesian evidence. In section 4 we show there is an optimum batch size which maximizes the test set accuracy, and in section 5 we derive scaling rules between the optimum batch size, learning rate, training set size and momentum coefficient. Throughout this work, "generalization gap" refers to the gap in test accuracy between small and large batch SGD training, not the gap in accuracy between training and test sets. Bayesian model comparison was first applied to neural networks in BID22. We provide a brief tutorial here, since the theory is central to the remainder of the paper. For simplicity we first consider a classification model M with a single parameter ω, training inputs x and training labels y. We can infer a posterior probability distribution over the parameter by applying Bayes theorem, DISPLAYFORM0 The likelihood, P (y|ω, DISPLAYFORM1, where H(ω; M) = − i ln (P (y i |ω, x i ; M)) denotes the cross-entropy of unique categorical labels. We typically use a Gaussian prior, P (ω; M) = λ/2πe−λω 2 /2, and therefore the posterior probability density of the parameter given the training data, P (ω|y, x; M) ∝ λ/2πe −C(ω;M), where C(ω; M) = H(ω; M) + λω 2 /2 denotes the L2 regularized cross entropy, or "cost function", and λ is the regularization coefficient. The value ω 0 which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label y t of a new input x t, we should compute the integral, P (y t |x t, y, x; M) = dω P (y t |ω, x t ; M)P (ω|y, x; M) DISPLAYFORM2 However these integrals are dominated by the region near ω 0, and since P (y t |ω, x t ; M) is smooth we usually approximate P (y t |x t, x, y; M) ≈ P (y t |ω 0, x t ; M). Having minimized C(ω; M) to find ω 0, we now wish to compare two different models and select the best one. The probability ratio, DISPLAYFORM3 The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs. BID10 showed that maximizing the evidence (or "marginal likelihood") minimizes a PAC-Bayes generalization bound. To compute it, we evaluate the normalizing constant of equation 1, DISPLAYFORM4 Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. Since this integral is dominated by the region near the minimum ω 0, we can estimate the evidence by Taylor expanding DISPLAYFORM5 Within this "Laplace" approximation, the evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. Thus far we have considered models of a single parameter; in realistic models with many parameters DISPLAYFORM6 ω0, where |∇∇C(ω)| ω0 is the determinant of the Hessian, and p denotes the number of model parameters BID17. The determinant of the Hessian is simply the product of its eigenvalues, (p i=1 λ i), and thus, DISPLAYFORM7 The contribution (λ DISPLAYFORM8 ω0) is often called the "Occam factor", because it enforces Occam's razor; when two models describe the data equally well, the simpler model is usually better BID12. Minima with low curvature are simple, because the parameters do not have to be finetuned to fit the data. Intuitively, the Occam factor describes the fraction of the prior parameter space consistent with the data. Since this fraction is always less than one, we propose to approximate equation 9 away from local minima by only performing the summation over eigenvalues λ i ≥ λ. The evidence can be reframed in the language of information theory, whereby Occam's factor penalizes the amount of information the model must learn about the parameters to accurately model the training data BID13 BID0 BID24.In this work, we will compare the evidence against a null model which assumes the labels are entirely random, assigning equal probability to each class. This unusual model has no parameters, and so the evidence is controlled by the likelihood alone, P (y|x; N U LL) = (1/n) N = e −N ln (n), where n denotes the number of model classes and N the number of training labels. Thus the evidence ratio, DISPLAYFORM9 Where DISPLAYFORM10 is the log evidence ratio in favor of the null model. Clearly, we should only assign any confidence to the predictions of our model if E(ω 0) < 0.The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. BID6 showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam's factor arises from the log ratio, ln (λ i /λ), these two effects cancel out 2. It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this "degeneracy". We argue these issues are not a major limitation, since the intuition we build studying the evidence in simple cases will be sufficient to explain the of both BID27 and BID19. BID27 showed that deep neural networks generalize well on training inputs with informative labels, and yet the same model can drastically overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep networks, let's consider a far simpler model; logistic regression. We form a small balanced training set comprising 800 images from MNIST, of which half have true label "0" and half true label "1". Our test set is also balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. Since the images contain 784 pixels, our model has just 784 weights and 1 bias. We show the accuracy of the model predictions on both the training and test sets in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the observations of BID27 is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels. Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the crossentropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well. Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully generalizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the DISPLAYFORM0 Figure 2: The cross-entropy and log evidence ratio, evaluated on random (a) or informative (b) labels. The evidence, evaluated on the training set, is strongly correlated with the test cross-entropy.log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evidence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength. Bayesian model comparison has explained our in a logistic regression. Meanwhile, BID20 showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the of BID27 in linear models where we can compute the evidence, and qualitatively consistent with their in deep networks where we cannot. BID7 recently demonstrated the of BID27 can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima. We showed above that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Consequently Bayesians often add isotropic Gaussian noise to the gradient BID26.In appendix A, we show this drives the parameters towards broad minima whose evidence is large. The noise introduced by small batch training is not isotropic, and its covariance matrix is a function of the parameter values, but empirically Keskar et al. FORMULA0 found it has similar effects, driving the SGD away from sharp minima. This paper therefore proposes Bayesian principles also account for the "generalization gap", whereby the test set accuracy often falls as the SGD batch size is increased (holding all other hyper-parameters constant). Since the gradient drives the SGD towards deep minima, while noise drives the SGD towards broad minima, we expect the test set performance to show a peak at an optimal batch size, which balances these competing contributions to the evidence. We were unable to observe a generalization gap in linear models (since linear models are convex there are no sharp minima to avoid). Instead we consider a shallow neural network with 800 hidden units and RELU hidden activations, trained on MNIST without regularization. We use SGD with a momentum parameter of 0.9. Unless otherwise stated, we use a constant learning rate of 1.0 which does not depend on the batch size or decay during training. Furthermore, we train on just 1000 images, selected at random from the MNIST training set. This enables us to compare small batch to full batch training. We emphasize that we are not trying to achieve optimal performance, but to study a simple model which shows a generalization gap between small and large batch training. In figure 3, we exhibit the evolution of the test accuracy and test cross-entropy during training. Our small batches are composed of 30 images, randomly sampled from the training set. Looking first at figure 3a, small batch training takes longer to converge, but after a thousand gradient updates a clear generalization gap in model accuracy emerges between small and large training batches. Now consider figure 3b. While the test cross-entropy for small batch training is lower at the end Figure 5: a) The test set accuracy as a function of batch size, for a range of learning rates. The performance peak shifts to the right as we increase, but the overall performance falls once 3. b) The best observed batch size is proportional to the learning rate across two orders of magnitude. of training; the cross-entropy of both small and large training batches is increasing, indicative of over-fitting. Both models exhibit a minimum test cross-entropy, although after different numbers of gradient updates. Intriguingly, we show in appendix B that the generalization gap between small and large batch training shrinks significantly when we introduce L2 regularization. From now on we focus on the test set accuracy (since this converges as the number of gradient updates increases). In figure 4a, we exhibit training curves for a range of batch sizes between 1 and 1000. We find that the model cannot train when the batch size B 10. In figure 4b we plot the mean test set accuracy after 10000 training steps. A clear peak emerges, indicating that there is indeed an optimum batch size which maximizes the test accuracy, consistent with Bayesian intuition. The of BID19 focused on the decay in test accuracy above this optimum batch size. We showed above that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. We argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. We now identify this SGD "noise scale", and use it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. A gradient update, DISPLAYFORM0 (a) (b) Figure 6: a) The test accuracy as a function of batch size, for a range of training set sizes. To reduce noise, we average each curve over five experiments. The performance peak shift to the right as we increase the size of the training set. Unsurprisingly, the overall model performance also improves.b) The best observed batch size is proportional to the size of the training set once N 20000.where is the learning rate, N the training set size, DISPLAYFORM1 dCi dω the true gradient, and DISPLAYFORM2 dCi dω the estimated gradient evaluated on a mini-batch. The expected gradient of a single example, DISPLAYFORM3 is a matrix describing the gradient covariances, which are a function of the current parameter values. We adopt the central limit theorem and model the gradient error α = (dĈ dω − dC dω) with Gaussian random noise (We discuss this approximation briefly in appendix C). It is easy to show that α = 0, while α 2 = N (DISPLAYFORM4 To continue, we interpret equation 11 as the discrete update of a stochastic differential equation BID21 BID9, DISPLAYFORM5 Where t is a continuous variable, η(t) represents noise, η(t) = 0 and η(t)η(t) = gF (ω)δ(t−t). The constant g controls the scale of random fluctuations in the dynamics. To relate this differential equation to the SGD, we compute a gradient update ∆ω = DISPLAYFORM6 Finally, to measure g, we equate the variance in this gradient update to the variance in equation 11, DISPLAYFORM7 Rearranging, the SGD noise scale g = (DISPLAYFORM8 The noise scale falls when the batch size increases, consistent with our earlier observation of an optimal batch size B opt while holding the other hyper-parameters fixed. Notice that one would equivalently observe an optimal learning rate if one held the batch size constant. A similar analysis of the SGD was recently performed by BID23, although their treatment only holds near local minima where the covariances F (ω) are stationary. Our analysis holds throughout training, which is necessary since BID19 found that the beneficial influence of noise was most pronounced at the start of training. When we vary the learning rate or the training set size, we should keep the noise scale fixed, which implies that B opt ∝ N. In figure 5a, we plot the test accuracy as a function of batch size after (10000/) training steps, for a range of learning rates. Exactly as predicted, the peak moves to the right as increases. Additionally, the peak test accuracy achieved at a given learning rate does not begin to fall until ∼ 3, indicating that there is no significant discretization error in integrating the stochastic differential equation below this point. Above this point, the discretization error begins to dominate and the peak test accuracy falls rapidly. In figure 5b, we plot the best observed batch size as a function of learning rate, observing a clear linear trend, B opt ∝. The error bars indicate the distance from the best observed batch size to the next batch size sampled in our experiments. This scaling rule allows us to increase the learning rate with no loss in test accuracy and no increase in computational cost, simply by simultaneously increasing the batch size. We can then exploit increased parallelism across multiple GPUs, reducing model training times BID11. A similar scaling rule was independently proposed by and BID3, although neither work identifies the existence of an optimal noise scale. A number of authors have proposed adjusting the batch size adaptively during training BID8 BID2 BID5, while BID1 proposed linearly coupling the learning rate and batch size within this framework. In Smith et al. FORMULA0, we show empirically that decaying the learning rate during training and increasing the batch size during training are equivalent. In figure 6a we exhibit the test set accuracy as a function of batch size, for a range of training set sizes after 10000 steps (= 1 everywhere). Once again, the peak shifts right as the training set size rises, although the generalization gap becomes less pronounced as the training set size increases. In figure 6b, we plot the best observed batch size as a function of training set size; observing another linear trend, B opt ∝ N. This scaling rule could be applied to production models, progressively growing the batch size as new training data is collected. We expect production datasets to grow considerably over time, and consequently large batch training is likely to become increasingly common. Finally, in appendix D we extend our analysis to SGD with momentum, identifying the noise scale, g ≈ N B(1−m), where m denotes the momentum coefficient. Notice that this reduces to the noise scale of conventional SGD as m → 0. When m > 0, we obtain an additional scaling rule B opt ∝ 1/(1 − m). This scaling rule predicts that the optimal batch size will increase when the momentum coefficient is increased. In FIG2 we plot the test set performance as a function of batch size after 10000 gradient updates (= 1 everywhere), for a range of momentum coefficients. In FIG2, we plot the best observed batch size as a function of the momentum coefficient, and fit our to the scaling rule above; obtaining remarkably good agreement. We propose a simple heuristic for tuning the batch size, learning rate and momentum coefficient in appendix E. Just like deep neural networks, linear models which generalize well on informative labels can memorize random labels of the same inputs. These observations are explained by the Bayesian evidence, which is composed of the cost function and an "Occam factor". The Occam factor penalizes sharp minima but it is invariant to changes in model parameterization. Mini-batch noise drives SGD away from sharp minima, and therefore there is an optimum batch size which maximizes the test accuracy. Interpreting SGD as the discretization of a stochastic differential equation, we predict this optimum batch size should scale linearly with both the learning rate and the training set size, B opt ∝ N. We derive an additional scaling rule, B opt ∝ 1/(1 − m), between the optimal batch size and the momentum coefficient. We verify these scaling rules empirically and discuss their implications. Instead of minimizing the cost function, Bayesian usually prefer to sample parameter values from the posterior , DISPLAYFORM0 where C(ω; M) is the regularized summed cost function, as shown in section 2 of the main text. It is well known that one can sample this posterior by simulating the overdamped Langevin equation BID9, which is described by the stochastic differential equation, DISPLAYFORM1 where t is a continuous variable, and η(t) describes Gaussian noise with mean η(t) = 0 and variance η(t)η(t) = 2T Iδ(t−t). The matrix I denotes the identity, while T is the "temperature". Notice this Langevin equation is extremely similar to the stochastic differential equation of SGD, discussed in section 5 of the main text. Indeed, if the gradient covariances F (ω) were stationary and proportional to the identity, then the SGD would integrate an overdamped Langevin equation with temperature proportional to the SGD noise scale g. As t → ∞, the probability of sampling any particular parameter vector ω from the Langevin equation, P (ω, t → ∞) ∝ e −C/T.We obtain posterior samples if T = 1. In order to draw posterior samples in practice, we repeatedly integrate the Langevin equation (at temperature T = 1), over a finite step t → t + /N, DISPLAYFORM2 where α denotes a Gaussian random variable with mean α = 0 and variance α 2 = 2 I/N, which introduces isotropic noise to the gradient update as described in section 4 of the main text. Note that, since C(ω; M) denotes the summed cost function, we chose to scale our step size by the training set size N. This also matches our treatment of SGD in section 5 of the main text. The larger the step size, the greater the discretization error, but if is sufficiently small and we iterate equation 17 sufficiently many times, we will obtain valid samples from the posterior. Since the probability of sampling any given parameter vector ω is proportional to the posterior, the probability of sampling a parameter vector belonging to any given local minimum is proportional to the integral of the posterior over the bowl of attraction D which surrounds that minimum. DISPLAYFORM3 Meanwhile we showed in section 2 of the main text that the evidence in favor of a model is proportional to the integral of the posterior over all parameter space. DISPLAYFORM4 As we discussed, this evidence is dominated by the contributions to the integral near local minima. In a convex model, there is only one such minimum; which allows us to accurately estimate the model evidence. Meanwhile, in non-convex models, there are many such minima, and so we can instead define the evidence as a sum over local evidences in favor of each minimum, DISPLAYFORM5 where we define the evidence in favor of a minimum as the integral over the local bowl of attraction, DISPLAYFORM6 Since the combined bowls of attraction of all the minima perfectly tile the entire parameter space, equations 19 and 20 are equivalent. Meanwhile, equating equations 18 and 21 we find that, when one performs Bayesian posterior sampling, the probability of sampling the parameter vector from a local minimum is proportional to the evidence in favor of that minimum. This demonstrates that bayesian posterior samples are biased in favor of local minima whose evidence is large, which explains why a single posterior sample ω p often achieves lower test error than the cost function minimum ω 0. In the experiments of section 4 of the main text, the L2 regularization coefficient λ = 0. In figure 8, we plot the evolution of the training curves when λ = 0.1, for both small batch and full batch training. Excluding the regularization parameter, these experiments are identical to figure 3. To our surprise, regularized full batch training took longer to converge than small batch training. In another surprise, regularization significantly reduced the size of the generalization gap. While large batch regularized training achieves slightly lower test set accuracy than unregularized small batch training, it also achieves lower test cross-entropy. The test cross-entropy of our regularized models does not degrade after many gradient updates, removing the need for early stopping. In section 5 of the main text, we approximated the difference between the full batch gradient and the mini-batch gradient estimate, α = (DISPLAYFORM0, by a Gaussian random variable. This enabled us to derive the scaling rules, which we verified empirically. We motivated this assumption by reference to the central limit theorem, which states that the gradient error will tend towards Gaussian noise as {N → ∞, B → ∞, B N}, so long as the distribution of gradients over individual training examples does not have heavy tails. In practice neither N nor B is infinite, and the gradient distribution may be heavy tailed, especially when gradients are sparse. Nonetheless the central limit theorem tends to be surprisingly robust in practice, and is consequently widely used. It is beyond the scope of this work to perform a thorough study of the gradient noise distribution in deep networks. However as a brief proof of principle, we present the distribution of the gradient immediately after random initialization in figure 9, for the shallow neural network discussed in sections 4 and 5 of the main text. In figure 9a, we present the distribution over the individual training examples, of the gradient of a single matrix element in the softmax output layer, chosen randomly. The distribution is double peaked and clearly not Gaussian. However in FIG2, we plot the distribution of the gradient of the same matrix element, when averaged over randomly sampled mini-batches of 30 images (without replacement). A single peak emerges, and while the distribution is still slightly skewed, it is clearly already approaching the Gaussian limit. We conclude that the Gaussian approximation is likely to be reasonable for commonly used mini-batch sizes. Momentum simulates a generalized Langevin equation (with structured fluctuations), DISPLAYFORM0 λ is the "damping coefficient" and η(t) describes Gaussian noise, whose statistics η(t) = 0 and η(t)η(t) = gλF (w)δ(t − t). As before, the coefficient g describes the scale of random fluctuations in the dynamics, and F (ω) describes the gradient covariances between parameters. We include a factor of λ in the noise variance to satisfy the fluctuation-dissipation theorem, which states that we can vary the damping coefficient without changing the probability of sampling any particular configuration of parameters in the limit t → ∞, if we proportionally increase the noise variance. To relate this Langevin equation to the usual momentum equations, we first re-express it as two coupled first order differential equations, dp dt = −λp − dC dω + η(t), DISPLAYFORM1 Integrating over a single step ∆t/N, DISPLAYFORM2 As observed in the main text, if we wish to keep the scale of random fluctuations constant, then we should scale the batch size B ∝ N. We also predict an additional scaling relation between the batch size and the momentum parameter, B ∝ 1/(1 − m). Note that one can also interpret ef f = /(1 − m) as the "effective learning rate". Here we propose a simple heuristic for tuning the batch size, learning rate and momentum parameter; in order to maximize both test accuracy and batch size (enabling parallel training across many machines). Note that this is only worthwhile if one expects to retrain a model many times.1. Set the learning rate to 0.1 and the momentum coefficient to 0.9. Run experiments at a range of batch sizes on a logarithmic scale, and identify the optimal batch size which maximizes the validation set accuracy. If training is not stable, reduce the learning rate, and repeat. 2. Repeatedly increase the batch size by a factor of 3, while scaling the learning rate ∝ B, until the validation set accuracy starts to fall. Then repeatedly increase the batch size by a factor of 3, while scaling the momentum coefficient (1 − m) ∝ 1/B, until either the validation set accuracy falls or the batch size reaches the limits of your hardware. 3. Having identified the final learning rate and momentum parameter, retune the batch size on a linear scale in the local neighborhood of the current batch size. We believe that this simple procedure will increase the test accuracy, reduce the cost of tuning hyperparameters, and significantly reduce the final number of gradient updates required to train a model.

Represent the TLDR for finding the corresponding scientific article

Generalization is strongly correlated with the Bayesian evidence, and gradient noise drives SGD towards minima whose evidence is large.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR

Question: In the industrial field, the positron annihilation is not affected by complex environment, and the gamma-ray photon penetration is strong, so the nondestructive detection of industrial parts can be realized. Due to the poor image quality caused by gamma-ray photon scattering, attenuation and short sampling time in positron process, we propose the idea of combining deep learning to generate positron images with good quality and clear details by adversarial nets. The structure of the paper is as follows: firstly, we encode to get the hidden vectors of medical CT images based on transfer Learning, and use PCA to extract positron image features. Secondly, we construct a positron image memory based on attention mechanism as a whole input to the adversarial nets which uses medical hidden variables as a query. Finally, we train the whole model jointly and update the input parameters until convergence. Experiments have proved the possibility of generating rare positron images for industrial non-destructive testing using countermeasure networks, and good imaging have been achieved. In recent years, with the advancement of science and technology, especially the rapid development of high-end manufacturing, in the field of industrial non-destructive testing, in many cases, it is necessary to perform defect detection without damaging or affecting the performance and internal structure of the device under test. Therefore, there is an increasing demand for corresponding detection devices. In complex industrial environments (such as aviation, internal combustion engines, chemical engineering, etc.), it is of great research significance to detect faults and defects in closed chambers. In this paper, the use of positron annihilation gamma photon imaging positron emission imaging technology for industrial nondestructive testing is studied. The occurrence of positron annihilation is not affected by factors such as high temperature, high pressure, corrosion, etc., so it can penetrate the dense metal material cavity, realize the undisturbed and non-destructive trace imaging of the detected object, and obtain the detected object after processing. Describe the image and perform a state analysis. Therefore, the quality of imaging technology directly affects the analysis of fault detection . Positron Emission Tomography (PET) was first used in medical imaging. The principle is that when a radioactive positron nucleus decays, a proton in the nucleus is converted into a neutron, and a positron and a neutron are released. The positron will quickly combine with the electrons in the material in a very short time, causing a positron annihilation phenomenon, producing a pair of gamma photon pairs with opposite directions and energy of 511KeV. Photon pairs are collected, identified, processed, and finally reconstructed to obtain medical images. Commonly used PET reconstruction algorithms are analytic method (K, 2000) and statistical method . The currently widely used algorithms are MLEM and OSEM. At present, PET technology has been widely used in the clinical diagnosis of human diseases. The advantages are quite obvious, the imaging quality is higher, and it shows great advantages in medical research. The principle of positron emission in industrial non-destructive fields is similar to medical imaging, but it has its own unique difficulties: the detection environment is more harsh, the sampling time is short, and at the same time, due to the phenomenon of scattering and attenuation of photons, industrial positron imaging is obtained. The image quality is even worse. Therefore, the reconstructed image needs to be further processed to obtain a higher quality image. In this paper, we propose adversarial networks of positron image memory module based on attention mechanism. Using medical images as basic data sets, introducing knowledge of migration learning, building memory module according to the contribution of detail features to images, a positron image generation network in the field of industrial non-destructive testing is obtained through joint training, thus achieving higher quality generation of industrial positron images. In summary, our main contributions in this paper are as follows: We are the first to advocate an idea of using Generative Adversarial Networks to enhance the detail of the positron image in the industrial non-destructive field, and realize the generation and processing of the scarce image data in the professional field. We use the medical CT image dataset as the basic training sample of the network framework, which is based on the idea of migration learning, and then extract the features of a small number of industrial non-destructively detected positron images, which can improve the details of the generated images, and make the network model have better applicability in the field of industrial non-destructive testing. We combine the attention-based mechanism in the professional domain image feature extraction. By constructing a memory module containing industrial positron image features, we can generate image generation in a specific domain, and finally conduct an industrial lossless positron image generation model. We train the whole network jointly, through the discriminant network of the antagonistic generation network, the front-end network was back-propagated, the input parameters were updated, and the model was optimized. Finally, the convergence was achieved and The Turing test was passed successfully. Model of GAN: Since GAN was proposed in 2014, it has become a research hotspot in the field of unsupervised learning, and related researches continue to increase. For the improvement of network structure, it mainly focuses on training stability and mode collapse. combine CNN with GAN to realize image generation, which makes GAN move from theory to practice, and establishes the framework and training mode of the whole GAN model. Arjovsky et al. M et al. propose Wasserstein GAN (WGAN), which use Earth-Mover instead of JS divergence to measure the distance between the real sample and the generated sample distribution, and the problem of pattern collapse was well solved. Mirza proposed Conditional Generative Adversarial Nets (CGAN) in 2014M & S, which transformed unsupervised tasks into supervised tasks, thus improving the stability of training. Mao, X et al. X et al. propose Least Squares Generative Adversarial Networks (LSGAN), which replace the loss function in the original network with the least squares loss function, alleviating the problem of training instability and insufficient diversity of generated images in GAN. Karras T et al. T et al. of NVIDIA team propose a progressive structure model to realize the transition from low-resolution to high-resolution image, and the generative model of high definition image can be trained smoothly. DeepMind team (A et al., 2018) introduce the idea of orthogonal regularization into GAN, which greatly improve the generation performance of GAN by truncating the input prior distribution Z in time, and contribute a lot of experience of parameter adjustment in the process of training mode. EBGAN (J et al., 2016) regards the discriminative network in GAN as an energy function and adds reconstruction error to it, which can solve the problem of mode collapse. Unrolled GAN (L et al., 2016) modifies the loss function of the generative network so that the generator can take into account the changes of discriminator after training K times and avoid the mode collapse caused by switching between different modes. DRAGAN (N et al., 2017) introduces the "no regret algorithm" in game theory, and construct gradient penalty scheme to avoid unwanted partial equilibrium. It can solve mode collapse problem and improve training efficiency. Domain Application: The greatest advantage of GAN is that it can generate real-like samples without any explicit modeling of data distribution in the whole generation process. Therefore, GAN has achieved good in many fields such as image, text, voice and so on. Image generation (P et al., 2016) (T et al., 2017) (Z et al., 2017), per resolution processing of image (C et al., 2017), object detection (Y et al., 2018) (J et al., 2017), object transfiguration (S et al., 2017), joint image generation , video generation , text to image (H et al., 2017)(A et al., 2017), text generation (J et al., 2018), speech conversion (C et al., 2017), domain adaptation (J et al., 2017), and so on. These researches provide more possibilities for the practical application of GAN in the field. to realize the batch generation of realistic medical images(prostatic lesion plaques, lung cancer cakes, retina), and the resolution reaches 16*16,56*56 and 64*64 respectively, the of which can pass the Turing test successfully. (C et al., 2018a) uses DCGAN to learn higher resolution MR images distribution from a small number of samples, and compare them with the real images, so as to achieve the effect of "false and true". (C et al., 2018b) uses PGGAN to synthesize skin lesion images, and it successfully showed amazing and highly realistic synthetic images. Conditional medical image generation studies are as follows: (D et al., 2017) realizes CT image synthesis from MR images by means of 3D full convolution network. The loss function of the network is composed of reconstruction loss and gradient loss, and it is also trained by combining additional confrontation network to improve the reality of the synthesized images. (P et al., 2017) compress the images of vascular tree into a multivariate normal distribution by means of the antagonistic automatic encoder. The normal distribution is sampled to synthesize any high resolution vascular tree image, and then the end-to-end framework for high resolution retinal image synthesis is obtained by using image conversion model (P et al., 2016). (T et al., 2017) proposes a two-stage GAN, the first one of which is to synthesize vessel tree images from noise training, and the second network uses image translation framework to generate real, high-resolution pairs of groundtruth vessel segmentation and the corresponding eye fundus image. (W et al., 2018) uses CGAN as the framework to synthesize 200*200 PET images by using CT image and binary label graph. The researchs contribution lies in the construction of a two-channel generation network to achieve a more realistic global output and it called multi-channel GAN. (F & D, 2018) sets up a multi-stage generator to obtain speckle images, low-resolution images, and high-resolution images in turn by intra-vascular ultrasound simulation of tissue maps according to different generating networks. (A & G, 2017) conducts joint learning by adding a specific task network to CGAN, and then obtain a network model that retain specific task characteristics. (M et al., 2018) uses WGAN as the network framework, and use noise and attribute vectors as inputs to generating high-resolution 3D images. In addition, in the aspects of image segmentation, reconstruction, detection, denoising, recognition, classification, etc, and the use of GAN model has also achieved some research , which provides feasibility for the application of GAN in professional and specific field. Traditional positron image processing in the field of industrial nondestructive processing focuses on the improvement of image reconstruction algorithm, but due to the inherent characteristics of small sampling points and short time, the quality of the obtained image is poor, and the stability of the image affected by the field is poor. Therefore, in this paper, we innovatively introduce the knowledge of neural network for positron image processing. And through the experimental verification, a high quality image is obtained. The structure of the whole positron image countermeasure generation network based on memory module is shown in the following figure 1: The original GAN model consists of two parts: the discriminative nets and the generative nets. Through training, the generator generates pseudo-data as close as possible to the real data distribution. The initial input of the network is random noise signal z, then map it to a new data space by generator to get the generated data G(z), after that the discriminator outputs a probability value through a binary classification of real data and generated data. The confidence level indicates the performance of the generator. Through continuous iteration optimization during the whole training process, the default optimization is achieved when the two sets of data cannot be distinguished. The basic idea of the GAN comes from the zero-sum game theory. The two networks against each other, in which the discriminator aims to distinguish the real distribution and the generative distribution as far as possible, while the generator aims to make the generated data consistent with the real data in the feature space and make the discriminator indistinguishable, finally the whole model reaches the optimum. The mathematical model is expressed as formula equation 1: Where x represents real images, z represents noise to the generator, and G(Z) represents the generated images, D(x) represents the probability of judging whether a real image is true. GAN is used to directly fit the distribution of n-dimensional vectors in images, and sample with random noise, the whole training process is in an unsupervised state, so the direction of data generation is uncontrollable unless the process traverses all initial distributions. At the same time, the generated images may not be the true expression of image meaning due to the excessive pursuit of quality. Therefore, when using the adversarial model to generate positron images in the industrial nondestructive testing, we choose to add a prior to restrict the data generation, so that the generation model can be trained for specific areas. Based on the scarcity of industrial positron image data, in this paper, we introduce the knowledge of migration learning and use medical images as training data to construct an encoder, which is based on the variational auto-encoder. Firstly, we sample medical image data X to get a series of sample points x 1, x 2, x 3, x n, which makes all the sample data in X fit successfully and obtains a distribution p(x), is described as formula equation 2: In order to achieve this goal, the distribution fitting of data sample X is finally realized with the help of implicit variable Z. It is assumed that p(x) describes a probability distribution of X generated by Z, which satisfies the Gauss distribution. Therefore, the whole encoder can be expressed as sampling Z from the standard normal distribution. In the process, we can get Variance and Mean of Sample Data X Sampled by Neural Network. The clustering process can be parametrized as formula equation 3: Where the mean and variance of the normal distribution which is exclusive to X k can be obtained. Then Z k can be sampled from this exclusive distribution. In order to obtain a more suitable generation model for positron image in the industrial field, we proposes an image feature memory module based on attention mechanism, which is used to extract domain image features. This network is a research focus of this chapter. The basic flow of the whole network is as follows:1) use neural network to extract the feature of the rare positron images, and obtain the images feature vectors. 2) combine the vector and the hidden variable get in Section 3.2 based on attention mechanism to obtain an image memory model. 3)use the memory model as the input of adversarial nets and train jointly with the whole network to obtain positron image generator for the industrial field. Positron image feature extraction We use the principal component analysis (H et al., 2015) is used to extract the positron sample data and the vector space transformation is used to reduce the dimensionality of higher dimensional positron data. Firstly, transform the original data into a new coordinate system by projection according to the new coordinate vector. Secondly, the variance of the first principal component of the projection data in the new coordinate system is the largest. As the dimension increases, the variance decreases in turn and the dimension decreases. is described as formula equation 4: The data matrix Y is de-averaged so that the mean value of each dimension is 0. Then we should find the most important feature vectors in the images, that is, the data on the coordinate axis represented by the feature fluctuates the most and the sum of squares of all samples projected on unit vector µ is the largest. Then we get the value of µ using Lagrange theorem. The mathematical expression is as follows equation 5: The nets use convolution neural network(CNN) to construct image feature extraction network, the network structure is divided into three layers, namely two convolution layers and one non-linear output layer. Firstly, small image slices are extracted from sample images and the dimension of the slices is the same as convolution core. Then traverse all the pixels in them and perform twolevel convolution operation. Finally, hashing operation and histogram statistics are carried out in the output layer to print the feature vector. Memory Module Based on Attention Mechanism The obtained positron eigenvectorŶ is fused with the hidden variable Z of the medical image obtained based on the attention mechanism to get the input nets. The purpose is to make the prior knowledge contained in the nets more focused on positron features, so that the features of scarce data can be more applied in the whole training process. The basic idea is the global attention and the focus in our model is to extract all positron image features. The specific realization is to align image data vectors, directly use medical images as query vectors, and input positron image feature vectors as hidden state to calculate their weights equation 6. Where z t is the medical image distribution, y s are feature vectors extracted from positron images, and score (z t, y s) is the scoring criterion for the operation. We get a constant and normalize it. And the contribution degree of each feature of positron image to the network can be obtained. So the image feature can be fused according to the weight ratio. Finally, the vector containing prior knowledge in the field is obtained as overall input of adversarial nets. Generative model The generative network is constructed based on DenseNet (G et al., 2017), and the positron image features can be requisitioned repeatedly in the model. The network can also strengthen the contribution of the characteristics of scarce data so that the generated images closer to real industrial positron images in detail. The generative model is as follows: the output of the memory model in chapter 3.3 as a whole input to the net, and the input of each layer is related to the output of all the previous layers, not only related to upper layer. It can be expressed as formula equation 7: X 0, X 1, · · ·, X (l−1) is the concatenation to the net. We can group all output feature maps from layer X 0 to X (l−1) according to different channels and the structure is used to reduce the parameters without losing features randomly, so that the initial input can enter each layers convolution calculation to realize the feature reuse. The basic structure is 33 convolution layer, Batch Normalization (S & C, 2015) and ReLU non-linear activation layer. Feature maps of all previous layers need to be cat in the network. In order to perform the down sampling operation, the net is divided into several Denseblocks and transition layers are used between different them. Referring to the original network, it consists of Batch Normalization layer, 1*1 convolution network and 2*2 average-pooling. In the same Denseblock, the state of each layer is associated with all previous layers, and the training of each layer is aimed at the global state feedback of the network to update the parameters. Discriminative model The discriminative net is used to discriminate specific images in a specific domain, in which domain image features can be used as the evaluation criteria for network classification as much as possible. The net uses Markov Model based on PatchGAN, which is composed of full convolution layers. The output is an n-dimensional matrix, and the mean of the matrix is used as the output of the discriminative network, so that each receptive field in the image can be judged, which is equivalent to the convolution discriminant in batches by layers, and finally fed back to the whole network. The calculation of receptive field is equation 8: In the model, the real input samples are medical data sets. Therefore, in order to make the generated data better characterize positron image features, we need to add an additional attention perception loss function to the net. The loss function of the whole net consists of two parts: L G AN and L A P G, the attention loss function L A P G is used to characterize the feature contribution of positron image, which is realized to measure the distribution distance between the generated data and the positron images. The loss function is described as equation 9: W i represents number of elements in each layer, and s is the number of layers. The loss function of the whole net can be described as equation 10, and the L G AN is similar as the original GAN. 4 EXPERIMENTAL We design the model firstly by using encoder to obtain the medical images hidden vectors and using principal component analysis to reduce positron images dimensionality and extract the main feature. Train memory module and adversarial nets jointly, and in the process of backpropagation, the identification network updates the parameters of the front-end network, so that the feature extraction network extracts the features repeatedly until the whole network achieves the optimal model. Finally, the positron image generator for industrial non-destructive testing is obtained. The discriminator refers to pixel and each batch is 70*70.The learning rate is 0.0002 in the whole net. The model are trained iteratively using Adam algorithm(=0.5). DeepLession The dataset (K et al., 2018) contains more than 32,000 lesions from more than 10,000 case studies. The data is developed by National Institutes of Health Clinical Center(NIHCC), and developed by mining historical medical data from image archives and communication systems. It is now maybe the largest set of CT medical image data available to all. In the experiment, 150,000 images were selected for our training and the image pixels are 256*256. Positron images The dataset is obtained by the Geant4 Application for Tomographic Emission(GATE). GATE is a simulation software which can simulate the physical process of PET imaging system, such as geometric size, material composition, detector movement and so on. In the model design, the anisotropic tube made of aluminium metal is filled with positron nuclide solution. Its activity is 600 Bq, the number of detectors is 184 *64, the sampling time is 0.1 s, the energy resolution is 15 percent, the time resolution is 300 ps, the energy window is 350-650 keV, and the time window is 10 ns. The design sampling time is 0.1s to meet the needs of rapid sampling in In this section, we compare our approach with the commonly used generation model, aiming at the generation of industrial positron images. Quantitative are presented in Table 1. The metrics to evaluate the performance of our method, namely multi-scale structural similarity(MS-SSIM) (A et al., 2017) and Frchet Inception Distance (FID) (M et al., 2017). The SSIM is used to measure the similarity of two images and FID is to measure the Frchet distance of two distributions in feature space. By comparing the experimental data, we can clearly see that the confrontation network constructed in this paper has a better effect on the generation of positron images for professional fields, and the generated images are closer to the real images. In this paper, we introduce an application of GAN in the field of nondestructive testing for specific industries. We combine the knowledge of transfer learning to make up the problem of insufficient data. The key point is to introduce attention mechanism to construct a positron image feature memory module, which can reuse image features under the condition of scarce data. At the same time, an attention loss function is added to the discriminative net to further improve the generator performance. Experiments show that compared with the start-of-the-art generation methods in deep learning, the model in our paper has an obvious improvement in the quality of industrial positron image generation. In the future, our focus is to further study the application of generative adversarial networks in industrial positron image processing, and to further improve the quality of domain images.

adversarial nets, attention mechanism, positron images, data scarcity

We introduced a novel, simple, and efficient data augmentation method that boosts the performances of existing GANs when training data is limited and diverse.