Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The show that our proposed method not only learns near-optimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms. Reinforcement learning (RL) has emerged as a promising tool for solving complex decision-making and control tasks from predefined high-level reward functions BID23. However, defining an optimizable reward function that inculcates the desired behavior can be challenging for many robotic applications, which include learning social-interaction skills BID17, dexterous manipulation BID5, and autonomous driving BID10.Inverse reinforcement learning (IRL) BID14 addresses the problem of learning reward functions from expert demonstrations, and it is often considered as a branch of imitation learning BID2 ). The prior work in IRL includes maximum-margin BID0 BID18 and maximum-entropy BID24 formulations. Currently, maximum entropy (MaxEnt) IRL is a widely used approach towards IRL, and has been extended to use non-linear function approximators such as neural networks in scenarios with unknown dynamics by leveraging sampling-based techniques BID3 BID5 BID9. However, designing the IRL algorithm is usually complicated as it requires, to some extent, hand engineering such as deciding domain-specific regularizers BID5.Rather than learning reward functions and solving the IRL problem, imitation learning (IL) learns a policy directly from expert demonstrations. Prior work addressed the IL problem through behavior cloning (BC), which learns a policy from expert trajectories using supervised learning BID15. Although BC methods are simple solutions to IL, these methods require a large amount of data because of compounding errors induced by covariate shift BID19. To overcome BC limitations, a generative adversarial imitation learning (GAIL) algorithm BID8 was proposed. GAIL uses the formulation of Generative Adversarial Networks (GANs) BID7, i.e., a generator-discriminator framework, where a generator is trained to generate expert-like trajectories while a discriminator is trained to distinguish between generated and expert trajectories. Although GAIL is highly effective and efficient framework, it does not recover transferable/portable reward functions along with the policies, thus narrowing its use cases to similar problem instances in similar environments. Reward function learning is ultimately preferable, if possible, over direct imitation learning as rewards are portable functions that represent the most basic and complete representation of agent intention, and can be re-optimized in new environments and new agents. Reward learning is challenging as there can be many optimal policies explaining a set of demonstrations and many reward functions inducing an optimal policy BID14 BID24. Recently, an adversarial inverse reinforcement learning (AIRL) framework BID6, an extension of GAIL, was proposed that offers a solution to the former issue by exploiting the maximum entropy IRL method BID24 whereas the latter issue is addressed through learning disentangled reward functions by modeling the reward as a function of state only instead of both state and action. However, AIRL fails to recover the ground truth reward when the ground truth reward is a function of both state and action. For example, the reward function in any locomotion or ambulation tasks contains a penalty term that discourages actions with large magnitudes. This need for action regularization is well known in optimal control literature and limits the use cases of a state-only reward function in most practical real-life applications. A more generalizable and useful approach would be to formulate reward as a function of both states and actions, which induces action-driven reward shaping that has been shown to play a vital role in quickly recovering the optimal policies BID13.In this paper, we propose the empowerment-regularized adversarial inverse reinforcement learning (EAIRL) algorithm 1. Empowerment BID20 ) is a mutual information-based theoretic measure, like state-or action-value functions, that assigns a value to a given state to quantify the extent to which an agent can influence its environment. Our method uses variational information maximization BID12 to learn empowerment in parallel to learning the reward and policy from expert data. Empowerment acts as a regularizer to policy updates to prevent overfitting the expert demonstrations, which in practice leads to learning robust rewards. Our experimentation shows that the proposed method recovers not only near-optimal policies but also recovers robust, transferable, disentangled, state-action based reward functions that are near-optimal. The on reward learning also show that EAIRL outperforms several state-of-the-art IRL methods by recovering reward functions that leads to optimal, expert-matching behaviors. On policy learning, demonstrate that policies learned through EAIRL perform comparably to GAIL and AIRL with non-disentangled (state-action) reward function but significantly outperform policies learned through AIRL with disentangled reward (state-only) and GAN interpretation of Guided Cost Learning (GAN-GCL) BID4. We consider a Markov decision process (MDP) represented as a tuple (S, A, P, R, ρ 0, γ) where S denotes the state-space, A denotes the action-space, P represents the transition probability distribution, i.e., P: S × A × S →, R(s, a) corresponds to the reward function, ρ 0 is the initial state distribution ρ 0: S → R, and γ ∈ is the discount factor. Let q(a|s, s) be an inverse model that maps current state s ∈ S and next state s ∈ S to a distribution over actions A, i.e., q: S × S × A →. Let π be a stochastic policy that takes a state and outputs a distribution over actions such that π: S × A →. Let τ and τ E denote a set of trajectories, a sequence of state-action pairs (s 0, a 0, · · · s T, a T), generated by a policy π and an expert policy π E, respectively, where T denotes the terminal time. Finally, let Φ(s) be a potential function that quantifies a utility of a given state s ∈ S, i.e., Φ: S → R. In our proposed work, we use an empowerment-based potential function Φ(·) to regularize policy update under MaxEnt-IRL framework. Therefore, the following sections provide a brief on MaxEnt-IRL, adversarial reward and policy learning, and variational information-maximization approach to learn the empowerment. MaxEnt-IRL BID24 ) models expert demonstrations as Boltzmann distribution using parametrized reward r ξ (τ) as an energy function, i.e., DISPLAYFORM0 where r ξ (τ) = T t=0 r ξ (s t, a t) is a commutative reward over given trajectory τ, parameterized by ξ, and Z is the partition function. In this framework, the demonstration trajectories are assumed to be sampled from an optimal policy π *, therefore, they get the highest likelihood whereas the suboptimal trajectories are less rewarding and hence, are generated with exponentially decaying probability. The main computational challenge in MaxEnt-IRL is to determine Z. The initial work in MaxEnt-IRL computed Z using dynamic programming BID24 whereas modern approaches BID5 a; BID6 present importance sampling technique to approximate Z under unknown dynamics. This section briefly describes Adversarial Inverse Reinforcement Learning (AIRL) BID6 algorithm which forms a baseline of our proposed method. AIRL is the current state-of-the-art IRL method that builds on GAIL BID8, maximum entropy IRL framework BID24 and GAN-GCL, a GAN interpretation of Guided Cost Learning BID5 a). GAIL is a model-free adversarial learning framework, inspired from GANs BID7, where the policy π learns to imitate the expert policy behavior π E by minimizing the JensenShannon divergence between the state-action distributions generated by π and the expert state-action distribution by π E through following objective DISPLAYFORM0 where D is the discriminator that performs the binary classification to distinguish between samples generated by π and π E, λ is a hyper-parameter, and H(π) is an entropy regularization term E π [log π]. Note that GAIL does not recover reward; however, BID4 shows that the discriminator can be modeled as a reward function. Thus AIRL BID6 ) presents a formal implementation of BID4 and extends GAIL to recover reward along with the policy by imposing a following structure on the discriminator: DISPLAYFORM1 where f ξ,ϕ (s, a, s) = r ξ (s) + γh ϕ (s) − h ϕ (s) comprises a disentangled reward term r ξ (s) with training parameters ξ, and a shaping term F = γh ϕ (s) − h ϕ (s) with training parameters ϕ. The entire D ξ,ϕ (s, a, s) is trained as a binary classifier to distinguish between expert demonstrations τ E and policy generated demonstrations τ. The policy is trained to maximize the discriminative reward DISPLAYFORM2 consists of free-parameters as no structure is imposed on h ϕ (·), and as mentioned in BID6, the reward function r ξ (·) and function F are tied upto a constant (γ − 1)c, where c ∈ R; thus the impact of F, the shaping term, on the recovered reward r is quite limited and therefore, the benefits of reward shaping are not fully realized. Mutual information (MI), an information-theoretic measure, quantifies the dependency between two random variables. In intrinsically-motivated reinforcement learning, a maximal of mutual information between a sequence of K actions a and the final state s reached after the execution of a, conditioned on current state s is often used as a measure of internal reward BID12, known as Empowerment Φ(s), i.e., DISPLAYFORM0 where p(s |a, s) is a K-step transition probability, w(a|s) is a distribution over a, and p(a, s |s) is a joint-distribution of K actions a and final state s 2. Intuitively, the empowerment Φ(s) of a state s quantifies an extent to which an agent can influence its future. Thus, maximizing empowerment induces an intrinsic motivation in the agent that enforces it to seek the states that have the highest number of future reachable states. Empowerment, like value functions, is a potential function that has been previously used in reinforcement learning but its applications were limited to small-scale cases due to computational intractability of MI maximization in higher-dimensional problems. Recently, however, a scalable method BID12 was proposed that learns the empowerment through the moreefficient maximization of variational lower bound, which has been shown to be equivalent to maximizing MI BID1. The lower bound was derived (for complete derivation see Appendix A.1) by representing MI in term of the difference in conditional entropies H(·) and utilizing the non-negativity property of KL-divergence, i.e., DISPLAYFORM1 where DISPLAYFORM2 is a variational distribution with parameters φ and w θ (·) is a distribution over actions with parameters θ. Finally, the lower bound in Eqn. 5 is maximized under the constraint H(a|s) < η (prevents divergence, see BID12) to compute empowerment as follow: DISPLAYFORM3 where β is η dependent temperature term. BID12 also applied the principles of Expectation-Maximization (EM) BID1 to learn empowerment, i.e., alternatively maximizing Eqn. 6 with respect to w θ (a|s) and q φ (a|s, s). Given a set of training trajectories τ, the maximization of Eqn. 6 w.r.t q φ (·) is shown to be a supervised maximum log-likelihood problem whereas the maximization w.r.t w θ (·) is determined through the functional derivative ∂I/∂w = 0 under the constraint a w(a|s) = 1. The optimal w * that maximizes Eqn. 6 turns out to be 1 DISPLAYFORM4, where Z(s) is a normalization term. Substituting w * in Eqn. 6 showed that the empowerment Φ(s) = 1 β log Z(s) (for full derivation, see Appendix A.2).Note that w * (a|s) is implicitly unnormalized as there is no direct mechanism for sampling actions or computing Z(s). BID12 introduced an approximation w * (a|s) ≈ log π(a|s) + Φ(s) where π(a|s) is a normalized distribution which leaves the scalar function Φ(s) to account for the normalization term log Z(s). Finally, the parameters of policy π and scalar function Φ are optimized by minimizing the discrepancy, l I (s, a, s), between the two approximations (log π(a|s) + Φ(s)) and β log q φ (a|s, s)) through either absolute (p = 1) or squared error (p = 2), i.e., DISPLAYFORM5 3 EMPOWERED ADVERSARIAL INVERSE REINFORCEMENT LEARNINGWe present an inverse reinforcement learning algorithm that learns a robust, transferable reward function and policy from expert demonstrations. Our proposed method comprises (i) an inverse model q φ (a|s, s) that takes the current state s and the next state s to output a distribution over actions A that ed in s to s transition, (ii) a reward r ξ (s, a), with parameters ξ, that is a function of both state and action, (iii) an empowerment-based potential function Φ ϕ (·) with parameters ϕ that determines the reward-shaping function F = γΦ ϕ (s) − Φ ϕ (s) and also regularizes the policy update, and (iv) a policy model π θ (a|s) that outputs a distribution over actions given the current state s. All these models are trained simultaneously based on the objective functions described in the following sections to recover optimal policies and generalizable reward functions concurrently.3.1 INVERSE MODEL q φ (a|s, s) OPTIMIZATION As mentioned in Section 2.3, learning the inverse model q φ (a|s, s) is a maximum log-likelihood supervised learning problem. Therefore, given a set of trajectories τ ∼ π, where a single trajectory is a sequence states and actions, i.e., τ i = {s 0, a 0, · · ·, s T, a T} i, the inverse model q φ (a|s, s) is trained to minimize the mean-square error between its predicted action q(a|s, s) and the action a taken according to the generated trajectory τ, i.e., DISPLAYFORM6 Empowerment will be expressed in terms of normalization function Z(s) of optimal w * (a|s), i.e., DISPLAYFORM7. Therefore, the estimation of empowerment Φ ϕ (s) is approximated by minimizing the loss function l I (s, a, s), presented in Eqn. 7, w.r.t parameters ϕ, and the inputs (s, a, s) are sampled from the policy-generated trajectories τ. To train the reward function, we first compute the discriminator as follow: DISPLAYFORM0 where r ξ (s, a) is the reward function to be learned with parameters ξ. We also maintain the target ϕ and learning ϕ parameters of the empowerment-based potential function. The target parameters ϕ are a replica of ϕ except that the target parameters ϕ are updated to learning parameters ϕ after every n training epochs. Note that keeping a stationary target Φ ϕ stabilizes the learning as also mentioned in BID11. Finally, the discriminator/reward function parameters ξ are trained via binary logistic regression to discriminate between expert τ E and generated τ trajectories, i.e., DISPLAYFORM1 3.4 POLICY OPTIMIZATION POLICY π θ (a|s)We train our policy π θ (a|s) to maximize the discriminative rewardr(s, a, s) = log(D(s, a, s) − log(1 − D(s, a, s))) and to minimize the loss function l I (s, a, s) = β log q φ (a|s, s) − (log π θ (a|s) + Φ ϕ (s)) p which accounts for empowerment regularization. Hence, the overall policy training objective is: DISPLAYFORM2 where policy parameters θ are updated using any policy optimization method such as TRPO BID21 or an approximated step such as PPO BID22.Algorithm 1 outlines the overall training procedure to train all function approximators simultaneously. Note that the expert samples τ E are seen by the discriminator only, whereas all other models are trained using the policy generated samples τ. Furthermore, the discriminating rewardr(s, a, s) boils down to the following expression (Appendix B.1): DISPLAYFORM3 where f (s, a, s) = r ξ (s, a) + γΦ ϕ (s) − Φ ϕ (s). Thus, an alternative way to express our policy training objective is E τ [log π θ (a|s)r π (s, a, s)], where r π (s, a, s) =r(s, a, s) − λ I l I (s, a, s), DISPLAYFORM4 Update θ i to θ i+1 using natural gradient update rule (i.e., TRPO/PPO) with the gradient: DISPLAYFORM5 After every n epochs sync ϕ with ϕ Fig (b) represents a problem where environment structure is modified during testing, i.e., a reward learned on a maze with left-passage is transferred to a maze with right-passage to the goal (green).which would undoubtedly yield the same as Eqn. 11, i.e., maximize the discriminative reward and minimize the loss l I. The analysis of this alternative expression is given in Appendix B to highlight that our policy update rule is equivalent to MaxEnt-IRL policy objective BID4 except that it also maximizes the empowerment, i.e., DISPLAYFORM6 where, λ and γ are hyperparameters, andĤ(·) is the entropy-regularization term depending on π(·) and q(·). Hence, our policy is regularized by the empowerment which induces generalized behavior rather than locally overfitting to the limited expert demonstrations. Our proposed method, EAIRL, learns both reward and policy from expert demonstrations. Thus, for comparison, we evaluate our method against both state-of-the-art policy and reward learning techniques on several control tasks in OpenAI Gym. In case of policy learning, we compare our method against GAIL, GAN-GCL, AIRL with state-only reward, denoted as AIRL(s), and an augmented version of AIRL we implemented for the purposes of comparison that has state-action reward, denoted as AIRL(s, a). In reward learning, we only compare our method against AIRL(s) and AIRL(s, a) as GAIL does not recover rewards, and GAN-GCL is shown to exhibit inferior performance than AIRL BID6. Furthermore, in the comparisons, we also include the expert The performance of policies obtained from maximizing the learned rewards in the transfer learning problems. It can be seen that our method performs significantly better than AIRL BID6 and exhibits expert-like performance in all five randomly-seeded trials which imply that our method learns near-optimal, transferable reward functions.performances which represents a policy learned by optimizing a ground-truth reward using TRPO BID21. The performance of different methods are evaluated in term of mean and standard deviation of total rewards accumulated (denoted as score) by an agent during the trial, and for each experiment, we run five randomly-seeded trials. To evaluate the learned rewards, we consider a transfer learning problem in which the testing environments are made to be different from the training environments. More precisely, the rewards learned via IRL in the training environments are used to re-optimize a new policy in the testing environment using standard RL. We consider two test cases shown in the FIG0.In the first test case, as shown in FIG0, we modify the agent itself during testing. We trained a reward function to make a standard quadruped ant to run forward. During testing, we disabled the front two legs (indicated in red) of the ant (crippled-ant), and the learned reward is used to reoptimize the policy to make a crippled-ant move forward. Note that the crippled-ant cannot move sideways (Appendix C.1). Therefore, the agent has to change the gait to run forward. In the second test case, shown in FIG0, we change the environment structure. The agent learns to navigate a 2D point-mass to the goal region in a simple maze. We re-position the maze central-wall during testing so that the agent has to take a different path, compared to the training environment, to reach the target (Appendix C.2). TAB0 summarizes the means and standard deviations of the scores over five trials. It can be seen that our method recovers near-optimal reward functions as the policy scores almost reach the expert scores in all five trials even after transfering to unseen testing environments. Furthermore, our method performs significantly better than both Next, we considered the performance of the learned policy specifically for an imitation learning problem in various control tasks. The tasks, shown in FIG3, include (i) making a 2D halfcheetah robot to run forward, (ii) making a 3D quadruped robot (ant) to move forward, (iii) making a 2D swimmer to swim, and (iv) keeping a friction less pendulum to stand vertically up. For each algorithm, we provided 20 expert demonstrations generated by a policy trained on a ground-truth reward using TRPO BID21. TAB2 presents the means and standard deviations of policy learning performance scores, over the five different trials. It can be seen that EAIRL, AIRL(s, a) and GAIL demonstrate similar performance and successfully learn to imitate the expert policy, whereas AIRL(s) and GAN-GCL fails to recover a policy. This section highlights the importance of empowerment-regularized MaxEnt-IRL and modeling rewards as a function of both state and action rather than restricting to state-only formulation on learning rewards and policies from expert demonstrations. In the scalable MaxEnt-IRL framework BID4 BID6, the normalization term is approximated by importance sampling where the importance-sampler/policy is trained to minimize the KL-divergence from the distribution over expert trajectories. However, merely minimizing the divergence between expert demonstrations and policy-generated samples leads to localized policy behavior which hinders learning generalized reward functions. In our proposed work, we regularize the policy update with empowerment i.e., we update our policy to reduce the divergence from expert data distribution as well as to maximize the empowerment (Eqn.12). The proposed regularization prevents premature convergence to local behavior which leads to robust state-action based rewards learning. Furthermore, empowerment quantifies the extent to which an agent can control/influence its environment in the given state. Thus the agent takes an action a on observing a state s such that it has maximum control/influence over the environment upon ending up in the future state s.Our experimentation also shows the importance of modeling discriminator/reward functions as a function of both state and action in reward and policy learning under GANs framework. The re-ward learning show that state-only rewards (AIRL(s)) does not recover the action dependent terms of the ground-truth reward function that penalizes high torques. Therefore, the agent shows aggressive behavior and sometimes flips over after few steps (see the accompanying video), which is also the reason that crippled-ant trained with AIRL's disentangled reward function reaches only the half-way to expert scores as shown in TAB0. Therefore, the reward formulation as a function of both states and actions is crucial to learning action-dependent terms required in most real-world applications, including any autonomous driving, robot locomotion or manipulation task where large torque magnitudes are discouraged or are dangerous. The policy learning further validate the importance of the state-action reward formulation. TAB2 shows that methods with state-action reward/discriminator formulation can successfully recover expert-like policies. Hence, our empirical show that it is crucial to model reward/discriminator as a function of state-action as otherwise, adversarial imitation learning fails to learn ground-truth rewards and expert-like policies from expert data. We present an approach to adversarial reward and policy learning from expert demonstrations by regularizing the maximum-entropy inverse reinforcement learning through empowerment. Our method learns the empowerment through variational information maximization in parallel to learning the reward and policy. We show that our policy is trained to imitate the expert behavior as well to maximize the empowerment of the agent over the environment. The proposed regularization prevents premature convergence to local behavior and leads to a generalized policy that in turn guides the reward-learning process to recover near-optimal reward. We show that our method successfully learns near-optimal rewards, policies, and performs significantly better than state-of-the-art IRL methods in both imitation learning and challenging transfer learning problems. The learned rewards are shown to be transferable to environments that are dynamically or structurally different from training environments. In our future work, we plan to extend our method to learn rewards and policies from diverse human/expert demonstrations as the proposed method assumes that a single expert generates the training data. Another exciting direction would be to build an algorithm that learns from sub-optimal demonstrations that contains both optimal and non-optimal behaviors. For completeness, we present a derivation of presenting mutual information (MI) as variational lower bound and maximization of lower bound to learn empowerment. As mentioned in section 2.3, the variational lower bound representation of MI is computed by defining MI as a difference in conditional entropies, and the derivation is formalized as follow. ≥ −E w(a|s) log w(a|s) + E p(s |a,s)w(a|s) [log q(a|s, s)] DISPLAYFORM0 The empowerment is a maximal of MI and it can be formalized as follow by exploiting the variational lower bound formulation (for details see BID12). w,q E p(s |a,s)w(a|s) [− 1 β log w(a|s) + log q(a|s, s)]As mentioned in section 2.3, given a training trajectories, the maximization of Eqn. 13 w.r.t inverse model q(a|s, s) is a supervised maximum log-likelihood problem. The maximization of Eqn. 13 w.r.t w(a|s) is derived through a functional derivative ∂I w,q /∂w = 0 under the constraint a w(a|s) = 1. For simplicity, we consider discrete state and action spaces, and the derivation is as follow: By using the constraint a w(a|s) = 1, it can be shown that the optimal solution w * (a|s) = 1 Z(s) exp(u(s, a)), where u(s, a) = βE p(s |a,s) [log q(a|s, s)] and Z(s) = a u(s, a). This solution maximizes the lower bound since ∂ 2 I w (s)/∂w 2 = − a 1 w(a|s) < 0. DISPLAYFORM0 In this section we derive the Empowerment-regularized formulation of maximum entropy IRL. Let τ be a trajectory sampled from expert demonstrations D and p ξ (τ) ∝ p(s 0)Π T −1 t=0 p(s t+1 |s t, a t) exp r ξ (st,at) be a distribution over τ. As mentioned in Section 2, the IRL objective is to maximize the likelihood: DISPLAYFORM0 Furthermore, as derived in BID6, the gradient of above equation w.r.t ξ can be written as: DISPLAYFORM1 where r ξ (·) is a parametrized reward to be learned, and p ξ,t = s t =t,a t =t p ξ (τ) denotes marginalization of state-action at time t. Since, it is unfeasible to draw samples from p ξ, BID4 proposed to train an importance sampling distribution µ(τ) whose varience is reduced by defining µ(τ) as a mixture of polices, i.e., µ(a|s) = 1 2 (π(a|s) +p(a|s)), wherep is a rough density estimate over demonstrations. Thus the above gradient becomes: DISPLAYFORM2 We train our importance-sampler/policy π to maximize the empowerment Φ(·) for generalization and to reduce divergence from true distribution by minimizing DISPLAYFORM3, the matching terms of π(τ) and p ξ (τ) cancel out, ing into entropy-regularized policy update. Furthermore, as we also include the empowerment Φ(·) in the policy update to be maximized, hence the overall objective becomes: DISPLAYFORM4 Our discriminator is trained to minimize cross entropy loss as mention in Eqn. 10, and for the proposed structure of our discriminator Eqn. 9, it can be shown that the discriminator's gradient w.r.t its parameters turns out to be equal to Equation 14 (for more details, see BID6). On the other hand, our policy training objective is DISPLAYFORM5 In the next section, we show that the above policy training objective is equivalent to Equation 15. We train our policy to maximize the discriminative rewardr(s, a, s) = log(D(s, a, s) − log(1 − D(s, a, s))) and minimize the information-theoretic loss function l I (s, a, s). The discriminative rewardr(s, a, s) simplifies to: DISPLAYFORM0 where f (s, a, s) = r(s, a) + γΦ(s) − Φ(s). The entropy-regularization is usually scaled by the hyperparameter, let say λ h ∈ R, thusr(s, a, s) = f (s, a, s) − λ h log π(a|s). Hence, assuming single-sample (s, a, s), absolute-error for l I (s, a, s) = | log q φ (a|s, s) − (log π(a|s) + Φ(s))|, and l i > 0, the policy is trained to maximize following: DISPLAYFORM1 = r(s, a) + γΦ(s) − Φ(s) − λ h log π(a|s) − log q(a|s, s) + log π(a|s) + Φ(s) = r(s, a) + γΦ(s) − λ h log π(a|s) − log q(a|s, s) + log π(a|s)Note that, the potential function Φ(s) cancels out and we scale the leftover terms of l I with a hyperparameter λ I. Hence, the above equation becomes: r π (s, a, s) = r(s, a, s) + γΦ(s) + (λ I − λ h) log π(a|s) − λ I log q(a|s, s)We combine the log terms together as: r π (s, a, s) = r(s, a) + λ I Φ(s) + λĤ(·)where λ is a hyperparameter, andĤ(·) is an entropy regularization term depending on q(a|s, s) and π(a|s). Therefore, it can be seen that the Eqn. 17 is equivalent/approximation to Eqn. 15. The following figures show the difference between the path profiles of standard and crippled Ant. It can be seen that the standard Ant can move sideways whereas the crippled ant has to rotate in order to move forward. The following figures show the path profiles of a 2D point-mass agent to reach the target in training and testing environment. It can be seen that in the testing environment the agent has to take the opposite route compared to the training environment to reach the target. We use two-layer ReLU network with 32 units in each layer for the potential function h ϕ (·) and Φ ϕ (·), reward function r ξ (·), discriminators of GAIL and GAN-GCL. Furthermore, policy π θ (·) of Figure 5: The top and bottom rows show the path followed by a 2D point-mass agent (yellow) to reach the target (green) in training and testing environment, respectively.all presented models and the inverse model q φ (·) of EAIRL are presented by two-layer RELU network with 32 units in each layer, where the network's output parametrizes the Gaussian distribution, i.e., we assume a Gaussian policy. For all experiments, we use the temperature term β = 1. We evaluated both mean-squared and absolute error forms of l I (s, a, s) and found that both lead to similar performance in reward and policy learning. We set entropy regularization weight to 0.1 and 0.001 for reward and policy learning, respectively. The hyperparameter λ I was set to 1.0 for reward learning and 0.001 for policy learning. The target parameters of the empowerment-based potential function Φ ϕ (·) were updated every 5 and 2 epochs during reward and policy learning respectively. Although reward learning hyperparameters are also applicable to policy learning, we decrease the magnitude of entropy and information regularizers during policy learning to speed up the policy convergence to optimal values. Furthermore, we set the batch size to 2000-and 20000-steps per TRPO update for the pendulum and remaining environments, respectively. For the methods BID6 BID8 presented for comparison, we use their suggested hyperparameters. We also use policy samples from previous 20 iterations as negative data to train the discriminator of all IRL methods presented in this paper to prevent the parametrized reward functions from overfitting the current policy samples.

Represent the short summary for finding the corresponding scientific article (long)

Our method introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies from expert demonstrations.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs;), which additively combine tensor products of vectors representing roles (e.g., sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations. By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated byTPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations Compositional symbolic representations are widely held to be necessary for intelligence BID8 ), particularly in the domain of language BID7. However, neural networks have shown great success in natural language processing despite using continuous vector representations rather than explicit symbolic structures. How can these continuous representations yield such success in a domain traditionally believed to require symbol manipulation?One possible answer is that neural network representations implicitly encode compositional structure. This hypothesis is supported by the spatial relationships between such vector representations, which have been argued to display geometric regularities that parallel plausible symbolic structures of the elements being represented (Mikolov et al. 2013 ; see Figure 1).Analogical relationships such as those in Figure 1 are special cases of linearity properties shared by several methods developed in the 1990s for designing compositional vector embeddings of symbolic structures. The most general of these is tensor product representations (TPRs; BID22 . Symbolic structures are first decomposed into filler-role bindings; for example, to represent the sequence, the filler 5 may be bound to the role of first element, the filler 2 may be bound to the role of second element, and so on. Each filler f i and -crucially -each role r i has a vector embedding; these two vectors are combined using their tensor product f i ⊗ r i, and these tensor products are summed to produce the representation of the sequence: f i ⊗ r i. This linear combination can predict the linear relations between sequence representations illustrated in Figure 1. (a) (b) (c) Figure 1: Plots of the first two principal components of (a) word embeddings BID14, (b) digit-sequence embeddings learned by an autoencoder (Section 2), and (c) sentences (InferSent: Conneau et al. 2017). All demonstrate systematicity in the learned vector spaces. In this article, we test the hypothesis that vector representations of sequences can be approximated as a sum of filler-role bindings, as in TPRs. We introduce the Tensor Product Decomposition Network (TPDN) which takes a set of continuous vector representations to be analyzed and learns filler and role embeddings that best predict those vectors, given a particular hypothesis for the relevant set of roles (e.g., sequence indexes or structural positions in a parse tree).To derive structure-sensitive representations, in Section 2 we look at a task driven by structure, not content: autoencoding of sequences of meaningless symbols, denoted by digits. The focus here is on sequential structure, although we also devise a version of the task that uses tree structure. For the representations learned by these autoencoders, TPDNs find excellent approximations that are TPRs. In Section 3, we turn to sentence-embedding models from the contemporary literature. It is an open question how structure-sensitive these representations are; to the degree that they are structuresensitive, our hypothesis is that they can be approximated by TPRs. Here, TPDNs find less accurate approximations, but they also show that a TPR equivalent to a bag-of-words already provides a reasonable approximation; these suggest that these sentence representations are not robustly structure-sensitive. We therefore return to synthetic data in Section 4, exploring which architectures and training tasks are likely to lead RNNs to induce structure-sensitive representations. To summarize the contributions of this work, TPDNs provide a powerful method for interpreting vector representations, shedding light on hard-to-understand neural architectures. We show that standard RNNs can induce compositional representations that are remarkably well approximated by TPRs and that the nature of these representations depends, in intrepretable ways, on the architecture and training task. Combined with our finding that standard sentence encoders do not seem to learn robust representations of structure, these findings suggest that more structured architectures or more structure-dependent training tasks could improve the compositional capabilities of existing models. The Tensor Product Decomposition Network (TPDN), depicted in FIG0, learns a TPR that best approximates an existing set of vector encodings. While TPDNs can be applied to any structured space, including embeddings of images or words, this work focuses on applying TPDNs to sequences. The model is given a hypothesized role scheme and the dimensionalities of the filler and role embeddings. The elements of each sequence are assumed to be the fillers in that sequence's representation; for example, if the hypothesized roles are indexes counting from the end of the sequence, then the hypothesized filler-role pairs for would be (4:last, 2:second-to-last, 5:third-to-last).The model then learns embeddings for these fillers and roles that minimize the distance between the TPRs generated from these embeddings and the existing encodings of the sequences. Before the comparison is performed, the tensor product (which is a matrix) is flattened into a vector, and a linear transformation M is applied (see Appendix B for an ablation study showing that this transformation, which was not a part of the original TPR proposal, is necessary). The overall function computed by the architecture is thus M (flatten( i r i ⊗ f i)). To establish the effectiveness of the TPDN at uncovering the structural representations used by RNNs, we first apply the TPDN to sequence-to-sequence networks trained on an autoencoding objective: they are expected to encode a sequence of digits and then decode that encoding to reproduce the same sequence (FIG0). In addition to testing the TPDN, this experiment also addresses a scientific question: do different architectures (specifically, unidirectional, bidirectional, and tree-based sequence-to-sequence models) induce different representations? Digit sequences: The sequences consisted of the digits from 0 to 9. We randomly generated 50,000 unique sequences with lengths ranging from 1 to 6 inclusive and averaging 5.2; these sequences were divided into 40,000 training sequences, 5,000 development sequences, and 5,000 test sequences. Architectures: For all sequence-to-sequence networks, we used gated recurrent units as the recurrent units. We considered three encoder-decoder architectures: unidirectional, bidirectional, and tree-based. 3 The unidirectional encoders and decoders follow the setup of BID26: the encoder is fed the input elements one at a time, left to right, updating its hidden state after each element. The decoder then produces the output sequence using the final hidden state of the encoder as its input. The bidirectional encoder combines left-to-right and right-toleft unidirectional encoders BID19; for symmetry, we also create a bidirectional decoder, which has both a left-to-right and a right-to-left unidirectional decoder whose hidden states are concatenated to form bidirectional hidden states from which output predictions are made. Our final topology is tree-based RNNs BID17 BID24, specifically the Tree-GRU encoder of BID6 and the tree decoder of. These architectures require a tree structure as part of their input; we generated a tree for each sequence using a deterministic algorithm that groups digits based on their values (see Appendix C). To control for initialization effects, we trained five instances of each architecture with different random initializations. Role schemes: We consider 6 possible methods that networks might use to represent the roles of specific digits within a sequence; see FIG1 for examples of these role schemes.1. Left-to-right: Each digit's role is its index in the sequence, counting from left to right. 2. Right-to-left: Each digit's role is its index in the sequence, counting from right to left. 3. Bidirectional: Each digit's role is an ordered pair containing its left-to-right index and its right-to-left index (compare human representations of spelling, Fischer-Baum et al. 2010). 4. Wickelroles: Each digit's role is the digit before it and the digit after it BID35. 5. Tree positions: Each digit's role is its position in a tree, such as RRL (left child of right child of right child of root). The tree structures are given by the algorithm in Appendix C.6. Bag-of-words: All digits have the same role. We call this a bag-of-words because it represents which digits ("words") are present and in what quantities, but ignores their positions. We hypothesize that RNN autoencoders will learn to use role representations that parallel their architectures: left-to-right roles for a unidirectional network, bidirectional roles for a bidirectional network, and tree-position roles for a tree-based network. Evaluation: We evaluate how well a given sequence-to-sequence network can be approximated by a TPR with a particular role scheme as follows. First, we train a TPDN with the role scheme in question (Section 1.1). Then, we take the original encoder/decoder network and substitute the fitted TPDN for its encoder FIG0 ). We do not conduct any additional training upon this hybrid network; the decoder retains exactly the weights it learned in association with the original encoder, while the TPDN retains exactly the weights it learned for approximating the original encoder (including the weights on the final linear layer). We then compute the accuracy of the ing hybrid network; we call this metric the substitution accuracy. High substitution accuracy indicates that the TPDN has approximated the encoder well enough for the decoder to handle the ing vectors. Performance of seq2seq networks: The unidirectional and tree-based architectures both performed the training task nearly perfectly, with accuracies of 0.999 and 0.989 (averaged across five runs). Accuracy was lower (0.834) for the bidirectional architecture; this might mean that the hidden size of 60 becomes too small when divided into two 30-dimensional halves, one half for each direction. Quality of TPDN approximation: For each of the six role schemes, we fitted a TPDN to the vectors generated by the trained encoder, and evaluated it using substitution accuracy (Section 2.1). The , in FIG1, show that different architectures do use different representations to solve the task. The tree-based autoencoder can be well-approximated using tree-position roles but not using any of the other role schemes. By contrast, the unidirectional architecture is approximated very closely (with a substitution accuracy of over 0.99 averaged across five runs) by bidirectional roles. Left-to-right roles are also fairly successful (accuracy = 0.87), and right-to-left roles are decidedly unsuccessful (accuracy = 0.11). This asymmetry suggests that the unidirectional network uses mildly bidirectional roles: while it is best approximated by bidirectional roles, it strongly favors one direction over the other. Though the model uses bidirectional roles, then, roles with the same left-to-right position (e.g.,, and) can be collapsed without much loss of accuracy. Finally, the bidirectional architecture is not approximated well by any of the role schemes we investigated. It may be implementing a role scheme we did not consider, or a structure-encoding scheme other than TPR. Alternately, it might simply not have adopted any robust method for representing sequence structure; this could explain why its accuracy on the training task was relatively low (0.83). Will the TPDN's success with digit-sequence autoencoders extend to models trained on naturally occurring data? We explore this question using sentence representations from four models: InferSent , a BiLSTM trained on the Stanford Natural Language Inference (SNLI) corpus BID3; Skip-thought , an LSTM trained to predict the sentence before or after a given sentence; the Stanford sentiment model (SST) BID25, a tree-based recursive neural tensor network trained to predict movie review sentiment; and SPINN BID4, a tree-based RNN trained on SNLI. More model details are in Appendix E. We now fit TPDNs to these four sentence encoding models. We experiment with all of the role schemes used in Section 2 except for Wickelroles; for sentence representations, the vocabulary size |V | is so large that the Wickelrole scheme, which requires |V | 2 distinct roles, becomes intractable. Preliminary experiments showed that the TPDN performed poorly when learning the filler embeddings from scratch, so we used pretrained word embeddings; for each model, we use the word embeddings used by that model. We fine-tuned the embeddings with a linear transformation on top of the word embedding layer (though the embeddings themselves remain fixed). Thus, what the model has to learn are: the role embeddings, the linear transformation to apply to the fixed filler embeddings, and the final linear transformation applied to the sum of the filler/role bindings. We train TPDNs on the sentence embeddings that each model generates for all SNLI premise sentences BID3. For other training details see Appendix E. Table 1a shows the mean squared errors (MSEs) for various role schemes. In general, the MSEs show only small differences between role schemes, except that tree-position roles do noticeably outperform other role schemes for SST. Notably, bag-of-words roles perform nearly as well as the other role schemes, in stark contrast to the poor performance of bag-of-words roles in Section 2. MSE is useful for comparing models but is less useful for assessing absolute performance since the exact value of this error is not very interpretable. In the next section, we use downstream tasks for a more interpretable evaluation. Tasks: We assess how the tensor product approximations compare to the models they approximate at four tasks that are widely accepted for evaluating sentence embeddings: Stanford Sentiment Treebank (SST), rating the sentiment of movie reviews BID25; Microsoft Research Evaluation: We use SentEval to train a classifier for each task on the original encodings produced by the sentence encoding model. We freeze this classifier and use it to classify the vectors generated by the TPDN. We then measure what proportion of the classifier's predictions for the approximation match its predictions for the original sentence encodings.4Results: For all tasks besides SNLI, we found no marked difference between bag-of-words roles and other role schemes (Table 2a). For SNLI, we did see instances where other role schemes outperformed bag-of-words (Table 2b). Within the SNLI , both tree-based models (SST and SPINN) are best approximated with tree-based roles. InferSent is better approximated with structural roles than with bag-of-words roles, but all structural role schemes perform similarly. Finally, Skip-thought cannot be approximated well with any role scheme we considered. It is unclear why Skip-thought has lower than the other models. Overall, even for SNLI, bag-of-words roles provide a fairly good approximation, with structured roles yielding rather modest improvements. Based on these , we hypothesize that these models' representations can be characterized as a bag-of-words representation plus some incomplete structural information that is not always encoded. This explanation is consistent with the fact that bag-of-words roles yield a strong but imperfect approximation for the sentence embedding models. However, this is simply a conjecture; it is possible that these models do use a robust, systematic structural representation that either involves a role scheme we did not test or that cannot be characterized as a tensor product representation at all. We now complement the TPDN tests with sentence analogies. By comparing pairs of minimally different sentences, analogies might illuminate representational details that are difficult to discern in individual sentences. We construct sentence-based analogies that should hold only under certain role schemes, such as the following analogy (expressed as an equation as in Mikolov et al. 2013): DISPLAYFORM0 A left-to-right role scheme makes equivalent to (f :r denotes the binding of filler f to role r): In, both sides reduce to now:2, so holds for representations using left-to-right roles. However, if instead used right-to-left roles, it would not reduce in any clean way, so would not hold. We construct a dataset of such role-diagnostic analogies, where each analogy should only hold for certain role schemes. For example, works for left-to-right roles or bag-of-words roles, but not the other role schemes. The analogies use a vocabulary based on to ensure plausibility of the constructed sentences. For each analogy, we create 4 equations, one isolating each of the four terms (e.g. I see = I see now -you know now + you know). We then compute the Euclidean distance between the two sides of each equation using each model's encodings. The are in Table 1b. InferSent, Skip-thought, and SPINN all show most consistent with bidirectional roles, while SST shows most consistent with tree-based or bidirectional roles. The bag-of-words column shows poor performance by all models, indicating that in controlled enough settings these models can be shown to have some more structured behavior even though evaluation on examples from applied tasks does not clearly bring out that structure. These analogies thus provide independent evidence for our from the TPDN analysis: these models have a weak notion of structure, but that structure is largely drowned out by the non-structure-sensitive, bag-of-words aspects of their representations. However, the other possible explanations mentioned above−namely, the possibilities that the models use alternate role schemes that we did not test or that they use some structural encoding other than tensor product representation−still remain. The previous section suggested that all sentence models surveyed did not robustly encode structure and could even be approximated fairly well with a bag of words. Motivated by this finding, we now investigate how aspects of training can encourage or discourage compositionality in learned representations. To increase interpretability, we return to the setting (from Section 2) of operating over digit sequences. We investigate two aspects of training: the architecture and the training task. Teasing apart the contribution of the encoder and decoder: In Section 2, we investigated autoencoders whose encoder and decoder had the same topology (unidirectional, bidirectional, or treebased). To test how each of the two components contributes to the learned representation, we now expand the investigation to include networks where the encoder and decoder differ. We crossed all three encoder types with all three decoder types (nine architectures in total). The are in TAB6 in Appendix D. The decoder largely dictates what roles are learned: models with unidirectional decoders prefer mildly bidirectional roles, models with bidirectional decoders fail to be well-approximated by any role scheme, and models with tree-based decoders are best approximated by tree-based roles. However, the encoder still has some effect: in the tree/uni and tree/bi models, the tree-position roles perform better than they do for the other models with the same decoders. Though work on novel architectures often focuses on the encoder, this finding suggests that focusing on the decoder may be more fruitful for getting neural networks to learn specific types of representations. The contribution of the training task: We next explore how the training task affects the representations that are learned. We test four tasks, illustrated in Table 3a: autoencoding (returning the input sequence unchanged), reversal (reversing the input), sorting (returning the input digits in ascending order), and interleaving (alternating digits from the left and right edges of the input). Table 3b gives the substitution accuracy for a TPDN trained to approximate a unidirectional encoder that was trained with a unidirectional decoder on each task. Training task noticeably influences the learned representations. First, though the model has learned mildly bidirectional roles favoring the left-to-right direction for autoencoding, for reversal the right-to-left direction is far preferred over left-to-right. For interleaving, the model is approximated best with strongly bidirectional roles: that is, bidirectional roles work nearly perfectly, while neither unidirectional scheme works well. Finally, for sorting, bag-of-words roles work nearly as well as all other schemes, suggesting that the model Table 3: (a) Tasks used to test for the effect of task on learned roles (Section 4). (b) Accuracy of the TPDN applied to models trained on these tasks with a unidirectional encoder and decoder. All numbers are averages across five random initializations. has learned to discard most structural information since sorting does not depend on structure. These experiments suggest that RNNs only learn compositional representations when the task requires them. This might explain why the sentence embedding models do not seem to robustly encode structure: perhaps the training tasks for these models do not heavily rely on sentence structure (e.g. BID12 achieved high accuracy on SNLI using a model that ignores word order), such that the models learn to ignore structural information, as was the case with models trained on sorting. There are several approaches for interpreting neural network representations. One approach is to infer the information encoded in the representations from the system's behavior on examples targeting specific representational components, such as semantics BID13; BID16 or syntax . Another approach is based on probing tasks, which assess what information can be easily decoded from a vector representation BID20 BID2 Kádár et al. 2017; Ettinger et al. 2018; compare work in cognitive neuroscience, e.g. BID9 ). Our method is wider-reaching than the probing task approach, or the Mikolov et al. FORMULA0 analogy approach: instead of decoding a single feature, we attempt to exhaustively decompose the vector space into a linear combination of filler-role bindings. The TPDN's successful decomposition of sequence representations in our experiments shows that RNNs can sometimes be approximated with no nonlinearities or recurrence. This finding is related to the of , who argued that LSTMs dynamically compute weighted sums of their inputs; TPRs replace the weights of the sum with the role vectors. also showed that recurrence is largely unnecessary for practical applications. BID33 report very good performance for a sequence model without recurrence; importantly, they find it necessary to incorporate sequence position embeddings, which are similar to the left-to-right roles discussed in Section 2. Methods for interpreting neural networks using more interpretable architectures have been proposed before based on rules and automata BID10 BID34.Our decomposition of vector representations into independent fillers and roles is related to work on separating latent variables using singular value decomposition and other factorizations BID29 BID0. For example, in face recognition, eigenfaces BID21 BID30 and TensorFaces (; BID32 use such techniques to disentangle facial features, camera angle, and lighting. Finally, there is a large body of work on incorporating explicit symbolic representations into neural networks (for a recent review, see BID1 ; indeed, tree-shaped RNNs are an example of this approach. While our work is orthogonal to this line of work, we note that TPRs and other filler-role representations can profitably be used as an explicit component of neural models (; BID11 ; BID28 BID18 . What kind of internal representations could allow simple sequence-to-sequence models to perform the remarkable feats they do, including tasks previously thought to require compositional, symbolic representations (e.g., translation)? Our experiments show that, in heavily structure-sensitive tasks, sequence-to-sequence models learn representations that are extremely well approximated by tensorproduct representations (TPRs), distributed embeddings of symbol structures that enable powerful symbolic computation to be performed with neural operations BID23. We demonstrated this by approximating learned representations via TPRs using the proposed tensor-product decomposition network (TPDN). Variations in architecture and task were shown to induce different types and degrees of structure-sensitivity in representations, with the decoder playing a greater role than the encoder in determining the structure of the learned representation. TPDNs applied to mainstream sentence-embedding models reveal that unstructured bag-of-words models provide a respectable approximation; nonetheless, this experiment also provides evidence for a moderate degree of structuresensitivity. The presence of structure-sensitivity is corroborated by targeted analogy tests motivated by the linearity of TPRs. A limitation of the current TPDN architecture is that it requires a hypothesis about the representations to be selected in advance. A fruitful future research direction would be to automatically explore hypotheses about the nature of the TPR encoded by a network. Here we analyze how several aspects of the TPDN architecture contribute to our . For all of the experiements described in this section, we used TPDNs to approximate a sequence-to-sequence network with a unidirectional encoder and unidirectional decoder that was trained to perform the reversal task (Section 4); we chose this network because it was strongly approximated by right-toleft roles, which are relatively simple (but still non-trivial). One area where our model diverges from traditional tensor product representations is in the presence of the final linear layer (step 5 in FIG0). This layer is necessary if one wishes to have freedom to choose the dimensionality of the filler and role embeddings; without it, the dimensionality of the representations that are being approximated must factor exactly into the product of the dimensionality of the filler embeddings and the dimensionality of the role embedding (see FIG0 . It is natural to wonder whether the only contribution of this layer is in adjusting the dimensionality or whether it serves a broader function. TAB3 shows the of approximating the reversal sequence-tosequence network with and without this layer; it indicates that this layer is highly necessary for the successful decomposition of learned representations. (Tables follow all appendix text.) Two of the parameters that must be provided to the TPDN are the dimensionality of the filler embeddings and the dimensionality of the role embeddings. We explore the effects of these parameters in FIG7. For the role embeddings, substitution accuracy increases noticeably with each increase in dimensionality until the dimensionality hits 6, where accuracy plateaus. This behavior is likely due to the fact that the reversal seq2seq network is most likely to employ right-to-left roles, which involves 6 possible roles in this setting. A dimensionality of 6 is therefore the minimum embedding size needed to make the role vectors linearly independent; linear independence is an important property for the fidelity of a tensor product representation BID22. The accuracy also generally increases as filler dimensionality increases, but there is a less clear point where it plateaus for the fillers than for the roles. The body of the paper focused on using the tensor product (f i ⊗ r i, see FIG0) as the operation for binding fillers to roles. There are other conceivable binding operations. Here we test two alternatives, both of which can be viewed as special cases of the tensor product or as related to it: circular convolution, which is used in holographic reduced representations BID15, and elementwise product (f i r i). Both of these are restricted such that roles and fillers must have the same embedding dimension (N f = N r). We first try setting this dimension to 20, which is what was used as both the role and filler dimension in all tensor product experiments with digit sequences. Red indicates accuracy under 1%; dark blue indicates accuracy over 80%. The models whose substitution accuracies are displayed are all TPDNs trained to approximate a sequence-to-sequence model that was trained on the task of reversal. We found that while these dimensions were effective for the tensor product binding operation, they were not effective for elementwise product and circular convolution TAB4 ). When the dimension was increased to 60, however, the elementwise product performed roughly as well as as the tensor product; circular convolution now learned one of the two viable role schemes (right-to-left roles) but failed to learn the equally viable bidirectional role scheme. Thus, our preliminary experiments suggest that these other two binding operations do show promise, but seem to require larger embedding dimensions than tensor products do. At the same time, they still have fewer parameters overall compared to the tensor product because their final linear layers (of dimensionality N) are much smaller than those used with a tensor product (of dimensionality N 2). When inputting digit sequences to our tree-based model, the model requires a predefined tree structure for the digit sequence. We use the following algorithm to generate this tree structure: at each timestep, combine the smallest element of the sequence (other than the last element) with its neighbor immediately to the right, and replace the pair with that neighbor. If there is a tie for the smallest digit, choose the leftmost tied digit. For example, the following shows step-by-step how the tree for the sequence 523719 would be generated:• 5 2 3 7 1 9• 5 2 3 7 DISPLAYFORM0 Section 4 summarized the of our experiments which factorially varied the training task, the encoder and the decoder. Here we report the full of these experiments in two tables: TAB5 shows the accuracies achieved by the sequence-to-sequence models at the various training tasks, and TAB6 shows the substitution accuracies of TPDNs applied to the trained sequence-to-sequence models for all architectures and tasks. As much as possible, we standardized parameters across all sequence-to-sequence models that we trained on the digit-sequence tasks. For all decoders, when computing a new hidden state, the only input to the recurrent unit is the previous hidden state (or parent hidden state, for a tree-based decoder), without using any previous outputs as inputs to the hidden state update. This property is necessary for using a bidirectional decoder, since it would not be possible to generate the output both before and after each bidirectional decoder hidden state. We also inform the decoder of when to stop decoding; that is, for sequential models, the decoder stops once its output is the length of the sequence, while for tree-based models we tell the model which positions in the tree are leaves. Stopping could alternately be determined by some action of the decoder (e.g., generating an end-of-sequence symbol); for simplicity we chose the strategy outlined above instead. For all architectures, we used a digit embedding dimensionality of 10 (chosen arbitrarily) and a hidden layer size of 60 (this hidden layer size was chosen because 60 has many integer factors, making it amenable to the dimensionality analyses in Appendix B.2). For the bidirectional architectures, the forward and backward recurrent layers each had a hidden layer size of 30, so that their concatenated hidden layer size was 60. For bidirectional decoders, a linear layer condensed the 60-dimensional encoding into 30 dimensions before it was passed to the forward and backward decoders. The networks were trained using the Adam optimizer with the standard initial learning rate of 0.001. We used negative log likelihood, computed over the softmax probability distributions for each output sequence element, as the loss function. Training proceeded with a batch size of 32, with loss on the held out development set computed after every 1,000 training examples. Training was halted when the loss on the heldout development set had not improved for any of the development loss checkpoints for a full epoch of training (i.e. 40,000 training examples). Once training completed, the parameters from the best-performing checkpoint were reloaded and used for evaluation of the network. When applying TPDNs to the digit-based sequence-to-sequence models, we always used 20 as both the filler embedding dimension and the role embedding dimension. This decision was based on the experiments in Appendix B.2; we selected filler and role embedding dimensions that were safely above the cutoff needed to lead to successful decomposition. The TPDNs were trained with the same training regimen as the sequence-to-sequence models, except that, instead of using negative log likelihood as the loss function, for the TPDNs we used mean squared error between the predicted vector representation and the actual vector representation from the original sequence-to-sequence network. The TPDNs were given the sequences of fillers (i.e. the digits), the roles hypothesized to go with those fillers, the sequence embeddings produced by the RNN, and the dimensionalities of the filler embeddings, role embeddings, and final linear transformation. The parameters that were updated by training were the specific values for the filler embeddings, the role embeddings, and the final linear transformation. For all four sentence encoding models, we used publicly available and freely downloadable pretrained versions found at the following links:• we use the SPINN-PI-NT version, which is equivalent to a tree-LSTM BID27 with 300-dimensional hidden states. For training a TPDN to approximate the sentence encoding models, the filler embedding dimensions were dictated by the size of the pretrained word embeddings; these dimensions were 300 for InferSent and SPINN, 620 for Skip-thought, and 25 for SST. The linear transformation applied to the word embeddings did not change their size. For role embedding dimensionality we tested all role dimensions in {1, 5, 10, 20, 40, 60}. The best-performing dimension was chosen based on preliminary experiments and used for all subsequent experiments; we thereby chose role dimensionalities of 10 for InferSent and Skip-thought, 20 for SST, and 5 for SPINN. In general, role embedding dimensionalities of 5, 10, and 20 all performed noticeably better than 1, 40, and 60, but there was not much difference between 5, 10, and 20.The training regimen for the TPDNs on sentence models was the same as for the TPDNs trained on digit sequences. The TPDNs were given the sequences of fillers (i.e. the words), the roles hypothesized to go with those fillers, the sequence embeddings produced by the RNN, the initial pretrained word embeddings, the dimensionalities of the linearly-transformed filler embeddings, the role embeddings, and the final linear transformation. The parameters that were updated by training were the specific values for the role embeddings, the linear transformation that was applied to the pretrained word embeddings, and the final linear transformation. The sentences whose encodings we trained the TPDNs to approximate were the premise sentences from the SNLI corpus BID3. We also tried instead using the sentences in the WikiText-2 corpus but found better performance with the SNLI sentences. This is plausibly because the shorter, simpler sentences in the SNLI corpus made it easier for the model to learn the role embeddings without distraction from the fillers. For each TPDN trained to approximate a sentence encoder, we evaluate it on four downstream tasks: (i) Stanford Sentiment Treebank (SST), which is labeling the sentiment of movie reviews BID25; this task is further subdivided into SST2 (labeling the reviews as positive or negative) and SST5 (labeling the reviews on a 5-point scale, where 1 means very negative and 5 means very positive). The metric we report for both tasks is accuracy.(ii) Microsoft Research Paraphrase Corpus (MRPC), which is labeling whether two sentences are paraphrases of each other . For this task, we report both accuracy and F1. (iii) Semantic Textual Similarity Benchmark (STS-B), which is giving a pair of sentences a score on a scale from 0 to 5 indicating how similar the two sentences are BID5. For this task, we report Pearson and Spearman correlation coefficients. (iv) Stanford Natural Language Inference (SNLI), which involves labeling a pair of sentences to indicate whether the first entails the second, contradicts the second, or neither BID3. For this task, we report accuracy as the evaluation metric. The first we report for the TPDN approximations of sentence encoders is similar to the substitution accuracy used for digit encoders. Here, we use SentEval Table 9: The proportion of times that a classifier trained on a sentence encoding model gave the same downstream-task predictions based on the original sentence encoding model and based on a TPDN approximating that model, where the TPDN uses the role schemes indicated by the column header. For all tasks but STS-B, these numbers show the proportion of predictions that matched; chance performance is 0.5 for SST2 and MRPC, 0.2 for SST5, and 0.33 for SNLI. For STS-B, the metric shown is the Pearson correlation between the TPDN's similarity ratings and the original model's similarity ratings; chance performance here is 0.0. Table 10: Downstream task performance for classifiers trained and tested on the TPDNs that were trained to approximate each of the four applied models. The rightmost column indicates the performance of the original model (without the TPDN approximation).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

RNNs implicitly implement tensor-product representations, a principled and interpretable method for representing symbolic structures in continuous space.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: We address the problem of teaching an RNN to approximate list-processing algorithms given a small number of input-output training examples. Our approach is to generalize the idea of parametricity from programming language theory to formulate a semantic property that distinguishes common algorithms from arbitrary non-algorithmic functions. This characterization leads naturally to a learned data augmentation scheme that encourages RNNs to learn algorithmic behavior and enables small-sample learning in a variety of list-processing tasks. Since the earliest days of neural network research, some of the most important questions about neural models have focused on their ability to capture the crispness, systematicity and compositionality that characterize symbolic computation and human cognition BID2 BID11, and to do so with a human-like number of examples BID10. While recent studies have demonstrated promising in training recurrent neural networks (RNNs) to approximate symbolic algorithms in domains like list manipulation BID4 BID7, binary arithmetic BID8, graph traversal BID3, and planar geometry BID12, the question of sample efficiency remains very much open. Difficult algorithmic problems may require tens or hundreds of thousands of labelled training examples, and even simple tasks on small inputs seem to require more data than should be necessary BID9.Our goal in this paper is to teach RNNs to approximate list-processing algorithms f:: DISPLAYFORM0. Inspired by the idea of parametricity BID13 ) from type theory and functional programming, we hypothesize that a feature that distinguishes many algorithms from arbitrary functions is that they commute with some family of element-wise changes to their inputs. We describe a method for learning this family from the training set D, and show how this learned information can be used to create an augmented training set for an RNN. Our experiments show that this augmentation scheme makes it possible to approximate algorithms from small training sets, in some cases requiring only a single example per input list length. RNN inductive biases. Our data augmentation approach is motivated by the failure patterns of unaugmented training. The confusion matrix in FIG0 shows the performance of an RNN (an LSTM BID5 with 128 hidden units) trained with ten examples to copy lists of two elements. The failure mode is clear: the model acts as an interpolating lookup table: the model tends to map the regions of input space around each training input x i to the training output f (x i). This is an entirely appropriate function model for classification, but a lookup table is clearly a poor example to follow for algorithm learning. Our approach for the remainder of this paper will be to formulate a semantic property that distinguishes algorithms from lookup tables, and then use data augmentation to nudge an RNN in an algorithmic direction. The confusion matrix for an RNN trained with ten examples to implement the copy algorithm. The inputs (rows) and outputs (columns) are each pairs of digits 0-9. A red pixel in cell (i, j) indicates that the model predicted output sequence j for input sequence i; perfect performance for this copy task would fill the diagonal of the heatmap with red pixels. The gray stripes indicate the ten training examples. Parametricity. Any computable function is technically an algorithm, but we have the intuition that some functions are more "algorithm-y" than others: an algorithm is a function that "does the same thing" to any input fed to it, while a nonalgorithmic function like a lookup table has different, idiosyncratically defined behavior for each possible input. Put another way, a change to the input of an algorithm should translate systematically to a change to its output. In our copy example, f =. If we modify the input by replacing the'9' token with a'3', then making the same substitution on the output side produces the correct equation We can make this intuition quantitative by drawing on a family of in type theory and programming language theory based on the idea of type parametricity, and often called "theorems for free" BID13. The main is the following: Any parametrically polymorphic (for brevity, we will simply say "polymorphic" henceforth) function f:: DISPLAYFORM0 where a is a type parameter 1, commutes with element-wise application of any function g:: a → a: DISPLAYFORM1 An illustrative example: doubling each of the elements of an integer list and then reversing the gives the same output as reversing and then doubling. Parametricity captures some intuitions about what makes a function algorithmic; it accounts for the copy example above, for instance, and in general, if a function commutes with element-wise transformations of its inputs, it cannot depend in a lookup-y way on the details of its input's elements. Since our interest is not limited to polymorphic functions, some generalization of equation 1 is required. The function drop evens, which removes the even elements from a list of integers, for instance, is a clearly legitimate algorithm that fails to obey equation 1; g: x → 2x is a counterexample. But while drop evens does not commute with all element-wise transformations, it does commute with a subset of them, namely those that preserve parity. So this motivates a hypothesis: an algorithm f:: [Int] → [Int] should have some systematicity in its definition that is reflected in its commuting with some set of element-wise transformations DISPLAYFORM2 We can draw a high-level analogy here with techniques from math and physics that characterize an object in terms of its symmetries. If we can learn G, we can use it to augment our training set to help learn f: given an input-output training example (x, f (x)), then for any g ∈ G, (g(x), g(f (x)) will also be a valid training example, and can be use for data augmentation. This section describes a technique for learning G from f's training data. We parameterize each g as a collection of swaps DISPLAYFORM0, each of which replaces each instance of an s token with the corresponding t token. For instance, if g = {3 → 4, 7 → 1}, then g =. Our approach to learning g will be to train a classifier C that predicts whether or not a given collection of swaps should commute with f. Given two training input-output pairs (x i, f (x i)) and (x j, f (x j)), chosen so that x i and x j have the same length, we first determine whether or not x i and x j are related by a set of swaps. Supposing there is a swap set g with g(x i) = x j, we then see whether g also relates the output lists g(f (x i))? = f (x j): if it does, then we have a positive training example of a swap set that commutes with f, and if it does not, then we have a negative example. Repeating this process for each pair of length-matched lists in our training set, we obtain a collection of training examples of positive (commutes with f) and negative (does not commute with f) swap sets. A promising feature of this setup is that while the original learning problem (learning f) has n training examples, the problem of learning G has O(n 2) examples. To support small-sample learning, we make the simplifying factorization assumption that each swap contributes independently to the commutativity of the whole swap set. Our classifier C acts on a swap set g = {g 1, . . ., g m} = {s 1 → t 1, . . ., s m → t m} by C(g) = smoothed min m i=1 c(g i), where c classifies individual swaps, and smoothed min(v) = −γlog i e −vi/γ BID1. For the experiments in this paper, c consisted of a bilinear layer that combined ten-dimensional embeddings of s i and t i, followed by a RELU nonlinearity and a linear layer with a scalar output. Per-sequence classification. We have assumed that a given swap can be classified independently of the input list to which it is to be applied. This assumption is violated by functions like sort: g = {3 → 6} commutes with sort for the input list, but not for the list, for example. To deal with this, we distinguish per-task classification, as described above, from persequence classification, which depends on the input list elements in addition to the swap set. Extending the factorization assumption from the per-task model, for a swap set g = {g 1, . . . g m} and input sequence x = [x 1, . . . x k], the per-sequence classifier predicts C(g, x) = smoothed min({c(g i, x j) | i = 1,... m, j = 1,... k}. Here, c extends the per-task classifier by adding a bilinear combination of the sequence element with the combination of the swap elements. Augmentation. To use a trained classifier C to generate an augmented training set, we randomly generate a large collection of candidate swap sets g, recording for each the classifier score C(g). Each combination of a swap set g and training pair (x, f (x)) gives a candidate augmentation example (g(x), g(f (x))). Any given candidate augmentation example could be generated in multiple ways by starting from different input-output pairs, so we assign to each candidate example the sum of the scores of the swap sets g that could be used to generate it. We take our augmentation training set to be the 1000 top-scoring candidates for each list length. For both the per-sequence and per-task models, if our classifier's training set is entirely missing either positive or negative examples, the model reverts to random augmentation. We compare four models: the per-task and per-sequence augmentation models described above, a random augmentation model that samples swap sets uniformly at random, and a no-augmentation baseline. We evaluate each on the list-processing functions shown in figure 2. We divide these functions into three types: polymorphic, token-dependent (depends on the identities of individual tokens), and order-dependent (depends on the order relations between tokens). These function categories are used in post-hoc analysis only; they are not explicitly made available to our models during training or testing. Expecting problem difficulty to scale to some extent with output sequence length, we also note for each function the length of its output when evaluated on an input list of length n. Functions that implement a filtering operation will have variable length outputs, but for simplicity, we group them with functions like copy in the "≈ n" output-length category. For each target function, we trained our models on even-length lists of lengths 2, 4, 6 and 8, and tested on all lengths 2-8. All list items were integers in the range. To evaluate our models' sample efficiency, we considered learning problems with 1, 5, 10, 20, 30, 40, and 50 input-output examples per input list length. For all problems, our model was an LSTM with two hidden layers of 128 units. Results. On the polymorphic target functions, all augmentation models, including the random one, substantially outperformed the non-augmented baseline. This performance is to be expected, as equation 1 guarantees that these functions commute with all element-wise input transformations. For functions with output length 1 and n, the augmented models are able to achieve close to perfect accuracy with only a single example per input list length. Moving to the non-polymorphic token-dependent functions, random augmentation not only ceases to suffice for good performance, but in fact often delivers lower accuracy than the non-augmented baseline, while both learned augmentation models continue to perform well on most target functions. For the order-dependent functions, the analysis in section 2 suggests that per-sequence augmentation should outperform per-task. In practice, the two learned models achieve roughly equal accuracy, perhaps reflecting the fact that the more expressive per-sequence model requires more data to train correctly. Moreover, while the learned augmentation models outperform the non-augmented baseline on the triangle task and on the very few-shot versions of the sorting tasks, their advantage over the non-augmented baseline is much less marked than on the other function types. Still, the learned models largely avoid the destructive effects of applying random augmentation. Future directions. Our augmentation schemes are model-agnostic; it will be interesting in the future to pair them with models like pointer networks or memory-augmented RNNs. It will also be interesting to extend the techniques of this paper to domains beyond list processing, for instance to the geometric algorithms studied in BID12.

Represent the TLDR for finding the corresponding scientific article

Learned data augmentation instills algorithm-favoring inductive biases that let RNNs learn list-processing algorithms from fewer examples.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.