Question: Represent the article for finding a single-sentence TLDR

Question: Learning communication via deep reinforcement learning has recently been shown to be an effective way to solve cooperative multi-agent tasks. However, learning which communicated information is beneficial for each agent's decision-making remains a challenging task. In order to address this problem, we introduce a fully differentiable framework for communication and reasoning, enabling agents to solve cooperative tasks in partially-observable environments. The framework is designed to facilitate explicit reasoning between agents, through a novel memory-based attention network that can learn selectively from its past memories. The model communicates through a series of reasoning steps that decompose each agent's intentions into learned representations that are used first to compute the relevance of communicated information, and second to extract information from memories given newly received information. By selectively interacting with new information, the model effectively learns a communication protocol directly, in an end-to-end manner. We empirically demonstrate the strength of our model in cooperative multi-agent tasks, where inter-agent communication and reasoning over prior information substantially improves performance compared to baselines. Communication is one of the fundamental building blocks for cooperation in multi-agent systems. The ability to effectively represent and communicate information valuable to a task is especially important in multi-agent reinforcement learning (MARL). Apart from learning what to communicate, it is critical that agents learn to reason based on the information communicated to them by their teammates. Such a capability enables agents to develop sophisticated coordination strategies that would be invaluable in application scenarios such as search-and-rescue for multi-robot systems , swarming and flocking with adversaries , multiplayer games (e.g., StarCraft, , DoTA, , and autonomous vehicle planning, Building agents that can solve complex cooperative tasks requires us to answer the question: how do agents learn to communicate in support of intelligent cooperation? Indeed, humans inspire this question as they exhibit highly complex collaboration strategies, via communication and reasoning, allowing them to recognize important task information through a structured reasoning process, (; ;). Significant progress in multiagent deep reinforcement learning (MADRL) has been made in learning effective communication (protocols), through the following methods: (i) broadcasting a vector representation of each agent's private observations to all agents , (ii) selective and targeted communication through the use of soft-attention networks, , that compute the importance of each agent and its information, , and (iii) communication through a shared memory channel , which allows agents to collectively learn and contribute information at every time instant. The architecture of implements communication by enabling agents to communicate intention as a learned representation of private observations, which are then integrated in the hidden state of a recurrent neural network as a form of agent memory. One downside to this approach is that as the communication is constrained in the neighborhood of each agent, communicated information does not enrich the actions of all agents, even if certain agent communications may be critical for a task. For example, if an agent from afar has covered a landmark, this information would be beneficial to another agent that has a trajectory planned towards the same landmark. In contrast, Memory Driven Multi-Agent Deep Deterministic Policy Gradient (MD-MADDPG), , implements a shared memory state between all agents that is updated sequentially after each agent selects an action. However, the importance of each agent's update to the memory in MD-MADDPG is solely decided by its interactions with the memory channel. In addition, the sequential nature of updating the memory channel restricts the architecture's performance to 2-agent systems. Targeted Multi-Agent Communication (TarMAC), , uses soft-attention for the communication mechanism to infer the importance of each agent's information, however without the use of memory in the communication step. The paradigm of using relations in agent-based reinforcement learning was proposed by through multi-headed dot-product attention (MHDPA) . The core idea of relational reinforcement learning (RRL) combines inductive logic programming (; Džeroski et al., 2001) and reinforcement learning to perform reasoning steps iterated over entities in the environment. Attention is a widely adopted framework in Natural Language Processing (NLP) and Visual Question Answering (VQA) tasks (b; a;) for computing these relations and interactions between entities. The mechanism generates an attention distribution over the entities, or more simply a weighted value vector based on importance for the task at hand. This method has been adopted successfully in state-of-the-art for Visual Question Answering (VQA) tasks (b), (a), and more recently , demonstrating the robustness and generalization capacity of reasoning methods in neural networks. In the context of multi-agent cooperation, we draw inspiration from work in soft-attention to implement a method for computing relations between agents, coupled with a memory based attention network from Compositional Attention Networks (MAC) , yielding a framework for a memory-based communication that performs attentive reasoning over new information and past memories. Concretely, we develop a communication architecture in MADRL by leveraging the approach of RRL and the capacity to learn from past experiences. Our architecture is guided by the belief that a structured and iterative reasoning between non-local entities should enable agents to capture higherorder relations that are necessary for complex problem-solving. To seek a balance between computational efficiency and adaptivity to variable team sizes, we exploit the soft-attention as the base operation for selectively attending to an entity or information. To capture the information and histories of other entities, and to better equip agents to make a deliberate decision, we separate out the attention and reasoning steps. The attention unit informs the agent of which entities are most important for the current time-step, while the reasoning steps use previous memories and the information guided by the attention step to extract the shared information that is most relevant. This explicit separation in communication enables agents to not only place importance on new information from other agents, but to selectively choose information from its past memories given new information. This communication framework is learned in an end-to-end fashion, without resorting to any supervision, as a of task-specific rewards. Our empirical study demonstrates the effectiveness of our novel architecture to solve cooperative multi-agent tasks, with varying team sizes and environments. By leveraging the paradigm of centralized learning and decentralized execution, alongside communication, we demonstrate the efficacy of the learned cooperative strategies. We consider a team of N agents and model it as a cooperative multi-agent extension of a partially observable Markov decision process (POMDP) . We characterize this POMDP by the set of state values, S, describing all the possible configurations of the agents in the environment, control actions {A 1, A 2, ..., A N}, where each agent i performs an action A i, and set of observations {O 1, O 2, ..., O N}, where each agent i's local observation, O i is not shared globally. Actions are selected through a stochastic policy π π π θi: with policy parameters θ i, and a new state is generated by the environment according to the transition function T: S × A 1 ×... × A N → S. At every step, the environment generates a reward, r i: S × A i → R, for each agent i and a new local observation o i: S → O i. The goal is to learn a policy such that each agent maximizes the total expected return where T is the time horizon and γ is the discount factor. We choose the deterministic policy gradient algorithms for all our training. In this framework, the parameters θ of the policy, µ µ µ θ, are updated such that the objective J(θ) = E s∼p π π π,a∼µ θ [R(s, a)] and the policy gradient, (see Appendix A), is given by: Deep Deterministic Policy Gradient (DDPG) is an adaptation of DPG where the policy µ µ µ and critic Q µ µ µ are approximated as neural networks. DDPG is an off-policy method, where experience replay buffers, D, are used to sample system trajectories which are collected throughout the training process. These samples are then used to calculate gradients for the policy and critic networks to stabilize training. In addition, DDPG makes use of a target network, similar to Deep Q-Networks (DQN) , such that the parameters of the primary network are updated every few steps, reducing the variance in learning. Recent work proposes a multi-agent extension to the DDPG algorithm, socalled MADDPG, adapted through the use of centralized learning and decentralized execution. Each agent's policy is instantiated similar to DDPG, as µ µ µ θi (a i |o i) conditioned on its local observation o i. The major underlying difference is that the critic is centralized such that it estimates the joint action- We operate under this scheme of centralized learning and decentralized execution of MADDPG, , as the critics are not needed during the execution phase. We introduce a communication architecture that is an adaptation of the attention mechanism of the Transformer network, , and the structured reasoning process used in the MAC Cell, . The framework holds memories from previous time-steps separately for each agent, to be used for reasoning on new information received by communicating teammates. Through a structured reasoning, the model interacts with memories and communications received from other agents to produce a memory for the agent that contains the most valuable information for the task at hand. An agent's memory is then used to predict the action of the agent, such that policy is given by µ µ µ θ: O i × M i → A i, where M i is the memory of agent i. To summarize, before any agent takes an action, the agent performs four operations via the following architectural features: Thought Unit, where each agent encodes its local observations into appropriate representations for communication and action selection, Question Unit, which is used to generate the importance of all information communicated to the agent, Memory Unit, which controls the final message to be used for predicting actions by combining new information from other agents with its own memory, through the attention vector generated in the Question unit, and Action Unit, that predicts the action. In Figure 1 we illustrate our proposed Structured Attentive Reasoning Network (SARNet). The thought unit at each time-step t transforms an agent's private observations into three separate vector representations: query, q Figure 1: SARNet consists of a thought unit, question unit, memory unit and action unit. The thought unit operates over the current observation, to represent the information for the communication and action step. The question unit attends to the information from communicating agents, representing the importance of each neighboring agent's information. The memory unit, guided by the Question Unit, performs a reasoning operation to extract relevant information from communicated information, which is then integrated into memory. The Action Unit, processes the information from the memory and the observation encoding to predict an action for the current time-step. The query is used by each agent i to inform the Question Unit which aspects of the communicated information are relevant to the current step. The key and value are broadcast to all communicating agents. The key vector is used in the Question Unit to infer the relevance of the broadcasting agent to the current reasoning step, and the value vector is subsequently used to update the information into the memory of agent i. The ing information broadcasted by each agent i to all the cooperating agents is then: This component is designed to capture the importance of each agent in the environment, including the reasoning agent i, similar to the self-attention mechanism in . In the attention mechanism used in , the attention computes a weight for each entity through the use of the sof tmax. However, we generate the attention mechanism over all individual representations in the vector for each entity, using Eq. 5. This allows the agent to compute the importance of each individual communicated information from other agents for a particular timestep. This is performed through a soft attention-based weighted average using the query generated by agent i, and the set of keys, K, that contain the keys, {k} from all agents. The recipient agent, i, upon receiving the set of keys, K, from all agents, computes the interaction with every agent through a Hadamard product,, of its own query vector, q i and all the keys, k j, in the set K. qh, is then applied to every interaction, qh t ij, that defines the query targeted for each communicating agent j, including self, to produce a scalar defining the weight of the particular agent. A sof tmax operation is then used over the new scalars for each agent to generate the weights specific to each agent. The use of the linear transformation in Eq. 5 allows the model to specify an importance not only for each individual agent, but more specifically it learns to assign an importance to each element in the information vector, as compared to the approach used in standard soft-attention based networks, such as Transformer , which only perform a dot-product computation between the query and keys. The memory unit is responsible for decomposing the set of new values, V, that contain, {v into relevant information for the current time-step. Specifically, it computes the interaction of this new knowledge with the memory aggregated from the preceding time-step. The new retrieved information, from the memory and the values, is then measured in terms of relevance based on the importance of each agent generated in the Question unit. As a first step, an agent computes a direct interaction between the new values from other agents, v j ∈ V, and its own memory, m i . This step performs a relative reasoning between newly received information and the memory from the previous step. This allows the model to potentially highlight information from new communications, given information from prior memory. The new interaction per agent j is evaluated relative to the memory, mi t ij, and current knowledge, V, is then used to compute a new representation for the final attention stage, through a feed-forward network, W . This enables the model to reason independently on the interaction between new information and previous memory, and new information alone. Finally, we aggregate the important information, mr ij, based on the weighting calculated in the Question unit, in. This step generates a weighted average of the new information, mr ij, gathered from the reasoning process, based on the attention values computed in. A linear transformation, W, is applied to the of the reasoning operation to prepare the information for input to the action cell. The action unit, as the name implies, predicts the final action of the agent, i, based on the new memory, Eq. 10, computed from the Memory unit and an encoding, e t i, Eq. 2, of its local observation, o i, from the Thought unit. where ϕ a θ a i is a multi-layer perceptron (MLP) parameterised by θ a i. We incorporate the centralized learning-decentralized execution framework from to implement an actor-critic model to learn the policy parameters for each agent. We use parameter sharing across agents to ease the training process. The policy network µ θi produces actions a i, which is then evaluated by the critic Q µ θ i, which aims to minimize the loss function., are the observations when actions {a 1, a 2, ..., a N} are performed, the experience replay buffer, D, contains the tuples (x, x, m 1, m 2, ..., m N, a 1, a 2, ...a N, r 1, r 2, ...r N), and the target Q-value is defined as y. To keep the training stable, delayed updates to the target network Q µ θ i is implemented, such that current parameters of Q µ θ i, are only copied periodically. The goal of the loss function, L(θ i) is to minimize the expectation of the difference between the current and the target action-state function. The gradient of the ing policy, with communication, to maximize the expectation of the rewards, J(θ i) = E[R i], can be written as: We evaluate our communication architecture on OpenAI's multi-agent particle environment, , a two-dimensional environment consisting of agents and landmarks with cooperative tasks. Each agent receives a private observation that includes only partial observations of the environment depending on the task. The agents act independently and collect a shared global reward for cooperative tasks. We consider different experimental scenarios where a team of agents cooperate to complete tasks against static goals, or compete against non-communicating agents. We compare our communication architecture, SARNet, to communication mechanisms of CommNet, , TarMAC and the non-communicating policy of MADDPG, . In this environment, N agents need to cooperate to reach L landmarks. Each agent observes the relative positions of the neighboring agents and landmarks. The agents are penalized if they collide with each other, and positively rewarded based on the proximity to the nearest landmark. At each time-step, the agent receives a reward of −d, where d is the distance to the closest landmark, and penalized a reward of −1 if a collision occurs with another agent. In this cooperative task, all agents strive to maximize a shared global reward. Performance is evaluated per episode by average reward, number of collisions, and occupied landmarks. Our model outperforms all the baselines achieving a higher reward through lower metrics of average distance to the landmark and collisions for N = L = 3 and N = L = 6 agents as shown in Table 1. We hypothesize that in an environment with more agents, the effect of retaining a memory of previous communications from other agents allows the policy to make a more informed decision. This leads to a significant reduction in collisions, and lower distance to the landmark. Our architecture outperforms TarMAC, which uses a similar implementation of soft-attention, albeit without a memory, and computing the communication for the next time-step, unlike SARNet, where the communication mechanism in time t, shapes the action, a t i for the current time-step. We also show the attention metrics for agent 0 at a single time-step during the execution of the tasks with N = 6 agents in Fig. 3a. Table 1: Partially observable cooperative navigation. For N = L = 3, the agents can observe the nearest 1 agent and 2 landmarks, and for N = L = 6, the agents can observe, 2 agents and 3 landmarks. Number of collisions between agents, and average distance at the end of the episode are measured. This task involves a slower moving team of N cooperating agents chasing M faster moving agents in an environment with L static landmarks. Each agent receives its own local observation, where it can observe the nearest prey, predator, and landmarks. Predators are rewarded by +10 every time they collide with a prey, and subsequently the prey is penalized −10. Since the environment is unbounded, the prey are also penalized for moving out of the environment. Predators are trained using the SARNet, TarMAC, CommNet, and MADDPG and prey are trained using DDPG. Due to the nature of dynamic intelligent agents competing with the communicating agents, the complexity of the task increases substantially. As shown in Fig. 3b, we observe that agent 0 learns to place a higher importance on agent 1's information over itself. This dynamic nature of the agent, in selecting which information is beneficial, coupled with extracting relevant information from the memory, enables our architecture to substantially outperform the baseline methods, Table 2a. Table 2: In 2a, N = L = 3, M = 1, the agents can observe the nearest 1 predator, 1 prey and 2 landmarks, and for N = L = 6, M = 2, the agents can observe, 3 predators, 1 prey and 3 landmarks. Number of prey captures by the predators per episode is measured. For Table 2b, we measure the avg. success rate of the communicating agents N, to reach the target landmark, and the same for the adversary M. Larger values for N are desired, and lower for M. (a) Partially observable predatory-prey Table 2b, both TarMAC and CommNet agents choose to stay far away from the target landmark, such that the adversarial agent, follow them. In contrast, SARNet and MADDPG, learn to spread out over all the landmarks, with a higher adversarial score, but achieving an overall higher mean reward for the complete task. (a) Attention for agent 0, at a single time-step for cooperative navigation environment for N = L = 6. (b) Attention for agent 0, at a single time-step in a predator-prey environment, for N = 6, and M = 2. Figure 3: Visualizing attention predicted by agent i over all agents during the initial stages of the task. We observe that agents learn to devalue their own information if it is more advantageous to place importance on information from other agents when reading and writing to the memory unit. We perform additional benchmarks showcasing the importance of memory in our communication architecture. By introducing noise in the memory at each time-step, we evaluate the performance of SARNet on partially-observable cooperative navigation and predator-prey environments. We find a general worsening of the for both tasks as demonstrated by the metrics in Table 3. However, we note that in spite of the corrupted memory, the agent's policy is robust to adversarial noise, as it learns to infer the important information from the thorough reasoning process in the communication. Table 3: Performance metrics when a Gaussian Noise of mean 0 and variance 1 is introduced in the memory (MC-SARNet) during execution of the tasks for predator-prey and cooperative navigation. We have introduced a novel framework, SARNet, for communication in multi-agent deep RL which performs a structured attentive reasoning between agents to improve coordination skills. Through a decomposition of the representations of communication into reasoning steps, our agents exceed baseline methods in overall performance. Our experiments demonstrate key benefits of gathering insights from its own memories, and the internal representations of the information available to agent. The communication architecture is learned end-to-end, and is capable of computing taskrelevant importance of each piece of computed information from cooperating agents. While this multi-agent communication mechanism shows promising , we believe that we can further adapt this method to scale to a larger number of agents, through a gating mechanism to initiate communication, and decentralized learning. Policy gradient (PG) methods are the popular choice for a variety of reinforcement learning (RL) tasks in the context described above. In the PG framework, the parameters θ of the policy are directly adjusted to maximize the objective J(θ) = E s∼p π π π,a∼π π π θ [R], by taking steps in the direction of ∇ θ J(θ), where p π π π, is the state distribution, s is the sampled state and a is the action sampled from the stochastic policy. Through learning a value function for the state-action pair, Q π π π (s, a), which estimates how good an optimal action a is for an agent in state s, the policy gradient is then written as, : Several variations of PG have been developed, primarily focused on techniques for estimating Q π π π. For example, the REINFORCE algorithm uses a rather simplistic method of sample return calculated as a cumulative expected reward for an episode with a discount factor γ, R t = T i=t γ i−t r i. When temporal-difference learning is used, the learned function Q π π π (s, a) is described as the critic, which leads to several different actor-critic algorithms , , where the actor could be a stochastic π π π θ or deterministic policy µ µ µ θ for predicting actions. Hyperparameters We use batch synchronous method for off-policy gradient methods, , for all the experiments. Each environment instance has a separate replay buffer of size 10 6. All policies are trained using the Adam optimizer with a learning rate of 5 × 10 −4, a discount factor, γ = 0.96. and τ = 0.001, for the soft update of the target network. The query, key and values, share a common first layer of size 128, and subsequently are linearly projected to 32-d. Batch normalization and dropouts in the communication channel are implemented. The observation, and action units have two hidden layers of size 128 units with ReLU as the activation function. The critic is implemented as a 2-layer MLP, with 1024 and 512 units. We use a batch-size of 128, and updates are initiated after accumulating experiences for 1280 episodes. Exploration noise is implemented through Orhnstein-Uhlenbeck process with θ = 0.15 and σ = 0.3. All experimental are averaged over 3 separate runs, with different random seeds. Baseline Hyperparameters Policies for TarMAC, CommNet and MADDPG are instantiated as an MLP, similar to SARNet. All layers in the policy network are of size 128 units, while the critic is a 2-layer network with 1024, 512 units. Both TarMAC and CommNet are implemented with 1-stage communication. For TarMAC, the query's, and key's have 16 units, and values are 32 units as described in . All other training parameters are kept similar to the SARNet implementation.

Novel architecture of memory based attention mechanism for multi-agent communication.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.


Question: Represent the article for finding a single-sentence TLDR

Question: In this paper, we introduce Symplectic ODE-Net (SymODEN), a deep learning framework which can infer the dynamics of a physical system from observed state trajectories. To achieve better generalization with fewer training samples, SymODEN incorporates appropriate inductive bias by designing the associated computation graph in a physics-informed manner. In particular, we enforce Hamiltonian dynamics with control to learn the underlying dynamics in a transparent way which can then be leveraged to draw insight about relevant physical aspects of the system, such as mass and potential energy. In addition, we propose a parametrization which can enforce this Hamiltonian formalism even when the generalized coordinate data is embedded in a high-dimensional space or we can only access velocity data instead of generalized momentum. This framework, by offering interpretable, physically-consistent models for physical systems, opens up new possibilities for synthesizing model-based control strategies. In recent years, deep neural networks have become very accurate and widely used in many application domains, such as image recognition , language comprehension , and sequential decision making . To learn underlying patterns from data and enable generalization beyond the training set, the learning approach incorporates appropriate inductive bias by promoting representations which are simple in some sense. It typically manifests itself via a set of assumptions, which in turn can guide a learning algorithm to pick one hypothesis over another. The success in predicting an outcome for previously unseen data then depends on how well the inductive bias captures the ground reality. Inductive bias can be introduced as the prior in a Bayesian model, or via the choice of computation graphs in a neural network. In a variety of settings, especially in physical systems, wherein laws of physics are primarily responsible for shaping the outcome, generalization in neural networks can be improved by leveraging underlying physics for designing the computation graphs. Here, by leveraging a generalization of the Hamiltonian dynamics, we develop a learning framework which exploits the underlying physics in the associated computation graph. Our show that incorporation of such physics-based inductive bias offers insight about relevant physical properties of the system, such as inertia, potential energy, total conserved energy. These insights, in turn, enable a more accurate prediction of future behavior and improvement in out-of-sample behavior. Furthermore, learning a physically-consistent model of the underlying dynamics can subsequently enable usage of model-based controllers which can provide performance guarantees for complex, nonlinear systems. In particular, insight about kinetic and potential energy of a physical system can be leveraged to synthesize appropriate control strategies, such as the method of controlled Lagrangian and interconnection & damping assignment , which can reshape the closed-loop energy landscape to achieve a broad range of control objectives (regulation, tracking, etc.). Inferring underlying dynamics from time-series data plays a critical role in controlling closed-loop response of dynamical systems, such as robotic manipulators and building HVAC systems . Although the use of neural networks towards identification and control of dynamical systems dates back to more than three decades , recent advances in deep neural networks have led to renewed interest in this domain. learn dynamics with control from highdimensional observations (raw image sequences) using a variational approach and synthesize an iterative LQR controller to control physical systems by imposing a locally linear constraint. and adopt a variational approach and use recurrent architectures to learn state-space models from noisy observation. SE3-Nets learn SE transformation of rigid bodies from point cloud data. use partial information about the system state to learn a nonlinear state-space model. However, this body of work, while attempting to learn state-space models, does not take physics-based priors into consideration. The main contribution of this work is two-fold. First, we introduce a learning framework called Symplectic ODE-Net (SymODEN) which encodes a generalization of the Hamiltonian dynamics. This generalization, by adding an external control term to the standard Hamiltonian dynamics, allows us to learn the system dynamics which conforms to Hamiltonian dynamics with control. With the learned structured dynamics, we are able to synthesize controllers to control the system to track a reference configuration. Moreover, by encoding the structure, we can achieve better predictions with smaller network sizes. Second, we take one step forward in combining the physics-based prior and the data-driven approach. Previous approaches require data in the form of generalized coordinates and their derivatives up to the second order. However, a large number of physical systems accommodate generalized coordinates which are non-Euclidean (e.g., angles), and such angle data is often obtained in the embedded form, i.e., (cos q, sin q) instead of the coordinate (q) itself. The underlying reason is that an angular coordinate lies on S 1 instead of R 1. In contrast to previous approaches which do not address this aspect, SymODEN has been designed to work with angle data in the embedded form. Additionally, we leverage differentiable ODE solvers to avoid the need for estimating second-order derivatives of generalized coordinates. Lagrangian dynamics and Hamiltonian dynamics are both reformulations of Newtonian dynamics. They provide novel insights into the laws of mechanics. In these formulations, the configuration of a system is described by its generalized coordinates. Over time, the configuration point of the system moves in the configuration space, tracing out a trajectory. Lagrangian dynamics describes the evolution of this trajectory, i.e., the equations of motion, in the configuration space. Hamiltonian dynamics, however, tracks the change of system states in the phase space, i.e. the product space of generalized coordinates q = (q 1, q 2, ..., q n) and generalized momenta p = (p 1, p 2, ..., p n). In other words, Hamiltonian dynamics treats q and p on an equal footing. This not only provides symmetric equations of motion but also leads to a whole new approach to classical mechanics . Hamiltonian dynamics is also widely used in statistical and quantum mechanics. In Hamiltonian dynamics, the time-evolution of a system is described by the Hamiltonian H(q, p), a scalar function of generalized coordinates and momenta. Moreover, in almost all physical systems, the Hamiltonian is the same as the total energy and hence can be expressed as where the mass matrix M(q) is symmetric positive definite and V (q) represents the potential energy of the system. Correspondingly, the time-evolution of the system is governed bẏ where we have dropped explicit dependence on q and p for brevity of notation. Moreover, sincė the total energy is conserved along a trajectory of the system. The RHS of Equation is called the symplectic gradient of H, and Equation shows that moving along the symplectic gradient keeps the Hamiltonian constant. In this work, we consider a generalization of the Hamiltonian dynamics which provides a means to incorporate external control (u), such as force and torque. As external control is usually affine and only influences changes in the generalized momenta, we can express this generalization as where the input matrix g(q) is typically assumed to have full column rank. For u = 0, the generalized dynamics reduces to the classical Hamiltonian dynamics and the total energy is conserved; however, when u = 0, the system has a dissipation-free energy exchange with the environment. Once we have learned the dynamics of a system, the learned model can be used to synthesize a controller for driving the system to a reference configuration q. As the proposed approach offers insight about the energy associated with a system, it is a natural choice to exploit this information for synthesizing controllers via energy shaping . As energy is a fundamental aspect of physical systems, reshaping the associated energy landscape enables us to specify a broad range of control objectives and synthesize nonlinear controllers with provable performance guarantees. If rank(g(q)) = rank(q), the system is fully-actuated and we have control over any dimension of "acceleration" inṗ. For such fully-actuated systems, a controller u(q, p) = β β β(q) + v(p) can be synthesized via potential energy shaping β β β(q) and damping injection v(p). For completeness, we restate this procedure using our notation. As the name suggests, the goal of potential energy shaping is to synthesize β β β(q) such that the closed-loop system behaves as if its time-evolution is governed by a desired Hamiltonian H d. With this, we have where the difference between the desired Hamiltonian and the original one lies in their potential energy term, i.e. In other words, β β β(q) shape the potential energy such that the desired Hamiltonian H d (q, p) has a minimum at (q, 0). Then, by substituting Equation and Equation into Equation, we get Thus, with potential energy shaping, we ensure that the system has the lowest energy at the desired reference configuration. Furthermore, to ensure that trajectories actually converge to this configuration, we add an additional damping term 2 given by However, for underactuated systems, potential energy shaping alone cannot 3 drive the system to a desired configuration. We also need kinetic energy shaping for this purpose . Remark If the desired potential energy is chosen to be a quadratic of the form the external forcing term can be expressed as This can be interpreted as a PD controller with an additional energy compensation term. In this section, we introduce the network architecture of Symplectic ODE-Net. In Subsection 3.1, we show how to learn an ordinary differential equation with a constant control term. In Subsection 3.2, we assume we have access to generalized coordinate and momentum data and derive the network architecture. In Subsection 3.3, we take one step further to propose a data-driven approach to deal with data of embedded angle coordinates. In Subsection 3.4, we put together the line of reasoning introduced in the previous two subsections to propose SymODEN for learning dynamics on the hybrid space R n × T m. Now we focus on the problem of learning the ordinary differential equation (ODE) from time series data. Consider an ODE:ẋ = f (x). Assume we don't know the analytical expression of the right hand side (RHS) and we approximate it with a neural network. If we have time series data X = (x t0, x t1, ..., x tn), how could we learn f (x) from the data? introduced Neural ODE, differentiable ODE solvers with O-memory backpropagation. With Neural ODE, we make predictions by approximating the RHS function using a neural network f θ and feed it into an ODE solver x t1,x t2,...,x tn = ODESolve(x t0, f θ, t 1, t 2, ..., t n) We can then construct the loss function L = X−X 2 2 and update the weights θ by backpropagating through the ODE solver. In theory, we can learn f θ in this way. In practice, however, the neural net is hard to train if n is large. If we have a bad initial estimate of the f θ, the prediction error would in general be large. Although |x t1 −x t1 | might be small,x t N would be far from x t N as error accumulates, which makes the neural network hard to train. In fact, the prediction error ofx t N is not as important asx t1. In other words, we should weight data points in a short time horizon more than the rest of the data points. In order to address this and better utilize the data, we introduce the time horizon τ as a hyperparameter and predict x ti+1, x ti+2,..., x ti+τ from initial condition x ti, where i = 0,..., n − τ. One challenge toward leveraging Neural ODE to learn state-space models is the incorporation of the control term into the dynamics. Equation has the formẋ = f (x, u) with x = (q, p). A function of this form cannot be directly fed into Neural ODE directly since the domain and range of f have different dimensions. In general, if our data consist of trajectories of (x, u) t0,...,tn where u remains the same in a trajectory, we can leverage the augmented dynamics 2 if we have access toq instead of p, we useq instead in Equation 3 As gg T is not invertible, we cannot solve the matching condition given by Equation 7. 4 Under review as a conference paper at ICLR 2020 With this improvisation, we can match the input and output dimension off θ, which enables us to feed it into Neural ODE. The idea here is to use different constant external forcing to get the system responses and use those responses to train the model. With a trained model, we can apply a timevarying u to the dynamicsẋ = f θ (x, u) and generate estimated trajectories. When we synthesize the controller, u remains constant in each integration step. As long as our model interpolates well among different values of constant u, we could get good estimated trajectories with a time-varying u. The problem is then how to design the network architecture off θ, or equivalently f θ such that we can learn the dynamics in an efficient way. Suppose we have trajectory data consisting of (q, p, u) t0,...,tn, where u remains constant in a trajectory. If we have the prior knowledge that the unforced dynamics of q and p is governed by Hamiltonian dynamics, we can use three neural nets -M −1 θ1 (q), V θ2 (q) and g θ3 (q) -as function approximators to represent the inverse of mass matrix, potential energy and the control coefficient. Thus, where The partial derivative in the expression can be taken care of by automatic differentiation. by putting the designed f θ (q, p, u) into Neural ODE, we obtain a systematic way of adding the prior knowledge of Hamiltonian dynamics into end-to-end learning. In the previous subsection, we assume (q, p, u) t0,...,tn. In a lot of physical system models, the state variables involve angles which reside in the interval [−π, π). In other words, each angle resides on the manifold S 1. From a data-driven perspective, the data that respects the geometry is a 2 dimensional embedding (cos q, sin q). Furthermore, the generalized momentum data is usually not available. Instead, the velocity is often available. For example, in OpenAI Gym Pendulum-v0 task, the observation is (cos q, sin q,q). From a theoretical perspective, however, the angle itself is often used, instead of the 2D embedding. The reason being both the Lagrangian and the Hamiltonian formulations are derived using generalized coordinates. Using an independent generalized coordinate system makes it easier to solve for the equations of motion. In this subsection, we take the data-driven standpoint. We assume all the generalized coordinates are angles and the data comes in the form of (x 1 (q), x 2 (q), x 3 (q), u) t0,...,tn = (cos q, sin q,q, u) t0,...,tn. We aim to incorporate our theoretical prior -Hamiltonian dynamics -into the data-driven approach. The goal is to learn the dynamics of x 1, x 2 and x 3. Noticing p = M(x 1, x 2)q, we can write down the derivative of x 1, x 2 and x 3, where "•" represents the elementwise product (Hadamard product). We assume q and p evolve with the generalized Hamiltonian dynamics Equation. Here the Hamiltonian H(x 1, x 2, p) is a function of x 1, x 2 and p instead of q and p. Then the right hand side of Equation can be expressed as a function of state variables and control (x 1, x 2, x 3, u). Thus, it can be fed into the Neural ODE. We use three neural nets -M 2 ) and g θ3 (x 1, x 2) -as function approximators. Substitute Equation and Equation into Equation, then the RHS serves as f θ (x 1, x 2, x 3, u). where In Subsection 3.2, we treated the generalized coordinates as translational coordinates. In Subsection 3.3, we developed a method to better deal with embedded angle data. In most of physical systems, these two types of coordinates coexist. For example, robotics systems are usually modelled as interconnected rigid bodies. The positions of joints or center of mass are translational coordinates and the orientations of each rigid body are angular coordinates. In other words, the generalized coordinates lie on R n × T m, where T m denotes the m-torus, with T 1 = S 1 and In this subsection, we put together the architecture of the previous two subsections. We assume the generalized coordinates are q = (r, φ φ φ) ∈ R n × T m and the data comes in the form of (x 1, x 2, x 3, x 4, x 5, u) t0,...,tn = (r, cos φ φ φ, sin φ φ φ,ṙ,φ φ φ, u) t0,...,tn. With similar line of reasoning, we use three neural nets -M with Hamiltonian dynamics, we havė where theṙ andφ φ φ come from Equation. Now we obtain a f θ which can be fed into Neural ODE. Figure 1 shows the flow of the computation graph based on Equation-. In real physical systems, the mass matrix M is positive definite, which ensures a positive kinetic energy with a non-zero velocity. The positive definiteness of M implies the positive definiteness of 4 In Equation, the derivative of M −1 θ 1 (x1, x2) can be expanded using chain rule and expressed as a function of the states., where L θ1 is a lower-triangular matrix. The positive definiteness is ensured if the diagonal elements of M −1 θ1 are positive. In practice, this can be done by adding a small constant to the diagonal elements of M θ1. It not only makes M θ1 invertible, but also stabilize the training. We use the following four tasks to evaluate the performance of Symplectic ODE-Net model -(i) Task 1: a pendulum with generalized coordinate and momentum data (learning on R 1); (ii) Task 2: a pendulum with embedded angle data (learning on S 1); (iii) Task 3: a cart-pole system (learning on R 1 × S 1); and (iv) Task 4: an acrobot (learning on T 2). Model Variants. Besides the Symplectic ODE-Net model derived above, we consider a variant by approximating the Hamiltonian using a fully connected neural net H θ1,θ2. We call it Unstructured Symplectic ODE-Net (Unstructured SymODEN) since this model does not exploit the structure of the Hamiltonian. Baseline Models. In order to show that we can learn the dynamics better with less parameters by leveraging prior knowledge, we set up baseline models for all four experiments. For the pendulum with generalized coordinate and momentum data, the naive baseline model approximates Equation -f θ (x, u) -by a fully connected neural net. For all the other experiments, which involves embedded angle data, we set up two different baseline models: naive baseline approximates f θ (x, u) by a fully connected neural net. It doesn't respect the fact that the coordinate pair, cos φ φ φ and sin φ φ φ, lie on T m. Thus, we set up the geometric baseline model which approximatesq andṗ with a fully connected neural net. This ensures that the angle data evolves on T m. Data Generation. For all tasks, we randomly generated initial conditions of states and subsequently combined them with 5 different constant control inputs, i.e., u = −2.0, −1.0, 0.0, 1.0, 2.0 to produce the initial conditions and input required for simulation. The simulators integrate the corresponding dynamics for 20 time steps to generate trajectory data which is then used to construct the training set. The simulators for different tasks are different. For Task 1, we integrate the true generalized Hamiltonian dynamics with a time interval of 0.05 seconds to generate trajectories. All the other tasks deal with embedded angle data and velocity directly, so we use OpenAI Gym simulators to generate trajectory data. One drawback of using OpenAI Gym is that not all environments use the Runge-Kutta method (RK4) to carry out the integration. OpenAI Gym favors other numerical schemes over RK4 because of speed, but it is harder to learn the dynamics with inaccurate data. For example, if we plot the total energy as a function of time from data generated by Pendulum-v0 environment with zero action, we see that the total energy oscillates around a constant by a significant amount, even though the total energy should be conserved. Thus, for Task 2 and Task 3, we use Pendulum-v0 and CartPole-v1, respectively, and replace the numerical integrator of the environments to RK4. For Task 4, we use the Acrobot-v1 environment which is already using RK4. We also change the action space of Pendulum-v0, CartPole-v1 and Acrobot-v1 to a continuous space with a large enough bound. Model training. In all the tasks, we train our model using Adam optimizer with 1000 epochs. We set a time horizon τ = 3, and choose "RK4" as the numerical integration scheme in Neural ODE. We vary the size of the training set by doubling from 16 initial state conditions to 1024 initial state conditions. Each initial state condition is combined with five constant control u = −2.0, −1.0, 0.0, 1.0, 2.0 to produce initial condition and input for simulation. Each trajectory is generated by integrating the dynamics 20 time steps forward. We set the size of minibatches to be the number of initial state conditions. We logged the train error per trajectory and the prediction error per trajectory in each case for all the tasks. The train error per trajectory is the mean squared error (MSE) between the estimated trajectory and the ground truth over 20 time steps. To evaluate the performance of each model in terms of long time prediction, we construct the metric of prediction error per trajectory by using the same initial state condition in the training set with a constant control of u = 0.0, integrating 40 time steps forward, and calculating the MSE over 40 time steps. The reason for using only the unforced trajectories is that a constant nonzero control might cause the velocity to keep increasing or decreasing over time, and large absolute values of velocity are of little interest for synthesizing controllers. In this task, we use the model described in Section 3.2 and present the predicted trajectories of the learned models as well as the learned functions of SymODEN. We also point out the drawback of treating the angle data as a Cartesian coordinate. The dynamics of this task has the following forṁ with Hamiltonian H(q, p) = 1.5p 2 + 5(1 − cos q). In other words M (q) = 3, V (q) = 5(1 − cos q) and g(q) = 1. In Figure 2, The ground truth is an unforced trajectory which is energyconserved. The prediction trajectory of the baseline model does not conserve energy, while both the SymODEN and its unstructured variant predict energy-conserved trajectories. For SymODEN, the learned g θ3 (q) and M −1 θ1 (q) matches the ground truth well. V θ2 (q) differs from the ground truth with a constant. This is acceptable since the potential energy is a relative notion. Only the derivative of V θ2 (q) plays a role in the dynamics. Here we treat q as a variable in R 1 and our training set contains initial conditions of q ∈ [−π, 3π]. The learned functions do not extrapolate well outside this range, as we can see from the left part in the figures of M −1 θ1 (q) and V θ2 (q). We address this issue by working directly with embedded angle data, which leads us to the next subsection. In this task, the dynamics is the same as Equation but the training data are generated by the OpenAI Gym simulator, i.e. we use embedded angle data and assume we only have access toq instead of p. We use the Under review as a conference paper at ICLR 2020 model described in Section 3.3 and synthesize an energy-based controller (Section 2.2). Without true p data, the learned function matches the ground truth with a scaling β, as shown in Figure 3. To explain the scaling, let us look at the following dynamicṡ with Hamiltonian H = p 2 /(2α) + 15α(1 − cos q). If we only look at the dynamics of q, we havë q = −15 sin q+3u, which is independent of α. If we don't have access to the generalized momentum p, our trained neural network may converge to a Hamiltonian with a α e which is different from the true value, α t = 1/3, in this task. By a scaling β = α t /α e = 0.357, the learned functions match the ground truth. Even we are not learning the true α t, we can still perform prediction and control since we are learning the dynamics of q correctly. We let V d = −V θ2 (q), then the desired Hamiltonian has minimum energy when the pendulum rests at the upward position. For the damping injection, we let K d = 3. Then from Equation and, the controller we synthesize is Only SymODEN out of all models we consider provides the learned potential energy which is required to synthesize the controller. Figure 4 shows how the states evolve when the controller is fed into the OpenAI Gym simulator. We can successfully control the pendulum into the inverted position using the controller based on the learned model even though the absolute maximum control u, 7.5, is more than three times larger than the absolute maximum u in the training set, which is 2.0. This shows SymODEN extrapolates well. The CartPole system is an underactuated system and to synthesize a controller to balance the pole from arbitrary initial condition requires trajectory optimization or kinetic energy shaping. We show that we can learn its dynamics and perform prediction in Section 4.6. We also train SymODEN in a fully-actuated version of the CartPole system (see Appendix E). The corresponding energy-based controller can bring the pole to the inverted position while driving the cart to the origin. The Acrobot is an underactuated double pendulum. As this system exhibits chaotic motion, it is not possible to predict its long-term behavior. However, Figure 6 shows that SymODEN can provide reasonably good short-term prediction. We also train SymODEN in a fully-actuated version of the Acrobot and show that we can control this system to reach the inverted position (see Appendix E). In this subsection, we show the train error, prediction error, as well as the MSE and total energy of a sample test trajectory for all the tasks. Figure 5 shows the variation in train error and prediction error with changes in the number of initial state conditions in the training set. We can see that SymODEN yields better generalization in every task. In Task 3, although the Geometric Baseline Model yields lower train error in comparison to the other models, SymODEN generates more accurate predictions, indicating overfitting in the Geometric Baseline Model. By incorporating the physics-based prior of Hamiltonian dynamics, SymODEN learns dynamics that obeys physical laws and thus provides better predictions. In most cases, SymODEN trained with a smaller training dataset performs better than other models in terms of the train and prediction error, indicating that better generalization can be achieved even with fewer training samples. R2-C4 R2-C5 in the training set. Both the horizontal axis and vertical axis are in log scale. Figure 6 shows the evolution of MSE and total energy along a trajectory with a previously unseen initial condition. For all the tasks, MSE of the baseline models diverges faster than SymODEN. Unstructured SymODEN performs well in all tasks except Task 3. As for the total energy, in Task 1 and Task 2, SymODEN and Unstructured SymODEN conserve total energy by oscillating around a constant value. In these models, the Hamiltonian itself is learned and the prediction of the future states stay around a level set of the Hamiltonian. Baseline models, however, fail to find the conservation and the estimation of future states drift away from the initial Hamiltonian level set. Here we have introduced Symplectic ODE-Net which provides a systematic way to incorporate prior knowledge of Hamiltonian dynamics with control into a deep learning framework. We show that SymODEN achieves better prediction with fewer training samples by learning an interpretable, physically-consistent state-space model. Future works will incorporate a broader class of physicsbased prior, such as the port-Hamiltonian system formulation, to learn dynamics of a larger class of physical systems. SymODEN can work with embedded angle data or when we only have access to velocity instead of generalized momentum. Future works would explore other types of embedding, such as embedded 3D orientations. Another interesting direction could be to combine energy shaping control (potential as well as kinetic energy shaping) with interpretable end-to-end learning frameworks. Tianshu Wei, Yanzhi Wang, and Qi Zhu. Deep Reinforcement Learning for Building HVAC Control. In Proceedings of the 54th Annual Design Automation Conference (DAC), pp. 22:1-22:6, 2017. The architectures used for our experiments are shown below. For all the tasks, SymODEN has the lowest number of total parameters. To ensure that the learned function is smooth, we use Tanh activation function instead of ReLu. As we have differentiation in the computation graph, nonsmooth activation functions would lead to discontinuities in the derivatives. This, in turn, would in an ODE with a discontinuous RHS which is not desirable. All the architectures shown below are fully-connected neural networks. The first number indicates the dimension of the input layer. The last number indicates the dimension of output layer. The dimension of hidden layers is shown in the middle along with the activation functions. Task 1: Pendulum • Input: 2 state dimensions, 1 action dimension • Baseline Model (0.36M parameters): 2 -600Tanh -600Tanh -2Linear • Unstructured SymODEN (0.20M parameters): • SymODEN (0.13M parameters): Task 2: Pendulum with embedded data • Input: 3 state dimensions, 1 action dimension • Naive Baseline Model (0.65M parameters): 4 -800Tanh -800Tanh -3Linear • Geometric Baseline Model (0.46M parameters):, where L θ1: 1 -300Tanh -300Tanh -300Tanh -1Linear -approximate (q,ṗ): 4 -600Tanh -600Tanh -2Linear, where Task 3: CartPole • Input: 5 state dimensions, 1 action dimension • Naive Baseline Model (1.01M parameters): 6 -1000Tanh -1000Tanh -5Linear, where L θ1: 3 -400Tanh -400Tanh -400Tanh -3Linear -H θ2: 5 -500Tanh -500Tanh -1Linear -g θ3: 3 -300Tanh -300Tanh -2Linear, where L θ1: 3 -400Tanh -400Tanh -400Tanh -3Linear -V θ2: 3 -300Tanh -300Tanh -1Linear -g θ3: 3 -300Tanh -300Tanh -2Linear Task 4:Acrobot • Input: 6 state dimensions, 1 action dimension • Naive Baseline Model (1.46M parameters): 7 -1200Tanh -1200Tanh -6Linear, where L θ1: 4 -400Tanh -400Tanh -400Tanh -3Linear -approximate (q,ṗ): 7 -800Tanh -800Tanh -4Linear, where L θ1: 4 -400Tanh -400Tanh -400Tanh -3Linear -H θ2: 6 -600Tanh -600Tanh -1Linear -g θ3: 4 -300Tanh -300Tanh -2Linear, where L θ1: 4 -400Tanh -400Tanh -400Tanh -3Linear -V θ2: 4 -300Tanh -300Tanh -1Linear -g θ3: 4 -300Tanh -300Tanh -2Linear The energy-based controller has the form u(q, p) = β β β(q) + v(p), where the potential energy shaping term β β β(q) and the damping injection term v(p) are given by Equation and Equation, respectively. If the desired potential energy V q (q) is given by a quadratic, as in Equation, then and the controller can be expressed as The corresponding external forcing term is then given by which is same as Equation in the main body of the paper. The first term in this external forcing provides an energy compensation, whereas the second term and the last term are proportional and derivative control terms, respectively. Thus, this control can be perceived as a PD controller with an additional energy compensation. In Hamiltonian Neural Networks (HNN), incorporate the Hamiltonian structure into learning by minimizing the difference between the symplectic gradients and the true gradients. When the true gradient is not available, which is often the case, the authors suggested using finite difference approximations. In SymODEN, true gradients or gradient approximations are not necessary since we integrate the estimated gradient using differentiable ODE solvers and set up the loss function with the integrated values. Here we perform an ablation study of the differentiable ODE Solver. Both HNN and the Unstructured SymODEN approximate the Hamiltonian by a neural network and the main difference is the differentiable ODE solver, so we compare the performance of HNN and the Unstructured SymODEN. We set the time horizon τ = 1 since it naturally corresponds to the finite difference estimate of the gradient. A larger τ would correspond to higher-order estimates of gradients. Since there is no angle-aware design in HNN, we use Task 1 to compare the performance of these two models. We generate 25 training trajectories, each of which contains 45 time steps. This is consistent with the HNN paper. In the HNN paper , the initial conditions of the trajectories are generated randomly in an annulus, whereas in this paper, we generate the initial state conditions uniformly in a reasonable range in each state dimension. We guess the reason the authors of HNN choose the annulus data generation is that they do not have an angle-aware design. Take the pendulum for example; all the training and test trajectories they generate do not pass the inverted position. If they make prediction on a trajectory with a large enough initial speed, the angle would go over ±2π, ±4π, etc. in the long run. Since these are away from the region where the model gets trained, we can expect the prediction would be poor. In fact, this motivates us to design the angle-aware SymODEN in Section 3.3. In this ablation study, we generate the training data in both ways. Table 1 shows the train error and the prediction error per trajectory of the two models. We can see Unstructured SymODEN performs better than HNN. This is an expected . To see why this is the case, let us assume the training loss per time step of HNN is similar to that of Unstructured SymODEN. Since the training loss is on the symplectic gradient, the error would accumulate while integrating the symplectic gradient to get the estimated state values, and MSE of the state values would likely be one order of magnitude greater than that of Unstructured SymODEN. Figure 7 shows the MSE and total energy of a particular trajectory. It is clear that the MSE of the Unstructured SymODEN is lower than that of HNN. The MSE of HNN periodically touches zero does not mean it has a good prediction at that time step. Since the trajectories in the phase space are closed circles, those zeros mean the predicted trajectory of HNN lags behind (or runs ahead of) the true trajectory by one or more circles. Also, the energy of the HNN trajectory drifts instead of staying constant, probably because the finite difference approximation is not accurate enough. Incorporating the differential ODE solver also introduces two hyperparameters: solver types and time horizon τ. For the solver types, the Euler solver is not accurate enough for our tasks. The adaptive solver "dopri5" lead to similar train error, test error and prediction error as the RK4 solver, but requires more time during training. Thus, in our experiments, we choose RK4. Time horizon τ is the number of points we use to construct our loss function. Table 2 shows the train error, test error and prediction error per trajectory in Task 2 when τ is varied from 1 to 5. We can see that longer time horizons lead to better models. This is expected since long time horizons penalize worse long term predictions. We also observe in our experiments that longer time horizons require more time to train the models. CartPole and Acrobot are underactuated systems. Incorporating the control of underactuated systems into the end-to-end learning framework is our future work. Here we trained SymODEN on Under review as a conference paper at ICLR 2020 fully actuated versions of Cartpole and Acrobot and synthesized controllers based on the learned model. For the fully-actuated CartPole, Figure 8 shows the snapshots of the system of a controlled trajectory with an initial condition where the pole is below the horizon. Figure 9 shows the time series of state variables and control inputs. We can successfully learn the dynamics and control the pole to the inverted position and the cart to the origin. For the fully-actuated Acrobot, Figure 10 shows the snapshots of a controlled trajectory. Figure 11 shows the time series of state variables and control inputs. We can successfully control the Acrobot from the downward position to the upward position, though the final value of q 2 is a little away from zero. Taking into account that the dynamics has been learned with only 64 different initial state conditions, it is most likely that the upward position did not show up in the training data. Here we show statistics of train, test, and prediction per trajectory in all four tasks. The train errors are based on 64 initial state conditions and 5 constant inputs. The test errors are based on 64 previously unseen initial state conditions and the same 5 constant inputs. Each trajectory in the train and test set contains 20 steps. The prediction error is based on the same 64 initial state conditions (during training) and zero inputs.

Represent the short summary for finding the corresponding scientific article (long)

This work enforces Hamiltonian dynamics with control to learn system models from embedded position and velocity data, and exploits this physically-consistent dynamics to synthesize model-based control via energy shaping.

Represent the TLDR for finding the corresponding scientific article

A structured latent-variable approach that adds discrete control states within a standard autoregressive neural paradigm to provide arbitrary grounding of internal model decisions, without sacrificing any representational power of neural models.