Neural Interaction Energy for Multi-Agent Trajectory Prediction
Kaixin Shen Ruijie Quan Linchao Zhu Jun Xiao Yi Yang



















































  Maintaining temporal stability is crucial in multi-agent trajectory prediction. Insufficient regularization to uphold this stability often results in fluctuations in kinematic states, leading to inconsistent predictions and the amplification of errors. In this study, we introduce a framework called Multi-Agent Trajectory prediction via neural interaction Energy (MATE). This framework assesses the interactive motion of agents by employing neural interaction energy, which captures the dynamics of interactions and illustrates their influence on the future trajectories of agents. To bolster temporal stability, we introduce two constraints: inter-agent interaction constraint and intra-agent motion constraint. These constraints work together to ensure temporal stability at both the system and agent levels, effectively mitigating prediction fluctuations inherent in multi-agent systems. Comparative evaluations against previous methods on four diverse datasets highlight the superior prediction accuracy and generalization capabilities of our model. 
  









































Neural Interaction Energy for Multi-Agent Trajectory Prediction
    Kaixin Shen, Ruijie Quan, Linchao Zhu, Jun Xiao, Yi Yang 

Zhejiang University
    April 29, 2024
==================================================================================



§ INTRODUCTION


Multi-agent trajectory prediction is a fundamental research task with many applications such as autonomous driving <cit.>, interactive robotics <cit.>, particle simulation <cit.>, and team sports <cit.>. It refers to predicting the future positions of multiple agents in a dynamic environment simultaneously, based on their historical positions and agent interactions. This task is critical in many real-world scenarios where a group of agents interacts with each other, giving rise to complicated behavior patterns at the level of both individuals and the whole system. For this reason, it is more challenging to simultaneously predict the future positions of multiple agents compared to single-agent trajectory prediction <cit.>.




Temporal stability is one essential aspect of multi-agent systems. 

It indicates that the system's state maintains consistent dynamics over time and converges into an equilibrium distribution <cit.>. Also, 
it implies that agents do not have rapid or incoherent transitions of motion states (position and velocity). 
Establishing a multi-agent trajectory prediction framework that preserves temporal stability offers two main benefits. First, the framework enables more accurate predictions of multi-agent trajectories. Second, it is adaptable to a wide range of system environments and agent configurations.
Temporal stability is one essential aspect of multi-agent systems to achieve trajectory prediction.
It indicates that the system's state maintains consistent dynamics over time and converges into an equilibrium distribution <cit.>. Also, 
it implies that agents do not have rapid or incoherent transitions of motion states (position and velocity). 
Establishing a multi-agent trajectory prediction framework that preserves temporal stability offers two main benefits. First, since temporal stability ensures the coherence of change in agents' motion states, the framework enables more accurate predictions of multi-agent trajectories. Second, the framework is adaptable to a wide range of system environments and agent configurations because temporal stability exists in various systems.






One of the major obstacles in multi-agent trajectory prediction lies in the frequent fluctuations of kinematic states across consecutive time intervals. These fluctuations pose a significant challenge as they can lead to inconsistencies in the trajectory predictions for each agent. The variability in kinematic states, such as sudden changes in position or velocity, further causes error amplification in sequential predictions <cit.>. Such fluctuations can be mainly attributed to the absence of proper regularization to temporal dynamics in multi-agent systems.


Motivated by this, we aim to regularize the multi-agent trajectory prediction systems to be temporally stable, enhancing the prediction robustness across diverse scenarios.
Temporal stability refers to the property that the system's kinematic states do not change abruptly or inconsistently over time, but rather approach a steady-state distribution <cit.>. 
Temporal stability offers two advantages for trajectory prediction in multi-agent systems: 1) Improved prediction accuracy. By enforcing temporal stability in the multi-agent systems, the prediction results would be more resilient to observation noises and error propagation.
2) Enhanced adaptability to diverse environments. 

The presence of temporal stability in multi-agent systems results in more uniform and coordinated movement patterns, enabling the system to adapt seamlessly to a variety of environmental conditions.










The reason lies in their implicit interaction representation and incoherent motion prediction. (1) Implicit interaction representation. Previous work <cit.> for multi-agent trajectory prediction typically leverages graph neural networks (GNNs) to represent the interactions between agents, where the interactions are considered as latent of graph edges. However, this work fails to directly constrain these implicit representations.

(2) Incoherent motion change. Existing methods take the previous step's position of agents as input and leverage the prediction model to predict the change in position, then they get the position of the next step. The prediction model without extra constraint may lead to an incoherent motion change. 



In this paper, we propose a framework named MATE for multi-agent trajectory prediction. 

We enhance the temporal stability of the multi-agent system


with an inter-agent constraint and an intra-agent constraint. These two constraints help to preserve temporal stability at the system level and agent level, respectively.










First, we introduce the inter-agent interaction constraint to enhance the stability of agent interactions at the system level. We represent the interactions between agents as the changes of their neural interaction energy, while the system-level dynamics can be considered as the aggregation of neural interaction energy from all agents.
Specifically, the neural interaction energy quantifies agents' motion interactiveness at each step. It also determines the action of agents in future steps. The agents with low neural interaction energy may exhibit reduced movement speed, whereas those with high energy are prone to be more dynamic.


The inter-agent interaction constraint minimizes the change of system-level neural interaction energy, stabilizing the multi-agent movement patterns and ensuring a more coherent evolution of the multi-agent system over time.













Second, we introduce the intra-agent motion constraint to enforce the agent-level stability. 
For each agent, we leverage a temporal motion variance term to measure the consistency of their movement patterns over time.
We derive an approximated motion based on the agent's past kinematic states and its interactions with surrounding agents.
The temporal motion variance term quantifies the discrepancy between the approximated motion and the predicted motion, which helps to identify the agent movement inconsistency. By minimizing this variance, it ensures that the predicted motion aligns closely with the approximated motion, restricting the kinematic states of agents to be coherent and may vary within a plausible range. 


















Our model outperforms the state-of-the-art methods in multi-agent trajectory prediction across four datasets, i.e., PHASE <cit.>, Socialnav <cit.>, Charged <cit.>, and NBA datasets. We further validate the advantages of our method in zero-shot generalization to unseen scenarios.

In summary, the main contributions of this paper are threefold:  

    
  * We propose to use neural interaction energy to model the interactions between agents, providing an effective framework to capture the complexities of agent interactions.
    
    
  * We establish an inter-agent interaction constraint and an intra-agent motion constraint, which well preserve the temporal stability at the system level and agent level.
    
  * 
    Extensive experiments demonstrate that our model accurately predicts future trajectories and improves the generalization abilities in unseen scenarios.


































§ RELATED WORK


Trajectory Prediction. Trajectory prediction approaches fall into model-based <cit.> and model-free methods <cit.>. Before the deep learning era, previous works employ various physical models to conduct trajectory prediction. Social force <cit.> and energy <cit.> are commonly used to model pedestrian motion. Some works <cit.> exploit the reciprocal velocity obstacles (RVO) model for trajectory prediction.  <cit.> predicts transitions in pedestrians by applying spatiotemporal graphs, where nodes and edges are represented by RNNs.  <cit.> computes joint motion predictions based on the time of possible collision between pairs of agents.  <cit.> propose the Multi-Agent Tensor Fusion encoding, which fuses contextual images of the environment with sequential trajectories of agents. We will include the above discussion and references in the revision.  
The rise of deep learning <cit.> has introduced model-free methods into this field. Such a method shifts the focus of modeling agents' trajectories to fitting the distribution of data <cit.>, which improves computational efficiency while reducing the model interpretability. 
GAT-LSTM <cit.> employs graph attention layers along with LSTMs for prediction. NRI <cit.> designs a VAE model with recurrent GNN networks to encode and predict trajectories. EvolveGraph <cit.> and fNRI <cit.> expand the framework of NRI and introduce additional modules to improve performance. RFM <cit.> uses a recurrent GNN model with a supervised loss to realize trajectory prediction. IMMA <cit.> introduces a multiplex graph and proposes a progressive layer training strategy. GRIN <cit.> employs generative models to learn the distribution of trajectories.
Inspired by traditional dynamics, we combine a novel concept named neural interaction energy and sequential models to predict the changes in agents' trajectories and introduce an inter-agent interaction constraint and an intra-agent motion constraint for preserving temporal stability.

Graph Neural Network(GNN). GNNs <cit.> are a type of neural network designed to operate on graph-structured data. GNNs are especially useful in tasks involving graph analysis, such as node classification, link prediction, and graph clustering. The key idea behind GNNs is to learn a function that maps the features of each node and edge to a new feature representation that captures the structure of the graph. This function is typically learned through an iterative message-passing process. One popular type of GNN is GCN <cit.>, which is based on the idea of convolving filters over the graph. Another type is GAT <cit.>, which uses attention mechanisms to weigh the importance of different nodes and edges in the graph. We borrow the idea of GAT to model the interaction between agents with a multi-edge graph and normalize according to the importance of neighbor nodes on each type of edge.

 
 
Differential Equations Informed Neural Network. Recently the combination of deep learning and differentiable equations has earned significant interest. Related research <cit.> can be categorized into different subfields such as deep learning assisted DE <cit.>, differentiable physics, neural differential equations <cit.>, and physics-informed neural networks (PINNs) <cit.>. PINNs are based on the physical prior that constrains the output by calculating the means of a partial differential equation (PDE) system with the assistance of a neural network. Inspired by such research, we propose novel neural differential equations for modeling interactions and establishing motion constraints for agents to predict coherent future trajectories.




§ METHODOLOGY







 §.§ Problem Formulation.


Given an observed trajectory 𝐗_i^T = {x_i^1, x_i^2, ..., x_i^T_obs} of an agent i ∈ N across period T_obs, our goal is to predict the future trajectories of all agents simultaneously. x_i^t ∈ℝ^2 is the 2D coordinate position of the agent i at time step t.
We denote the future trajectory across T_pred steps as 𝐘_i^T = {x_i^T_obs+1, x_i^T_obs+2, ..., x_i^T}, where T=T_obs+T_pred. Note that for each agent, we only predict a single future trajectory across T_pred steps.
In this paper, kinematic states include position and velocity. At each step, the position x of an agent indicates its location in the 2D coordinate system. In addition, to represent movement speed, we calculate the velocity v of agents at each step based on their 
position and time interval Δ t.


















 §.§ Inter-agent Interaction Modeling




Predicting future trajectories in multi-agent systems necessitates an accurate modeling of agent interactions. Existing approaches often rely on graph neural networks (GNNs) to depict these interactions as latent features of graph edges. However, this representation ignores ensuring the stability of agent interactions. Also, it does not focus on promising a temporally coherent evolution of the system. 
To address these overlooked aspects, we employ a neural interaction energy mechanism to model the interactions of agents, further preserving the temporal stability at the system level. 



Neural Interaction Energy E. 
We first present the neural interaction energy to quantify the agent's motion interactiveness, thereby capturing the dynamics of multi-agent interactions and illustrating
how interactions influence agents' future trajectories. 





Neural interaction energy quantifies the motion interactiveness at each step and affects the agent movement in future steps. This motion interactiveness is a function concerning both an agent's kinematic states (position and velocity) and its interaction representation. 
For clarity, we use E_i^t to denote the neural interaction energy scalar for agent i at step t and use E_i^t to denote the corresponding neural interaction energy features in the neural networks.



Intuitively, the reduction of neural interaction energy will decrease the agent interactiveness, while agents with high neural interaction energy are more likely to move.

In a multi-agent system, the motion of the target agent is affected by other agents. The change in neural interaction energy, reflecting each agent's interactions with others, acts as a measure of the dynamic exchanges within the system. Moreover, the overall dynamics of the multi-agent system can be viewed as the aggregation of the neural interaction energies of all participating agents, providing a comprehensive view of the system's interactive behavior.







Inter-agent Interaction Constraint.
We introduce an inter-agent interaction constraint to preserve system-level temporal stability based on the interaction between agents. Considering a multi-agent system, agents are moving concurrently and their behaviors are mutually influenced through complex social interactions.
The social interactions among agents indicate the transfer of neural interaction energy, accelerating or decelerating agents. However, due to the complexity of interactions between agents in systems, the neural interaction energy transfer process becomes difficult to analyze from the perspective of the agent level. 

Nevertheless, we model the interactions at the system level. We aim to preserve system-level temporal stability. It means the changes in its neural interaction energy maintain temporal continuity. The target formulation for system-level temporal stability is,

    E_i^t+Δ t - E_i^t→ 0,   when Δ t → 0.

Therefore, over a short time interval, the aggregated neural interaction energy of the system should experience negligible changes, resulting in a nearly constant sum of neural interaction energy changes of agents. That is,
 
    Δ∑_i=1^NE_i = ∑_i=1^NE_i^t+Δ t - ∑_i=1^NE_i^t→ 0,   when Δ t → 0.

Note that the magnitude of change in neural interaction energy differs among different types of agents, necessitating the assignment of weights to achieve normalization. 
Considering that different types of agents have different ranges of positional changes within the same time interval, we employ Δx as the normalization weight. We have

    ∑_i=1^NΔ E_i/Δx_i→ 0,   when Δ t → 0.

Δ E_i and Δx_i denote the change in neural interaction energy and the change in position of agent i, respectively. 



















Consequently, we can model the interactions between agents while preserving the temporal stability at the system level. Formally, the system would have minimal neural interaction energy change in its aggregated interactions of all agents over a short period. Since the time interval is short, Δx tends to be zero. Not that we use the derivative of the neural interaction energy features concerning the position ∇_xE_i(x) in implementation to replace the expression in Eq. (<ref>). That is 

    ∑_i=1^N∇_xE_i(x) =  ∑_i=1^Nlim_Δ x → 0ΔE_i/Δx_i→ 0.

We omit the superscript t for simplification. 


In summary, we leverage Eq. (<ref>) to ensure the temporal stability of a multi-agent system by minimizing the aggregated changes in neural interaction energy over a short period.
Therefore, by incorporating this inter-agent interaction constraint into model optimization, we preserve system-level temporal stability.
We add this constraint in the form like

    ℒ_E = 1/N1/T∑_i=1^N∑_t=1^T∇_xE_i^t(x)_2.





 §.§ Intra-agent Motion Modeling






This section introduces to improve intra-agent motion coherence and avoid catastrophic erroneous predictions, preserving agent-level temporal stability.

If the value of the target agent's kinematic states deviates from the normal range (e.g., a pedestrian moves at the speed of a car), it indicates unstable changes and a potential prediction error. We aim to mitigate the prediction discrepancy at each step.
Technically, we define a temporal motion variance term and regularize this variance to be coherent.

This helps to restrict the value of the agent's kinematic states to vary within a plausible range.



















The kinematic states of the target agent in the current time step (x^t, v^t) can be determined by
the kinematic states from the previous time step

and the neural interaction energy.


We calculate the derivative of the predicted position x^t and the change in neural interaction energy ∇_xE^t(x) concerning the previous step velocity v^t-1. We formulate the following temporal motion variance u for the target agent,

    u^t=γ·∂∇_xE^t(x)/∂v^t-1 +  β·∂x^t/∂v^t-1 + α ,

where γ and β are scaling scalars and α is the bias term representing a constant correlation between the position and velocity of agents. For clarity, we omit the subscripts i of the variables as agents share the same formula.  
Within a multi-agent environment, an agent's position is not solely determined by its own intention but is also influenced by the interactions with other agents. Consequently, in addition to ∂x^t/∂v^t-1, our motion variance u introduces an additional term to encapsulate external influences on the agent. Hyperparameters γ and β are introduced to weigh the significance of these two terms, catering to scenarios where agent behaviors vary based on differing extents of external influences. This refined approach renders our proposed constraints more adaptable for trajectory prediction within multi-agent systems.

 


In the temporal motion variance u, we use the derivative of the predicted output (∇_xE^t(x) and x^t) concerning v^t-1 to measure the prediction's sensitivity.

A high sensitivity means that a minor miscalculation in estimating v^t-1 can magnify into a prominent prediction error. The detailed mathematical derivation of Eq. (<ref>) and determination of hyperparameters γ, β, and α can be found in the Appendix. 

By minimizing the temporal motion variance u:

    ℒ_D =1/N1/T∑_i=1^N∑_t=1^Tu_i^t_2,

we reduce the prediction sensitivity and penalize inaccurate or incoherent predictions, achieving agent-level temporal stability.
We add this intra-agent motion constraint to model optimization.








 













Above, we have introduced the concept of neural energy to explain the interactions between agents explicitly. Now we propose an additional motion constraint that facilitates the integration of agents' previous and current step states. 

In the ideal simulated case, the agents' trajectories should follow a Markovian property, meaning, p(x^t+1|x^1, ..., x^t, z) = p(x^t+1|x^t, z), and we can predict the position of each step merely through calculation. However, this assumption does not hold in most real-world systems, as the agent's position in the current step is influenced by all its past trajectories. In order to establish a unified framework applicable to both simulation scenarios (where the current position is only influenced by the previous step) and real-world situations (where the current position is influenced by all previous steps), we made a trade-off in utilizing past trajectory information. Specifically, we adopted an RNN model as the foundational framework and proposed a soft constraint to amplify the influence of the agent's state at the previous step on its current position. Drawing reference from the form of the motion equation, we will express the constraint by combining the neural energy we defined with the assistance of partial differential equations (PDEs). That is

    x^t = x^t-1 + Δx s.t.Δx = G_θ(E_i^t, v_i^t-1, x_i^t-1),

where Δ t is timestep, x^t, v^t denote the position and velocity of the current step, x^t-1, v^t-1 denote corresponding position and velocity of the previous step, and Δ x can be considered as the result of a function G related to the position, velocity, and neural energy. For clarity, we omit the subscripts of the variables as agents share the same formula. Inspired by PINN <cit.>, we leverage partial differential equations (PDEs) to express the relation between the outputs and inputs of each step. That is, for all t ∈ T, we have

    u(x^t, x^t-1, v^t-1, E^t) = α + γ·∂E^t/∂v^t-1∂x^t-1 - ∂x^t/∂v^t-1,

where α and γ are hyperprameters. Refer to supplementary for more details on formula derivation. So far, we explicitly establish the relation u between the previous and the current step state of agents and indicate how past trajectories influence future trajectories. To strengthen the influence of the previous step's state on the current step for better prediction, we add such physical soft constraints to the model optimization by comprising the following PDE expression as part of the loss function.

    ℒ_D =1/N1/T∑_i=1^N∑_t=1^Tu(x_i^t, x_i^t-1, v_i^t-1, E_i^t) - 0^2.





 §.§ Model Architecture




MATE-Encoder with Multi-Edge Graph. Previous research employs a single-edge GNN to represent interactions between agents. In such a framework, the interaction latent z produced by the GNN is normalized and then utilized as a one-hot vector to symbolize interactions for proceeding prediction. However, multiple types of interactions frequently coexist between agents in real scenarios, and a simple one-hot vector fails to encapsulate the intricate properties of these interactions. For instance, in social event systems, factors like familiarity, status, and identities concurrently influence agents' interactions with others.

Addressing this, as shown in Fig. <ref>, we present a multi-edge GNN, where multiple edges are assigned between agents. Each edge symbolizes a distinct type of interaction and is separately encoded through a unique Multi-Layer Perceptron (MLP) layer. We hypothesize that all types of interactions influence the agents' neural interaction energy and their future trajectories. This fosters a richer representation of the interactions in a given multi-agent system.

Mathematically, given the observation of all agents X^1:T_obs, we employ a GNN model to learn the interaction latent as z_ij = GNN(X_i, X_j, θ). Here X_i and X_j represent the observed trajectory of agent i and j respectively, θ denotes model parameters, and z_ij represents the interaction latent between agent i and agent j. The interaction latent z_ij has K unnormalized dimensions, each indicating a type of interaction. We assume that agents can maintain diverse interactions and that the neighboring agents of the target agent have different importance considering each type of interaction. Consequently, instead of normalizing the dimensions of interaction latent z_ij, we normalize the same dimension of interaction latent between agent i and all its neighbors j as:

    z_ij^k = exp(z_ij^k)/∑_j∈N_iexp(z_ij^k),  ∀ k∈K.

N_i denotes the neighbor set of agent i, and K represents the number of dimensions of the latent code z_ij, we take the z = { z_ij| 1 ≤ i, j ≤ N } as the output of the MATE-Encoder.















MATE-Decoder with Neural Interaction Energy Module.

To optimize the prediction of the future trajectory, our MATE-Decoder incorporates a sequential model, incorporating our proposed constraints. Specifically in the MATE-Decoder, we introduce a Neural Interaction Energy Module (NIEM) unit (depicted in Fig. <ref>) that processes the previous position x, velocity v, interaction latent z, and previous hidden state h, yielding the new position, velocity, and hidden state for the predicted step. To enhance the performance of the MATE-Decoder, we adopt a burn-in strategy <cit.> that initiates the decoding process from the first observation step during training, thereby prolonging the prediction horizon and enhancing model robustness. 





Structure of the NIEM. Initially, a message passing (MSG) block is leveraged to integrate the information between the hidden state h and interaction latent z. This is followed by the processing of the embedding from the previous step's kinematic states along with the output from the MSG block through a GRU module. The GRU module results in the generation of an updated hidden state. Subsequently, an Energy Module calculates the neural interaction energy features E for each agent. These features serve as inputs for subsequent networks, facilitating the generation of agent trajectories. Additionally, a Constraint Module exploits these neural interaction energy features to formulate the inter-agent interaction constraint ℒ_E and the intra-agent motion constraint ℒ_D. It is noteworthy that, in our implementation, the inter-agent interaction constraint is integrated into the trajectory prediction through two distinct methods. For each dataset, the method yielding the most favorable result is selected for final result presentation. Refer to the Appendix for more details about the NIEM.





 §.§ Loss for Training

We employ an end-to-end training method. In our approach, we compute the mean squared error (MSE), denoted as ℒ_P, between the predicted position and their corresponding ground truth across period T. That is,

    ℒ_P   = 1/N1/T∑_i=1^N∑_t=1^Tx_i^t-d_i^t_2,

where d_i^t means ground truth position for agent i at time step t. Moreover, we integrate the proposed constraints L_E and L_D into our optimization. Above all, the loss function for our framework is shown below and λ_1, λ_2 are hyperparameters for scale balance, 

    ℒ = ℒ_P + λ_1 ℒ_E + λ_2  ℒ_D.















§ EXPERIMENTS




 §.§ Datasets


We validate our method on four datasets: PHASE <cit.>, Socialnav <cit.>, Charged <cit.>, NBA. For simulated datasets, we set the same simulation configurations with previous work for fair Comparison. The detailed description each dataset are as follows:

PHASE. The PHASE is a dataset of physically grounded abstract social events that resemble a wide range of real-life social interactions by including social concepts such as helping another agent. For comparison with previous work, we choose the “collaboration” task in that two agents collaborate to move one of the balls to a pre-set position. Note that we additionally take the label information indicating the difference between agents and balls as part of the input. We do the same data augmentation on the training set as previous work, by flipping the environment and rotating it by 90 degrees, 180 degrees, and 270 degrees, resulting in a training set with 8x more instances. We use 80% for training, 10% for validation, and 10% for testing, and predict 10 steps based on the observation of 24 steps. The timestep of this dataset is 0.25.

Social Navigation Environment. The Social Navigation Environment (Socialnav) dataset includes annotations for each trajectory, and it also includes annotations for social interactions between agents, such as whether two agents are nearby or whether one agent is following another. Complex unseen trajectories can be easily generated by varying different parameters and configurations. We follow the environmental configurations of previous work. In this simulation, agents are controlled by ORCA policy. We set the radius of agents to 0.3, the radius of the circular environment to 8, and the preferred speed of agents to 1. For Social Navigation we use 100k multi-agent trajectories, in total for training, validation, and testing. We also simulate 50k, 25k, 12.5k, and 10k samples to validate the performance of our model in different dataset sizes. We use 80% for training, 10% for validation, and 10% for testing, and predict 10 steps based on the observation of 24 steps. The timestep of this dataset is 0.25. 

Charged. The Charged datasets are usually used to illustrate the performance of graph relation prediction in previous work, as the relation type between agents can be set manually. We simulate a dynamic system controlled by physical laws. There are 5 charged particles in each scene, each particle has a positive or negative charge with equal probability, attracting the particles with different charges and vice versa. We experiment with the generalization of models on the Charged dataset by modifying configurations. We generate 50k trajectories for training, 10k trajectories for validation, and 10k trajectories for testing. In the basic environment, we set the side length of the box square to 5, the number of particles to 5, and the sampling timestep to 0.2. To compare with GRIN and previous work, we use our model to predict 20 steps based on the observation of 80 steps. The timestep of this dataset is 0.2.

NBA. The National Basketball Association (NBA) uses the SportVU tracking system to collect player-tracking data, where each frame contains the location of all ten players and the ball at each time step. We use 300k multi-agent trajectories of players and the ball in total for training, validation, and testing. We use 80% for training, 10% for validation, and 10% for testing, and predict 10 steps based on the observation of 24 steps. The timestep of this dataset is 0.4.




 §.§ Setup

Baselines. Following IMMA <cit.> and GRIN <cit.>, we compare our results with approaches conducting multi-agent trajectory prediction. In specific, we choose MLP, GAT-LSTM <cit.>, NRI <cit.>, EvolveGraph <cit.>, RFM <cit.>, fNRI <cit.>, IMMA, FQA <cit.>, dNRI <cit.>, Social-GAN <cit.>, and GRIN as baselines in quantitative evaluation. We choose GAT-LSTM and IMMA for qualitative analysis.

Hyperparameter Choice. For all datasets, we set λ_1 to 1 and λ_2 to 0.001 in Eq. (<ref>). λ_2 is set to a smaller value to balance the regularization term. When comparing with baselines, we only report this set of hyperparameters as they have the highest average performance across four datasets.

Evaluation Metrics. We quantitatively compare our model with other methods using the following metrics:
1) Average Displacement Error (ADE) is the L2 distance between the ground truth trajectories and predicted trajectories. 2) Final Displacement Error (FDE) measures the L2 distance between the ground truth of the destination and the predicted destination. 3) Graph Accuracy  (Graph.) measures the consistency between the predicted interaction graph and the input interaction graph. To ensure a fair comparison with previous methods <cit.>, we use Graph Accuracy to evaluate our model on the PHASE and Socialnav datasets since these datasets have the corresponding ground-truth social interaction graph. 

Implementation Details.
For generalization in the Socialnav dataset, we change the environment by setting the number of agents to 10 (doubling the number of agents) and setting the preferred speed of agents to 2 (doubling the speed of agents). For generalization in the Charged dataset, we change the environment by setting the number of agents to 10 (doubling the number of agents), setting the time step to 0.1 (halving the sampling timestep), setting the side length of the box square to 3.5 (halving the environment space size).

We do experiments for PHASE, Socialnav, and Charged datasets with Nvidia 2080Ti GPU, with Ubuntu 18.04 system. Our networks were implemented using the PyTorch framework. For Socialnav and Charged datasets, the input dimension is 2 (position for each agent) and it is 4 for the PHASE dataset (position and label for each agent). We train the model with Adam optimizer with learning rate 1 × 10^-3 and decay the learning rate by a factor of 0.9 if the validation performance has not improved in 5 epochs. We use a hidden size of 96, 128, and 128 for the Social Navigation Environment, PHASE, and the Charged dataset, and we use 4 latent graph layers for PHASE and Socialnav datasets (the dimension of edge latent z is 4) and 4 latent graph layers for the Charged dataset (the dimension of edge latent z is 2). 

As the size of the NBA dataset is much larger than the other three datasets, we do an experiment using Nvidia 3090 GPU, with Adam optimizer with learning rate 5 × 10^-4 and decay the learning rate by a factor of 0.9 if the validation performance has not improved in 5 epochs. We use a hidden size of 256 and 5 latent graph layers for the NBA dataset (the dimension of edge latent z is 5).

For implementing IMMA, RFM, RFM (skip 1), and GAT models for comparison in the Charged dataset, we use the default configurations mentioned in IMMA, that is, we use Adam optimizer with an initial learning rate of 1e-6 and decay the learning rate by
a factor of 0.9 if the validation performance has not improved in 5 epochs, and we use 2 latent graph layers (the dimension of edge latent z is 2).



 §.§ Results for Trajectory Prediction


Quantitative Evaluation. In Table <ref>, we report ADE, FDE, and Graph Accuracy for each model on PHASE and Socialnav datasets. Our model achieves the best performance on both datasets. Compared with the IMMA model, we achieved a relative improvement by 17.6%  13.7% in ADE  FDE on the PHASE dataset, respectively. On the Socialnav dataset, the improvement is 20.1%  17.9% in ADE  FDE. In the aspect of the Graph Accuracy metric, we compare the ability of each model to predict social interaction graphs. The results show that our model achieves an accuracy of over 90% on both datasets. It means that our method can not only accurately predict the future trajectories of each agent, but also clearly express the relationships between agents, which helps explain how agents affect other’s trajectories.
Our model and other models are also tested on Charged dataset and NBA dataset. Results are presented in Table <ref> and Table <ref>. As can be seen, our model yields the best performance compared with existing methods, indicating that our method is robust to both simulated and real-world scenarios.

Qualitative Analysis. Fig. <ref> visualizes the trajectory prediction results of our model on Socialnav dataset. Note that each agent has a target agent and moves towards its target. It can be seen that our model accurately predicts the future trajectory of agents with the assistance of their past information and interactions with others. 
We also visualize the prediction results of our method and others in Fig. <ref> on Charged dataset. It can be seen that interactions exist between charges that repel and attract each other. We accurately predict such interactions and future trajectories of charges. We visualize the results on NBA dataset in Fig. <ref> and show the trajectories of the ball and the players in different colored lines. 
See more visualizations of our model in the Appendix.












































 §.§ Ablation Study
 Table <ref> shows the results of our ablation study on PHASE dataset, where we verify the effect of different components of our model. Specifically, we compare the performance of our model with different variants: 1) MATE (base model), 2) MATE + ℒ_E (inter-agent interaction constraint), 3) MATE + ℒ_D (intra-agent motion constraint), and 4) MATE + ℒ_E + ℒ_D. We depict the results in Table <ref>. Compared with MATE (base model), our model performs better with the assistance of the inter-agent interaction constraint on PHASE dataset, indicating that modeling interactions between agents through neural interaction energy and constraining it as prior knowledge preserves a temporally stable system, improving the prediction accuracy and robustness of our model. Moreover, adding the intra-agent motion constraint further boosts the performance, indicating that punishing the incoherent motion prediction and connecting the relation between interactions and motions of agents helps to preserve the agent-level temporal stability.



 §.§ Generalization to Unseen Scenarios


Our method focuses on preserving temporal stability, which exists in various multi-agent systems. Consequently, our approach is robust to unseen scenarios. We conduct zero-shot experiments to present the generalization ability of our method. 
Table <ref> and Table <ref> depict the zero-shot experiments on Socialnav and Charged datasets, where we test the trajectory prediction accuracy of our model and other existing models to unseen situations.

On Socialnav dataset, we create two new environments by 1) doubling the number of agents and 2) doubling the speed of agents. As shown in Table <ref>, our model outperforms other methods in both environments, demonstrating that our model is generalizable to different scenarios of complexity and dynamics in the social navigation environment. 

On Charged dataset, we test our model on three novel situations: 1) doubling the number of charges, 2) halving the sampling timestep, and 3) halving the environment space size. As shown in Table <ref>, our model achieves the best performance in most situations. In Fig. <ref>, we also visualize the results of our model and others in the unseen situation by halving the environment space size. Though the trajectory prediction accuracy of each model drops, our model performs the best. This indicates that our model has a better generalization ability.



 §.§ Results of Different Dataset Sizes


To investigate the impact of different size datasets on our models, we simulate datasets of 50k, 25k, 12.5k, and 10k samples, and use the same network structure and hyperparameters, The result is in Fig. <ref>. Compared with the strongest baseline (IMMA), our model outperforms on all datasets of different sizes, and the prediction performance remains stable. 





§ CONCLUSION


In this work, we proposed a framework named MATE for multi-agent trajectory prediction, aiming at eliminating the prediction fluctuations and preserving the temporal stability of multi-agent systems. Our framework introduces the concept of neural interaction energy to model the interactions and motions of agents in a dynamic environment. Furthermore, we employ partial differential equations to establish the inter-agent interaction constraint and the intra-agent motion constraint, which preserve the temporal stability of the multi-agent system and improve the prediction accuracy of our model. Through quantitative and qualitative experiments, we evaluate our framework on four datasets and demonstrate that it outperforms the state-of-the-art methods in trajectory prediction. We also show the advantages of our framework in zero-shot generalization capacity. Our work opens up new possibilities for representing interactions between agents for multi-agent systems and illustrating the mechanism of how interactions influence agents' future trajectories.




































































































































































































































































































































































































































































































































ACM-Reference-Format



