Delay-Aware Multi-Agent Reinforcement Learning for Cooperative Adaptive Cruise Control with Model-based Stability Enhancement
Jiaqi Liu Ziran Wang Peng Hang Jian Sun


Spatial segregation of massive clusters in a simulation of colliding dwarf galaxies
    Bruce G. Elmegreen1,
Natalia Lahén2
    April 29, 2024
===================================================================================

empty
empty



Cooperative Adaptive Cruise Control (CACC) represents a quintessential control strategy for orchestrating vehicular platoon movement within Connected and Automated Vehicle (CAV) systems, significantly enhancing traffic efficiency and reducing energy consumption. 
In recent years, the data-driven methods, such as reinforcement learning (RL), have been employed to address this task due to their significant advantages in terms of efficiency and flexibility. However, the delay issue, which often arises in real-world CACC systems, is rarely taken into account by current RL-based approaches.
To tackle this problem, we propose a Delay-Aware Multi-Agent Reinforcement Learning (DAMARL) framework aimed at achieving safe and stable control for CACC. We model the entire decision-making process using a Multi-Agent Delay-Aware Markov Decision Process (MADA-MDP) and develop a centralized training with decentralized execution (CTDE) MARL framework for distributed control of CACC platoons. An attention mechanism-integrated policy network is introduced to enhance the performance of CAV communication and decision-making. Additionally, a velocity optimization model-based action filter is incorporated to further ensure the stability of the platoon. Experimental results across various delay conditions and platoon sizes demonstrate that our approach consistently outperforms baseline methods in terms of platoon safety, stability and overall performance.







§ INTRODUCTION

Connected and Automated Vehicles (CAVs) are poised to bring revolutionary changes to the entire transportation system. Cooperative Adaptive Cruise Control (CACC) represents a pivotal technology within CAVs, where each CAV is required to formulate an independent driving plan<cit.>. Through coordination with other CAVs, the technology aims to avoid potential collisions and achieve the objective of string stability in CACC platoons. By adaptively coordinating the platoon through vehicle-to-vehicle communication, CACC can significantly reduce the following distances and speed variations while maintaining safety, thereby enhancing traffic efficiency, alleviating congestion, and reducing energy consumption<cit.>.

In recent years, CACC technology has garnered considerable attention from researchers, leading to extensive exploration. Common methodologies include classical control theory and optimization-based approaches <cit.>.  
Specifically, some studies have focused on car-following models <cit.> and string stability <cit.>, modeling CACC within the context of two-vehicle systems.
Other researchers have conceptualized CACC as an optimal control problem <cit.>. These methods rely on precise system modeling, which is often unavailable <cit.>.

On the other hand, CACC platoon control has also been conceptualized as a sequential decision-making problem and modeled using reinforcement learning (RL) <cit.>.
RL has experienced rapid advancements in recent years <cit.>, originally proposed in the control domain within the Markov Decision Process (MDP) framework for optimal stochastic control under uncertainty <cit.>. 
This approach demonstrates remarkable efficiency and flexibility, making it a promising candidate for CACC platoon control<cit.>.


Peake et al. <cit.> have pioneered the application of deep reinforcement learning for CACC, enabling platoon vehicles to adopt a robust communication protocol and facilitating vehicle training through LSTM-based policy networks. Concurrently, Wang et al. explored the utilization of the policy iteration method to deduce parameters for the classical Proportional-Integral (PI) controller, sidestepping the need for direct longitudinal control <cit.>. Further, a novel information reward design was proposed to bolster the safety and robustness of the Q-learning technique <cit.>, while <cit.> introduced a policy gradient RL strategy aimed at preserving safe longitudinal distances between vehicles. 


However, it is noteworthy that, despite the significant attention and research on RL methods for CACC control problems, these approaches still rely on rather idealized assumptions in their modeling, such as perfect or delay-free communication and decision-making. This impedes the further deployment and application of these methods in the real world. In reality, communication delays within CACC, sensor latency, and delays in the execution of decision actions are common and inevitable<cit.>. Existing RL methods rarely consider these time-delay issues. Particularly for multi-agent problems like CACC, delays can exacerbate issues, as a delay in one agent may propagate to other coupled agents. This not only can lead to a decline in agent performance but also may disrupt the stability of the dynamic system, potentially causing catastrophic failures in safety-critical systems<cit.>.


To address these challenges, we propose a Multi-Agent Reinforcement Learning (MARL) framework that incorporates delay awareness and an enhanced attention mechanism to model the collaborative decision-making problem in CACC systems. Initially, we consider the multiple delays present within the CACC platoon and construct a Multi-Agent Delay-Aware  Markov Decision Process (MADA-MDP) to model this problem. Subsequently, based on the MADA-MDP framework, we employ a MARL approach characterized by Centralized Training with Decentralized Execution (CTDE) to model the entire CACC decision-making process. In this framework, each CAV is controlled by an independent policy network, with complete information sharing during centralized training, while allowing for independent decision-making by each CAV beyond necessary platoon communication to enhance platoon safety. 
The attention mechanism is employed in the policy network to enhance the information processing ability of CAVs. Furthermore, to address the complexities of platoon decision-making, we introduce an action filter based on a velocity optimization model to the MARL's decision output, ensuring platoon stability.

We validated our approach on CACC platoons of various sizes. Compared with several baseline methods, our method effectively handles internal platoon delay issues and demonstrates superior performance in terms of platoon safety and stability.

Our contributions can be summarized as follows:

    
  * We propose a Multi-Agent Markov decision process that considers the internal delays within CACC, and based on this process, we formulate the CACC  problem within a CTDE MARL framework.
    
  * We incorporate an attention mechanism-based decision-making policy network for CAVs and design a model-based action filter for each CAV to enhance the safety and stability of the platoon.
    
  * Our method is tested on fleets of various sizes within simulation environments, and compared to benchmark algorithms, it demonstrates outstanding performance.














§ PRELIMINARY
  

In this section, Markov Decision Processes(MDPs) under delay conditions and multi-agent scenarios are introduced.



 §.§ Delay-Aware Markov Decision Process


For decision systems where delay is not a consideration, the traditional Markov process (delay-free) can be employed for modeling. A traditional MDP is typically defined by a quintuple: MDP = ⟨ S, A, ρ, p, r ⟩<cit.>. Here, S denotes the state space, A is the action space, ρ represents the initial state distribution, p is the transition distribution, and r is the reward function. In RL, the MDP framework is often utilized to model the decision-making problem, with π representing the policy of an agent. The objective of RL is to find the optimal policy π^* that maximizes the expected cumulative reward ∑_t=0^Tγ^t r(s_t, a_t).
However, the traditional MDP framework's assumption of immediate action may lead to policies far from optimal in decision systems with delays, potentially causing significant performance degradation or safety issues.

To mitigate this problem and restore the Markov property, which means future states depend only on the current state and action), we introduce the Delay-Aware Markov Decision Process (DAMDP) to accommodate delays<cit.>.
The DAMDP represents an augmented variant of the traditional MDP, formally defined by the tuple ⟨𝒳, 𝒜, ρ̃, p̃, r̃⟩. Here, 𝒳 = 𝒮×𝒜^k encapsulates the state space, augmented to incorporate the action sequence to be executed over the next k steps, with k ∈ℕ representing the delay duration. The action space 𝒜 remains consistent with that of the original MDP. The augmented reward function is defined as r̃(𝐱_t, ã_t) = r(s_t, a_t), and the transition distribution is given by:

    p̃(𝐱_t+1|𝐱_t, ã_t) =     p(s_t+1 | s_t, a_t^(t)) ∏_i=1^k-1δ(a_t+i^(t+1) - a_t+i^(t)) ·
       δ(a_t+k^(t+1) - ã_t),

where the state vector 𝐱 of DAMDP includes an action sequence to be executed in the forthcoming k steps. The notation a_t_1^(t_2) denotes that the action is an element of 𝐱_t_2, with the subscript indicating the time of action execution, and the superscript specifying the reference time frame. δ is the Dirac delta function. If y ∼δ(· - x) then y=x with probability one.

The action ã_t, selected at time t within a DAMDP context, is slated for execution at time t+k, thereby accommodating the k-step action delay, such that ã_t= a_t+k. Furthermore, the policy network π for the agent is under continuous refinement to ascertain the optimal policy, adjusting dynamically to the intricacies introduced by the delay-aware framework.



 §.§ Multi-Agent Delay-Aware Markov Decision Process

Given the prevalence of various types of delays in real-world decision systems, we define the Multi-Agent Delay-Aware Markov Decision Process (MADA-MDP) based on DAMDP framework for subsequent modeling of CACC vehicle platoons. The MADA-MDP is articulated as the tuple ⟨𝒳, 𝒜, ρ, p, r ⟩, where 𝒳 = 𝒮×𝒜_1^k_1×…×𝒜_N^k_N represents the augmented state space, incorporating the action sequence to be executed over the next k_i steps for agent i, thereby denoting the delay step. The action space 𝒜 aligns with that in the traditional MDP.

The transition distribution is defined as:

    p(𝐱_t+1|𝐱_t, ã_t) = 
        p(s_t+1, a^1, (t+1)_t+1, …, a^1, (t+1)_t+k_1,…, a^N, (t+1)_t+1, …, a^N, (t+1)_t+k_N. 
       . | s_t, a^1, (t)_t, …, a^1, (t)_t+k_1-1,…, a^N, (t)_t, …, a^N, (t)_t+k_N-1, ã_t ) 
    
    =     p(s_t+1 | s_t, a^1, (t)_t, …, a^N, (t)_t) 
       ∏_i=1^N∏_j=1^k_i-1δ(a^i, (t+1)_t+j - a^i, (t)_t+j) ∏_i=1^Nδ(a^i, (t+1)_t+k_i - ã_t^i),

where the policy input for agent i at time t comprises two components: õ_t^i = (o_t, obs^i, o_t, act^i), with o_t, obs^i being the observation of the environment and o_t, act^i representing a planned action sequence of length k_i for agent i, to be executed from the current time step: o_t, act^i = (a_t^i, …, a_t+k_i-1^i).

Agents i take actions a_i ∈𝒜_i based on a policy π_i : 𝒳_i×𝒜_i→ [0,1], interacting with the MADA-MDP environment, which is augmented due to the extended dimensionality of state vectors resulting from action and observation delays. Notably, action and observation delays constitute similar mathematical challenges, as they both introduce a mismatch between current observations and executed actions, as proven in literature <cit.>. 
To ensure conciseness in our study, we have considered and transformed both the observation delay and action delay into a mathematical formula representing action delays.




§ PROBLEM FORMULATION FOR CACC


In this section, we introduce the vehicle dynamics model utilized for modeling CACC systems and formalize the CACC problem using the MADA-MDP.




 §.§ Vehicle Dynamics


As depicted in Fig.<ref>, we consider a platoon comprising 𝒱 CAVs traveling along a straight path. For the sake of simplicity, it is assumed that all CAV in the system possess identical characteristics, such as maximum permissible acceleration and deceleration. The platooning system is orchestrated by a platoon leader vehicle (PL, 1st vehicle), while the platoon member vehicles (PMs, i ∈{2, …, 𝒱}) follow the PL. Each PM i maintains a desired inter-vehicle distance (IVD) h_i and velocity v_i relative to its preceding vehicle i-1, adhering to its distinct spacing policy <cit.>. The one-dimensional dynamics for vehicle i are given by:

rCl
ḣ_i    =    v_i-1 - v_i,

v̇_i    =    u_i,
  

where v_i-1 and u_i denote the velocity of the preceding vehicle and the acceleration of vehicle i, respectively. Following the framework established in <cit.>, the discretized vehicle dynamics, with a sampling interval Δ t, are described by:

rCl
h_i, t+1    =    h_i, t + ∫_t^t + Δt (v_i-1, τ - v_i, τ) dτ,

v_i, t+1    =    v_i, t + u_i, t Δt.
  


To ensure both comfort and safety, each vehicle must adhere to the following constraints <cit.>:

rCl
h_i, t    ≥    h_min,

0    ≤    v_i,t ≤v_max,

u_min    ≤    u_i,t ≤u_max,
  

where h_min, v_max, u_min, and u_max represent the minimum safe headway, maximum speed, deceleration, and acceleration limits, respectively.



 §.§ Problem Formulation

Considering the prevalent issue of delays stemming from communication or decision-making processes within CACC systems in the real world, we model this problem using the MADA-MDP. The process is represented by the tuple ⟨𝒳, 𝒜,  ρ, p, r ⟩, where 𝒳 denotes the augmented state space, 𝒜 represents the action space, and the state transition distribution p characterizes the system's intrinsic dynamics.
Furthermore, we assume that each CAV is equipped with the capability to communicate with the vehicles ahead and behind via V2V communication channels.



  §.§.§ State Space

The state space encapsulates the environmental description. Following the definition of MADA-MDP, the observation spaces of all CAVs are augmented as 𝒳 = 𝒮×𝒜_1^k_1×…×𝒜_N^k_N. Here, the original state of agent i, 𝒮_i, is defined by [v, v_diff, v_h, h, u], where v denotes the current normalized vehicle speed. v_diff = (v_i-1,t - v_i,t) represents the vehicle speed difference with its leading vehicle. v_h = (v^∘(h) - v_i,t), h = (h_i,t + (v_i-1, t - v_i,t) Δ t - h^*) / h^*, and u = u_i, t / u_max are the headway-based velocity, normalized headway distance, and acceleration, respectively.



  §.§.§ Action Space

In the considered CACC system, the action a_t ∈𝒜 directly pertains to the longitudinal control. The action decision output for CAV i includes three components a_i,t=(α_i,t,β_i,t,û_i,t), where û_i,t is the acceleration control value output by the policy network, and α_i,t,β_i,t are control parameters for the action filtering layer, which will be elaborated upon in Section <ref>. The overall action space comprises the combined actions of all CAVs, i.e., 𝒜 = 𝒜_1×𝒜_2×⋯×𝒜_|𝒱|.



 §.§ Reward Function


To ensure that CAVs within the platoon simultaneously prioritize safety, efficiency, and comfort, our reward function is designed as follows:

    R_i, t = 1/C[ w_1 (h_i, t - h^*)^2 + w_2 (v_i, t - v^*)^2 + w_3 u^2_i, t],

where w_1, w_2, and w_3 are weighting coefficients, and C is the scaling coefficient. In this equation, the first two terms, (h_i, t - h^*)^2 and (v_i, t - v^*)^2, penalize deviations from the desired headway and speed, respectively, encouraging the agent to closely achieve these objectives. The third term, u^2_i, t, is included to minimize sudden accelerations, thereby promoting a smoother and more comfortable ride for passengers.



§ DELAY-AWARE MARL FOR CACC PLATOONS




In this section, we first introduce the framework of Delay-Aware MARL. Subsequently, we provide a detailed description of the policy network and the stability enhancement module.







 §.§ Delay-Aware Multi Agent Reinforcement Learning

Drawing from the findings of<cit.>, it is understood that MDPs under delay conditions do not necessitate direct modeling but can instead be directly solved through RL. Hence, based on MADA-MDP, we propose the Delay-Aware Multi-Agent Reinforcement Learning-CACC (DAMARL-CACC) decision framework. This approach employs a centralized training and decentralized execution paradigm, meaning that the vehicle platoon does not require a central controller, effectively mitigating the non-stationarity problem posed by multi-agent decision-making.

For each CAV in the platoon, we introduce a centralized Q-function Critic, based on global information, and an independent Actor that only requires communication and partial observations. The framework is illustrated in Fig.<ref>. In this framework, each CAV i is controlled by an independent policy network π_i, with π={π_1,…,π_N} representing the set of all policies, parameterized by θ={θ_1,…,θ_N}. The objective function's gradient for CAV i is updated using the deterministic policy gradient algorithm as follows:







    ∇_θ_i J(μ_i) =  
       𝔼_x, a ∼𝒟[∇_θ_iμ_i(a_i| o_i) ∇_a_i Q^μ_i (x, a_1, ..., a_N)|_a_i= μ_i (o_i)],

where policies π, states x, and observations o are augmented based on the MADA-MDP. In the delay-aware scenario, 𝒳 includes all CAVs' observations and the action sequences of all agents in the near future, 𝒳 = (o_1, ..., o_N), where o_i is the input to CAV i's policy π_i and consists of two parts: o_i = (o_obs^i, o_act^i). Here, o_obs^i is the i^th CAV's observation of the environment, and o_act^i is the planned action sequence of length k_i for agent i starting from the current timestep, e.g., at time t, o_t, act^i = a_t:t+k_i-1^i. o_act^i is obtained from the action buffer, serving as a bridge between the agent and the environment.

The replay buffer 𝒟 is used to record the historical experiences of all CAVs. The centralized Q function Q^μ_i for CAV i is updated as<cit.>:


    ℒ(θ_i) = 𝔼_x, a, r, x'[(Q^μ_i(x, a_1,…, a_N) - y)^2],

where y = r_i + γ Q^μ'_i(x', a_1',…, a_N')|_a_j'= μ'_j(o_j), and μ' = {μ_θ'_1, …, μ_θ'_N} representing the target policy set with softly updated parameters θ'_i to stabilize training.



 §.§ Attention-based Policy Network

Recent advancements have underscored the efficacy of attention mechanisms in enabling neural networks to prioritize the processing of the most pertinent input information, thereby augmenting the model's computational efficiency <cit.>. In light of these findings, we have developed an attention-based policy network for each CAV, as delineated in Fig. <ref>. This network is structured around three core components: the encoder block, the attention block, and the decoder block.


Encoder Block: The features 𝒳 of agent i are transformed into high-dimensional vectors through the application of a Multilayer Perceptron (MLP), with a uniform set of weights shared across all vehicles:

    𝒵_i = MLP(𝒳_i)

Subsequently, the feature matrix is channeled into the attention block, which is composed of N_head stacked attention heads.

Attention Block: Each self-vehicle generates a singular query Q_i=[q_0] ∈R ^1× d_k, aimed at identifying a subset of vehicles based on the environmental context, where d_k denotes the output dimensionality of the encoder layer. This query is subjected to a linear projection and juxtaposed against a set of keys K_i=[k^0_i,k^1_i, …,k^N_i]∈R^(N+1)× d_k, each encapsulating descriptive features of individual vehicles. The similarity is computed using the dot product q_0 k_i^T. The components Q_i, K_i, and V_i are determined as follows:

    Q_i = W^Q 𝒵_i 
        K_i = W^K 𝒵_i 
        V_i = W^V 𝒵_i

where the dimensions for W^Q and W^K are configured as (d_k× d_N), and that for W^V is (d_v × d_h).
The computation of the attention matrix involves scaling the dot product by the inverse square root of the dimension 1/√(d_k) and its normalization using a softmax function σ. This process facilitates the aggregation of a set of output values V_i=[v^0_i,…,v^N_i], wherein each v^j_i represents a feature derived through a shared linear projection L_v ∈R^d_x × d_k. The attention calculation for each head is encapsulated by:

    ω_i^m = σ (  Q_i K^T_i/√(d_k))V_i


The aggregated output from all M heads is unified through a linear layer:

    ω_i = ∑_m=1^Mω_i^m


The resultant attention vector ω_i=[ω_i,1, ω_i,2,…, ω_i,|𝒩_i|] quantifies the degree of attention agent i allocates to the surrounding vehicle j, conforming to the cumulative summation condition: ∑^|𝒩_i|_j=1ω_i,j = 1.

Decoder Block: Leveraging the attention-weighted features ω_i alongside the encoded state information 𝒵_i, the decoder employs a MLP to deduce the final action vector:

    a_i = MLP(𝒵_i, ω_i)

Then the action a_i is checked by an action filter to ensure the security and stability, which is described in the next subsection.



 §.§ Platoon Stability Enhancement with Model-based Controller


Given MARL's data-driven essence, we introduce an action filter layer comprising a velocity optimization model to enhance the safety and robustness of CACC platoons by ensuring safe acceleration for trailing CAVs.

The velocity optimization model computes a safe acceleration, termed the ideal acceleration u'_i,t for the i^th CAV, based on control parameters:

    u'_i,t = α_i (v^∘ (h_i, t; h^s, h^g) - v_i, t) + β_i (v_i-1, t - v_i,t),

where α_i and β_i denote the headway gain and relative speed gain, respectively, influencing vehicle acceleration. Here, h^s and h^g symbolize the stopped and full-speed headway distances.

The headway-based velocity strategy v^∘ is defined as:

    v^∘ (h) ≜
    0,    if  h < h^s,
    1/2 v_max (1 - cos(πh - h^s/h^g - h^s)),    if  h^s ≤ h ≤ h^g,
    
    v_max,    if  h > h^g.

This function outlines an optimal velocity based on the current headway, with velocity set to zero for headways ≤ h^s to prevent collisions, gradually increases within h^s to h^g, and is capped at v_max for headways ≥ h^g, ensuring traffic flow safety and efficiency.

The action output by each CAV at time t, a_i,t = (α_i,t, β_i,t, û_i,t), comprises the CAV's acceleration control value û_i,t, determined by the policy network, and α_i,t, β_i,t, the control parameters for the action filter layer.
Upon action generation by the policy network, the action filter layer computes the ideal acceleration u'_i,t using parameters α_i,t, β_i,t and Equation (<ref>). It then selects the acceleration action u_i,t^* that maximizes benefits according to the reward function:

    u_i,t^* =  
        u'_i,t   if  R(u'_i,t) ≥ R(û_i,t), 
    û_i,t   otherwise,

where R(·) denotes the reward function specified in Equation (<ref>).










For a detailed description of the entire algorithm, see Algorithm<ref>.





§ EXPERIMENTS AND ANALYSIS




 §.§ Simulation Settings

Scenarios. 
In our study, we devise two distinct scenarios, namely Catchup and Slowdown, to evaluate the efficacy of our methods. In the Catchup scenario, PMs indexed by i = 2, …, 𝒱 are initialized with velocities v_i, 0 = v^*_t and headways h_i, 0 = h^*_t. Conversely, the PL begins with v_1, 0= v^*_t and h_1, 0 = a · h^*_t, where a represents a random variable uniformly distributed between 3 and 4. The Slowdown scenario, on the other hand, initializes all vehicles, indexed by i = 1, …, 𝒱, with initial velocities v_i, 0 = b · v^*_t and headways h_i, 0 = h^*_t, with b uniformly distributed between 1.5 and 2.5. Herein, v^*_t linearly decreases to 15 m/s within the first 30 seconds and subsequently remains constant. For the dynamics model of CACC in subsection <ref>, we set h_min = 1 m, v_max=30 m/s, u_min = -2.0 m/s  2, and u_max = 2.0 m/s 2.

Delay Setting.  
In the real world, the predominant delays encountered in CACC encompass communication delay τ_1, sensor delay τ_2, and actuator delay τ_3. We approximate the total delay τ as the cumulative sum of these components: τ=τ_1+τ_2+τ_3. Based on settings from the literature regarding communication and sensor delays<cit.>, we configure the total delay τ to be 0.5 seconds.

Baselines.
For comparative analysis, we select three cutting-edge MARL algorithms as baselines, including MADDPG<cit.>, ConseNet<cit.>, and NeurComm<cit.>. The critics for all these algorithms are fully decentralized. ConseNet and NeurComm specifically incorporate designed communication protocols to facilitate enhanced decision-making among agents.



 §.§ Implementation Details

We set the hyperparameters w_1, w_2, and w_3 in Equation (<ref>) to -1, -1, and -0.2, respectively, with C=15. The simulation spans T=60 seconds with an interaction period Δ t=0.1 seconds, allowing the environment to progress Δ t seconds per MDP step. And the delay planned action sequence length for each CAV i is calculated as: k_i =⌊τ/Δ t⌋=5.
The Encoder and Decoder in the attention-based policy network are MLP with two 64 × 64 linear layers. The Attention Layer has two heads and a 128 feature size. For the model-based controller, h^s = 5m and h^g = 35m.
We train over 1M steps for each model, with the discount factor γ of 0.99. The learning rates are set to 5.0 × 10^-4 for the actor and 2.5 × 10^-4 for the critic.
Each algorithm undergoes tripartite training with different random seeds for generalization. All experiments run on a platform with an Intel Xeon Silver 4214R CPU and NVIDIA GeForce RTX 3090 GPU.



 §.§ Experiment Results



  §.§.§ Performance Analysis

Fig.<ref> compares the learning curves of our method against other baseline methods for a CACC platoon with a size of 8. ConseNet demonstrates relatively better performance among the baseline algorithms, whereas MADDPG and NeurComm struggle to converge stably in the Catchup scenario. In contrast, our method exhibits higher learning efficiency, more stable decision-making, and superior overall performance across both scenarios.

After training completion, we assess all algorithms over 50 trials under varying initial conditions, using metrics such as average headway, average velocity, and collision counts. Tab.<ref> presents these evaluation results, showing our method consistently outperforms the other baselines across different scenarios. Notably, in the more challenging Slowdown scenario, MADDPG and NeurComm fail to ensure safety throughout the tests, ConseNet performs relatively better, but our method achieves the lowest collision rate.

Moreover, taking the Catchup scenario as an example, we plot the headway and velocity changes of CACC vehicles controlled by different methods. NeurComm leads to collisions around 25 seconds, causing task failure. MADDPG and ConseNet manage some level of control, yet they fall short of achieving the optimal control target within 60 seconds. Our method, however, achieves the target headway and velocity in the shortest time with the best stability.


















  §.§.§ Impact of Platoon Size

We also investigate the impact of platoon size on our model's performance under delay considerations. 
The experiments are conducted across different platoon sizes (sizes={5,8,10}). 

Tab. <ref> displays the performance of various methods during the testing phase.

It is observed that with an increase in platoon size, which amplifies the difficulty and complexity of coordination among CAVs, most algorithms exhibit a performance decline in both scenarios. 

However, our algorithm consistently outperforms all other baseline methods in every scenario, with its advantage being even more pronounced in the Slowdown scenario.





§ CONCLUSION


CACC technologies are crucial for enhancing traffic efficiency and reducing energy consumption. Existing reinforcement learning-based CACC control methods have not accounted for the widespread issue of delays. To address this challenge, we proposed a delay-aware MARL framework. Leveraging a Delay-aware Markov Decision Process, we introduced a CTDE MARL framework to implement distributed control of CAVs within CACC platoons. An attention mechanism-integrated Policy network and a velocity optimization model-based action filter were designed to improve the model's performance. Our approach was experimentally validated on platoons of various sizes, demonstrating superior performance in scenarios with delays compared to baseline methods.

In the future, we aim to delve deeper into the specific types of delays within CACC platoons, such as communication and control delays, and design personalized modules to address these delays separately. Additionally, we plan to conduct experiments in more complex mixed traffic scenarios to enhance the adaptability of the model.

IEEEtran 




