A Survey on Intermediate Fusion Methods for Collaborative Perception Categorized by Real World Challenges
Melih Yazgan Thomas Graf Min Liu J. Marius Zoellner












A Survey on Intermediate Fusion Methods for Collaborative Perception Categorized by Real World Challenges
    Melih Yazgan23⁎,
Thomas Graf3⁎,
Min Liu3⁎, Tobias Fleck23 and J. Marius Zöllner23

2FZI Research Center for Information Technology, Germany

3Karlsruhe Institute of Technology, Germany
 
    25 April 2024
==============================================================================================================================================================================================



⁎ These authors contributed equally. 















This survey analyzes intermediate fusion methods in collaborative perception for autonomous driving, categorized by real-world challenges. We examine various methods, detailing their features and the evaluation metrics they employ. The focus is on addressing challenges like transmission efficiency, localization errors, communication disruptions, and heterogeneity. Moreover, we explore strategies to counter adversarial attacks and defenses, as well as approaches to adapt to domain shifts. The objective is to present an overview of how intermediate fusion methods effectively meet these diverse challenges, highlighting their role in advancing the field of collaborative perception in autonomous driving.









Autonomous driving, collaborative perception, intermediate fusion, V2X communication






§ INTRODUCTION



Highly automated vehicles greatly benefit from their many cameras and 3D sensors. While individual perception integrates outputs from various sensors on the ego vehicle for independent object detection<cit.>, it faces constraints in complex scenarios characterized by occlusions, distant objects, and sensor noise<cit.>. Collaborative perception can mitigate these limitations by gathering additional perceptual information from other vehicles and infrastructure, a process facilitated by Vehicle-to-Everything (V2X) communication methods<cit.>. This includes Vehicle-to-Vehicle (V2V)<cit.> and interactions with roadside infrastructures, referred to as Vehicle-to-Infrastructure (V2I)<cit.>, both of which contribute to the optimization of safety, reliability, and efficiency in autonomous driving<cit.>.

To implement collaborative perception, it is crucial to integrate data from collaborators into the ego vehicle through fusion techniques. There are three main fusion types: Early Fusion, Late Fusion, and Intermediate Fusion.
Early fusion involves sharing raw sensing data at the input level into a unified coordinate system<cit.>. Although this approach ensures a high level of accuracy, the volume of data to be transmitted is excessively large and sensitive to noise and delay <cit.>. Realizing robust raw data fusion by using current V2X communication capabilities is impossible.
Late fusion methods transmit detection results, representing the output of deep learning models <cit.>. This approach minimizes communication overhead due to the reduced data size, but the associated gains in detection accuracy are constrained.
In intermediate fusion method, feature maps generated by deep learning models are shared <cit.>. This feature-level sharing approach enables achieving a remarkably high level of accuracy, accompanied by a reduction in communication cost <cit.>. Moreover, the shared intermediate features exhibit increased resilience to GPS inaccuracies and communication latencies<cit.>.

Previous research <cit.> have provided overviews of various methods and datasets related to collaborative perception. However, these works cover a broad spectrum of topics and deal with individual concepts only on the surface.

This work focuses on intermediate-level collaborative perception methods and provides a categorized overview of these methods into real-world challenges.

The paper is organized as follows: Section <ref> systematically analyzes the challenges, and presenting state-of-the-art collaboration methods in Tables <ref>-<ref>. Section <ref> concludes with a summary and discussion about open challenges for future research.




Another reason for considering more sensors is that they all
have different characteristics. This means that their accuracy
may change under different conditions. For example, our
results showed that camera can detect non-DSRC equipped
vehicles. Yet, the accuracy of the deceleration estimate was not
very precise. Similarly, if we want to extend the concept to
vehicles that are outside of our line of travel, the localization
accuracy of the camera may not be very reliable either. The
reason is that camera is very good at estimating lateral (leftright), but not as good at longitudinal (distance) offset. On the
other hand, radar is exactly opposite. Thus, fusing the results
from those two sensors would be more accurate than either one
individually. 

OLD:

Autonomous driving
Disadvantage Single Agent
Advantage Collaborative Perception
Collaborative Perception explanation
collaboration methods (V2V, V2X, V2I)
fusion methods
disadvantage early fusion, late fusion
advantage intermediate fusion
ideal scenarios vs real world issues
contribution: other surveys, whats new
sectionwise

 Also, if you want to work offline or in your own IDE, you can simply 'git clone' this project via the menu on the left. Test of all the citations
<cit.>
<cit.>
<cit.>

If you refer to figures/tables and sections, please use the commands ref and secref as shown here: <ref> and sec:acknowledgment. If you want to link to your open-source repository, please use a href

If you want to include tables, the <https://www.tablesgenerator.com/> is a helpful tool. When you use acronyms, the acronym package automatically makes sure that every acronym is spelled out the first time you use it: First time: ad and second time: ad.

Research Gap.//Present Technology limitations//
Gap statement/research question//Why autonomous vehicles aren't still working//. A research gap is a question or a problem that has not been answered by any existing studies or research within your field. Sometimes, a research gap exists when there is a concept or new idea that hasn't been studied at all. The literature review is a core element of your thesis and shows you can work scientifically. As you explain what other researchers have found on your topic, the reader will realize that you know this topic extremely well. This will build trust that you can provide a piece of work yourself that is scientifically relevant.

Contribution.
 Contribution. Identify the core contribution. Before you start writing anything, it’s important to identify the single core contribution that your paper makes to the field. Anyone who spends even very little time reading the paper needs to understand the contribution.




an image here could be nice and represent one or more challenges, like <ref> or look at img "V2XBasedPerception" in img Folder. But isn't a must.

One key advantage of intermediate fusion over sharing raw point clouds is the marginal accuracy loss after compression <cit.>

Tested with four state-of-the-art LiDAR-based 3D object detectors, intermediate fusion can
outperform all other fusion methods and achieve state-of-the-art performance even under large compression rates. <cit.>

So far, most of the deep feature extraction is conducted by CNN, such as [13], [14], [75], because the CNN-based feature is highly related to the local spatial information. Very recently, some studies have applied transformers as the deep feature extractor [19], [78] due to their capability for panoramic feature extraction. <cit.>

So far there are different surveys like <cit.> <cit.> <cit.> <cit.> <cit.>

Towards... investigate the model robustness against various noises, including communication latency, localization errors, lossy communication, and mixed noises, where noises are generated t <cit.>

There are also hybrid collaboration methods like <cit.>, <cit.>, <cit.>, and <cit.> as well as other forms like <cit.>

- Collaborative Perception: Was ist CP
- Single Agent vs Collaborative (Warum CP)
- Standards V2V, V2X, V2I (welche Arten)
- Warum Intermediate (Vorteile I, Nachteile E und L)
- Contributions
- Sectionwise (Erklärung was gemacht, Tabelle erwähnen)

CHAPTER RELATED WORKS

3 Types of Sensor fusion


  * Input-Level Sharing (Early Collaboration) 

  * Output Level Sharing(Late Collaboration)

  * Feature Level Sharing(Intermediate)




 §.§ Input Level Sharing



This paradigm involves the exchange of raw sensing data, serving as the primary input to deep learning models. Although offering substantial accuracy gains, the abundance of data leads to notable communication overhead. Strategies to mitigate this overhead are discussed.



 §.§ Output Level Sharing



Sharing detection results, the output of deep learning models, characterizes this paradigm. Due to the inherently reduced data size, this approach minimizes communication overhead. However, the associated accuracy gains are relatively constrained.



 §.§ Feature Level Sharing



The intermediate feature maps generated within the deep learning models constitute the essence of this sharing approach. A remarkable potential for improvement in accuracy is identified through an effective fusion of feature maps, accompanied by a reduction in communication overhead.

Fusion with Reinforcement Learning
https://arxiv.org/abs/2004.10927Cooperative perception with deep reinforcement learning for connected vehicles

Early Methods: F-Cooper, FS-COD

The authors of Cooper however proposed an extrapolation of the deep fusion scheme in F-Cooper in which the raw data from a laser scanner start being processed in a neural network. The authors tried using the feature at a different level: the voxel feature level and the spatial feature level. The first one shares a 3D grid containing the result of the VoxelNET neural network while the other one shares a higher-level feature from the fusion of spatial features maps. While the first one generates a large amount of data, the spatial feature level is sparser, thus lighter, facilitating the exchanges in a bandwidthlimited environment. Similarly, Marvasti et al. in FS-COD propose a method to share deep features from an intermediary layer of a neural network. However, such an approach brings the question of standardization of the perception pipeline among every user especially on the evolution of the neural network <cit.>

PillarGrid is the first deep-fusion-based cooperative perception approach that combines point cloud data from an onboard LiDAR and a roadside LiDAR


There are also hybrid collaboration methods like <cit.>, <cit.>, <cit.>, and <cit.> as well as other forms like <cit.>
Also Hydro-3D

Benchmarking cite: Hydro-3D/V2X-ViT/CoBEVT





§ CHALLENGES AND GAPS





 §.§ Transmission Efficiency




Addressing transmission issues due to bandwidth and computing limitations in collaborative perception is crucial. Recent advancements, presented in Table <ref>, focus on reducing communication bandwidth and transmission latency while maintaining detection performance, achieved through compression methods, selective communication, or a combination of both. Collaborative perception generally requires trade-offs between detection quality and the needed transmission bandwidth.

Compression Methods try to condense the raw input data into an intermediate representation that still encodes the necessary information to perform a downstream task, e.g., object detection. Compression models are typically evaluated by investigating two metrics in parallel: a downstream task metric like the object detection quality and the data transmission size. While data transmission size is sometimes represented on an absolute(logarithmic) scale for visualization purposes<cit.>, the actual metrics of importance are the Average Byte (AB) and the compression ratio. The AB metric, calculated by representing each element in the transmitted data as a 32-bit float, offers a precise measure of the transmission cost, excluding ancillary data like calibration files and timestamps<cit.>. Alongside this is the compression ratio, which is the ratio between the input data size and the output size of the compression model.
V2VNet <cit.> employs a compression approach where sensor data is first transformed into a condensed intermediate form. This data is then compressed using a Convolutional Neural Network (CNN) and adapted variational image compression algorithm. The process concludes with quantization and entropy encoding. Li et al.<cit.> presents a student-teacher approach named DiscoGraph. A notable aspect of DiscoGraph is its operational efficiency during inference, requiring only the student model, DiscoNet. It reduces communication volume by employing 1×1 convolutional autoencoder and outperforms V2VNet<cit.> by higher compression rates in detection accuracy. AttFusion <cit.> maintains performance at high compression rates. An encoder architecture, featuring multiple 2D convolutions and max pooling, is applied to compress information into a feature map. Experimental results indicate that at a compression rate of 4096, the performance decreases by approximately 3%, which remains superior to early and late fusion methods. V2X-ViT <cit.> utilize multiple 1×1 convolutions for efficient feature map compression, demonstrating high detection performance at various compression rates against <cit.>.
FFNet<cit.> incorporates a feature flow prediction module with compression. Two compressors are employed for the feature and its derivative, each utilizing three Conv-Bn-ReLU blocks. It reduces the transmission cost by maintaining better detection accuracy than early, late fusion, and intermediate methods such as <cit.>.
VIMI <cit.> framework utilizes a two-stage compression approach. It employs a compression module incorporating both channel and spatial compression blocks. The transmission cost decreases to a level comparable to late fusion. Scalable Feature Compression <cit.> further enhances transmission efficiency by employing scalable multiple channel reduction modules. It compresses intermediate features into a layered bitstream. These modules include a 1×1 convolution layer for channel reduction before entropy coding and a 5×5 convolution layer for channel and spatial dimension reduction of the lower scalable layers.

Selective Strategies try to optimize the bandwidth and channel usage by carefully selecting agents to communicate with and what type of information to send to other agents. The evaluation metrics except compression ratio are the same as we described in compression. 
Who2com <cit.> introduces a three-stage handshake communication mechanism that divides the communication process into request, match, and connect stages. While Who2com operates on the premise that every agent must consistently communicate with one other agent, which leads to inefficient bandwidth usage, When2com<cit.> presents a model that learns to choose when to communicate and to construct communication groups. In the case of a single requesting agent, When2com outperforms Who2com by achieving a 50% reduction in bandwidth requirements and a higher detection accuracy. Where2comm <cit.> minimizes bandwidth using a spatial confidence-aware system. The system utilizes the spatial confidence map to identify the most informative spatial areas in the feature map and determines the optimal collaboration partners. Additionally, it only communicates non-zero features along with their corresponding indices. It shows better detection accuracy-bandwidth performance compared to methods<cit.>.
Cooperverse <cit.> employs dynamic feature sharing to optimize communication bandwidth by adjusting shared feature cells based on specific bandwidth requirements. Additionally, it filters to prioritize features based on their spatial distance from the sensor. CoCa3D <cit.> optimizes communication efficiency by selecting the optimal messages and demonstrates superior performance against methods <cit.>. Learn2com <cit.> learns to represent feature importance through attention weights, utilizing these weights during inference to determine the most suitable infrastructure for communication. A further framework that reduces communication volume by choosing the messages that contain the most relevant information is How2comm<cit.>. This method introduces a mutual information-aware communication framework that implements spatial-channel attention queries. A spatial query examines the features, while a channel query selects the most valuable channels. It then identifies the required messages and integrates them into a feature map. It shows better detection accuracy-bandwidth performance compared to methods <cit.>.
Differently, CenterCoop <cit.> addresses overlapping sensor observations by encoding Bird's Eye View (BEV) frames into compact representations. The method prevents duplicate detections arising from overlapping sensor features. Compared to Where2comm<cit.>, it shows slightly better detection performance in drastically reduced bandwidth. Another multi-model approach, BM2CP <cit.>, achieves communication efficiency by selectively transmitting only the most critical features identified through a binary confidence mask. It shows better performance than the methods <cit.>.
UMC <cit.> reduces communication costs by employing an entropy-based selection mechanism that transmits only the most informative regions from multi-resolution data. This approach ensures efficient bandwidth usage by filtering out lower-quality regions, effectively balancing the transmission of broad-scale and detailed information while maintaining superior detection accuracy-bandwidth performance compared to other methods.

Combined Approaches aim to enhance both efficiency and
performance by integrating various methods. In our review, we explore the integration of compression and selection. MASH<cit.> combine agent selection and compression, using a 2D spatial handshaking mechanism based on Who2com<cit.> for information exchange and bandwidth management, while 1x1 convolutional encoders minimize communication volume. FPV-RCNN <cit.> combines feature selection from generated proposals with 3D data compression, transmitting less data than a 2D BEV fusion while maintaining superior detection performance. By employing a channel-wise encoder and decoder to reduce feature maps from 128 to 32 channels, Slim-FCP <cit.> effectively filters out non-essential features. It combines attention-weight and semantic-information channel selection to minimize redundancy and an irrelevant feature remover to eliminate non-critical features. What2comm <cit.> framework, a feature decoupling mechanism is employed to separate data into exclusive and common features. Following this, importance maps identify the most critical regions within these features. After this identification process, a compression module is integrated to compress these selected regions efficiently. Compared to model Where2comm<cit.>, it attains a superior performance-bandwidth trade-off across all tested bandwidth options.



   


 §.§ Localization and Pose Error

 


Localization and pose errors in collaborative perception can lead to feature map misalignment and performance drops. These effects result from errors of the used localization systems, e.g., GPS inaccuracies, sensor noise, temporal misalignment, synchronization errors, or pose drift. Methods addressing these issues are categorized in Table <ref>. All these models are typically evaluated using a quality metric of a downstream task, e.g., object detection accuracy, by comparing it to the detection quality under noise. Noisy Settings are generally generated from a Gaussian distribution. Only in RobustV2Vnet<cit.> is rotational noise generated with von Mises distribution.

Error Correction is a categorization focused on methods that proactively address and rectify errors at each agent or sensor level before the information fusion. Vadivelu et al.<cit.> proposed RobustV2VNet, an end-to-end learnable pose error correction network. This framework features a pose regression module designed to correct pose errors by learning a parameter for adjusting noisy relative transformations. Additionally, they introduce a consistency module that employs a Markov random field to refine these predictions, ensuring a globally consistent absolute pose across all agents. FPV-RCNN<cit.> introduced a localization error correction mechanism
that classifies keypoints into various environmental elements and selectively shares their coordinates. The crucial part of the module is the maximum consensus algorithm, used to identify corresponding vehicle centers and pole points, aiding in precise localization error estimation. In contrast, CoALign<cit.> introduced an agent-object pose graph optimization without requiring accurate pose supervision in training. It aligns relative poses among agents and detected objects, promoting pose consistency. It operates without requiring accurate pose supervision in training, showcasing strong generalization to various levels of pose errors.
FeaCo<cit.> presents a pose-error rectification module to combat pose errors in noisy settings. The module operates in two pivotal stages. The first, the localization stage, involves creating a spatial confidence map to facilitate the selection of critical regions for proposal matching on the feature map, a crucial step for alignment precision. In the second stage, the matching phase, it calculates a fine-grained transformation matrix, which is essential for aligning feature maps and ensuring consistent synchronization among agents. Most recently, BM2CP<cit.> introduced BM2CP-robust, which adjusts the relative pose prior to sensor fusion. It calculates local bounding boxes and their uncertainty for each agent based on LiDAR voxel features. Following this, it utilizes the agent-object pose graph optimization from CoAlign<cit.> for each agent to enhance the refinement of the relative pose.

Spatio Temporal Collaboration plays a crucial role in error correction and addressing misalignments, particularly at the information fusion stage. This category of methodologies is designed to tackle the challenges of integrating data from multiple sources, ensuring that spatial and temporal discrepancies are effectively corrected. V2X-ViT<cit.> addresses GPS-related localization errors using its Multi-Scale window attention module (MSwin), which captures diverse visual information for spatial accuracy. Additionally, its Spatio-Temporal Correction Module (STCM) corrects global misalignments by warping feature maps for better data alignment. Differently, Select2Col<cit.> introduced a Historical Prior Hybrid Attention (HPHA) information fusion algorithm. It utilizes a short-term attention module to capture the correlation of semantic information from the temporal dimension. HPHA determines incorrect features using historical feature information, reducing spatial localization noise.
Another method that uses historical information is SCOPE<cit.>. It ensures that the ego agent integrates confidence-aware spatial information by utilizing a cross-attention module. Concurrently, an adapted LSTM module extracts contextual dependencies from historical frames. Continuing this trend, What2comm<cit.> introduced a Spatial Attention Integration (SAI) and a Temporal Context Aggregation (TCA) module. The SAI component, leveraging a cross-attention module, effectively aligns position embeddings of features with their associated importance maps. This process allows for the extraction of perceptually crucial information. Concurrently, the TCA module integrates historical feature representations via a gating mechanism, substantially mitigating collaboration noise. Its robustness presented compared to V2X-ViT<cit.>, V2VNet<cit.>, and Where2Comm<cit.>. Unlike previous methods that relied heavily on historical data, How2comm<cit.> presented a Spatio-Temporal Collaboration Transformer (STCFormer) along with a flow-guided delay compensation strategy. The STCFormer integrates spatial and temporal data from multiple agents, enhancing perception accuracy even amidst spatial misalignments caused by localization errors. The flow-guided strategy predicts future states of collaborating agents to maintain feature alignment.



 §.§ Communication Issues


    

    
Communication delays and interruptions, such as network congestion and signal loss, are critical challenges in collaborative perception. Various methods, shown in Table <ref>, have been developed to address these issues, focusing on latency, Lossy Communication (LC), and Non-Line-of-Sight (NLOS) conditions.
    
Latency is defined by the time difference between recording the sensor data in one agent and the time after de-serialization is finished at the second agent. Latency is highly influenced by (de-)serialization speed and inference speed of models. Since latency is inevitable, methods that register multiple sources with different latency are of particular interest. The performance of the introduced models is typically measured by a downstream task quality metric, e.g., object detection accuracy. The method is considered valid if the downstream task metric improves when compensating latency.
V2X-ViT <cit.>, besides transmission efficiency and localization errors, integrates adaptive delay-aware positional encoding, capturing temporal information to ensure accurate synchronization despite delays. Differently, SyncNet<cit.> introduces a novel latency-aware collaborative perception system that allows it to serve as a plugin latency compensation module for various collaborative perception methods. It synchronizes perceptual features across agents to a standard time stamp, which enhances collaboration. SyncNet simultaneously estimates real-time features and collaboration attention. This marks a distinctive departure from traditional methods. Historical information is harnessed to achieve synchronization, with dual-branch pyramid Long Short-Term Memory (LSTM) networks capturing spatial features at multiple scales. In addition to historical information, flow maps are utilized to address latency. FFNet<cit.> proposes a self-supervised approach to train a feature flow generator. The generator predicts future features and compares them with the ego vehicle's sensor data. It employs a linear operation for prediction, effectively addressing temporal fusion errors across different latencies and compensating for uncertainties in latency. Experiments demonstrate the robustness of FFNet to various latencies. To correct the feature errors caused by latency, CoBEVFlow<cit.> generates a flow map from BEV. In this process, related regions of interest are matched based on an irregularly sampled sequence of messages sent by the same agent. Using the BEV flow map, the estimated feature map is then determined to reallocate the asynchronous features. Compared to SyncNet<cit.>, V2X-ViT<cit.>, CoBEVFlow achieves the best performance under various time delays.
    
LC refers to the data transmission resulting in lost or altered information, which is another significant challenge. LC can lead to losing a data frame from a specific timestamp or just blocks of the data, e.g., spatially correlated subblocks of a feature map or grid structure.
V2VAM+LCRN <cit.> introduces an LC-aware repair network and a V2V attention module, focusing on recovering incomplete shared features and enhancing vehicle interaction. The effectiveness of the proposed method is shown compared to V2X-ViT<cit.> using object detection accuracy in lossy conditions. V2X-INCOP <cit.> uses historical data to recover information lost during communication interruptions, stabilizing the model's training under imperfect communication conditions. The 3D detection performance of the model in various packet drop rates outperforms the methods such as V2X-ViT <cit.> and Where2Comm <cit.>.

NLOS conditions, are characterized by the absence of direct visual contact between the transmitter and receiver, are addressed by APMM+MoHeD <cit.>. It combines matrix matching, which transforms and broadcasts sensor data into a matrix with vehicle coordinates, enabling the detection of overlapping blind zones and an optimized relay node selection strategy. This strategy prioritizes nodes based on mobility and height, enhancing packet reception, reducing collision risks, and minimizing the need for relay re-selection.



 §.§ Heterogeneity



Heterogeneity within collaborative perception presents a multifaceted challenge that spans models, tasks, sensors. While enriching the system with a broad spectrum of information and capabilities, this diversity also introduces significant complexities in integrating and interpreting varied sources and types of information. Methods addressing these issues are summarized in Table <ref>.

Model Heterogeneity necessitates the alignment of intermediate states from different models before their information can be fused due to the inherent differences in the characteristics of these states across models. Such models are evaluated using a downstream task, e.g., object detection accuracy. 
The MPDA<cit.> framework addresses model heterogeneity by introducing a learnable resizer and sparse cross-domain transformer. The resizer dynamically adjusts feature maps from different agents, aligning spatial resolutions and channel dimensions. The transformer then harmonizes these features, ensuring domain-invariant representations through adversarial training.

    
Task Heterogeneity is a particular case of model heterogeneity where the alignment has to compensate for different target objectives of the different model tasks. Evaluation of such models can be done by using the downstream task objectives. STAR<cit.> presents a task-agnostic strategy using a spatio-temporal autoencoder. It enables the sharing of spatially subsampled and temporally mixed encoded information, facilitating scene reconstruction from shared data. This method provides a unified base for various tasks, overcoming task discrepancies. On the other hand, CORE<cit.> adopts a collaborative reconstruction approach. It combines a compression module for efficient feature broadcasting, an attention-based collaboration module for integrating messages, and a reconstruction module synthesizing these inputs into a comprehensive environmental understanding.
    
Sensor Heterogeneity occurs when different sensor modalities are to be fused, but also if different sensor intrinsic properties (e.g., beam count and resolution for LiDAR, intrinsic parameters for a camera) of a single modality are of interest. Performing better than single modalities or single sensors on the downstream task is the evaluation method for these models.
The first study, led by Liu et al.<cit.>, introduces the HPL-ViT, designed for V2V scenarios with diverse LiDAR systems. It employs graph-attention transformers to extract and fuse features specific to each agent, initiated by creating BEV feature maps. Another significant contribution comes from the HM-ViT<cit.>, which spatially transforms and updates features from different sensors through local and global attention mechanisms and outputs final predictions through a vision transformer. Complementing this, the BM2CP<cit.> leverages LiDAR-guided feature selection, integrates depth information from LiDAR and camera for enhanced accuracy, and demonstrates adaptability by maintaining performance despite sensor absence. In addressing the challenges of sensor heterogeneity from a different angle, DI-V2X<cit.> focuses on aligning feature representations from various LiDAR sensors, using domain-mixing instance augmentation and domain-adaptive fusion for consistent 3D object detection. FedBEVT<cit.> introduces a federated learning framework tailored for BEV perception in road traffic systems. It addresses data variability with camera-specific adjustments and adaptive multi-camera masking. This approach is particularly effective when data privacy is crucial and camera setups vary widely among vehicles.



 §.§ Adversarial Attack and Defense

    In collaborative perception systems, defending against malicious attacks is vital for maintaining data integrity and system trust. Table <ref> summarizes methods addressing these issues.
    Tu et al.<cit.> showed that adversarial attacks reduce multi-agent system performance. However, their impact lessens with more collaborative non-attacker agents, highlighting the need for enhanced neural network robustness. ROBOSAC<cit.> introduces a consensus-based defense effective against unknown attack models, especially with more collaborative agents. Zhang et al.<cit.> focus on vehicular systems, proposing an anomaly detection system to detect data fabrication attacks by identifying inconsistencies. Complementing these, MADE<cit.> employs a semi-supervised anomaly detection method to identify and remove malicious agents, proving effective against unknown attacks and controlling false positives.
    

    


 §.§ Domain Shift



    In addition to the previously delineated challenges, collaborative perception systems encounter the often understated domain shift problem due to the domain gap between simulation-based and real-world data. Table <ref> summarizes the methods addressing these issues. The S2R-ViT<cit.> takes a novel approach to this issue, concentrating on deployment and feature gaps. It integrates a vision transformer and a feature adaptation method, both adept at handling uncertainties in data transition from simulations to real-world scenarios. The vision transformer focuses on identifying and managing uncertainties, while the feature adaptation technique aligns and adapts simulated data features for real-world applicability. This integration has proven effective in 3D object detection within simulation-to-reality scenarios, surpassing frameworks such as AttFuse<cit.>, V2VNet<cit.>, and V2X-ViT<cit.>.
    Differently from  S2R-ViT, DUSA<cit.> expands the horizon by addressing sim-to-real adaptation and innovatively tackling domain discrepancies between different sensor setups. It employs an approach that adapts to location-specific characteristics of real-world scenarios for integrating essential features alongside a method that ensures consistency of these features when shared among diverse agents. Its effectiveness is further validated in various collaborative 3D detection methods such as V2X-ViT<cit.>, DiscoNet<cit.>.



    


§ CONCLUSION


We have categorized various intermediate fusion methods for collaborative perception, highlighting their potential and limitations. A predominant focus observed is on transmission efficiency, often evaluated within simulated environments. However, these simulations rarely capture the complexities of real-world scenarios, raising concerns about the real-time capabilities of sophisticated architectures like those based on transformers. Simulations often fail to model accurately the significant fluctuations in latency caused by real-world environmental and infrastructural factors, such as signal obstructions in dense urban areas. Additionally, as the number of agents in a network increases, so does the data traffic, which can lead to congestion and further latency if the network infrastructure lacks scalability.

This situation underscores the critical need for research focused on enhancing the scalability of networks to accommodate the expected rise in Connected and Autonomous Vehicles (CAVs) on the roads. The anticipated increase in CAVs presents both opportunities and challenges. More CAVs could enhance the quality and quantity of data available for collaborative perception, yet the effective selection and utilization of this data remains underexplored. Future research should focus on developing dynamic collaboration graphs to identify the most informative agents and exploring parallel information fusion processing.

The datasets currently used to evaluate these methods often feature only a small number of collaborative agents, failing to represent the scale of real-world operations adequately. Compounded by challenges such as labeling, privacy concerns, and the significant investments required to gather comprehensive real-world datasets, domain shift remains a formidable barrier<cit.>. Additionally, the integration of diverse sensor configurations poses another critical issue. Different vehicle manufacturers use various sensor setups, leading to a broad spectrum of data types and formats. Most current methods rely on homogeneous sensor configurations for validation, which fails to address the diversity seen in actual deployment scenarios, as indicated in 
Tables <ref>-<ref>. The complexity of integrating these varied data sources is compounded by the different models used for autonomous navigation. This increases the difficulty of interpreting and integrating shared feature maps. This necessitates more sophisticated data integration methods for effective collaboration across diverse sensor and system architectures.

In conclusion, while remarkable progress has been made in intermediate fusion for collaborative perception, the journey toward fully unleashing its potential is fraught with challenges. Addressing these issues demands a multidisciplinary approach, leveraging expertise from communication technology, data fusion, security, and machine learning. We hope our findings facilitate further research on intermediate-level collaborative perception.












§ ACKNOWLEDGMENT


This work is developed within the framework of the "Shuttle2X" project, funded by the Federal Ministry for Economic Affairs and Climate Action (BMWK) and the European Union, under the funding code 19S22001B. The authors are solely responsible for the content of this publication.


IEEEtran



