--- abstract: 'With autonomous driving developing in a booming stage, accurate object detection in complex scenarios attract wide attention to ensure the safety of autonomous driving. Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods. Firstly, we introduce the tasks, evaluation criteria and datasets of object detection for autonomous driving. Then, the process of mmWave radar and vision fusion is divided into three parts: sensor deployment, sensor calibration and sensor fusion, which are reviewed comprehensively. Especially, we classify the fusion methods into data level, decision level and feature level fusion methods. Besides, we introduce the fusion of lidar and vision in autonomous driving in the aspects of obstacle detection, object classification and road segmentation, which is promising in the future. Finally, we summarize this article.' author: - 'Zhiqing Wei, Fengkai Zhang, Shuo Chang, Yangyang Liu, Huici Wu, Zhiyong Feng [^1]' title: 'MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review' --- Autonomous Driving, Millimeter Wave Radar and Vision Fusion, Object Detection, Data Level Fusion, Decision Level Fusion, Feature Level Fusion, Lidar GLOSSARY {#glossary .unnumbered} ======== <span>m<span>0.1</span>m<span>0.4</span></span> AI & Artificial Intelligence\ IoT & Internet of Things\ DARPA & Defense Advanced Research Projects Agency\ ALV &Autonomous Land Vehicle\ SAE &Society of Automotive Engineers\ ADS &Automated Driving System\ ADAS & Advanced Driver Assistance System\ mmWave &millimeter wave\ 2D & Two-dimensional\ 3D & Three-dimensional\ AP & Average Precision\ AR & Average Recall\ PR & Precision-Recall\ mAP & mean Average Precision\ IoU & Intersection over Union\ <span>m<span>0.1</span>m<span>0.4</span></span> ACC & Autonomous Cruise Control\ FOV & Field Of View\ EKF & Extended Kalman Filter\ ROI & Region of Interest\ GVF & Gradient Vector Flow\ MILN & Multilayer In-place Learning Network\ CNNs & Convolutional Neural Networks\ MTT & Multi-Target Tracking\ RPN & Regional Advice Network\ FCN & Fully Convolutional Neural Network\ CMGGAN & Conditional Multi-Generator Generative Adversarial Network\ DCNN & Deep Convolutional Neural Network\ SAF & Spatial Attention Fusion\ Introduction ============ The main causes of traffic accidents are the complicated road conditions and a variety of human errors. Autonomous driving technology can leave the judgment of road conditions and human manipulation to the vehicle itself, which can improve traffic efficiency and driving safety. In recent years, with the significant development of artificial intelligence (AI), Internet of Things (IoT), and mobile communication, etc., autonomous driving has developed rapidly. There is no doubt that in the foreseeable future, autonomous driving vehicles will enter people's daily life and become a common AI product. The earliest research on autonomous driving was in the 1980s when Defense Advanced Research Projects Agency (DARPA) organized the autonomous land vehicle (ALV) project @001. In 2009, Google announced the establishment of a research team to start the research of autonomous driving technology @002. In 2013, Baidu launched the research of "baidu nomancar". In the same year, Audi, Nissan and other traditional automobile manufacturers began to layout autonomous driving cars. In 2014, Society of Automotive Engineers (SAE) released j3016 standard, which defined the standards of autonomous driving level from L0 (no driving automation) to L5 (full driving automation) @004. L4 and L5 can completely get rid of manual operation during driving. However, there are more challenges to guarantee the safety of driving @003. For safety reasons, the anti-collision function is an essential part of automated driving system (ADS), and the core task of the anti-collision system is obstacle detection. Therefore, one of the challenges for high-level autonomous vehicles is accurate object detection in complex scenarios. Object detection is the most popular research in the field of vision image processing. In the early stage of research, due to the lack of efficient image feature representation methods, object detection algorithms rely on the construction of manual features. Representative traditional algorithms include HOG @005, DPM @006 and so on. In recent years, as deep learning has developed, its great potential in object detection has been discovered. Detection algorithms based on deep learning are divided into two types: one-stage and two-stage. The former contains YOLO @007, SSD @008, RetinaNet @009, and the latter contains R-CNN @010, Fast R-CNN @011, Faster R-CNN @012, etc. However, the current visual object detection algorithms have hit a performance ceiling because the detection algorithms face very complicated situations in practice. For autonomous driving scenarios, the obstacles include pedestrians, cars, trucks, bicycles, and motorcycles, and obstacles within the visual range have different scales and length-to-width ratios. In addition, there may be varying degrees of occlusion between obstacles, and the appearance of the obstacles may be blurred because of the extreme weather such as rainstorm, heavy snow and fog, which result in a great reduction in detection performance @013. Study has shown that CNN has poor generalization ability for untrained distortion types @014. Compared with vision sensors, the detection performance of millimeter wave (mmWave) radar is less affected by extreme weather @015 @016. In addition, the mmWave radar not only measures the distance, but also uses the Doppler effect of the signal reflected by the moving object to measure the velocity vector @017 @018. However, mmWave radar cannot provide the outline information of the object, and it is difficult to distinguish relatively stationary targets. In summary, the detection capabilities of the vision sensor and the mmWave radar could complement each other. The detection algorithms based on mmWave radar and vision fusion can significantly improve the perception of autonomous vehicles and help vehicles better cope with the challenge of accurate object detection in complex scenarios. The process of mmWave radar and vision fusion based object detection is shown in Fig. \[Process of mmWave radar and vision fusion\]. The processes of mmWave radar and vision fusion consist of three parts: sensor selection, sensor calibration and sensor fusion. In order to achieve the desired performance of object detection with mmWave radar and vision fusion, the following challenges need to be solved. - Spatiotemporal calibration: The premise of fusion is to be in the same time and space, which means that mmWave radar and vision information need to be calibrated. - Information fusion: Object detection algorithms that fuse the sensing information of different sensors to achieve the optimal performance are essential. In order to solve the above challenges, firstly, it is essential to analyze the characteristics of different sensors and select the appropriate sensor. Secondly, it is necessary to study the sensor calibration, which involves the coordinate transformation between different sensors, the filtering of invalid sensing information, and the error calibration. Last but not least, it is necessary to study the sensing information fusion and realize the improvement of sensing capability through the complementary of different sensors, which involve the mmWave radar and vision fusion schemes. ![Process of mmWave radar and vision fusion<span data-label="Process of mmWave radar and vision fusion"></span>](ProcessOfFusion.pdf){width="80.00000%"} In recent years, there have been some surveys on object detection, but most of them focus on visual solutions. @019 summarized the vision-based object detection solutions, and conducts a comprehensive and detailed survey for object detection algorithms before and after the advent of deep learning. It is helpful for understanding vision-based object detection, but its detection targets are not only vehicles and there is no relevant content about fusion of radar and vision. @020 summarized the functions of each sensor module in the advanced driver assistance system (ADAS), but lacks the explanation of the fusion between radar and vision. @021 summarized the multi-sensor fusion and vehicle communication technology for autonomous driving, involving the fusion of cameras, mmWave radar, lidar, GPS and other sensors and V2X technology. It focuses on the classification of sensor fusion schemes, but lacks sufficient and in-depth review on the radar and vision fusion. In this article, we focus on the low-cost mmWave radar and vision fusion solutions. The process of mmWave radar and vision fusion consists of sensor selection, sensor calibration and sensor fusion, each of which is detailed reviewed. Especially, we classified the fusion methods into three-level fusion schemes including data level, decision level and feature level schemes. As shown in Fig. \[The organization of this paper\], the rest of this article is organized as follows. Section introduces the tasks of object detection, evaluation criteria and the public dataset. Section reviews the sensor configuration schemes of some automobile manufacturers and the comparison of the sensors. Section reviews the sensors calibration technologies. Section reviews sensor fusion schemes. Section analyzes the development trend of fusion of radar and vision for autonomous driving. Finally, we summarize this paper in Section . ![The organization of this paper<span data-label="The organization of this paper"></span>](TheOrganizationOfThisPaper.pdf){width="50.00000%"} Tasks, Evaluation Criteria and Datasets ======================================= Tasks ----- Object detection is of vital importance in the field of autonomous driving. In addition to target classification, the detection tasks also include positioning of the existing objects in the input image. The goal of two-dimensional (2D) object detection is to select the detected obstacle targets in the vision image of the vehicle with 2D bounding box. Then, the targets are classified and positioned. The positioning here refers to the positioning of the targets in the image rather than the positioning of targets relative to the vehicle in the real world. In three-dimensional (3D) object detection, the processes include the selection of the detected obstacle targets in the vision image of the vehicle with 3D bounding box, the classification and positioning of the targets. The positioning is not only the positioning of the selected target in the image, but also determining the posture and position of the objects in the real world. Evaluation criteria ------------------- Average Precision (AP) and Average Recall (AR), representing accuracy and regression in object detection, are commonly used as evaluation index. The Precision-Recall (PR) curve can be obtained by taking Recall values and Precision values as the horizontal and vertical axes respectively. The mean Average Precision (mAP), which represents the consolidated results of the detection model, can be obtained by calculating the average AP value of all classes. Take the KITTI dataset, which is widely used in autonomous driving, as an example. For 2D object detection, the correctness of object positioning is determined by comparing whether the Intersection over Union (IoU) between detection bounding box and annotated bounding box is greater than the threshold @025. It uses the mAP model proposed in @022 for performance evaluation. The calculation method is to set 11 equally spaced thresholds in \[0, 1\]. For Recall value greater than each threshold, there is a maximum precision value, and the average value of 11 precise values is the final mAP value. However, 3D object detection currently is more attractive in the studies of autonomous driving. The KITTI officially stipulated that for vehicles, correct prediction requires that the predicted 3D box overlaps the true 3D box by more than 70%, while for pedestrians and bicycles, 50% overlap of the 3D bounding box is required. In addition, KITTI adopted the recommendations of the Mapillary team in @023, which proposed that 40 recall positions could calculate the mAP more accurately than the 11 recall positions method. Datasets -------- Datasets are very important for the research of object detection. This section will briefly introduce some representative datasets for autonomous driving. Table 1 shows the basic features of some widely used datasets. \[!t\] <span>|m<span>0.15</span>&lt;|m<span>0.07</span>&lt;|m<span>0.1</span>&lt;|m<span>0.1</span>&lt;|m<span>0.1</span>&lt;|</span> Dataset & Release Year & RGB image & Radar & Lidar\ Apolloscape & 2018 & Y & N & Y\ KITTI & 2012 & Y & N & Y\ Cityscapes & 2016 & Y & N & N\ Waymo Open Dataset & 2019 & Y & N & Y\ nuScense & 2019 & Y & Y & Y\ ### Apolloscape@024 ApolloScape is part of the Apollo Open Platform created by Baidu in 2017. It uses a Reigl lidar to collect point cloud. The 3D point cloud generated by Reigl is more accurate and denser than the point cloud generated by Velodyne. Nowadays, ApolloScape has opened 147,000 frames of pixel-level semantically annotated images, including perceptual classification and road network data, etc. These high-resolution images with pixel-by-pixel semantic segmentation and annotation come from a 10 km distance measurement across three cities. Moreover, each area was repeatedly scanned under different weather and lighting conditions. ### KITTI@025 The KITTI dataset is established by the Karlsruhe Institute of Technology in Germany and the Toyota American Institute of Technology in USA, which is currently the most commonly used autonomous driving dataset. The team used a Volkswagen car equipped with cameras and Velodyne lidar to drive around Karlsruhe, Germany for 6 hours to record the traffic information. The dataset provides the raw image and accurate 3D bounding box with class label for every sequence. The object classes include cars, vans, trucks, pedestrians, cyclists and trams. ### Cityscapes@026 The Cityscapes dataset is jointly provided by three German labs including Daimler, Max planck institute informatik and Technische Universität Darmstadt. It is a semantic understanding image dataset of urban street scenes. It mainly contains 5000 high-quality pixel-level annotated images of driving scenes in an urban environment (2975 for training, 500 for validation, 1525 for test, 19 categories in total) from over 50 cities. In addition, it has 20,000 rough-annotated images. ### Waymo Open Dataset@027 The Waymo dataset is an open source project of Waymo, an autonomous driving company under Alphabet Inc. It consists of labeled data collected by Waymo self-driving cars under various conditions, including more than 10 million miles of autonomous driving mileage data covering 25 cities. The dataset includes lidar point clouds and vision images. Vehicles, pedestrians, cyclists, and signs have been meticulously marked. The team totally captured more than 12 million 3D annotations and 1.2 million 2D annotations. ### nuScenes@028 The nuScenes dataset, established by nuTonomy, is the largest existing autonomous driving dataset. It is the first dataset which is equipped with the full autonomous vehicle sensors. This dataset not only provides camera and lidar data, but also contains radar data, which is currently the only dataset with radar data. Especially, the 3D bounding box annotation provided by nuScenes not only contains 23 classes, but also has 8 attributes including pedestrian pose, vehicle state, etc. Sensor Deployment ================= This section introduces the sensor deployment schemes for autonomous driving vehicles. Through the analysis of the sensor deployment for vehicles equipped with autonomous driving systems launched by several major automobile manufacturers, it can be found that mmWave radar, lidar, and cameras are the main sensors for vehicles to detect the obstacles. By comparing the characteristics of the three sensors, it can be concluded that the fusion scheme of mmWave radar and camera has the characteristics of low cost, high accuracy, and adaptability to severe environment. In the following sections, the sensor deployment schemes including sensor configuration, sensor comparison and selection, are reviewed in details. Sensor configuration -------------------- \[!t\] <span>|m<span>0.15</span>&lt;|m<span>0.15</span>&lt;|m<span>0.15</span>&lt;|m<span>0.15</span>&lt;|</span> Company & Autonomous driving system & Sensor configuration & SAE autonomous driving level\ Tesla & Autopilot & 8 cameras, 12 ultrasonic radars mmWave radar & Level 2\ Baidu & Apollo & Lidar mmWave radar Camera & Level 4\ NIO & Aquila & Lidar 11 cameras 5 mmWave radars 12 ultrasonic radars & Level 3\ Xpeng & XPILOT & 6 cameras 2 mmWave radars 12 ultrasonic radars & Level 3\ Audi & Traffic Jam Pilot & 6 cameras 5 mmWave radars 12 ultrasonic radars Lidar & Level 3\ Mercedes Benz & Drive Pilot & 4 panoramic cameras Lidar mmWave radar & Level 3\ According to Table 2, it can be discovered that the overwhelming majority of automobile manufacturers have adopted a sensor configuration scheme combining radar and camera. In addition to Tesla, other manufacturers have used the fusion sensing technology combining lidar, mmWave radar and camera. It can be concluded that the sensing solution using the fusion of radar and vision is the current mainstream trend in the field of obstacle detection for autonomous driving vehicles. The reason is that the radar and the camera have complementary characteristics. The specific details will be explained in Section 3.2. Sensor comparison ----------------- @038 investigated the advantages and disadvantages of mmWave radar, lidar, and cameras in various applications. This article compares the characteristics of the three sensors, as shown in Table 3. According to Table 3, it can be seen that the three sensors have complementary advantages. \[!t\] <span>|m<span>0.15</span>&lt;|m<span>0.15</span>&lt;|m<span>0.15</span>&lt;|m<span>0.15</span>&lt;|</span> Sensor type & mmWave radar & Lidar & Camera\ Range resolution & \*\*\*\* & \*\*\*\*\* & \*\*\ Angle resolution & \*\*\*\* & \*\*\*\*\* & \*\*\*\*\*\*\ Speed detection & \*\*\*\*\* & \*\*\*\* & \*\*\*\ Detection accuracy & \*\* & \*\*\*\*\* & \*\*\*\*\*\*\ Anti-interference performance & \*\*\*\*\* & \*\*\*\*\* & \*\*\*\*\*\*\ Requirements for weather conditions & \* & \*\*\*\* & \*\*\*\*\ Operating hours & All weather & All weather & Depends on light conditions\ Cost and processing overhead & \*\* & \*\*\*\* & \*\*\*\ ### mmWave radar and Lidar As a common and necessary sensor on autonomous driving vehicles, mmWave radar has the characteristics of long range detection, low cost and detectability of dynamic targets. Due to these advantages, the vehicle's sensing ability and safety have been improved @035. Compared with lidar, the advantages of mmWave radar are mainly reflected in the aspects of coping with severe weather and low deployment cost @038. In addition, it also has the following advantages. - The mmWave radar can detect obstacles within 250 meters, which is of vital importance to the security of autonomous driving, while the detection range of lidar is within 150 meters @040. - The mmWave radar can measure the relative velocity of the target vehicle based on the Doppler effect with the resolution of 0.1m/s, which is critical for vehicle decision-making in autonomous driving @040. However, mmWave radar has also the following shortcomings @mason02@example.com. - Compared with lidar, mmWave radar has relatively lower angle resolution and detection accuracy, and it cannot detect pedestrians. Besides, the mmWave radar data is sparser. - The measurement of mmWave radar lacks semantic information, making it unable to fully meet the sensing demands of high-level autonomous driving. - The clutter cannot be completely filtered out from the radar measurement, leading to errors in radar signal processing. Lidar applies laser beams to complete real-time dynamic measurement, establishing a 3D environment model, which support the prediction of the surrounding environment, and the position and velocity of targets. Due to the characteristics of the laser, its propagation is less affected by clutter, and the ranging is non-coherent and large-scale. According to @039, the detection range of lidar can reach more than 50m in the longitudinal direction. The detection circle radius in the periphery of the vehicle reaches more than 20m. The rear end of the vehicle reaches a detection range of more than 20m, with high resolution of angular and range. However, the field of view (FOV) of lidar is limited. The price is one of the factors limiting the massive application of lidar. However, with the development of lidar technology, its cost is showing a downward trend. The more important factor is that lidar is highly affected by bad weather condition. When encountering heavy fog, rain and snow, its performance will be greatly degraded due to the inability to identify obstacles @038. ### Radar and Camera Radar is the best sensor for detecting distance and radial speed. It has "all-weather" capability, especially considering that it can still work mason02@example.com. But radar cannot distinguish the color and has a poor capability of targets classification @038. Camera has good color perception and classification capabilities, and the angle resolution capability is not weaker than that of lidar @038. But it is limited in estimating speed and distance @039. In addition, image processing relies on the powerful computing power of the vehicular chip, while the information processing of mmWave radar is not required. Making full use of radar sensing information can greatly save computing resources @038. Comparing the characteristics of radar and camera, it can be found that there are many complementary features between them. Therefore, the use of radar and vision fusion perception technology in the field of obstacle detection can effectively improve the perception accuracy and enhance the object detection capability of autonomous vehicles. Sensor Calibration ================== Due to the difference in the spatial location and sampling frequency of different sensors, the sensing information of different sensors for the same target may not match. Hence, calibrating the sensing information of different sensors is necessary. The detection information returned by mmWave radar is radar points and the camera gets vision image. The calibrated radar and vision information is shown in Fig. \[Radar points are projected on the image\], radar points are projected on the corresponding objects on the vision image. Generally speaking, sensor calibration involves coordinate calibration @041@049 @051@061@062@077@079, radar point filtering @049@051 and error calibration @076@mason02@example.com. ![Radar points are projected on the image<span data-label="Radar points are projected on the image"></span>](RadarPointsAreProjectedOnTheImage.pdf){width="70.00000%"} Coordinate calibration ---------------------- The purpose of coordinate calibration is to match the radar points to the objects in the images. For coordinate calibration, the most common methods are classified into coordinate transformation method @049@077, sensor verification method @041@062@079 and vision based method @050@051, which are reviewed as follows. - Coordinate transformation method: The coordinate transformation method is to unify the radar information and vision information under the same coordinate system through matrix operations. In @077, space calibration was completed by the method of coordinate transformation according to the spatial position coordinates of mmWave radar and vision sensor. For the time inconsistency caused by different sensor sampling rates, the thread synchronization method is adopted to realize the acquisition of the image frame and mmWave radar data simultaneously. @049 used the point alignment method based on pseudo-inverse, and obtains the coordinate transformation matrix by using the least square method. The traditional coordinate transformation cannot generate the accurate position of the target, which brings errors to the final results. In @048, Wang et al. proposed a calibration experiment to project the real coordinates into the radar detection map without special tools and radar reflection intensity, which weaken the dependence on calibration errors. - Sensor verification method: The sensor verification method is to calibrate each other with the detection information of different sensors on the same object. In @041, the sensor verification consists of two steps. First, the target list is generated by radar, and then the list is verified by the vision information. In @079, after the coordinate transformation of radar, the image is firstly searched roughly and then compared with the radar information. The result of the comparison is dividing the targets into two types: matched target and unmatched target. In @062, Streubel et al. designed a fusion time slot to match the objects detected by radar and vision in the same time slot. - Vision based method: In @050, the motion stereo technology was used to achieve the matching of radar objects and image objects. In @051, Huang et al. used adaptive background subtraction to detect moving targets in the image, generate candidate areas, and verify the targets by judging whether the radar points are located in the candidate areas. Radar point filtering --------------------- The purpose of radar point filtering is to filter out noise and useless detection results to avoid misjudgments caused by these radar points. In @049, Guo et al. proposed a method for noise filtering and effective target extraction using intra-frame clustering and inter-frame tracking information. In @051, the radar points were filtered by the speed and angular velocity information obtained by mmWave radar. Then, the invalid radar points were filtered, which reduces the impact of stationary targets such as trees and bridges on mmWave radar. Error calibration ----------------- Due to errors in sensors or mathematical calculations, there may be errors in the calibrated radar points. Some articles have proposed methods to correct these errors. In @076, a method based on interactive fine-tuning was proposed to make ultimate rectification to the radar points projected on the vision image. @087 proposed an improved Extended Kalman Filter (EKF) algorithm to model the measurement errors of different sensors. In @085, the influence of various coordinates on detection results was analyzed and discussed, and a semi-integral Cartesian coordinate representation method was proposed to convert all information into a coordinate system that moves with the host vehicle. Vehicle Detection Based on Sensor Fusion ======================================== Generally speaking, there are three fusion levels for mmWave radar and vision, including data level, decision level and feature level. Data level fusion is the fusion of data detected by mmWave radar and camera, which has the minimum data loss and the highest reliability. Decision level fusion is the fusion of the detection results of mmWave radar and camera. Feature level fusion requires to extract radar feature information and then fuse it with image features. The comparison of the three fusion levels is provided in Table 4. \[!t\] <span>|m<span>0.15</span>&lt;|m<span>0.25</span>|m<span>0.25</span>|</span> Fusion level & &\ Data level & Minimum data loss and the highest reliability & Depending on the number of radar points\ Decision level & Making full use of sensor information & Modeling the joint probability density function of sensors is difficult\ Feature level & Making full use of feature information and achieving best detection performance & Computing is complicated, the overhead of radar information transformation\ Data Level Fusion ----------------- As shown in Table 5, data level fusion firstly generates the Region of Interest (ROI) based on radar points @041@049@mason02@example.com. Then, the corresponding region of the vision image is extracted according to the ROI. Finally, feature extractor and classifier are used to perform object detection on these images @042 @045@048@049@070@073-@075@mason02@example.com. The latest literatures use neural networks for object detection and classification @mason02@example.com. For data level fusion, the number of effective radar points has a direct impact on the final detection results. If there is no radar point in a certain part of the image, this part will be ignored. This scheme can narrow the searching space for object detection, save computational resources, and leave a hidden security danger at the same time. \[!t\] <span>|c|c|m<span>0.12</span>&lt;|m<span>0.35</span>|</span> & Reference &\ &@041 &Using radar points to increase the speed of ROI generation.\ &@049&The distance information determines the size of the ROI.\ &@071&Using ROI for overtaking detection.\ & & @045 @049 @072 &Using histogram equalization, grayscale variance and contrast normalization to preprocess the image.\ & & @048 @072 @074&Image segmentation with radar point as the reference center.\ & & @042 @043 @073 @074 @075 @078 &Using features such as symmetry and shadow to extract vehicle contours.\ & & @044 @045&Using Haar-like model for feature extraction.\ & & @045 & Adaboost algorithm for object classification.\ & & @070 @079 & SVM for object classification.\ & & @046 @072 & Neural network-based classifier for object classification.\ ### ROI generation ROI is a selected area in the image, which is the focus of target detection. Compared with a pure image processing scheme, the data-level fusion scheme uses radar points to generate ROI, which can significantly improve the speed of ROI generation @041. The size of initial ROI is determined by the distance between the obstacle and mmWave radar @049. The improved front-facing vehicle detection system proposed by @071 can detect overtaking with a high detection rate. This method focuses on the area where overtaking is about to occur. The overtaking is determined by checking two specific characteristics of vehicle speed and movement angle. In @075, a square with 3 meters of side length centered on the radar point is taken as the ROI. ### object detection Because of the uncertainty of the position and size of the objects in the image, the object detection based on vision often adopts sliding window and multi-scale strategy, which produces a fair amount of candidate boxes, resulting in low detection efficiency. mmWave radar and vision fusion scheme can avoid sliding window method, which reduces the computational cost and improves the detection efficiency. Object detection task focuses on the vision image processing, the task can be divided into three steps: image preprocessing @045@048@049@072@074, image feature extraction@042-@045@061@073-@075@078 and object classification@045@070@mason02@example.com. $\bullet$ Image preprocessing In order to remove the noise in the image and enhance the feature information, so as to facilitate the subsequent feature extraction and object classification tasks, image preprocess is necessary. The methods of image preprocessing mainly include histogram equalization, gray variance, contrast normalization and image segmentation. In @045, Bombini et al. conducted a series of tests on histogram equalization, grayscale variance and contrast normalization in order to obtain the invariance under different illumination conditions and cameras. They conclude that the contrast normalization achieved better performance. In @049, the gradient histogram method was used to preprocess the image, and an improved position estimation algorithm based on ROI was proposed, which can obtain a smaller potential object region and further improve the detection efficiency. In @072, median filtering, histogram equalization, wavelet transform and Canny operator were used in image preprocessing. In @048@072@074, radar points were taken as the reference center to segment the image, and then the object boundary was determined to improve the object detection speed. $\bullet$ Feature extraction The purpose of feature extraction is to transform the original image features into a group of features with obvious physical or statistical significance, which is convenient for object detection. In the stage of image feature extraction, the available vehicle features include symmetry and underbody shadow, etc. In @042@043@078, symmetry was used for ROI detection and @061 utilized shadow detection to obtain feature information, while in @073 and @075, vertical symmetry and underbody shadow characteristics were comprehensively utilized for vehicle detection. In @075, the gradient information of the image was used to locate the boundary effectively, and the method based on gradient vector flow (GVF) Snake was used to describe the accurate contour of vehicle. Since the color distribution of the object was stable during the movement, the histogram matching method was feasible track the vehicle. In @044, Kadow et al. applied Haar-like model for feature extraction, which is a classic feature extraction algorithm for face detection. In order to improve detection rate, mutual information was used for Haar-like feature selection @045. In @074, the visual selective attention mechanism and prior information on visual consciousness during human driving was proposed to extract features and identify object contour from segmented images. The shadow was extracted by histogram and binary image segment, and the pedestrian edge was detected by 3×3 image corrosion template. $\bullet$ Object classification In the object classification stage, Adaboost, support vector machine (SVM) and other object classification algorithms are used to select the final box of the vehicle in the vision image. In @045, Adaboost algorithm is used to scan the ROI projected on the image plane. In @070@079, SVM was used for object recognition and classification. @080 combined ROI image and Doppler spectrum information for object classification. @072 adopted object classification based on infrared image analysis. The authors classified the objects into point objects and regional objects according to the object area, and used a neural network-based classifier to classify the regional targets @072. @046 utilized Multilayer In-place Learning Network (MILN) as a classifier, which demonstrated superior accuracy in two-class classification task. ![Faster R-CNN @012<span data-label="Faster R-CNN"></span>](fasterrcnn.pdf){width="50.00000%"} \[!t\] ![image](yolo.pdf){width="100.00000%"} \[!t\] ![image](SSD.pdf){width="100.00000%"} Decision Level Fusion --------------------- The advantage of radar is longitudinal distance and the advantage of the vision sensor is horizontal field of view. Decision level fusion can take into account the advantages of both aspects and make full use of sensing information. The challenge of the decision-level fusion filtering algorithm is to model the joint probability density function of the two kinds of detection information. This is because the noise of the two kinds of detection information is different. The decision level fusion mainly includes two steps: sensing information processing @052@058@063@064@088 @090 and decision fusion@053-@mason02@example.com. ### Sensing information processing The processing of sensing information includes radar information and vision information. Radar detection results generate a list of objects and contain information such as the speed and distance of objects @mason02@example.com. Vision information processing performs object detection algorithm on images, which includes traditional feature extraction combined with classifier @058@063 and Convolutional Neural Network (CNN) @064@088 @090. $\bullet$ Radar information A mmWave radar and vision fusion system was proposed for autonomous driving navigation and lane change in @052, in which the radar sensor obtains the target distance through FFT, obtains the angular position of the target through digital wavefront reconstruction and beamforming, and then analyzes the target position. In @058, the results of mmWave radar detection is a list of possible objects in the radar FOV, and each element in the list includes the distance, azimuth angle and relative velocity of the detected object. $\bullet$ Image object detection In the visual detection of @058, histogram was used to calculate edge information. Then, gradient histogram feature extraction and Boosting based classifier were used to detect pedestrian. @063 proposed an auxiliary navigation method combining mmWave radar and RGB depth sensor, and used the MeanShift algorithm to detect objects in depth images. Furthermore, the average depth of ROI determined the distance of the detected object. @064 has improved the vision data processing algorithm in @063, and applied Mask R-CNN for object detection. In @088, YOLO V2 is used for vehicle detection, and the input is an RGB image with the size of 224×224. Preprocessing subtracts the average RGB value of training set images from each pixel. @090 also applied YOLO V2 for vehicle detection, and proved that YOLO V2 is superior to Faster R-CNN and SSD in both speed and accuracy, which is more suitable for vehicle detection tasks. @089 applied YOLO V3 in obstacle detection, and its weight is pre-trained in COCO dataset. \[!t\] ![image](retinanet.pdf){width="100.00000%"} ### Decision fusion The decision level fusion of vehicle detection fuses the detection results of different sensors. The mainstream filtering algorithms apply Bayesian theory @053@054, Kalman filtering framework @055@059@060, and Dempster Shafer theory @058. In some literatures, the list of radar detection targets was used to verify vision detection results @mason02@example.com. In addition, @086 proposed the Motion Stereo algorithm to adjust and refine the final detection results. $\bullet$ Fusion Methods based on Bayesian theory @053 proposed a method based on Bayesian theory to solve the problem of multi-sensor data fusion by using probabilistic reasoning method, which is called Bayesian programming. The traditional multi-sensor fusion algorithms are no longer applicable when new sensors are added. The fusion algorithm is modularized and generalized in @054, and a dynamic fusion scheme based on Bayesian network was proposed to improve the reusability of each fusion algorithm. $\bullet$ Fusion Methods based on Kalman filter Based on the EKF framework on Lie group, a decision level fusion filter using a special Euclidean group is proposed in @055. @059 proposed a fusion framework that can track the detection object simultaneously in 3D space and 2D image plane. An uncertainty driven mechanism similar to Kalman filter is used to equalize the sensing results of different qualities. In @060, the given image was firstly detected by radar to roughly search the target. Then, a trained spot detector was used to obtain the object's bounding box. The information fusion method based on Kalman filter was adopted, and the functional equivalence of centralized and decentralized information fusion schemes was proved. $\bullet$ Fusion Methods based on Dempster Shafer theory @058 proposed decision level fusion based on Dempster Shafer theory, taking the detection lists of multiple sensors as input, using one of them as a temporary evidence grid and fusing it with the current evidence grid, and finally performing clustering processing. The target was identified in the evidence grid. $\bullet$ Fusion Methods based on Radar validation @056 overlapped the target list generated by vision detection and radar detection to generate a unique vehicle list. The radar data was used to verify the vision detection results. If there was a target matching the vision detection results in the radar data, a blue box would be marked as a strong hypothesis. Otherwise, if there was no target, it would not be discarded, a green box would be marked as a weak hypothesis. @057 proposed an multi-target tracking (MTT) algorithm that can correct the tracked target list in real time by evaluating the tracking score of the radar scattering center. The stereo vision information is used to fit the contour of the target vehicle, and the radar target matched with the target vehicle is used to correct its position. ![The network structure in @065<span data-label="SSDBasednetwork"></span>](SSDBasednetwork.pdf){width="50.00000%"} Feature Level Fusion -------------------- Feature level fusion is a new scheme emerging in recent years. It is a common approach to use an additional radar input branch in feature level fusion methods @mason02@example.com. CNN-based objects detection model can effectively learn image feature information. By transforming radar detection information into image form, the detection model can learn radar and vision feature information simultaneously, and the feature level fusion will be realized. \[!t\] ![image](RVNet.pdf){width="100.00000%"} ### Object detection framework Convolutional neural networks (CNNs) are widely applied in object detection based on feature level fusion. At present, some algorithms have achieved the results of superior performance, such as Faster RCNN, YOLO (V3), SSD, RetinaNet, etc. $\bullet$ Detection framework based on CNN Faster-RCNN is a widely used detector, and its integration has been greatly improved compared with Fast-RCNN. Its main contribution is the introduction of the Regional Advice Network (RPN), which enables nearly costless area suggestion. The RPN can simultaneously predict the target bounding box and categorization score for each location. The network structure of Faster RCNN is shown in Fig. \[Faster R-CNN\]. YOLO is the first one-stage detector, which is the abbreviation of "You Only Look Once". It adopts a new detection idea: Using a single neural network to complete the detection task. Dividing the vision image into multiple regions and predicting bounding box for each region. Therefore, the detection speed has been remarkably improved. However, its positioning accuracy is reduced compared with the two-stage detector. Fig. \[YOLO\] shows the network structure. SSD is another excellent one-stage detector. It eliminates the bounding box suggestion and pixel or feature resampling stage, and fundamentally improves the detection speed. The detections in different scales are generated from the feature maps of different scales, and the detections are clearly separated by aspect ratio, such that the accuracy is significantly improved. As a one-stage detector with high detection speed, SSD also ensures the detection accuracy close to that of two-stage detectors. Fig. \[SSD\] illustrates the network structure. As to RetinaNet, the inventors discussed the reason why the accuracy of one-stage detector is lower than that of two-stage detector, and concluded that this phenomenon is caused by the extreme foreground background class imbalance. They proposed a new loss function named Focal loss to reshape the standard cross entropy loss, aiming to focus more attention on the objects that are difficult to classify during training. Fig. \[retinanet\] shows the network structure of RetinaNet. $\bullet$ Fusion framework based on CNN In @065, feature level fusion was firstly applied in mmWave radar and vision fusion. As shown in Fig. \[SSDBasednetwork\], its detection network was improved on the basis of SSD \[8\]. The radar branch was concatenated after the second ResNet18 block of the vision branch. In @066, a new sensor fusion framework called RVNet was proposed, which was similar to YOLO \[7\]. As is shown in Fig. \[RVNet\]. the input branches contain separate radar and vision branch and the output branches contain independent branch for small and large obstacles, respectively. The CNN used in @067 was built on RetinaNet \[9\] with a VGG backbone @091, named CameraRadarFusionNet (CRF-Net). The author expanded the network so that it can process additional radar input branch. Its input branches were radar feature and vision feature respectively, and the output results are the 2D regression of the coordinates of the objects and the classification fraction of the objects. The structure of CRF-Net is shown in Fig. \[CRFNet\]. ![The structure of CameraRadarFusionNet (CRF-Net) @067<span data-label="CRFNet"></span>](CRFNet.pdf){width="50.00000%"} \[!t\] ![image](SAF.pdf){width="100.00000%"} In @069, the author proposed a new detection network called spatial attention fusion-based fully convolutional one-stage network (SAF-FCOS), which was built on FCOS @092. The radar branch was improved from ResNet-50, and the vision image branch adopted a two-stage operation block similar to ResNet-50. In order to improve the detection accuracy, the final object detection comprehensively utilized FCOS @092 and RetinaNet \[9\]. Fig. \[SAF\] shows the structure of SAF-FCOS. ### Radar feature extraction The purpose of radar feature extraction is to transform radar information into image-like matrix information, because radar information cannot be fused with image information directly. Radar feature extraction mostly adopts the method of converting radar points to the image plane to generate radar image. The transformed radar image with multiple channels contains all the environmental features detected by the radar. Each channel represents a physical quantity such as distance, longitudinal speed, lateral speed and so on. @068 proposed a new Conditional Multi-Generator Generative Adversarial Network (CMGGAN), which takes the measured data of radar sensors to generate artificial, camera-like environmental images, including all environmental features detected by radar sensors. A new description method for radar features was proposed in @066, which is called radar sparse image. Radar sparse image is a 416×416 three-channel image, whose size directly corresponds to the size of the vision image. The three channels contain radar point velocity and depth feature information. In @069, Chang et al. converted the depth, horizontal and vertical information at the radar point into real pixel values of different channels. For the area without radar point, they set the pixel value to 0 and render the radar image with circles centered on radar points. Based on experimental results, the anti-noise performance of the radar image saved in the PNG format is better than that of the jpg format. In @067, considering the lack of height information in radar detection results, Nobis et al. stretched the projected radar points in the vertical direction in order to integrate them with the image better. And the features of radar information were stored in the enhanced image in the form of pixel values. In addition, a ground truth noise filter was proposed to filter the invalid radar points. ### Radar vision feature fusion The basic feature fusion methods can be divided into two kinds: concatenation method and element-wise addition method. The former concatenates the radar feature matrix and the image feature matrix into a multi-channel matrix while the latter adds two matrices into one. In @065, two fusion methods of concatenation and element-wise addition are set, and the experimental results show that both fusion methods have improved the detection performance. The element-wise addition method performs better on manually labeled test sets and the concatenation method performs better on generated test sets. @066 and @067 both adopt the concatenation method. In @069, a new block named spatial attention fusion (SAF) was proposed for sensor feature fusion. SAF block was used to generate an attention weight matrix to fuse radar and vision features. At the same time, @069 compared the SAF method with three methods of element-wise addition, multiplication and concatenation, and the result shows that SAF has the best performance. In addition, @069 conducted generalization experiments on Faster R-CNN, and the detection performance was also improved by SAF model. Future Trends ============= As the cost of lidar decreases, it has become a trend to gradually replace mmWave radar. The fusion of lidar and vision is becoming promising in autonomous driving. The main research includes object detection, object classification and road segmentation. Object detection ---------------- The proposed fusion method in @081 firstly applied lidar point cloud to generate suggestions for potential car positions in the image. Then, the position of the bounding box was refined by mining multiple layers of information in the proposal network. Finally, the object detection was performed by a detection network which shared part of the layers with the proposed network. @084 proposed a fusion method similar to the data level fusion of mmWave radar and vision. It utilized lidar instead of mmWave radar to generate initial recommendations for the position of objects. Then, multi-scale features are exploited to realize accurate detection of objects of different sizes. Object classification --------------------- @082 presented a lidar and vision fusion method for object classification, which combines Deep-CNN (DCNN) and upsampling theory. It can inherit the advantages of the two methods and avoid the shortcomings of each method. This method up-sampled the lidar information and converted it into depth information, which would be fused with the vision image data and sent into the DCNN for training. The experimental results show that this method exhibits superior effectiveness and accuracy of object classification. Road detection -------------- @083 proposed a method to combine lidar point clouds and vision image for road detection. Firstly, the method of point alignment was to project the lidar point cloud onto the vision image. Then, the vision image was upsampled to obtain a set of dense images containing spatially encoded information. Finally, they trained multiple Fully Convolutional Neural Networks (FCN) for road detection. The proposed FCN was evaluated on KITTI and got a MaxF score of 96.03%, which is one of the top-performing methods so far. Conclusion ========== Object detection is one of the most important tasks for autonomous driving. In this article, we provide an overview of mmWave radar and vision fusion for vehicle detection. Firstly, we introduced the tasks, evaluation criteria and datasets of autonomous driving. Secondly, we divided the mmWave radar and vision fusion process into three parts and the fusion algorithms were classified into three levels: data level, decision level and feature level. Finally, lidar and vision fusion, as a promising technology in the future, is reviewed in the perspectives of obstacle detection, object classification and road detection. <span>100</span> J. Towler and M. Bries, "Ros-military: Progress and promise," *Ground Vehicle Systems Engineering and Technology Symposium (GVSETS)*, 2018. WAYMO. "Journey" waymo.com <https://www.waymo.com/journey/>(accessed Jul. 23, 2021). K. Ren, Q. Wang and C. Wang, "The security of autonomous driving: Threats, defenses, and future directions," *Proceedings of the IEEE*, vol. 3, no. 2, pp. 357-372, Nov. 2019. SAE On-Road Automated Vehicle Standards Committee and others, "Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems," *SAE Standard J*, vol. 3016, pp. 1-16, 2014. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," *IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)*, vol. 1, pp. 886-893, Jun. 2015. P. Felzenszwalb, D. McAllester, and D. Ramanan, "A discriminatively trained, multiscale, deformable part model," *IEEE Conference on Computer Vision and Pattern Recognition*, pp. 1-8, Jun. 2008. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," *IEEE Conference on Computer Vision and Pattern Recognition*, pp. 1-8, Jun. 2008. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg, "SSD: Single shot multibox detector," *ECCV 2016: Computer Vision – ECCV 2016*, pp. 21–37, Sep. 2016. T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll´ar, "Focal loss for dense object detection," *IEEE transactions on pattern analysis and machine intelligence*, vol. 42, no. 2, pp. 318-327, Feb. 2020. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," *IEEE Conference on Computer Vision and Pattern Recognition*, pp. 580–587, Jun. 2014. K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 37, no. 9, pp. 346–361, Sep. 2015. S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 39, no. 6, pp. 91–99, Jun. 2017. C. Michaelis, B. Mitzkus and R. Geirhos, "Benchmarking robustness in object detection: Autonomous driving when winter is coming," *arXiv:1907.07484*, Jul. 2019. R. Geirhos, C. R. M. Temme and J. Rauber, "Generalisation in humans and deep neural networks," *arXiv:1808.08750*, Aug. 2018. R. Zhang and S. Cao, "Real-time human motion behavior detection via CNN using mmWave radar," *IEEE Sensors Letters*, vol. 3, no. 2, pp. 1-4, Feb. 2019. K. Yoneda, N. Hashimoto and R. Yanase, "Vehicle Localization using 76GHz Omnidirectional Millimeter-Wave Radar for Winter Automated Driving," *IEEE Intelligent Vehicles Symposium (IV)*, pp. 971-977, Jun. 2018. T. Nagasaku, K. Kogo and H. Shinoda, "77GHz Low-Cost Single-Chip Radar Sensor for Automotive Ground Speed Detection," *IEEE Compound Semiconductor Integrated Circuits Symposium*, pp. 1-4, Oct. 2008. M. E. Hines and S. A. Zelubowski, "Conditions affecting the accuracy of speed measurements by low power MM-wave CW Doppler radar," *Vehicular Technology Society 42nd VTS Conference-Frontiers of Technology*, vol. 2, pp. 1046-1050, May. 1992. Z. Zou, Z. Shi and Y. Guo, "Object detection in 20 years: A survey," *arXiv:1905.05055*, May. 2019. A. Ziebinski, R. Cupek and H. Erdogan, "A survey of ADAS technologies for the future perspective of sensor fusion," *International Conference on Computational Collective Intelligence*, vol. 9876, pp. 135-146, Sep. 2016. Z. Wang, Y. Wu and Q. Niu, "Multi-sensor fusion in automated driving: A survey," *IEEE Access*, vol. 8, pp. 2847-2868, Dec. 2019. M. Everingham, L. V. Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results," 2011. A. Simonelli, S. R. Bulò and L. Porzi, "Disentangling Monocular 3D Object Detection," *IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 1991-1999, Nov. 2019. X. Huang, X. Cheng and Q. Geng, "The apolloscape dataset for autonomous driving," *IEEE/CVF Conference on Computer Vision and Pattern
{'source': 'arXiv_out/2108'}