Combining Visible and Infrared Spectrum Imagery using Machine Learning for Small Unmanned Aerial System Detection
Vinicius G. Goecks Grayson Woods John Valasek
cs.CV cs.LG eess.IV
 

ON LOGARITHMIC HÖLDER CONTINUITY OF MAPPINGS ON THE
BOUNDARY
    EVGENY SEVOST'YANOV

    December 30, 2023
=============================================================





There is an increasing demand for technology and solutions to counter commercial, off-the-shelf small unmanned aerial systems (sUAS).
Advances in machine learning and deep neural networks for object detection, coupled with lower cost and power requirements of cameras, led to promising vision-based solutions for sUAS detection.
However, solely relying on the visible spectrum has previously led to reliability issues in low contrast scenarios such as sUAS flying below the treeline and against bright sources of light.
Alternatively, due to the relatively high heat signatures emitted from sUAS during flight, a long-wave infrared (LWIR) sensor is able to produce images that clearly contrast the sUAS from its background.
However, compared to widely available visible spectrum sensors, LWIR sensors have lower resolution and may produce more false positives when exposed to birds or other heat sources.
This research work proposes combining the advantages of the LWIR and visible spectrum sensors using machine learning for vision-based detection of sUAS. Utilizing the heightened background contrast from the LWIR sensor combined and synchronized with the relatively increased resolution of the visible spectrum sensor, a deep learning model was trained to detect the sUAS through previously difficult environments. More specifically, the approach demonstrated effective detection of multiple sUAS flying above and below the treeline, in the presence of heat sources, and glare from the sun.
Our approach achieved a detection rate of 71.2 ± 8.3%, improving by 69% when compared to LWIR and by 30.4% when visible spectrum alone, and achieved false alarm rate of 2.7 ± 2.6%, decreasing by 74.1% and by 47.1% when compared to LWIR and visible spectrum alone, respectively, on average, for single and multiple drone scenarios, controlled for the same confidence metric of the machine learning object detector of at least 50%.
With a network of these small and affordable sensors, one can accurately estimate the 3D position of the sUAS, which could then be used for elimination or further localization from more narrow sensors, like a fire-control radar (FCR).
Videos of the solution's performance can be seen at <https://sites.google.com/view/tamudrone-spie2020/>.













§ INTRODUCTION

  




Small unmanned aircraft systems (sUAS), commonly called drones, were once only thought of as military aircraft. 
We have seen images from the government or the news media showing comparatively larger aircraft such as the General Atomics MQ-9 Reaper, which are being controlled by an operator sitting hundreds of miles away. 
They are effective in performing surveillance but can also deploy weapons without risking its operator. 
Increasingly, the technology to control and operate such unmanned aircraft is becoming affordable and easily available. 
We are seeing a rise in the manufacturing of consumer-grade and commercial-grade sUAS.
Companies such as DJI<cit.>, Parrot<cit.>, and Autel Robotics<cit.> have drones equipped with high definition RGB and infrared cameras, sophisticated mission planners, and are capable of completing its mission autonomously, that can be bought for as little as $1000 USD.



This capability is posing crucial privacy and security threats to people and organizations around the world, specially when these drones are retrofitted with sophisticated cameras and possibly weapons, which sparked a need for counter-sUAS, or counter-drone, solutions.











One of the components of a counter-drone solution is the detection of the drone.
Due to the wide availability of cameras and advancements in deep learning and computer vision techniques, vision-based solutions became more appealing to address the detection problem.
Vision based-object detection techniques were mainly developed to detect objects on the visible spectrum while using RGB cameras.
However, accuracy strongly decreases when detecting objects under fog, rain, or in poorly illuminated environments.

As an alternative, long-wave infrared (LWIR) cameras are more robust to these conditions but are less accurate due to lower resolution and false positives when detecting objects with similar heat signatures such as differentiating at long distance drones from birds or light bulbs.



To leverage the advantages of each sensor, our proposed solution combines images captured from RGB and LWIR cameras to train a machine learning model for the detection of sUAS. Our main contribution is the insight of utilizing the heightened background contrast from the LWIR sensor combined and synchronized with the relatively increased resolution of the RGB sensor to train a deep learning model to detect sUAS through previously difficult environments. More specifically, the approach demonstrated effective detection of multiple sUAS flying above and below the treeline, in the presence of heat sources, and glare from the sun.







§ RELATED WORK


For large unmanned aerial systems (UAS) detection, the majority of techniques use radar imaging <cit.>. However, the size and speed of small unmanned aerial systems (sUAS) present challenges for radar detection, which must be able to scan a large area with discrimination to prevent false alarms from birds and moving ground objects <cit.>. Deep learning techniques have been developed to help detect and classify sUAS by recognizing patterns in the Doppler signature of micro UAS <cit.>. However, the confidence with which these systems can be practically employed for counter-sUAS is still up for debate.

Image-based sUAS detection provides another solution for the detection of sUAS.
Visual sensors have been used to track targets moving over complex backgrounds <cit.>. In addition, temporal information from subsequent images can be supplemented to make the tracking robust to when the background or target object is changing in appearance <cit.>. Detection of sUAS with visual sensors can even be employed when the sensors themselves are moving (for example, when onboard another sUAS <cit.>). Image processing techniques such as feature detection and corner detection can also be used to analyze an object or scene from two different positions and find features that would correspond to a real-world 3D object or scene <cit.>, offering an increased ability to classify the target object.


Fusing multi-modal sensor outputs such as RGB and infrared video streams are gaining prominence to detect objects under changing environmental conditions. If one sensing modality is impaired in a particular condition, other sensing modalities might not be affected. The fusion of sensor data for object detection is typically done in any of these three phases: at the pixel level, at the feature level, or at the decision level. RGB and infrared sensor data fused at a pixel level <cit.> can then be segmented using either parametric <cit.>techniques such as Gaussian Mixture Model or non-parametric techniques such as kernel-based density estimators. A new non-parametric technique based on codebook model <cit.> has shown enhanced object detection performance <cit.> when applied to pixel-level fused data. 
The image segmentation accuracy as measured using the detection rate metric is considerably higher for the fused data compared to RGB and infrared data separately<cit.>. Moreover the processing time for fused data was found to be less compared to the summation of the processing time required for RGB and infrared data individually. The detection accuracy is shown to be further improved if we allow a context-aware, quality-based fusion <cit.>. Context sensitive indicators are introduced <cit.> to weigh in the soft segmentation performed over individual RGB and infrared data.

More recently, machine learning approaches were developed to remove the necessity of hand-designing image features and corners to, instead, learn these patterns automatically from large datasets.
Simple Online and Realtime Tracking with a Deep Association Metric (Deep SORT) <cit.> trained a convolutional neural network on a large-scale person identification dataset to learn an association metric for each target identified. Paired with Kalman filtering in the image-space and frame-by-frame data association, this approach achieves favorable performance when tracking objects for longer periods of time under occlusion.
One of the most common approaches, You Only Look Once (YOLO) <cit.>, frames object detection as a regression problem where a neural network predicts, in real time, bounding boxes and class probabilities directly from full images in only one evaluation step.
Improving upon YOLO, EfficientNet <cit.> proposes the compound scaling method that uniformly scales depth, width, and resolution of convolution neural networks following a fixed ratio to achieve state-of-the-art object detection while using smaller and faster networks. 
Combined with machine learning, data from visual sensors can be used to not only track sUAS <cit.> but also distinguish them from birds <cit.> and other background objects.
Our approach, detailed below, utilizes many of these advancements to provide a robust framework for sUAS detection.






§ METHODS


This research work proposes combining the advantages of the LWIR and visible spectrum sensors using machine learning for vision-based detection of sUAS.
Due to the power and battery discharge rates required to fly small unmanned quadrotors, these vehicles hold a characteristic heat signature when imaged with LWIR sensors that contrasts with common cluttered backgrounds suck as buildings, trees, sky, and the terrain.
The same does not happen in the visible spectrum where changes in lighting and cluttered background can completely camouflage a flying small unmanned quadrotor.
Conversely, visible spectrum sensors are widely available in the market and have higher resolution, which aids detection at long-ranges assuming that the vehicle contrasts with the background.
Our prototype, as seen in Figure <ref> consists of two calibrated cameras aligned based on the position of their image capture sensor and attached to a custom mount system that allows them to be attached to a camera tripod.
The cameras, from left to right, are a FLIR infrared camera BOSON 640 coupled with 4.9mm lens and a FLIR RGB camera CM3-U3-50S5C-CS Chameleon3 USB3 coupled with a 3.5mm C Series fixed focal length lens. The camera specifications are shown in Table <ref>.
All camera data was streamed and recorded using ROS (Robot Operating System) <cit.>, synchronized to run at 30 Hz.
A sample RGB and respective LWIR frame demonstrating the data capture system can be seen in Figure <ref>.








The data collection process consisted of imaging two different sUAS, DJI Mavic Pro and DJI Phantom 4, while flying at different times of the day and weather conditions, such as during sunset and sunrise in a clear sky or during the afternoon of a cloudy day, and under different lighting conditions in order to create a diverse dataset to be used as training data for the machine learning model.
A total of 30 minutes of data at 30 frames per second were recorded for a total of 52800 frames for each camera, LWIR and RGB. From that, 1275 frames in total were manually labeled by drawing a bounding box surrounding the vehicle.
The main assumption during this process was that the person should label LWIR and RGB data independently so there is no information leakage between one spectrum to another.


To combine the calibrated and synchronized LWIR and RGB images, we first computed the homography transformation from the LWIR to the RGB image frame so each pixel in the infrared spectrum could be associated with one pixel in the visible spectrum.
The pixels were combined by computing the equally weighted sum between them so the resulting pixel carries information from both spectra.
Figure <ref> illustrates our proposed approach to combined LWIR+RGB data in comparison to LWIR and RGB alone.
The green box indicates that the machine learning object detector has reached the desired level of confidence to identify a drone in that location, to be explained in more detail in the following paragraphs.



In possession of a synchronized and annotated dataset of LWIR+RGB, LWIR, and RGB images, we trained a machine learning model for drone detection for each separate dataset using YOLOv3 <cit.> official implementation[YOLOv3 on Github: <https://github.com/pjreddie/darknet>.].
The LWIR and RGB models used their respective images and labels to train the YOLO object detector while the LWIR+RGB model used the combined frames, as seen in the leftmost image of Figure <ref>, and all LWIR and RGB labels.
As illustrated in Figure <ref>, YOLO splits the input image in S × S grid cells, predicting one object and a fixed number of bounding boxes per cell using dimension clusters as anchor boxes <cit.>.
Each prediction of a bounding box is accompanied by a box confidence score and each grid cell by a conditional class probability, which, in our case, represents the probability of containing or not a drone.
These scores are condensed with a dot product in a final class confidence score that measures the confidence in both the classification (correct class of containing or not a drone) and the localization of the object (coordinates of the corners of the bounding box surrounding the drone).
YOLO was initially designed to be used with RGB images but no modification in the architecture was required since the LWIR+RGB frames still have 3 image channels, similar to an RGB image. The same applies to the LWIR frames, displayed in colors, which also have 3 image channels.
Due to the modularity of this approach, it can be integrated into existing surveillance and camera systems capable of recording LWIR and RGB data.









§ RESULTS AND DISCUSSION


The proposed approach is evaluated in four drone detection scenarios: drones crossing above and below the horizon and treeline, drones flying in front of cameras pointed directly the sun during sunrise, drones flying together with other heat sources such as public street lighting, and drone detection at approximately 300 meters away, here called long-range since the sUAS is barely visible on the RGB spectrum and also to differentiate of the previous cases where the vehicles were flying at less than 300 meters away.
We compared our approach of combining LWIR and RGB images and detecting drones using a custom trained YOLO model against the same YOLO architecture but trained only using LWIR and RGB images alone. All YOLO models for the baselines and the proposed method were trained on the same scenarios and using the same labels, only changing the input video appropriately.
Performance is computed in terms of Detection Rate (DR) and False Alarm Rate (FAR) <cit.>. DR is computed as the ratio between true positives and the sum of true positives (TP) and false negatives (FN), as seen in Equation <ref>, and FAR is computed as the ratio between false positives (FP) and the sum of true and false positives, as seen in Equation <ref>.
Since YOLO is a probabilistic model, it outputs a confidence metric for each object detected in the image frame. In this work, we evaluate the performance metrics DR and FAR for all baselines and the proposed method for four confidence thresholds: 25, 50, 75, and 90%.
Each specific condition is measured, explained, and illustrated bellow and qualitative results can be seen in the complete prediction videos for each condition shown in the project webpage[Project page: <https://sites.google.com/view/tamudrone-spie2020/>.]. 



    DR = TP/TP+FN




    FAR = FP/TP+FP


Our first experiment, shown in Figure <ref>, has a drone crossing above and below the treeline and horizon.
This scenario is problematic for solutions that solely rely on the visible spectrum because of the drone’s low contrast relative to the background.
The infrared sensor, shown in the middle of Figure <ref>, has a significantly higher contrast compared to the RGB sensor, due to salient heat emission from the drone batteries and motors.
This allows the sensor to keep track of the drone even when it crosses above and below the skyline.
In comparison, our proposed method, LWIR+RGB, presents a better horizon contrast when compared to LWIR and RGB alone.



Figure <ref> illustrates our second experiment with the drone flying in front of cameras pointed directly at the sun during sunrise.
This is another scenario where the RGB sensor does not have enough contrast for the proper detection of the sUAS.
Using both the visible and infrared spectrum, our solution can detect the drone with much less difficulty, while also rejecting the false positives that arise from purely relying on the infrared spectrum.



Our third experiment, shown in Figure <ref>, illustrates the solution’s robustness to other heat sources such as public street lighting.
This scenario is problematic for solutions that solely rely on the infrared spectrum because of the similarities of drone's and lamp's heat signatures at low resolution.
By combining both LWIR and RGB images, our method is still capable to leverage the heat signature of the drone to improve detection but also filters out light bulbs because they are clearly distinct from drones in the visible spectrum.
In this scenario, RGB alone still fails when the drones are below the horizon and in front of the trees.



In our final experiment, shown in Figure <ref>, we demonstrate our proposed method capabilities detecting drones when the sUAS is barely visible on the RGB spectrum, here called long-range to differentiate of the previous cases where the vehicles were flying at less than 300 meters away.
This scenario is challenging to LWIR alone due to the reduced resolution and still challenging for RGB alone depending on how the drone contrasts with the background.
As before, our proposed method still presents higher performance metrics for being able to combine both sensors, as explained in the following paragraph.



Table <ref> and <ref> show the numerical results in terms of mean and standard deviation of Detection Rate (DR) and False Alarm Rate (FAR), respectively for each table, averaging the metrics for all scenarios and also showing them separately for cases where a single or multiple drones were flying at the same time, for all confidence threshold (CT) conditions of the machine learning model output.
In bold we highlighted the best performing method for each condition, showing that our method outperforms the baselines 87.5% of the time.
Improvements in the DR ranges from 12.9 to 880.4% while the reduction in FAR ranges from -15.4 to -100.0%, depending on the scenario.
Overall, setting CT to 50% presents a good balance of DR and FAR for all conditions investigated.
Qualitative results can be seen in the complete prediction videos for each condition shown in the project webpage[Project page: <https://sites.google.com/view/tamudrone-spie2020/>.]. 
















§ CONCLUSION


In this paper, we have demonstrated that using state-of-the-art deep learning algorithms for object detection with the insight of combining long-wave infrared (LWIR) and visible spectrum (RGB) sensors can significantly improve the detection of sUAS in previously difficult situations as, for example, when sUAS flies crossing above and below the treeline and horizon line, in the presence of other heat sources, with cameras pointed directly to the sun, and in long-range scenarios.
We compared our approach of combining LWIR and RGB images and detecting drones using a custom trained YOLO model against the same YOLO architecture but trained only using LWIR and RGB images alone.
Our approach achieved a detection rate of 71.2 ± 8.3%, improving by 69% when compared to LWIR and by 30.4% when visible spectrum alone, and achieved false alarm rate of 2.7 ± 2.6%, decreasing by 74.1% and by 47.1% when compared to LWIR and visible spectrum alone, respectively, on average, for single and multiple drone scenarios, controlled for the same confidence metric of the machine learning object detector of at least 50%.
We believe that the further development of this technology has serious benefits to the area of counter-sUAS and can supplement existing technology in practical applications and enable counter-sUAS systems to operate in a 24/7 continuous regime and prevent malicious use of the sUAS.






§ LIMITATIONS AND FUTURE WORK


There are still some inherent limitations to the vision-based approaches for object detection.
One of those limitations stems from scenarios where the drone is indistinguishable from objects with similar visual characteristics and heat signatures.
Although the multi-modal approach mitigated this issue, the system had difficulty distinguishing the drone from the birds during long-range detection since both objects have a similar heat signatures at low resolution.
Besides using higher-resolution sensors and collecting more training data, one avenue we are exploring to further mitigate this limitation uses a filtering algorithm, like Deep SORT <cit.>, to track drones and other objects over the individual frames.
From there, each object's flight characteristics are used as another input for detection and provide a more educated estimate for indiscernible heat signatures. 
In combination with other sensor modalities like a radar and spectrum analyzer, the solution presented could offer an accurate and robust solution to sUAS detection.




 
The authors would like to thank every other participant of the “A-Team", winning team of the A-Hack-of-the-Drones 2018, where this project started: Edan Coben, Emily Fojtik, Garrett Jares, Humberto Ramos, Nicholas Waytowich, Niladri Das, Sunsoo Kim, Vedang Deshpande, Venkata Tadiparthi, and Vernon Lawhern; Texas A&M University Professors: Dr. Daniel Ragsdale, Dr. John Hurtado, and Dr. Raktim Bhattacharya, for supporting the students in the event; Rodney Boehm from Texas A&M University Engineering Entrepreneurship program for supporting the work after the event; and MD5 and Army Future Commands for sponsoring the event and following up with the team to continue the work after the hackathon.


 
spiebib 

