Hyperspectral cameras can provide unique spectral signatures for consistently distinguishing materials that can be used to solve surveillance tasks. In this paper, we propose a novel real-time hyperspectral likelihood maps-aided tracking method (HLT) inspired by an adaptive hyperspectral sensor. A moving object tracking system generally consists of registration, object detection, and tracking modules. We focus on the target detection part and remove the necessity to build any offline classifiers and tune a large amount of hyperparameters, instead learning a generative target model in an online manner for hyperspectral channels ranging from visible to infrared wavelengths. The key idea is that, our adaptive fusion method can combine likelihood maps from multiple bands of hyperspectral imagery into one single more distinctive representation increasing the margin between mean value of foreground and background pixels in the fused map. Experimental results show that the HLT not only outperforms all established fusion methods but is on par with the current state-of-the-art hyperspectral target tracking frameworks.
@cite_30: We consider methods to address the optical feature-aided remote sensing tracking problem for vehicles in a challenging environment. Our approach is to apply the dynamic data driven application systems computing paradigm to implement control of an adaptive sensor. This adaptive sensor acquires a panchromatic image while simultaneously allowing the collection of visible-near infrared spectral data at specified pixels. This sensor holds the promise of delivering the increased accuracy of targeted spectral sensing without the enormous data volume of full spectral images. The target of interest is optimally imaged by the sensor based on the target's forecasted location and motion relative to the extracted content of the background. Background context is both extracted from the image and created from the OpenStreetMap road network. We describe the implementation of the tracking framework and testing of some of the components using simulated imagery created with the digital imaging and remote sensing image generation model. The Gaussian sum filter is employed to solve the data assimilation problem by forming a multimodel forecasting set that is used to increase the robustness and flexibility of tracking. For feature matching, we create an efficient sampling strategy that is informed by the viewing conditions to adaptively determine which pixels to measure spectrally in order to distinguish between different targets using a spectral distance measure.
@cite_22: Visual object tracking can be considered as a figure-ground classification task. In this paper, different features are used to generate a set of likelihood maps for each pixel indicating the probability of that pixel belonging to foreground object or scene background. For example, intensity, texture, motion, saliency and template matching can all be used to generate likelihood maps. We propose a generic likelihood map fusion framework to combine these heterogeneous features into a fused soft segmentation suitable for mean-shift tracking. All the component likelihood maps contribute to the segmentation based on their classification confidence scores (weights) learned from the previous frame. The evidence combination framework dynamically updates the weights such that, in the fused likelihood map, discriminative foreground background information is preserved while ambiguous information is suppressed. The framework is applied here to track ground vehicles from thermal airborne video, and is also compared to other state-of-the-art algorithms.
@cite_9: ABSTRACT Object tracking in urban environments is an important and challenging problemthat is traditionally tackled usingvisibleand nearinfraredwavelengths.By inserting extended datasuchasspectralfeatures of theobjectsonecan improve the reliability of the identication process.However, huge increase in data created by hyperspectralimaging is usually prohibitive.To overcome the complexity problem, we propose a persistent air-to-ground targettrackingsystem inspiredby a state-of-the-art,adaptive, multi-modal sensor.The adaptive sensoris capable ofprovidingpanchromatic images as wellasthespectraof desiredpixels. This addresses the datachallengeofhyperspectraltracking by onlyrecordingspectral dat aasneeded.Spectral likelihoods are integratedinto a dataassociationalgorithm in a #ayesian fashionto minimze the likelihoodof misidentication.A framework forcontrolling spectral data collection is developed by incorporating motion segmentation information and priorinformationfrom a GaussianSum lter(GSF) movement predictionsfrom a multi-modelforecasting set. An intersectionmask ofthesurveillancearea is extractedfrom OpenStreetMapsourceand incorporatedinto thetrackingalgorithm to performonline renement ofmultiplemodel set.The proposedsystem istestedusingchallenging and realisticscenarios generated in an adverse environment.Keywords: spectral likelihoods, persistent vehicle tracking, performance-driven sensor, adaptive forecasting
@cite_32: Hyperspectral imagery has the capability of capturing spectral features of interest that can be used to differentiate among similar materials. While hyperspectral imaging has been demonstrated to provide data that enable classification of relatively broad categories, there remain open questions as to how fine of discrimination is possible. An application of this fine discrimination question is the potential that spectral features exist in the surface reflectance of ordinary civilian vehicles that would enable tracking of a particular vehicle across repeated hyperspectral images in a cluttered urban area. To begin to explore this question a vehicle tracking experiment was conducted in the summer of 2005 on the Rochester Institute of Technology (RIT) campus in Rochester, New York. Several volunteer vehicles were moved around campus at specific times coordinated with over flights of RIT's airborne Modular Imaging Spectrometer Instrument (MISI). MISI collected sequential images of the campus in 70 spectral channels from 0.4 to 1.0 microns with a ground resolution of approximately 2.5 meters. Ground truth spectra and photographs were collected for the vehicles. These data are being analyzed to determine the ability to uniquely associate a vehicle in one image with its location in a subsequent image. Initial results have demonstrated that the spectral measurement of a specific vehicle can be used to find the same vehicle in a subsequent image, although this is not always possible and is very dependent upon the specifics of the situation. Additionally, efforts are presented that explore predicted performance for variations in scene and sensor parameters through an analytical performance prediction model.
@cite_24: Vehicle tracking from a moving aerial platform poses a number of unique challenges including the small number of pixels representing a vehicle, large camera motion, and parallax error. This paper considers a multi-modal sensor to design a real-time persistent aerial tracking system. Wide field of view (FOV) panchromatic imagery is used to remove global camera motion whereas narrow FOV hyperspectral image is used to detect the target of interest (TOI). Hyperspectral features provide distinctive information to reject objects with different reflectance characteristics from the TOI. This way the density of detected vehicles is reduced, which increases tracking consistency. Finally, we use a spatial data based classifier to remove spurious detections. With such framework, parallax effect in non-planar scenes is avoided. The proposed tracking system is evaluated in a dense, synthetic scene and outperforms other state-of-theart traditional and aerial object trackers.
@cite_27: We consider the optical remote sensing tracking problem for vehicles in a complex environment using an adaptive sensor that can take spectral data at a small number of locations. The Dynamic Data-Driven Applications Systems (DDDAS) paradigm is well-suited for dynamically controlling such an adaptive sensor by using the prediction of object movement and its interaction with the environment to guide the location of spectral measurements. The spectral measurements are used for target identification through feature matching. We consider several adaptive sampling strategies for how to assign locations for spectral measurements in order to distinguish between multiple targets. In addition to guiding the measurement process, the tracking system pulls in additional data from OpenStreetMap to identify road networks and intersections. When a vehicle enters a detected intersection, it triggers the use of a multiple model prediction system to sample all possible turning options. The result of this added information is more accurate predictions and analysis from data assimilation using a Gaussian Sum filter (GSF).
@cite_16: Tracking vehicles through dense environments is an important and challenging task that is mostly tackled using visible and near IR wavelengths. Hyperspectral imaging is known to improve the robustness of target identification, but the massive increase in data created is usually prohibitive for tracking many targets. We present a persistent real-time aerial target tracking system, taking advantage of an adaptive, multimodal sensor concept and blending the hyperspectral likelihoods with kinematic likelihoods in a multidimensional assignment framework. The adaptive sensor is capable of providing wide field of view panchromatic images as well as the spectra of small number of pixels. The proposed system does not require large amount of hyperspectral data collection as we focus on tracking fewer number of targets with higher persistency. This overcomes the data challenge of hyperspectral tracking by following dynamic data-driven application systems (DDDAS) principles to control hyperspectral data collection where most beneficial. The DDDAS framework for controlling hyperspectral data collection is developed by incorporating prior information from the filter movement predictions and information from motion detection. The proposed multidimensional hyperspectral feature-aided tracker is compared to a 2-D hyperspectral feature-aided tracker and another cascaded hyperspectral data based tracker by generating a synthetic, realistic, aerial video on a dense scene.
@cite_25: Very large format video or wide-area motion imagery (WAMI) acquired by an airborne camera sensor array is characterized by persistent observation over a large field-of-view with high spatial resolution but low frame rates (i.e. one to ten frames per second). Current WAMI sensors have sufficient coverage and resolution to track vehicles for many hours using just a single airborne platform. We have developed an interactive low frame rate tracking system based on a derived rich set of features for vehicle detection using appearance modeling combined with saliency estimation and motion prediction. Instead of applying subspace methods to very high-dimensional feature vectors, we tested the performance of feature fusion to locate the target of interest within the prediction window. Preliminary results show that fusing the feature likelihood maps improves detection but fusing feature maps combined with saliency information actually degrades performance.
@cite_17: Abstract Vehicle tracking in adverse environments is a challenging problem because of the high number of factors constraining their motion and possibility of frequent occlusion. In such conditions, identification rates drop dramatically. Hyperspectral imaging is known to improve the robustness of target identification by recording extended data in many wavelengths. However, it is impossible to transmit such a high rate data in real time with a conventional full hyperspectral sensor. Thus, we present a persistent ground-based target tracking system, taking advantage of a state-of-the-art, adaptive, multi-modal sensor controlled by Dynamic Data Driven Applications Systems (DDDAS) methodology. This overcomes the data challenge of hyperspectral tracking by only using spectral data as required. Spectral features are inserted in a feature matching algorithm to identify spectrally likely matches and simplify multidimensional assignment algorithm. The sensor is tasked for spectra acquisition by the prior estimates from the Gaussian Sum Filter and foreground mask generated by the background subtraction. Prior information matching the target features is used to tackle false negatives in the background subtraction output. The proposed feature-aided tracking system is evaluated in a challenging scene with a realistic vehicular simulation.
Our work is inspired by Yin, Porikli and Collins @cite_22 who proposed a single target tracker for aerial imagery and @cite_25 who used different set of visual features to extract feature likelihood maps and adaptively fuse them to obtain a single fusion map. While hyperspectral imagery has shown to have potential for vehicle tracking @cite_32 @cite_16 @cite_24 @cite_30 @cite_27 @cite_17 @cite_9 , the sensors are rare and expensive and hence most of the aerial tracking work has been done in infrared, single-band or RGB. In this section, we review the efforts done in the above sensor domains to solve vehicle detection and tracking and how hyperspectral imagery could possibly solve some of the problems.