Thermophysical Algebraic Invariants from Infrared Imagery for Object Recognition.
AbstractAn important issue in developing a model-based vision system is the specification of features that are invariant to viewing and scene conditions and also specific, i.e., the feature must have different values for different classes of objects. We formulate a new approach for establishing invariant features. Our approach is unique in the field since it considers not just surface reflection and surface geometry in the specification of invariant features, but it also takes into account internal object composition and state which affect images sensed in the nonvisible spectrum. A new type of invariance called Thermophysical Invariance is defined. Features are defined such that they are functions of only the thermophysical properties of the imaged objects. The approach is based on a physics-based model that is derived from the principle of the conservation of energy applied at the surface of the imaged object.
Introduction
Non-visible modalities of sensing have been shown to greatly increase the amount of information
that can be used for object recognition. A very popular and increasingly affordable sensor
modality is thermal imaging - where non-visible radiation is sensed in the long-wave infrared
(LWIR) spectrum of 8-m to 14-m. The current generation of LWIR sensors produce images
of contrast and resolution that compare favorably with broadcast television quality visible light
imagery. However, the images are no longer functions of only surface reflectance. As the
wavelength of the sensor transducer passband increases, emissive effects begin to emerge as
the dominant mode of electromagnetic energy exitance from object surfaces. The (primarily)
emitted radiosity of LWIR energy has a strong dependence on internal composition, properties,
and state of the object such as specific heat, density, volume, heat generation rate of internal
sources, etc. This dependence may be exploited by specifying image-derived invariants that
vary only if these parameters of the physical properties vary.
In this paper we describe the use of the principle of conservation of energy at the surface
of the imaged object to specify a functional relationship between the object's thermophysical
properties (e.g., thermal conductivity, thermal capacitance, emissivity, etc.), scene parameters
(e.g., wind temperature, wind speed, solar insolation), and the sensed LWIR image gray level.
We use this functional form to derive invariant features that remain constant despite changes in
scene parameters/driving conditions. In this formulation the internal thermophysical properties
play a role that is analogous to the role of parameters of the conics, lines and/or points that are
used for specifying geometric invariants (GI's) when analyzing visible wavelength imagery [1].
Thus, in addition to the currently available techniques of formulating features that depend only
on external shape [2] - [10], and surface reflectance properties [11] - [16], the phenomenology of
image generation can be used to establish new features that "uncover" the composition
and thermal state of the object, and which do not depend on surface reflectance characteristics.
An intuitive approach to thermo-physical interpretation of LWIR imagery is given in [17].
This approach rests upon the following observation, termed the "Thermal History Consistency
Constraint" and analogous to Lowe's well known Viewpoint Consistency Constraint [18]: "The
temperature of all target features for a passive target must be consistent with the heat flux
transfer resulting from exposure to the same thermal history." In [17] this constraint is exploited
by analyzing objects to locate components that are similar in terms of thermo-physical
properties and then examining a temporal sequence of calibrated LWIR data to experimentally
assess the degree to which such thermo-physically similar components exhibit similar temperature
state temporal behavior. Such analysis was shown to lead to formulation of simple intensity
ratio features exhibiting a strong degree of temporal stability that could be exploited provided:
(1) thermally homogeneous regions in the LWIR image corresponding to the thermo-physically
similar object components could be reliably segmented, and (2) a target-specific geometric reference
frame is available in order to correctly associate extracted regions with candidate object
components.
To avoid the difficulties inherent in assumptions (1) and (2) above an alternative technique
applicable to overall object signatures was suggested in [17]. An analysis of typical LWIR
'lumped parameter' object temperature modeling approaches suggests that over small time
scales object temperature can be crudely modeled by a small dimensional linear system with
algebraically separable spatial and temporal components. Ratios of spatial integrals of temperature
with a simple set of orthonormal 2D polynomials (obtained from applying Gramm-Schmidt
to of the resulting functions were nearly constant
with time when measured against 24 hours of LWIR imagery of a complex object (a tank)
but no experimentation was done with multiple objects to examine between and within-class
separation, so little can be drawn in the way of a substantive conclusion with respect to utility
as an object identification technique.
A physics-based approach that attempts to establish invariant features which depend only
on thermophysical object properties was reported in [19], [20]. A thermophysical model was
formulated to allow integrated analysis of thermal and visual imagery of outdoor scenes. This
method used estimations of the energy flux into and out of the surface of the object. The surface
orientation and absorptivity were obtained from the visual image using a simplified shape-from-
shading method. The surface temperature was estimated from the thermal image based on an
appropriate model of radiation energy exchange between the surface and the infrared camera.
A normalized feature, R, was defined to be the ratio of energy fluxes estimated at each pixel.
The value of R was lowest for vehicles, highest for vegetation and in between for buildings and
pavements. This approach is powerful in that it makes available features that are completely
defined by internal object properties. The computed value of R may be compared with accurate
ground truth values computed from known physical properties of test objects - one of the major
advantages of using physics-based/phenomenological models as compared to statistical models.
Classification of objects using this property value is discussed in [21], [22].
There are several factors that limit the performance of the above thermophysical approach.
The thermal and visual image pairs may not be perfectly registered. Also, segmentation errors
typically cause a large portion of an object to be included with small portions of a different
object in one region. This results in meaningless values of the surface energy estimates at/near
the region boundaries. These errors give rise to a significant number of inaccurate estimates
of the surface energy exchange components. The histogram of values of the ratio of energy
fluxes tends to be heavy-tailed and skewed. A statistically robust scheme has been proposed to
minimize this drawback [22]. However, the computational complexity for such a technique is
very high, and the following drawbacks below were not adequately overcome: (1) The value of
R was found to be only weakly invariant - while separation between classes was preserved, the
range of values of this feature for each class was observed to vary with time of day and season of
year, (2) The feature was able to separate very broad categories of objects, such as automobiles,
buildings, and vegetation - but lacked the specificity to differentiate between different models of
vehicles. Although the thermophysical feature, R, is limited in its application for recognition,
the energy exchange model on which it is based forms the groundwork for the derivation of
more powerful thermophysical invariant features. That approach is extended in this paper by
applying the concepts of algebraic invariance to the energy exchange model resulting in new
thermophysical invariant features for object recognition.
The derivation of thermophysical invariants (TI's) from non-visible wavelength imagery, the
evaluation of the performance of these invariants, and their use in object recognition systems
poses several advantages. The main advantage of this approach is the potential availability of
a number of new (functionally independent) invariants that depend on internal compositional
properties of the imaged objects. Note that it is possible to evaluate the behavior of thermophysical
invariants using ground truth data consisting of images of objects of known composition
and internal state. This additional information can be used to augment/complement the behavior
of GI's. One way in which GI's can be integrated with TI's for object recognition is as
follows: (1) Parametric curves and/or lines are extracted from an LWIR image. (2) The curves
are used to compute GI's which are in turn used to hypothesize object identity and pose, and
(3) TI's are computed for this hypothesis and compared to a stored model library for verifica-
tion. Some details of this approach are presented later. Although the TI's are used solely for
the verification of hypothesis generated by other means, this task is of primary importance in
a number applications such as site change detection, monitoring, and surveillance.
The ideas presented in this paper are continuations/extensions of previous and ongoing
research in thermophysical model-based interpretation of LWIR imagery. A brief description
of this thermophysical approach is presented in section 2. The formulation of a new method to
T , a
cv
Air T amb
W cnd
st

Figure

1: Energy exchange at the surface of the imaged object. Incident energy is primarily
in the visible spectrum. Surfaces loses energy by convection to air, and via radiation to the
atmosphere. An elemental volume at the surface is shown. Some of the absorbed energy raises
the energy stored in the elemental volume, while another portion is conducted into the interior
of the object.
derive thermophysical invariants is described in section 3. Section 4 describes a context based
approach that proposes the thermophysical feature in a hierarchical framework. Experimental
results of applying this new approach to real imagery are presented in section 5, which is
followed by a discussion of the behavior of the new method, issues to be considered in using
this method for object recognition, and issues that remain to be explored.
Thermophysical Approach to IR Image Analysis
Consider an infinitesimal volume at the surface of the imaged object (figure 1). Energy absorbed
by the surface equals the energy lost to the environment.
lost (1)
Energy absorbed by the surface (per unit surface area) is given by
where, W I is the incident solar irradiation on a horizontal surface per unit area and is given by
available empirical models (based on time, date and latitude of the scene) or by measurement
with a pyranometer, ' i is the angle between the direction of irradiation and the surface normal,
and ff s is the surface absorptivity which is related to the visual reflectance ae s by ff
Note that it is reasonable to use the visual reflectance to estimate the energy absorbed by the
surface since approximately 90% of the energy in solar irradiation lies in the visible wavelengths
[23].
The energy lost by the surface to the environment was given by
lost st
where, W cv denotes the energy (per unit surface area) convected from the surface to the air
which has temperature T amb and velocity V , W rad is the energy (per unit surface area) lost
by the surface to the environment via radiation and W cnd denotes the energy (per unit surface
area) conducted from the surface into the interior of the object. The radiation energy loss is
computed
amb
where, oe denotes the Stefan-Boltzman constant, T s is the surface temperature of the imaged
object, and T amb is the ambient temperature. Assume ffl for the atmosphere is equal to ffl for the
imaged object. This assumption is reasonable if the objects are not uncoated metals [24]. This
assumption may not hold if the imaged surface is exposed or unoxidized metal which is usually
rare.
The convected energy transfer is given by
where, h is the average convected heat transfer coefficient for the imaged surface, which depends
on the wind speed, thermophysical properties of the air, and surface geometry [23].
abs
st
R cv
R rad
W rad W cv
W cnd

Figure

2: The equivalent thermal circuit for the extended model that separates the stored
energy component and the conduction component to the interior of the object.
The equivalent thermal circuit for the surface is shown in figure 2. Lateral conduction from
the elemental volume at the surface is assumed negligible since the temperature of the material
adjacent to the surface volume under consideration may be assumed to be similar. In general,
the internal temperature of the material will be different from that at the surface. The energy
flow due to this gradient is expressed as the conducted energy, W cnd = \Gammak dT=dx, where k is
the thermal conductivity of the material, and x is distance below the surface. Since we are
considering an elemental volume at the surface this can be written in finite difference form:
\Deltax
, for infinitesimal \Deltax. W cnd is also expressed in units of energy flowing
across unit area.
Within the elemental volume, the temperature is assumed uniform, and the increase in the
stored energy given by W st
dTs
dt
, where C T is the thermal capacitance for the material of
the elemental surface volume. This is given by C is the density of the surface
material, V is the elemental volume, and c is the specific heat. Again, W st is expressed in units
of energy per unit surface area. The equivalent circuit (shown in figure 2) have resistances given
by:
and R
3 Thermophysical Algebraic Invariants, TAI's
The energy balance equation, W abs st +W cnd may be rewritten in the following
linear
a
Using the expressions for the various energy components as presented in the previous section
we can express each term in the above expression as:
dt
dx
a
a
a
Note that a calibrated LWIR image provides radiometric temperature. However, this requires
knowledge of the emissivity, ffl, of the surface. For common outdoor materials, and
common paints and surface coatings, the value of ffl is around 0.9 [25],[26]. Therefore, the radiometric
temperature, T s , may be computed based on the assumption, 0:9. Hence a 3 and
a 4 can be computed from the LWIR image alone (and knowledge of the ambient temperature),
while a 1 , a 2 and a 5 are known when the identity and pose of the object is hypothesized. The
"driving conditions", or unknown scene parameters that can change from scene to scene are
given by the x 5. Thus each pixel in the thermal image eqn (8) defines a point in
5-D thermophysical space.
Consider two different LWIR images of a scene obtained under different scene conditions and
from different viewpoints. For a given object, N points are selected such that (a) the points are
visible in both views, and (b) each point lies on a different component of the object which differs
in material composition and/or surface orientation. Assume (for the nonce) that the object pose
for each view, and point correspondence between the two views are available (or hypothesized).
A point in each view yields a measurement vector
defined by eqn (8) and a corresponding driving conditions vector -
a collection of N of these vectors compose a 5 \Theta N matrix, for the first
scene/image. These same points in the second scene will define vectors that compose a 5 \Theta N
). The driving condition matrix,
, from the
first scene and X
from the second scene, are each of size N \Theta 5.
Since the N points are selected to be on different material types and/or different surface
orientations, the thermophysical diversity causes the N vectors -a
also the vectors, -a 0
N . Without loss of generality, assume that five vectors -a
span ! 5 and also that -a 0
These five points in ! 5 specify the 5 \Theta 5 measurement
matrices, in the first scene and A
5 ), in the second scene.
The point selection process here is analogous to the selection of characteristic 3D points in
the construction of geometric invariants. Since A and A 0 are of full rank, there exists a linear
transformation
Since from (7),
we have,
Thus each driving condition vector also undergoes a linear transformation.
Consider the measurement vector -a of a point as defined in (8). From one scene to the next
we expect the two object properties - thermal capacitance, C T , and conductance, k - to remain
constant. Thus the transformation we need to consider is seen to be a subgroup of GL(5) which
has the form M
The transformation of a measurement vector from one scene to the next is given
a 2
a 3
a 4
a
The first two elements, the thermal capacitance and the thermal conductance, are held constant
and the other scene dependent elements are allowed to change. In general, we have
where A and A 0 are the 5 \Theta 5 matrices derived from the two scenes and for the chosen points.
Now that the transformation of the point configuration has been established in 5D thermophysical
space, we ask the question - what function of the coefficients, a i;j ; is invariant to
transformations of the form - equation (12)? In answer to this question we first determine the
expected number of invariant relationships. The transformation group represented by M has
parameters. The measurement matrix A has 25 degrees of freedom. The counting argument
described by Mundy [1] and others is that the number of invariant relationships is equal to "the
degrees of freedom of the configuration" minus "the number of transformation parameters".
Here a count yields 10 invariant relationships for a configuration of five points. However, the
counting argument also shows us that it is unnecessary to use all five points. Each point has
five degrees of freedom. In order for invariant relationships to exist, a minimum of four points
can be used. Using four points the counting argument gives
relations. Note that this further simplifies the point selection since we now require only four
points in the two views. The measurement vectors in each view being required to span ! 4 .
Algebraic elimination of the transformation parameters using four copies of the linear form
(7) subjected to the transformation (12) provides us with the invariants. This elimination may
be performed by using recently reported symbolic techniques [27].
The five invariant functions derived by this elimination process can be divided into two
types. Each is a ratio of determinants. Indeed, it is well known that absolute invariants
of linear forms are always ratios of powers of determinants [28]. The first type of invariant
function is a determinant formed from components of three of the four vectors.
a 1;3 a 2;3 a 4;3
a 1;4 a 2;4 a 4;4
a 1;5 a 2;5 a 4;5
a 2;3 a 3;3 a 4;3
a 2;4 a 3;4 a 4;4
a 2;5 a 3;5 a 4;5
where a i;j is a jth component of the ith vector (ith point).
The second type is formed from components of all four vectors.
a 1;1 a 2;1 a 3;1 a 4;1
a 1;2 a 2;2 a 3;2 a 4;2
a 1;4 a 2;4 a 3;4 a 4;4
a 1;5 a 2;5 a 3;5 a 4;5
a 1;1 a 2;1 a 3;1 a 4;1
a 1;3 a 2;3 a 3;3 a 4;3
a 1;4 a 2;4 a 3;4 a 4;4
a 1;5 a 2;5 a 3;5 a 4;5
where a i;j is a jth component of the ith vector (ith point). Since the 4 measurement vectors
we can assume without loss of generality that the denominator determinants in (14)
and (15) are non-zero. The number of independent functions that can be formed with four
points must add up to the expected number from the counting argument. The first type hasB B @3C C independent functions given four points and second type has one. The counting
argument is thus satisfied.
The derivation of thermophysical algebraic invariant features, as described above, relies on
a number of assumptions. These assumptions are summarized. (1) Four points are chosen
such that measurement vectors (in each scene) are linearly independent, i.e. one (or more) of
the four points have different material properties and/or surface normal. (2) The four points
are related by a linear transformation of the form (12) from one scene to another. (3) The
points are corresponded. (4) Object identity is hypothesized (which is verified or reputed by
the feature value). (5) The thermal capacitance and conductance of the object (surface) do
not change while other other scene variable are allowed to vary from one scene to another. (6)
Emissivity of the imaged surface in the 8-m \Gamma 12-m band is 0.9. The first two assumptions go
hand in hand, the vectors are linearly independent by proper choice of the points, i.e. model
formation. Most objects have sufficient diversity in surface material types to easily satisfy this
requirement. Given such proper model formation, the existence of linear transformation M is
trivially ensured. The point correspondence and the hypothesis assumptions are satisfied in
the method of application of the features, described in section 3.1. Assumption (6) is satisfied
except for bare metal surfaces and esoteric low emissivity coatings.
In order for the invariant feature to be useful for object recognition not only must the
values of the feature, be invariant to scene conditions but the value must be different if the
measurement vector is obtained from a scene that does not contain the hypothesized object,
and/or if the hypothesized pose is incorrect. Since the formulation above takes into account only
feature invariance but not separability, a search for the best set of points that both identifies
the object and separates the classes must be conducted over a given set of points identified on
the object. The search may be conducted over all the combination of the points in a set or
until an acceptable feature is found. We have examined all combinations, first rating each set
for their intra-class invariance, then further evaluating it for inter-class separability. Results on
real imagery are described in section 5.
3.1 Employing TAI's for Object Recognition
The feature computation scheme formulated above is suitable for use in an object recognition
system that employs a hypothesize-and-verify strategy. The scheme would consist of the following
steps: (1) extract geometric features, e.g., lines and conics, (2) for image region, r,
hypothesize object class, k, and pose using, for example, geometric invariants as proposed by
Forsyth, et al [2], (3) use the model of object k and project visible points labeled
onto image region r using scaled orthographic projection, (4) for point labeled i in the image
region, assign thermophysical properties of point labeled i in the model of object k, (5) using
gray levels at each point and the assigned thermophysical properties, compute the measurement
matrices A and A 0 , and hence compute the feature f k (r) using equation (14) or equation (15),
and finally, (6) compare feature f k (r) with model prototype -
f k to verify the hypothesis.
For example, consider two class of vehicles - a van and a car as shown in figure 3. Here,
the correct hypotheses (models) are shown in the top row. The bottom row indicate incorrect
hypotheses with model points being assigned to the image regions. The front and rear wheels are
detected and used to establish an object centered reference frame in the image to be interpreted.
The coordinates of the selected points are expressed in terms of this 2D object centered frame.
Thus, when a van vehicle is hypothesized for an image actually obtained of a car or some
unknown vehicle, the material properties of the van are used, but image measurements are
obtained from the image of the car at locations given by transforming the coordinates of the
van points (in the van center coordinate frame) to the image frame computed for the unknown
vehicle.
4. Varying Contextual Support
It is widely known that the explicit use of contextual knowledge can improve scene interpretation
performance. Such contextual knowledge has been, typically, in the form of spatial
and geometric relationships between scene objects or between different components of an ob-
ject. Contextual support for an image region (point) being labeled consists of the neighboring
regions (points), the class memberships of which influence the class assignment for the region
(point) under consideration. In the feature extraction scheme described in section 3, contextual
knowledge is used in an implicit manner. Recall that a set of points is hypothesized to belong
to a specific type of object, and this hypothesis is verified. Thus, the class identity of each of
Figure

3: Top row - the car and van object types with points selected on the surface with
different material properties and/or surface normals. For the van - Point 1: Vulcanized Rubber,
2: Aluminum Alloy, 3: Polystyrene-like polymer, 4: Steel, 5: Polypropylene-like polymer,
Steel, 7: Polypropylene-like polymer For the car - Point 1: Vulcanized Rubber, 2: Steel, 3: Steel,
4: Chromed Steel, 5: Steel, 7: Carbon Steel. Bottom row - assignment
of point labels under erroneous hypotheses. First, the two wheels of the vehicle are used to
establish a local, 2D, object-centered coordinate frame. Then, the 2D to 2D transformation
between model and image is determined, and the model points and labels are transformed to
the image.
these points is made in the context of the identities of the other points.
The computation of feature I2 described in section 3 requires 4 points while the computation
of I1 requires only 3 points. Thus, I1 and I2 require different amounts of contextual support.
In some situations the available, useful, contextual support may be even lower, since it may be
possible to extract only one or two points reliably, e.g., where the object is partially occluded
or if the specific aspect view contains only a homogeneous surface. The formulation of feature
I1 does allow for this reduction in contextual support as explained below.
The variation in context may also be viewed as a variation in the dimensionality of a
specific predetermined subspace of the thermophysical feature space. The dimensionality of
the subspace determines the number of measurements required from the thermal image, and
hence the degree of context used. As the number of image measurements is increased, fewer
dimensions are required in the precomputed subspace and more context is derived from the
imaged object. In the method presented below, a subspace in thermophysical measurement
space is precomputed for each invariant feature established for an object class.
Consider the formulation of feature I1 which can be expressed in the form
where the column vectors - b i that compose -
. For non-zero invariants, the column
vectors -
b i that compose B also span ! 3 . In the above, we have points in an n-D space,
3. The points are divided into two sets each consisting of n points, and the two sets
differ by one element, and share
The measurement matrices in (16) may be expressed, in general, as:
and -
Consider ~j, which spans the null space of B
0: Note that B
3. The feature value, jBj
; is non-infinite and
non-zero if and only if - b T
Furthermore, it can be shown that:
The proof of this statement is included in appendix A.
For convenience of terminology, let us call the vector ~j the null-space vector. The components
of the null-space vector can be found by,
Consider instances of the null-space vector defined above for different scene conditions.
Each scene condition, in which the object is imaged, results in a different null space vector ~j(j)
For a given object, it may be possible to find a collection of n
and a decomposition into two sets of n points each such that the corresponding the null-space
vectors, ~j(j), for the different scenes are tightly clustered in the measurement space, i.e., they
vary minimally from scene to scene. If this condition is met it is reasonable to use a single
predetermined average null-space vector, ~j   , that characterizes the object irrespective of the
scene conditions. Now, only two measurement vectors, -
bn and -
are needed to compute
the invariant feature - using the image and hypothesized object class, as described in section 3.
Equation (19) becomes,
where f   is the feature value obtained using the average (representative) null-space vector.
In general, one needs to search (during a training phase) for an appropriate set of n
points, and a decomposition of this set into two sets of n points each that share
with the constraint that the null space vectors vary minimally from scene to scene. This will
establish the optimal two-point invariants for the object.
In order for the null-space vector approach to be useful in an object recognition scheme,
separability between the object classes must be ensured. That is, for the correct object hypoth-
esis, the feature value given by equation (21) must be as expected and must remain invariant
to scene conditions while in the case that the object hypothesis is erroneous a value other than
the expected invariant feature value is obtained. Consider a two-class separation problem. The
null-space vectors from different scenes for each object will ideally form a tight cluster. Separability
between these clusters can be measured using any of the many established statistical
measures, e.g. the Mahalanobis distance [29].
The variation in contextual support offered by the above methods may be exploited in a
hierarchal system where the two-point formulation is used for a quick, initial classification, with
low missed detection rate but perhaps high false alarm rate, to eliminate fruitless branches in
a broad search tree. In the following section we present results illustrating the classification
ability of the method described in section 3 as well as that of the null-space vector approach
for multiple objects.
4 Experimental Results
The method of computing thermophysical affine invariants discussed above was applied to real
imagery acquired at different times of the day. Several types of vehicles were imaged:
A van containing mostly plastic (composite) body panels, a car made entirely of sheet-metal
body panels, a military tank, and two different military trucks (Figs. 3,4). Several points were
selected (as indicated in the figures) on the surfaces of different materials and/or orientation.
The measurement vector given by eqn (8) was computed for each point, for each image/scene.
Figure

4: Three of the vehicles used to test the object recognition approach, (from top left
clockwise) tank, truck 1 and truck 2. The axis superimposed on the image show the object
centered reference frames. The numbered points indicate the object surfaces used to form the
measurement matrices. These points are selected such that there are a variety of different
materials and/or surface normals within the set.
Hypothesis: Truck 1 Truck 1 Truck 1 Truck 1 Truck 1
Data From: Truck 1 Tank Van Car Truck 2
9 am -0.68 1.38 -1.00 -6.66e14 -7.03

Table

1: Values of the I1-type feature used to the identify the vehicle class, truck 1. The feature
consisted of point set f4; 7; 8; 10g, corresponding to the points labeled in figure 6. The feature
value is formed using the thermophysical model of truck 1 and the data from the respective
other vehicles. When this feature is applied to the correctly hypothesized data of the tank it
has a mean value of -0.57 and a standard deviation of 0.13. This I1-type feature produces a
good stability measure of 4.5, and good separability between correct and incorrect hypotheses.
The feature values for incorrect hypotheses are at least 3.32 standard deviations away from the
mean value for the correct hypothesis.
Hypothesis: Truck 1 Truck 1 Truck 1 Truck 1 Truck 1
Data From: Truck 1 Van Car Tank Truck 2
9 am -0.31 -454.87 -252.78 -9.38 29.9e5

Table

2: Values of the I2-type feature used to the identify truck 1. The feature consisted of point
set f2; 3; 5; 9g, corresponding to the points labeled in figure 4. The feature value is formed using
the thermophysical model of truck 1 and the data from the respective other vehicles. When
this feature is applied to the correctly hypothesized data of truck 1 it has a mean value of -0.49
and a standard deviation of 0.46. The feature values under an incorrect hypothesis are at least
15.2 standard deviations away from the mean value for the correct hypothesis.
Given N points on the object to form the feature of type I1, or of type I2, there are
Each of the nN features have a distribution of values from scene to scene. The non-zero variance
is due to many factors including: (a) how well the linear thermophysical model is satisfied, (b)
calibration errors, and (c) numerical computation errors. In our experiments the features were
computed for all the combinations of points for the different scenes. A large number of four-
point-sets yielded features with low variance from scene to scene. First, the point sets were
rated based on their stability/low variance, where the stability is defined as the average value
of the feature over time, divided by the standard deviation of the feature for that time period.
Next, the most stable features were evaluated for inter-class separation as explained below.
As mentioned in section 3 the feature is computed based on the hypothesized identity of the
object (and it's thermophysical properties). Hence, if the object identity (class membership) k
is hypothesized in the image region r, then the image measurements are used along with the
thermophysical properties of object class k to generate a feature value f k (r). Verification of this
hypothesis may be achieved by comparing f k (r) with a class prototype -
f k . This comparison
may be achieved using any of a number of available classifier rules, e.g., minimum distance
rule. Since it is important that erroneous hypotheses be refuted, one must consider inter-class
behavior as well as intra-class behavior of the feature. To experimentally investigate such
behavior using real imagery we adopted the following procedure. Given an image of a vehicle,
(1) assume the pose of the vehicle is known, then (2) use the front and rear wheels to establish
an object centered reference frame. The center of the rear wheel is used as the origin, and center
of the front wheel is used to specify the direction and scaling of the axes. The coordinates of
the selected points are expressed in terms of this 2D object centered frame. For example, when
a van vehicle is hypothesized for an image actually obtained of a car or some unknown vehicle,
the material properties of the van are used, but image measurements are obtained from the
image of the car at locations given by transforming the coordinates of the van points (in the
van center coordinate frame) to the image frame computed for the unknown vehicle.

Table

shows inter-class and intra-class variation for a feature of type I1 for the truck
object class - for images obtained at eight different times over two days. The behavior of
the invariant feature formed by one choice of a set of 4 points is shown in table 1 for correct
hypothesis. Also, table 1 shows the case where the hypothesized object is the truck 1 while
the imaged object is either a car, van, truck 2, or tank. As can be seen, the correct hypothesis
generates a feature value that is invariant and distant from the feature values generated by
erroneous hypotheses. Thus, the feature is shown to have good characteristics in both (high)
inter-class separability and (low) intra-class stability. Table 2 shows similar results for a feature
of type I2 also used for separating truck 1 from other objects. Table 3 describes the performance
of a feature of type I2 used to separate the tank object class from other classes. Features for
each of the other object classes were also examined, and performance similar to that described
above was observed. Also, each class produced a large number of features (of each type) that
demonstrated good inter-class separation and intra-class stability.
The method described in section 4 was used to compute two-point invariant features for the
5 types of classes examined above. The ten points chosen on the truck 1 vehicle corresponding
to the points in Fig. 4 provide 210 different features. For each selection of a pair of points, the
average null vector is computed over different images under the correct hypothesis. This vector
is then used to compute the feature under mistaken hypotheses. The ability of this feature to
separate correct hypotheses from mistaken hypotheses can be described by the measure
and -m are the mean values for the correct hypothesis and mistaken hypothesis,
respectively, and oe c and oe m are the corresponding standard deviations [30]. The performance
Hypothesis: Tank Tank Tank Tank Tank
Data From: Tank Van Car Truck 1 Truck 2
9 am 1.07 -2.42e5 1 -13.19 -5.31

Table

3: Values of a feature of type I2 used to the identify the tank, for correct and mistaken
hypothesis. The feature consisted of point set f3; 5; 7; 9g, corresponding to the points labeled
in figure 4. The feature value is formed using the thermophysical model of tank and the data
from the respective other vehicles. When this feature is applied to the correctly hypothesized
data of the tank it has a mean value of 2.52 and a standard deviation of 1.82. The feature value
under a mistaken hypothesis is at least 3.3 standard deviations away from the average value
under the correct hypothesis.
Hypothesis: Truck 1 Truck 1 Truck 1 Truck 1
Data From: Tank Van Car Truck 2
Feature 1: 78.13 14.89 3.55 9.33
Feature 2: 75.72 13.01 3.30 7.25
Feature 3: 29.46 6.46 3.29 6.97
Feature 4: 24.98 5.62 3.23 6.81
Feature 5: 16.29 5.42 3.02 5.86

Table

4: Separability measure for the two-point features derived in section 4. The features
were derived for separating the class Truck 1 from other classes. Larger values indicate better
separation between correct hypotheses and erroneous ones.
of the best five features identified for the truck 1 object class is given in table 4. Similar features
are available for the other object classes.
The approach described above is promising in that it makes available features that are (1)
invariant to scene conditions, (2) able to separate different classes of objects, and (3) based on
physics based models of the many phenomena that affect LWIR image generation.
It is important to note that although the derivation of the features explicitly used the
constraint that the value be invariant from one scene to another for a given object class, class
separation was not explicitly incorporated in the derivation of the features. Hence, practical
use of the approach for recognition requires searching all the possible features for the best
separation. It is not clear that a solution will always exist for a collection of object classes.
Note that different aspects of an object may be imaged - the set of visible points being different
for each aspect. The complexity of the search task is compounded by attempting to ensure inter-class
separation in the presence of erroneous pose hypothesis, which we have not considered in
this paper.
The hypothesis of object pose and identity is best achieved by employing geometrical invariance
techniques [2]. For example, conics may be fit to wheels which manifest high contrast
in LWIR imagery, and their parameter values may be used to compute GI's. This may be
employed to generate object identity and pose that may be verified by the thermophysical
invariance scheme described above. This approach is also being investigated at present.
Note that the formulation described in this paper assumes that the scene objects are passive,
i.e., internal thermal sources are not explicitly modeled. The experiments were conducted on
vehicles that had not been exercised prior to (or during) image acquisition. Reformulation of
the scheme presented in this paper to explicitly incorporate internal sources will be a interesting
area of future research.
The specification of optimal classifiers that use as input the features established in this
paper is another area that merits investigation. In our experiments, we have found that the
distribution of the features, especially under erroneous hypotheses, are poorly modeled by the
commonly used Gaussian distribution. We are investigating the use of Symmetric Alpha Stable
Distributions which appear to model the behavior of the features more closely. Our research
in this area is directed at realizing classifiers with low false alarm rates and high detection
probabilities that use features based on the the above physics based approach for a variety of
applications.
6

Acknowledgments

The authors thank Tushar Saxena and Deepak Kapur of the Institute for Logic and Program-
ming, Computer Science Dept, State University of New York at Albany, for making available
to us their algorithmic elimination methods. This research was supported by the AFOSR contract
F49620-93-C-0063, the AFOSR grant LRIR-93WL001, an AFOSR Laboratory Graduate
Fellowship, ARPA contract F33615-94-C-1529 and the National Science Foundation Contract
IRI-91109584.



--R

Geometric Invariance in Computer Vision

"Bayesian Inference in Model-Based Vision"
"View Variation of Point-Set and Line-Segment Features"
"Recognizing Planar Objects Using Invariant Image Features"
"Semi-Local Invariants"
"Direct Computation of Qualitative 3D Shape and Motion Invariants"
"Model-based Invariants for 3D Vision"
"Noise-Resistant Invariants of Curves"
"Quasi-Invariant Properties and 3D Shape Recovery of Non- Straight, Non-Constant Generalized Cylinders"
"Polarization-based material classification from specular reflection"
"Photometric Invariants Related to Solid Shape"
"Surface Descriptions from Stereo and Shading"
"3-D Stereo Using Photometric Ratios"
"Using Illumination Invariant Color Histogram Descriptors for Recognition"
"Reflectance Ratio: A Photometric Invariant for Object Recog- nition"
"Thermal Invariants for Infrared Target Recognition"
"The Viewpoint Consistency Constraint,"
"Integrated Analysis of Thermal and Visual Images for Scene Interpretation"
"A Phenomenological Approach to Multisource Data Integration: Analyzing Infrared and Visible Data"
"Thermal and Visual Information Fusion for Outdoor Scene Perception"
"Robust Physics-based Sensor Fusion"
Fundamentals of Heat Transfer
"Thermal Radiation,"
Thermal Radiative Properties of Coatings
Thermal Radiative Properties of Nonmetallic Solids
"Computing Invariants using Elimination Methods,"
"Foundations of the Theory of Algebraic Invariants"
"On the Generalized Distance in Statistics,"
Introduction to Statistical Pattern Recognition
--TR

--CTR
Horst W. Haussecker , David J. Fleet, Computing Optical Flow with Physical Models of Brightness Variation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.661-673, June 2001
Ronald Alferez , Yuan-Fang Wang, Geometric and Illumination Invariants for Object Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.21 n.6, p.505-536, June 1999
image understanding;thermal image;invariance;physics-based computer vision;model-based vision