Continuous Time Matching Constraints for Image Streams.
Corresponding image points of a rigid object in a discrete sequence of images fulfil the so-called multilinear constraint. In this paper the continuous time analogue of this constraint, for a continuous stream of images, is introduced and studied. The constraint links the Taylor series expansion of the motion of the image points with the Taylor series expansion of the relative motion and orientation between the object and the camera. The analysis is done both for calibrated and uncalibrated cameras. Two simplifications are also presented for the uncalibrated camera case. One simplification is made using an affine reduction and the so-called kinetic depths. The second simplification is based upon a projective reduction with respect to the image of a planar configuration. The analysis shows that the constraint involving second-order derivatives are needed to determine camera motion. Experiments with real and simulated data are also presented.
Introduction
A central problem in scene analysis is the analysis
of 3D-objects from a number of its 2D-images,
obtained by projections. In this paper, the case of
rigid point con-gurations with known correspondences
is treated. The objective is to calculate the
shape of the object using the shapes of the images
and to calculate the camera matrices, which give
the camera movement.
One interesting question is to analyse the multilinear
constraints that exist between corresponding
points in an image sequence. It is well
known that corresponding points in two images
ful-l a bilinear constraint, known as the epipo-
This work has been done within then ESPRIT Reactive
LTR project 21914, CUMULI and the Swedish Research
Council for Engineering Sciences (TFR), project 95-64-222
lar constraint (Stefanovic 1973, Longuet-Higgins
1981). This constraint is conveniently represented
by a 3 \Theta 3 matrix called the essential matrix
in the calibrated case and the fundamental matrix
in the uncalibrated case. The continuous analogue
of this constraint has also been treated in the
literature. In this case corresponding representations
involve the so called in-nitesimal epipole or
the focus of expansion and the axis of rotation.
This has been studied by photogrammetrists in
the calibrated case and recently by Faugeras and
Vieville in the uncalibrated case, cf. (Vieville and
Faugeras 1995). The continuous time analogue
can be derived from the bilinear constraint as the
limit when the time dioeerence tends to zero. Since
the bilinear constraint involves two time instants,
the continuous time analogue involves the Taylor
expansion to the -rst order, the so called one-jet.
2 -str-m and Heyden
The goal of this work is to emphasise the connection
between the discrete time constraints and
corresponding continuous time constraints, and to
derive the continuous time analogue of multilinear
constraints.
The multilinear camera-image motion constraints
have been treated in several recent conference pa-
pers, (Faugeras and Mourrain 1995, Heyden and
#str#m 1995, Triggs 1995). These derivations involve
dioeerent types of mathematical techniques
and are represented using dioeerent mathematical
objects, e.g. Grassman-Cayley algebra, Joint
Grassmannian etc. In this paper we will use a
matrix formulation, such that the image-motion
constraint will be of the type that the rank of a
certain matrix is less than full, i.e. that all subde-
terminants of this matrix are zero. The elements
of this matrix will depend on the motion of the
image point and the relative motion and orientation
between the camera and the object con-gu-
ration as expressed by the so called camera projection
matrix. There are other ways of formulating
the same constraint. However, this particular
choice of formulation has some nice advantages:
ffl Using this formulation, it is easier to show
that all multilinear constraints can be derived
from the bi- and trilinear constraints.
ffl The dioeerence and the similarities between
the uncalibrated and calibrated case is clearer

ffl The parameters in the multilinear constraints
are closely linked to the camera parameters
describing the projection of points in 3D onto
each image plane.
ffl The continuous time case can easily be derived
from the discrete time case.
In this paper we introduce and study the
continuous time analogue of these multilinear
constraints. The nth order constraint involve the
Taylor expansion up to order n of the image point
motion and a similar expansion for the motion of
the camera relative to the scene.
These constraints have the same applications as
their discrete counterparts, i.e. they can be used
as matching constraints to -nd points that are
moving rigidly with respect to the camera. They
can also be used to calculate relative motion with
no a priori knowledge about the scene.
We would like to emphasise the derivation of
these constraints. Some experimental validation
has been included, but needless to say, more experiments
are needed in order to validate the potential
of these types of constraints.
The paper is organised as follows. Some basic
notations are introduced in Section 2. Sections
3, 4 and 5 contain background material on
the ambiguity in the choice of coordinate system,
camera matrix parametrisation and a short derivation
of the discrete time constraints. The continuous
time analogue of these constraints is derived
in Section 6. These constraints are studied
in Sections 7, 8 and 9 with respect to coordinate
ambiguity, motion observability and motion esti-
mation. The latter is illustrated with simulated
and real data in Section 10. Then follows a short
discussion and conclusions in Section 11.
2. Camera Geometry and Notation
The pinhole camera model is used and formulated
using projective geometry.
A point in three dimensions with Euclidean co-ordinates
represented as a point in
the three-dimensional projective space using homogeneous
coordinates
\Theta
U x U y U z 1
T . In
projective geometry two representations are considered
as the same point if one is a multiple of
the other, (Faugeras 1993).
Projection onto the image plane is conveniently
represented in the camera projection matrix for-
mulation
where - is the unknown depth and u is the image
position, also in homogeneous coordinates
\Theta
possibly corrected for the internal
calibration if this is known. A priori knowledge
about the camera and the camera-object motion,
give information about the structure of the camera
projection matrix P .
\Theta
I

uncalibrated, (2)
\Theta
I

known A, (3)
\Theta
I

known
\Theta
T denotes the unknown
position of the camera focus, R a rotation matrix
describing the orientation of the camera, A a
matrix representing the unknown internal calibration
parameters and I the 3 \Theta 3 identity matrix.
Thus in the three situations above there are 11,
6, and 3 degrees of freedom respectively in each
camera matrix.
In the uncalibrated case when the internal calibration
matrix A is unknown it is convenient to
Continuous Time Matching Constraints for Image Streams 3
of the camera. In other words we think of the
camera as having a position T and a generalised
orientation -
Q. The position determines how object
points are projected onto the viewing sphere
and the orientation -
rearranges these directions.
Both orientation and position of the camera will
change over time. In the continuous time case we
will use the notation ( -
for the
orientation and position of the camera at time t,
. In the discrete
time case the camera at time t will be represented
by ( -
Z. The image position u(t) of
a point U at time t is thus
\Theta -

Alternatively, the notation
and T will be used. The projection is then expressed
as
I

Capital U is used to denote object points. Corresponding
lower case letter u is used to denote
the corresponding image point. Subscripts, e.g.
are used to denote dioeerent points. The superscripts
with parenthesis are used to denote co-
eOEcients of the Taylor series expansion, i.e.
Boldface 0 denotes a zero matrix, usually 3 \Theta 1 or
2 \Theta 1.
3. Uniqueness of Solutions
The overall goal is to calculate the camera matrices
P (t) and to reconstruct the coordinates of
the objects U given only the image positions u(t),
such that the camera equation
is ful-lled. The camera motion is assumed to be
smooth. The solution for P (t) and U can only
be given up to an unknown choice of coordinate
system in the world. This is easiest seen in the
uncalibrated case. Let B be any non-singular 4\Theta4
matrix. Change the projection matrices according
to
and change object points according to
Now
U) is another solution with dioeerent
coordinates that also ful-l the camera equation
For the case of uncalibrated cameras there are
15 degrees of freedom in choosing a projective co-ordinate
system.
For the case of known internal calibration there
is an arbitrary choice of orientation, origin and
scale (7 degrees of freedom).
In the case of known external and internal orientation
we are only allowed to make changes of
the type
\Deltay
i.e. one may choose the origin arbitrarily and also
the overall scale.
It is important to keep this problem of non-uniqueness
in mind. The question of choosing a
particular or canonical choice of coordinate sy-
stem, as discussed in (Luong and Vieville 1994),
can be important when designing numerical algo-
rithms. This is commented further upon in Section
7.
4. Simplifications in the Uncalibrated Case
In the uncalibrated case we can simplify the problem
by partially -xing the object and image co-ordinate
systems. Two such simpli-cations are
of special interest. These are explained in detail
in (Heyden and #str#m 1995, Heyden and
#str#m 1997), and will be brieAEy described here.
4.1. Affine Reduction
If the same three points can be seen in a sequence
of images, they can be used to simplify the problem
according to the following theorem.
Theorem 1. Let u 1 (t), u 2 (t) u 3 (t) be the images
of three points U 1
and U 3
. Choose an
object coordinate system where U
Choose image
coordinate systems at time t so that u 1
Then the camera projection matrices can be written
Q(t) is a diagonal
matrix and T (t) is the position of the camera
at time t.
4 -str-m and Heyden
Proof: The choice of coordinate systems gives
three constraints on the camera matrix P (t),
Since P (t)U j is the jth column of the matrix P (t),
it follows that the -rst three columns of P (t) form
a diagonal matrix, i.e.
By aOEne normalisation of three corresponding
points in an image sequence, the analysis of the
remaining points can be made almost as if the cameras
were calibrated and with the same rotation.
The unknown elements in the diagonal matrices
correspond to the so called kinetic depths of the
three image points relative to the camera motion,
cf. (Sparr 1994, Heyden 1995). This is basically
the same idea as the relative aOEne coordinates in
(Shashua and Navab 1996). Using projective geometry
one can think of this as de-ning the plane
through the three points as the plane at in-nity
and also partially locking the orientation of this
plane by these three points.
4.2. Projective Reduction
Further simpli-cations can be obtained if four or
more coplanar points, e.g. points belonging to a
planar curve, are detected in a subsequence of
images.
Theorem 2. Let u 1 (t), u 2 (t), u 3 (t) and u 4 (t) be
the images of four coplanar points U 1
and U 4
so that no three of them are collinear.
Choose an object coordinate system where U
and U Choose image coordinate
systems at time t so that u 1
Then the camera projection matrices can be written
\Theta
I

, where T (t) is the position
of the camera at time t.
Proof: The -rst 3 \Theta 3 block matrix -
Q(t) of P (t)
acts as the identity on
i.e.
where - denotes equality up to scale. According
to the assumptions, the four points
constitute a projective
basis for the projective plane. Therefore, it
follows that -
I .
By projective alignment of the images of at least
four coplanar points the problem can thus be
analysed as if both internal calibration and orientation
of the camera are known at all times.
The motion of the camera can be described by
the pair (Q(t); T (t)) or ( -
(or -
describes the generalised orientation of
the camera and T (t) (or -
describes the position
of the camera at time t. Depending on which
assumptions and simpli-cations we have made,
the matrices Q(t) (or -
Q(t)) lie on dioeerent manifolds

ffl Traditional uncalibrated setting: The matrices
Q(t) are arbitrary but nonsingular and
two matrices are considered equal if one is
a (positive) multiple of the other. There are
eight degrees of freedom.
AOEnely reduced setting: The matrices Q(t)
are diagonal and nonsingular and two matrices
are considered equal if one is a (positive)
multiple of the other. There are two degrees
of freedom.
Projectively reduced setting: The matrices
Q(t) are identity matrices. There are no
degrees of freedom. Alternatively we may
require: The matrices Q(t) are multiples of
the identity matrix and two matrices are considered
equal if one is a (positive) multiple of
the other.
Calibrated setting: The matrices Q(t) are ort-
hogonal. There are three degrees of freedom.
Alternatively we may require: The matrices
are multiples of orthogonal matrices and
two matrices are considered equal if one is a
(positive) multiple of the other.
These manifolds are non-linear and have a so
called Lie group structure under matrix multipli-
cation. The corresponding Lie algebra will be of
importance. Unlike the Lie group the Lie algebra
is a linear subspace of all 3 \Theta 3 matrices q:
ffl Traditional uncalibrated setting: The matrices
q(t) are arbitrary and two matrices are
considered equal if their dioeerence is a multiple
of the identity matrix. There are eight
degrees of freedom.
AOEnely reduced setting: The matrices q(t)
are diagonal and two matrices are conside-
Continuous Time Matching Constraints for Image Streams 5
red equal if their dioeerence is a multiple of
the identity matrix. There are two degrees of
Projectively reduced setting: The matrices
q(t) are zero matrices. There are no degrees
of freedom.
Calibrated setting: The matrices q(t) are an-
tisymmetric. There are three degrees of freedom

These Lie algebras are obtained from the Lie
groups using the exponential map
The dioeerent Lie groups and Lie algebras are summarised
in Table 1. Since two matrices in the Lie
Group are considered to be equal if one is a multiple
of the other, it is often convenient to choose
a speci-c representative. One such choice of representative
is to always scale the matrix so that
the determinant is one. Similarly two matrices in
the Lie algebra are considered to be equal if the
dioeerence is a multiple of the identity. A unique
representative can be chosen by demanding that
the trace of the matrix is zero. This -ts in nicely
with the exponential map since
where we have used the notation det as the determinant
and tr as the trace of a matrix.
5. Multilinear Constraints in the Discrete Time
Case
In order to understand the matching constraints
in the continuous time case, it is necessary to
take a look at the corresponding constraints in
the discrete time case. For a more thorough tre-
atment, see (Heyden and #str#m 1995). Alternative
formulations of the same type of constraints
can be found in (Faugeras and Mourrain 1995,
Triggs 1995).
We start with the de-nition.
De-nition 1. The nth order discrete multilinear
constraint is

Table

1. The Lie-Groups and their corresponding Lie-
Algebras in the four dioeerent settings.
Setting Lie Group Lie Algebra
Uncalibrated Arbitrary Arbitrary
Aoe. red. Diagonal Diagonal
Proj. red.
Calibrated Rotation Antisymmetric
Theorem 3. In a sequence of discrete images
corresponding points with coordinates u(t 0
obey the nth order discrete multilinear constraint.
This means that there exist a solution to (8) for
that holds for every corresponding
point sequence.
Proof: This can be seen by lining up each projection
constraint
in a linear matrix equation66P (t 0
\GammaU
Since this system has a non-trivial solution the
leftmost matrix cannot have full rank.
6. Continuous Time Analogue of Multilinear
Constraints
The multilinear constraints in the continuous time
case can be derived using Taylor series expansion
of the time dependent functions in (5),
Use the Taylor series expansions
Recall that superscript (i) denotes the ith coeOE-
cient in the Taylor series expansion, e.g. - (k) =k!
. Substituting the Taylor series expansions
6 -str-m and Heyden
into (10) gives
Identifying the coeOEcients of t i for
\GammaU
Since this set of equations has a non-trivial solution
the leftmost matrix cannot have full rank.
De-nition 2. The nth order continuous
constraint is
where
Theorem 4. In a continuous sequence of images
the image coordinates u (0) and their derivatives,
up to order n at the same instant of time
obey the nth order continuous constraint.
Proof: Follows from the derivation above.
Remember that -
, and similarly for
the other variables. This means in particular that
dt has the meaning of image point velocity.
The continuous time constraints can be simpli-
-ed somewhat by partially choosing a coordinate
system according to P (which implies
By multiplying the big
matrix M from the right by the matrix
we obtain
MS
Since the matrix S has full rank, it follows that
By elimination of the -rst
three rows and columns of the matrix MS in (16),
the constraint (15) is simpli-ed to
De-nition 3. The continuous analogue of the
bilinear and trilinear constraints are de-ned as
the nth order constraint above with
respectively. The analogue of the bilinear
constraint is the -rst order continuous constraint
rank
\Theta -

and analogue of the trilinear constraint is the second
order continuous constraint
rank
7. Choice of Coordinate System
In the previous sections we have used the option
to partially -x the coordinate system to simplify
the problem. Sometimes it is useful to choose
a canonical object coordinate frame to obtain a
canonical coordinate representation of the reconstructed
object and projection matrices.
In the calibrated case a speci-c coordinate system
can be determined by setting the initial orientation
to Q(t 0
overall scale by jT (1) (t 0
In the uncalibrated case there are four things
to consider when choosing a projective coordinate
system in the reconstruction.
1. The position of the plane at in-nity, 3 d
2. The individual points at the plane at in-nity,
3. The origin, 3 d
4. The scale, 1 d
Continuous Time Matching Constraints for Image Streams 7
One way of doing this in the discrete time case
is to lock individual points at the plane at innity
by lock the origin by T
and the scale by jT 1. The position
of the plane at in-nity can be chosen by
choosing a speci-c Q(t 1 ). This cannot, however,
be done arbitrarily. The matrix Q(t 1 ) ful-lls the
bilinear constraint, as discussed in (Luong and
Vieville 1994). The question of choosing a canonical
coordinate system (and thereby choosing a
speci-c plane at in-nity) is simpler in the aOEnely
and projectively reduced settings. The plane at
in-nity is determined by the three or more points
that are used in the reduction. A canonical coordinate
system can then be chosen by Q(t 0
similar to the
calibrated case.
Similarly in the continuous time case one possible
choice is to take P (which implies
discussed in the
simpli-cation above, and then take j -
tr -
This determines the choice of coordinate
system in the calibrated, aOEnely reduced and
in the projectively reduced settings. In the uncalibrated
case there is a further choice of the plane
at in-nity. This choice can be made by choosing
one -
that ful-lls the -rst order continuous
constraint.
Since Q(t) is undetermined up to a scalar factor
it is possible to enforce uniqueness if we require
det t. This condition is of course
ful-lled for I . A Taylor series expansion
of Q(t) gives
which can be seen by expanding the determinant.
Thus we have tr Q The coeOEcients of t k
in (20) are complicated expressions in Q (i) . Ho-
wever, it can be seen that the coeOEcient of t k can
be written as tr Q (k) plus terms involving Q (i) for
k. This means that we can ensure uniqueness
if we require
The price we have to pay for this simpli-cation is
that det Q(t) depends on t, det
in general for
Assume again that Q (0) has been chosen to be
the unit matrix. It also follows from the matrix
exponential
that Q has the properties
listed in Table 1. However, the relations between
q (i) and Q (i) for i ? 1 are more complicated.
8. Motion Observability
What can be said about the observability of the
full motion of the camera, e.g. is it possible to
determine camera motion uniquely up to a choice
of coordinate system using only the -rst order
continuous constraint?
8.1. Motion Observability from the First Order Continuous
Constraint
Does the -rst order continuous constraint determine
camera motion uniquely up to choice of camera
system? Using the -rst order continuous
constraint we can calculate T (1) (t) up to an unknown
scale factor and Q (1) (t) up to an arbitrary
choice of the plane at in-nity. Since only the
direction of T (1) (t) can be obtained at each time
instant, it is not possible to reconstruct T uni-
quely. Thus, the -rst order continuous constraint
is not enough to determine motion.
8.2. Motion Observability from the Second Order
Continuous Constraint
If T (1) (t) and Q (1) (t) are known and T (1) (t) 6= 0,
then T (2) (t) and Q (2) (t) are uniquely determined
by the second order continuous constraint (19).
One can think of this as T (2) (t) and Q (2) (t)
being a function of T (1) (t), Q (1) (t), T (t), Q(t) and
image motion at this time instant, i.e.
It is a well known fact that this kind of dioee-
rential equations can be solved at least locally,
given a set of initial conditions on T
These initial conditions are determined
by choosing a coordinate system and at
the same time ful-lling the -rst order continuous
constraint at Thus the full motion of
the camera is observable from the second order
continuous constraints if T (1) (t) 6= 0. Since the
8 -str-m and Heyden
full camera motion (T (t); Q(t)) can be calculated
uniquely up to a choice of coordinate system, it
is also possible to calculate derivatives of all or-
ders. Thus all continuous multilinear constraints
follow from the -rst and second order continuous
constraints.
9. Estimation of Motion Parameters using the
Continuous Time Constraints
Study again the simpli-cation of the continuous
time constraint in (17). One use of this constraint
is to calculate ( -
the motion of the
points in the image u (i) . Typically, the relative
noise increases with increasing orders of dioeeren-
tiation. We therefore expect u (2) to be noisier
than u (1) which in turn is noisier than u (0) . It
is therefore natural to estimate motion parameters
in steps. First estimate ( -
using the
-rst order continuous constraint (18). Then try
to estimate ( -
and the
second order continuous constraint (19). Repeat
this as long as the level of noise permits.
There are some numerical diOEculties with using
the continuous time constraint to estimate motion
parameters. First of all it can be quite diOEcult to
obtain good estimates of the image point positions
and their derivatives. Secondly, since these
estimates are noisy, this needs to be modelled and
taken into account. This aoeects the way motion
parameters should be estimated.
9.1. Using the First Order Continuous Constraint
The -rst order continuous constraint (18) involves
the -rst order derivative in camera motion, T (1)
and Q (1) .
One major dioeerence between the discrete time
case and the continuous time case is that the derivatives
of the orientation, Q (1) , lie on a linear
manifold. As an example take the calibrated case.
Whereas in the discrete case the motion parameters
involve a rotation matrix Q(t 1
), the continuous
time analogue involve an anti-symmetric
matrix Q (1) . It is easier to parametrise the set
of anti-symmetric matrices (since this is a linear
subspace of all matrices), than to parametrise the
set of rotation matrices (which is a non-linear ma-
nifold).
The velocity T (1) also lies on a linear manifold.
It can, however, only be determined up to scale.
If T (1) is known, the problem of determining Q (1)
is linear so it can quite easily be solved with linear
methods. On the other hand, if Q (1) is known the
problem of determining T (1) can be formulated
and solved in a linear fashion. This suggests a fast
two-step iterative method. Guess Q (1) . Holding
-xed solve for T (1) . Holding T (1) -xed solve
for Q (1) . This method has been tried and in most
cases it does seem to converge nicely. It should
be noted, however, that there are no guarantees
that this method will converge.
An alternative and more robust method is to
tessellate the sphere of directions T (1) 2 S 2 . For
each T (1) , solve for Q (1) linearly and store the
residual. Choose the pair (T (1) ; Q (1) ) that gave
the lowest residual.
Any of these two methods can be re-ned by
taking the motion parameters as an initial estimate
of (T (1) ; Q (1) ), in a non-linear least squares
minimisation. An advantage of this re-nement is
that it allows more sophisticated error measures,
e.g. the maximum likelihood estimate, that takes
into account the quality of the estimates of u and
u (1) . Another advantage of this re-nement is that
it allows for an analysis of the stability of the solution
through the analysis of the residuals at the
optimum.
9.2. Using the Second Order Continuous Constraint
The second order continuous constraint in (19) involve
the -rst and second order derivatives of the
camera motion. As for the -rst order continuous
constraint, it should be possible to use a two step
iterative method. Guess ( -
Holding
-xed solve for ( -
T (2) ). Holding
-xed solve for ( -
should be noted that the convergence properties
of these methods haven't been studied.
Another approach might by to estimate
using the -rst order continuous
constraint and while holding these -xed, estimate
using the second order continuous
constraint.
Close to a solution the method could be rened
by non-linear maximisation of a Likelihood
function.
10. Experiments
To illustrate the continuous constraints, we have
used iterative methods as described above. We
consider the -rst order continuous constraint
in 3D to 2D projection and the second order
constraint in an industrial application of 2D to
1D projection.
Continuous Time Matching Constraints for Image Streams 9
10.1. First Order Constraints on Image Streams
An image sequence of an indoor scene have been
used, see Figure 1, where one image in the sequence
is shown. The whole sequence contains
more than 200 images. To illustrate the applicability
of the continuous constraints, we have only
used 2 images. Points have been extracted using a
corner detector, and we have used 28 points with
correspondences. The aOEnely reduced coordinates
have been calculated from the images, giving
u(0) and u(h), where h denotes the time increment
between the dioeerent exposures. In this case
We have used u u(0). The derivatives
have been computed from image 1 and
image 2 using the dioeerence approximation
Using the iterative approach and the aOEnely
reduced setting and only 10 iterations, starting
from Q we obtain the following solution
ful-lling the -rst order constraint:
This solution can be compared to the solution obtained
in the discrete time case between u(0) and
Here -
Q(h) should be compared to -
I
and -
T (h) to -
T (1) . The angle
between -
T (1) to -
T (h) is 2:8 degrees.
10.2. Second Order Constraints on Angle Streams
The continuous multilinear constraints have an
interesting practical application to the vision system
of an autonomous vehicle. The vehicle is
equipped with a calibrated camera with a one-dimensional
retina. It can only see speci-c beacons
or points in a horizontal plane. Let (x; y)
be the coordinates of such a beacon and let as
Fig. 1. One image in the sequence used in the continuous
case.
before this point be represented by a vector
\Theta
T . Then the measured image point is a
direction vector according to the
camera equation
where the camera matrix P (t) is a 2 \Theta 3 calibrated
camera matrix
Let the Taylor expansion of the rotation matrix
R(t) and the vector -
T (t) be
ReAEector
Angle meter
a
Fig. 2. a: A laser guided vehicle. b: A laser scanner or
angle meter.
Using this notation the continuous time analogue
of the trilinear constraint (19) becomes
det
det
The matrix R is a 2\Theta2 rotation matrix. Assuming
that the Taylor expansion of the rotation angle
OE(t) is -rst
derivative of R has the form
r 1-
and the second coeOEcient of the Taylor expansion
has the form
R
The second order continuous constraint in (26)
does not, however, involve the term r 2in R (2) .
This can be seen by adding r 2times the last column
to the second column in (26). The second
order continuous constraint in (26) will now be
studied in more detail. Introduce the variables
a 1
a 2
and
The determinant of M can be written
\Theta
a 1
a 2
22
where
\Gammay 0
\Theta
a 1
a 2

and
\Theta
T . A tentative algorithm to -nd w and
z has been investigated.
Algorithm 1
1. Start with a crude estimate of r 1
and r 2
, for
example
\Theta
T .
2. For all image directions u i , calculate the corresponding
. The
vector w should be orthogonal to every vector
, the vector w can be found as
the left null space of the matrix
\Theta

The vector w is found by using a singular value
decomposition of M .
3. Once w is approximately known, z can be
found as the right null space of6wV 1
4. Repeat steps 2 and 3.
The algorithm has been implemented and
tested experimentally. The results are illustrated
with simulated data. In these simulations the
angles to -fteen beacons were calculated during a
period of one second. Gaussian noise of dioeerent
standard deviations 0:1; 0:5;
added to the angle measurements. In the real application
the standard deviation is approximately
0:2 mrad. The angle measurements in a two second
period were used to estimate the Taylor co-
eOEcients of the measured image direction u i using
standard regression techniques. These Taylor co-
eOEcients were then used to calculate the Taylor
coeOEcients of the motion (r using
the algorithm above. The experiment was repeated
100 times. The standard deviation of the
estimated motion parameters are shown in Table
2. The true values of the motion parameters,
in the simulation are r
Continuous Time Matching Constraints for Image Streams 11

Table

2. Standard deviation of estimated motion parameters
for dioeerent noise levels when using Algorithm 1.
11. Discussion and Conclusions
In this paper the simpli-ed formulation of the
multilinear forms that exist between a sequence of
images has been used to derive similar constraints
in the continuous time case. The new formulation
makes it easier to analyse the matching
constraints in image sequences. It has been shown
that these constraints contain information in the
-rst and second order only. This representation
is fairly close to the representation of the motion
and it is easy to generalise to dioeerent settings.
Four such settings calibrated, uncalibrated, aOE-
nely reduced and projectively reduced, are described
in the paper.
The continuous constraints can be used to
design -lters to estimate structure and motion
from image sequences. Using only the -rst order
constraint it is possible to estimate the direction
of the movement of the camera. Having only
this information it is not possible to build up the
whole camera movement. Using the second order
constraint it is possible to estimate the second
order derivative of the camera movement
with a scale consistent with the -rst order deri-
vative. This information can be used to build up
the camera movement. This is analogous to the
discrete time case, where the trilinearities are needed
in order to estimate the camera movement, if
only multilinear constraints between consecutive
images are used.
The -rst example above shows that the -rst order
continuous constraint is comparable to the discrete
case. However, the continuous constraints
are sensitive to noise because estimating the
image velocities ampli-es the noise present in the
images. The higher order continuous constraints
are even more sensitive, because they involve higher
order derivatives. Using -ltration techniques
to estimate the derivatives from image coordinates
in more than two images could reduce the in-
AEuence of noise.
The second example illustrates the applicability
of higher order constraints in an industrial
application. An industrial vehicle is guided by a
one-dimensional visual system. It is interesting
to note that the -rst order constraint in this case
gives no information about the motion. Higher
order constraints are essential.
We believe that the continuous constraints are
important theoretical tools. We also believe that
they are necessary to analyse image sequences
with high temporal sampling frequency.

Acknowledgements

The authors thank Tony Lindeberg and Lars
Bretzner at KTH for help with corner detection
and tracking.



--R



Workshop on Intelligent Autonomous Vehicles



on Computer Vision

the velocity case

Topological and Geometrical Aspects of
PhD thesis
Department of Medical and Physiological Physics

What can be seen in three dimensions
with an uncalibrated stereo rig?


On the geometry
and algebra of the point and line correspon between


Reconstruction from image sequences
by means of relative depths
Computer

for sequences of images
of Visual Scenes

forms for sequences of images.
in Image and Vision Computing.
A computer algorithm for
reconstructing a scene from two projections
Canonic representations
for the geometries of multiple projective views
Conf. on Computer Vision
Relative aOEne
structure: Canonical model for 3d from 2d geometry
and applications
Machine Intell
A common framework for kinetic depth
reconstruction and motion for deformable objects





Motion analysis
with a camera with unknown and possibly varying


--TR
multilinear constraint;calibrated camera;structure;motion;optical flow;uncalibrated