IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

1

Performance Characterization of Image Feature
Detectors in Relation to the Scene Content Utilizing
a Large Image Database

arXiv:1709.08202v2 [cs.CV] 13 Oct 2017

Bruno Ferrarini1 , Shoaib Ehsan1 , Ales̆ Leonardis2 , Naveed Ur Rehman3 , and Klaus D. McDonald-Maier1

Abstract—Selecting the most suitable local invariant feature
detector for a particular application has rendered the task of
evaluating feature detectors a critical issue in vision research.
Although the literature offers a variety of comparison works
focusing on performance evaluation of image feature detectors
under several types of image transformations, the influence of
the scene content on the performance of local feature detectors
has received little attention so far. This paper aims to bridge this
gap with a new framework for determining the type of scenes
which maximize and minimize the performance of detectors in
terms of repeatability rate. The results are presented for several
state-of-the-art feature detectors that have been obtained using a
large image database of 20482 images under JPEG compression,
uniform light and blur changes with 539 different scenes captured
from real-world scenarios. These results provide new insights into
the behavior of feature detectors.
Index Terms—Feature Detector, Comparison, Repeatability.

I. I NTRODUCTION
Local feature detection has been a challenging area of
interest for the computer vision community for some time.
A large number of different approaches have been proposed
so far, thus making evaluation of image feature detectors an
active research topic in the last decade or so. Most evaluations
available in the literature focus mainly on characterizing
feature detectors’ performance under different image transformations without analyzing the effects of the scene content in
detail. In [1], the feature tracking capabilities of some corner
detectors are assessed utilizing static image sequences of a
few different scenes. Although the results permit to infer a
dependency of the detectors’ performance on the scene content, the methodology followed is not specifically intended to
highlight and formalize such a relationship, as no classification
is assigned to the scenes. The comparison work in [2] gives
a formal definition for textured and structured scenes and
shows the repeatability rates of six feature detectors. The
results provided by [2] show that the content of the scenes
influences the repeatability but the framework utilized and
the small number of scenes included in the datasets [3] do
not provide a comprehensive insight into the behavior of the
feature detectors with different types of scenes. In [4], the
scenes are classified by the complexity of their 3D structures
1 University of Essex, School of Computer Science and Electronic Engineering, Colchester CO4 3SQ, UK
2 Dep. of Electrical Engineering, COMSATS Institute of Information Technology, Islamabad, Pakistan
3 School of Computer Science, University of Birmingham, Birmingham,
UK

in complex and planar categories. The repeatability results
reveal how detectors perform for those two categories. The
limit in the generality of the analysis done in [4] is due
to the small number and variety of the scenes employed,
whose content are mostly human-made. This paper aims to
help better understand the effect of the scene content on the
performance of several state-of-the art local feature detectors.
The main goal of this work is to identify the biases of these
detectors towards particular types of scenes, and how those
biases are affected by three different types and amounts of
transformations (JPEG compression, blur and uniform light
changes). The methodology proposed utilizes the improved
repeatability criterion presented in [5], as a measure of the
performance of feature detectors, and the large database [6]
of images consisting of 539 different real-world scenes containing a wide variety of different elements. This paper offers
a more complete understanding of the evaluation framework
first described in the conference version [7], providing further
background, description, insight, analysis and evaluation.
The remainder of the paper is organized as follows. Section II
provides an overview of the related work in the field of feature
detector evaluation and scene taxonomy. In Section III, the
proposed evaluation framework is described in detail. Section
IV is dedicated to the description of the image database
utilized for the experiments. The results utilizing the proposed
framework are presented and discussed in Section V. Finally,
Section VI presents the conclusions.
II. R ELATED W ORK
The contributions to the evaluation of local feature detectors
are numerous and vary based on: 1) the metric used for quantifying the detector performance, 2) the framework/methodology
followed and 3) the image databases employed. Repeatability
is a desirable property for feature detectors as it measures the
grade of independence of the feature detector from changes
in the imaging conditions. For this reason, it is frequently
used as a measure of performance of local feature detectors.
A definition of repeatability is given in [8] where, together
with the information content, it is utilized as a metric for
comparing six feature detectors. A refinement of the definition
of repeatability is given in [9], and used for assessing six
state-of-the-art feature detectors in [2] under several types
of transformations on textured and structured scenes. Two
criteria for improved repeatability measure are introduced in
[5] that provide results which are more consistent with the

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

2

Fig. 1: The reference image of three scenes and the effect of the application of 60% of light reduction, 98% of JPEG compression
rate and 4.5σ Gaussian blur.

actual performance of several popular feature detectors on the
widely-used Oxford datasets [3]. The improved repetability
criteria are employed in the evaluation framework proposed
in [10] and in [11], which presents a method to assess the
performance bounds of detectors. Moreover, repeatability is
used as a metric for performance evaluation in [12] and [4]
that utilize non-planar, complex and simple scenes.
The performance of feature detectors has also been assessed
employing metrics other than repeatability. The performance
measure in [13] is completeness, while feature coverage is
used as a metric in [14]. The feature detectors have also been
evaluated in the context of specific applications, such as in
[1], where corner detectors are assessed in the context of point
feature tracking applications.
III. T HE P ROPOSED E VALUATION F RAMEWORK
The proposed framework has been designed by keeping
in mind the objective of evaluating the influence of scene
content on the performance of a wide variety of state-of-the-art
feature detectors. A proper application of such a framework
requires a large image database I organized in a series of n
datasets. Each dataset needs to contain images from a single
scene with different amounts of image transformation. The
images included in such a database should be taken from a
large variety of different real-world scenarios. The proposed
framework consists of the steps discussed below.
A. Repeatability data

A = {1, 2, 3, ......m}

(2)

P = {1, 2, 3, ......n}

(3)

where m corresponds to the maximum amount of transformation and 1 relates to the reference image (zero transformation);
n is the total number of scenes and each scene is utilized to
build one separate dataset, thus finally resulting in n datasets
in total. Let Bkd be the set of repeatability rates computed
for step k (corresponding to k image transformation amount)
for a feature detector d across n datasets (which implies
repeatability values for n scenes):
Bkd = {B1kd , B2kd , ......Bnkd }

(4)

Each set Bkd contains n repeatability ratios, one for each
dataset. In particular, for the image database utilized in this
work for the experiments [6], n is 539 while maximum
value of k is 10 or 14 depending on which transformation
is considered. Thus, Bkd includes 539 values of repeatability
for the k th step.
B. Scene rankings

The framework is based on the repeatability criterion described in [5], whose consistency with the actual performance
of a wide variety of feature detectors has been proven across
well-established datasets [3]. As proposed in [5], the repeatability rate is defined as follows:
Repeatability =

where Nrep is the total number of repeated features and
Nref is the number of interest points in the common part
of the reference image. Let A and P be the sets of indices
representing the m discrete amount of transformation and the
scenes respectively.

Nrep
Nref

(1)

The top and lowest rankings for each detector d are built
selecting the j highest and the lowest repeatability scores at k
amount of image transformation. Let Tkd (j) and Wkd (j) the
sets containing the indices of the scenes whose repeatability
falls in the top and lowest ranking respectively:

Tkd (j) = Skd(1) , Skd(1) , ......Skd(j)

(5)

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

3

Fig. 2: Some images from the database utilized for the experiments. Each row shows images belonging to the same category:
outdoor, non-outdoor, human-made content, natural content, simple and complex edges.



Wkd (j) = Skd(n) , Skd(n−1) , ......Skd(n−j+1)

(6)

where Skd(i) ∈ P is the scene index corresponding to the ith
highest repeatability score obtained by the detector d for the
scene under k amount of transformation. Thus, in accordance
with this notation, Skd(1) is the scene for which the detector
scored the best repeatability score, Skd(2) corresponds to the
second highest repeatability rate, Skd(3) to the third highest
and so on, until Skd(n) which is for the lowest one.

scene as it includes a few edges delimiting well contrasted
areas. Although the main structures (broccolis borders) can be
identified in scene 76, the rough surface of the broccolis is
information rich that results in labeling this scene as complex.
Location
Type
Object
Type
Complexity

Outdoor
Indoor
Human-made
Natural
Simple
Complex

C. Scene classification
The scenes are attributed with three labels on the basis
of human judgment. As described in Table I, each label is
dedicated to a particular property of the scene and has been
assigned independently from the others. These attributes are:
the location type (f ), which may take the label outdoor or
indoor, the type of the elements contained (g), which may take
the label natural or human-made, and the perceived complexity
of the scene (h), which may take the label simple or complex.
Figure 2 shows a sample of the scenes from the image database
[6] utilized for the experiments grouped so that each row
shows scenes sharing the same value for one of the three
labels f , g and h. Scene 9 is tagged as outdoor and, along
with scene 76 and 17, contains natural elements. The scenes
40, 530 and 373 are labeled as human-made and the first is also
classified as indoor. The scene 530 is categorized as a simple

Indoor scene and close-up a single
or of a few objects.
The complement of above.
Elements are mostly artificial.
Elements are mostly natural.
A few edges with quite regular
shapes.
A large number of edges with
fractal-like shapes.

TABLE I: Classification labels and criteria

D. Ranking trait indices
The labels of the scenes included in the rankings, (5) and
(6), are examined in order to determine the dominant types
of scenes. For each ranking Tkd (j) and Wkd (j), the ratios
of scenes classified as outdoor, human-made and simple are
computed. Thus, three ratios are associated to each ranking
where higher values mean higher share of the scene type
associated:
P
Si .[f, g, h]
(7)
∀Si ∈ Tkd : Tkd .[F, G, H] =
j
P
Si .[f, g, h]
∀Si ∈ Wkd : Wkd .[F, G, H] =
(8)
j

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

These vectors contain three measures which represent the
extent of the bias of detectors. For example, if a top ranking
presents F = 0.1, G = 0.25 and H = 0.8, it can be
concluded that the detector, for the given amount of image
transformation, works better with scenes where its’ element
are mostly natural (low G), with simple edges (high H) and
that are not outdoor (low F ). As opposed to that, if the same
indices were for a lowest ranking it could be concluded that
the detector obtains its lowest results for non-outdoor (F ) and
natural (G) scenes with low edge complexity (H).
IV. I MAGE DATASET
The image database used for the experiments is discussed in
this section and is available at [6]. It contains a large number of
images, 20482, from real-world scenes. This database includes
a wide variety of outdoor and indoor scenes showing both
natural and human-made elements. The images are organized
in three groups of datasets, corresponding to the three transformations: JPEG compression, uniform light and Gaussian
blur changes. Each dataset includes a reference image for the
particular scene and several images of the same scene with
different amounts of transformation for a total of 10 images for
Gaussian blur and 14 for JPEG compression and uniform light
change transformations. Figure 2 provides both a sample of the
scenes available and an example of transformation applied to
an image.
Several well-established datasets, such as [3], are available
for evaluating local feature detectors, however are not suitable
for use with the proposed framework due to the relatively
small number and lesser variety of scenes included, and the
limited range of the amount of transformations applied. For
example, UBC dataset [3], which was used in [2], includes
images for JPEG compression ratios varying from 60% to
98% only. Among the Oxford datasets [3], Leuven offers
images under uniform light changes, however the number
of images in that dataset is only six. Although the database
employed in [15] for assessing several feature detectors under
different light conditions contains a large number of images,
the number of scenes are limited to 60. Moreover, these scenes
were captured in a highly controlled environment so they are
lesser representative of real-world scenario in comparison of
the image database used here.
The images included in the database utilized for this work
have a resolution of 717 × 1080 pixels and consist of 539
real-world scenes. Each transformation is applied in several
discrete steps to each of the scenes. The Gaussian blur amount
is varied in 10 discrete steps from 0 to 4.5σ (10 × 539 = 5390
images), JPEG compression ratio is increased from 0 to 98% in
14 steps. Similarly, the amount of light is reduced from 100%
to 10% (14×539 = 7546 images). Thus, the database includes
a dataset of 10 or 14 images for each of the 539 scenes for
a total of 20482 images. The ground truth homography that
relates any two images of the same scene is a 3 × 3 identity
matrix for all three transformations as there is no geometric
change involved.
Accordingly with the classification criteria introduced in
the Section III-C, 51% of the 539 scenes have been labeled

4

as outdoor, 65% contain mostly human made elements and
51% have been attributed with the simple label. Overall, the
database has reasonable balance between the content types
introduced by the proposed classification criteria, so that it
becomes possible to produce significant bias descriptors for
the local feature detectors.
V. R ESULTS
The proposed framework has been applied for producing
the top and lowest rankings for a set of eleven feature
detectors which are representative of a wide variety of different approaches [16] and includes the following: EdgeBased Region (EBR) [17], Harris-Affine (HARAFF), HessianAffine (HESAFF) [18], Maximally Stable Extremal Region
(MSER) [19], Harris-Laplace (HARLAP), Hessian-Laplace
(HESLAP) [9], Intensity-Based Regions (IBR) [20], SALIENT
[21], Scale-invariant Feature Operator (SFOP) [22], SpeededUp Robust Feature (SURF) [23] and SIFT [24].
The first subsection provides details on how the repeatability
data have been obtained and the second one is dedicated to
the discussion about the ranking traits of each local invariant
feature detector.
A. Repeatability Data
The repeatability data are obtained for each transformation
type utilizing the image database discussed in Section IV.
This data is collected using the authors’ original programs
with control parameter values suggested by the respective
authors. The feature detector parameters could be varied in
order to obtain a similar number of extracted features for
each detector. However, this has a negative impact on the
repeatability of a detector [9] and is therefore not desirable
for such an evaluation.
Utilizing the dataset described in detail in Section IV, 18865
repeatability rates have been computed for each local feature
detector with the exception of SIFT, which has been assessed
only under JPEG compression. It should be noted that SIFT
detects more than 20,000 features for some images in the
image database which makes it very time-consuming to do
such a detailed analysis for SIFT. In the case of JPEG image
database, it took more than two months to obtain results on
HP ProLiant DL380 G7 system with Intel Xeon 5600 series
processors. Therefore, results for SIFT are not provided in this
section.
The number of datasets is 539, the number of discrete step
of transformation amount k varies across the transformations
considered. We employed: k = 14 for JPEG compression and
uniform light change transformations and k = 10 for Gaussian
blur. Since the first step of transformation amount corresponds
to the reference image, the number of set Bkd (4) is 13 for
JPEG compression and light changes, and 9 for blurring for a
total of 2 × (13 × 539) + 9 × 539 = 18865 repeatability rate
values for each detector.
B. Trait Indices
In this section the trait indices for all the assessed image
feature detectors are presented and discussed. The trait indices

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

Fig. 3: Top and lowest trait indices of EBR in percentage for
different amounts of light reduction (a,d), JPEG compression
(b,e) and blurring (c,f).

have been designed to provide a measure of the bias of the
feature detector for any of the types of scene introduced by
the classification criterion described in the Section III-C.
In other words, they are indicative of the types of scene
for which a feature detector is expected to perform well.
Accordingly, with the definition provided in the Section
III-D, they represent the percentage of the scenes in the
top and lowest rankings of a particular type of scene. Thus,
they permit to characterize quantitatively the performance of
feature detectors from the point of view of the scene content.
The trait indices are built starting from the top and lowest
rankings of any feature detector. For obtaining the results
presented in this work, the evaluation framework has been
applied utilizing a ranking length of 20 (j = 20). Finally, the
related trait indices are computed by applying the equations
(7) and (8) presented in the Section III-D.
The results of all detectors are shown in the Figures 3–13
and discussed below. The results are presented utilizing
radar charts: the transformation amounts are shown on the
external perimeter and increase clockwise; the trait indices
are expressed in percentage with the value which increases
from the center (%0) to the external perimeter (%100) of the
chart.
1) EBR trait indices: All the available trait indices of
Edge-Based Regions (EBR) detector [17] are reported in
Figure 3. The performances of EBR are very sensitive to
uniform light reduction and Gaussian blur [10]. In particular,
for light reduction higher than 60% and Guassian blur equal
or greater than 3.0σ, there are more than 20 scenes for
which EBR scores a repeatability rate value of 0% making
impossible to form a scene ranking as described above
(Section V-B1). For this reason, in Figures 3.d and 3.e the
trait indices for the higher amounts of those transformation
are omitted.
EBR exhibits high values (around 80% - 90%) of G in
the top rankings and low (rarely above 25%) in the lowest
rankings, denoting a strong bias towards the scenes including
many human made elements. EBR performs generally well

5

Fig. 4: Top and lowest trait indices of HARLAP in percentage
for different amounts of light reduction (a,d), JPEG compression (b,e) and blurring (c,f).

Fig. 5: Top and lowest trait indices of HARAFF in percentage
for different amounts of light reduction (a,d), JPEG compression (b,e) and blurring (c,f).

on simple scenes as well, in particular under Gaussian blur,
with the share of simple scenes never below 70%. The values
assumed by F indices are not indicative of the EBR’s bias for
particular location types as they assume very similar values
between the top and lowest rankings.
2) HARLAP and HARAFF trait indices: The rankings of
HARLAP [9] and HARAFF [18] are very similar to each
other and so are the values of their trait indices. Both of
them are particularly prone to uniform light changes and the
trait indices for high level of transformation amounts are not
available. Figures 4.a and 5.a report the results for the top
rankings up to 80% of light reduction and up to 60% for the
lowest rankings.
HARLAP and HARAFF present a bias towards simple scenes,
which is particularly strong under uniform light reduction
and blurring as can be inferred by the high values, which
H assumes in the top twenties and the low values in the
relative lowest rankings. A clear preference of those detectors
for human made objects can be claimed under light changes,

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

6

Fig. 6: Top and lowest trait indices of HESLAP in percentage
for different amounts of light reduction (a,d), JPEG compression (b,e) and blurring (c,f).

Fig. 7: Top and lowest trait indices of HESAFF in percentage
for different amounts of light reduction (a,d), JPEG compression (b,e) and blurring (c,f).

however this is not the case under JPEG compression and
Gaussian blur whose related G indices are too close between
top and lowest rankings to draw any conclusion. The F
indices are extremely low (never above 20%) for the top
twenty rankings under Gaussian blur revealing that HARLAP
and HARAFF deal better with non-outdoor scenes under this
particular transformation.

significantly lower than ones at other compression amounts
and reach a minimum at 80% which are 10% for H and 25%
for G. Similar variations can be appreciated also for F in
both top and lowest rankings with values variations broad up
to 40%. While the G and H indices in many cases present
small differences between the top and lowest rankings, the
F indices are often inversely related. For example, at 30%
compression, F is equal to 10% for the top twenty and 60%
for the bottom twenty, G is 60% in both cases and H differs
for just 20% between the top and lowest rankings.
In conclusion, the classification criteria adopted in this work
permits to infer a strong dependency of SIFT on the JPEG
compression rate variations, however, it does not allow to
draw any conclusions about the general bias, if any exists,
towards a particular type of scene.

3) HESLAP and HESAFF trait indices: Due to the
similarities between the approach in localizing the interest
point in images, HESLAP and HESAFF present many
similarities between their trait indices. Similarly to HARLAP
and HARAFF, uniform light changes have a strong impact
on the HESLAP and HESAFF’s performance [10]. For that
reason, the Figures 6.a and 7.a show only the results for the
top rankings of up to 80% of light reduction and up to 60%
for the lowest rankings.
HESLAP and HESAFF perform better on scenes characterized
by simple elements and edges under blurring (especially
for high σ values) and uniform light decreasing. The same
indices, H, computed under JPEG compression present
fluctuations around 50% for both the top and lowest rankings
without bending towards simple or complex scenes. Both the
detectors perform well on scenes containing human-made
elements under light reduction, JPEG compression and up to
2.5σ of Gaussian blur. Although both HESLAP and HESAFF
do not have any bias for outdoor scenes, the HASLAP’s
F index decrease from 45% to 15% constantly over the
variation range of blurring amount.
4) SIFT: From the trait index data, it is not possible to
determine a clear bias in the performance of SIFT [24], as
the values of the trait indices fluctuate over the entire range
of the JPEG compression rate. Figure 8 confirms a bias
towards simple and human made objects only between 10%
and 60% and at at 98% of JPEG compression. whereas the
indices G and H present large fluctuations in the top twenty
scene rankings for the other compression rates. In particular,
between 70% and 90% of JPEG compression their values are

5) IBR: The uniform light change has a significant impact
on the performance of IBR [20][10]. This made impossible to
obtain the the lowest trait indices for brightness reduction at
85% and 90%. Following the same approach as Section V-B1,
those indices are set to 0 (Figure 9.a). Under light reduction,
the presence of a weak bias across the range of transformation
amount is evident for human-made objects: G indices are
never below 50% in top rankings while their counterpart

Fig. 8: Top and lowest trait indices of SIFT in percentage for
different amounts JPEG compression (a,b).

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

7

Fig. 9: Top and lowest trait indices of IBR in percentage for
different amounts of light reduction (a,d), JPEG compression
(b,e) and blurring (c,f).

Fig. 10: Top and lowest trait indices of MSER in percentage
for different amounts of light reduction (a,d), JPEG compression (b,e) and blurring (c,f).

in the lowest indices are never above 40%. A similar trend
can be observed for F : the share of outdoor scenes in the
top twenty is generally below 50%, while is generally never
below 50% for light reduction rates from 10% to 65%. Under
JPEG compression, IBR achieved better performances on
scenes, which are both simple and human made. Indeed,
the related G and H indices in the top twenties reached
very high values, which are never below 75% and 80%
respectively (Figure 9.b). The same kind of bias observed for
JPEG compression characterized IBR under blurring as well:
the top rankings are mainly populated by human made and
simple scenes, whereas the lowest rankings contain mostly
scenes with the opposite characteristics (Figure 9.c and 9.f).

and others lines present in a scene but makes low contrast
region harder to be identified. Thus, the application of the
uniform light transformation has the effect of populating the
top ranking of SALIENT with those images containing a few
but high contrast elements.
On the other hand, the result for Gaussian blur (Figure 11.c)
shows a completely opposite situation, in which the most
frequent scenes in the top rankings are those characterized by
simple structure or, in other words, scenes whose information
has relevant components at low frequencies. Indeed, as
indicated above, Gaussian blurring can be seen as a low
pass filter and applying it to an image results in loss of the
information at high frequencies. Under JPEG compression,
SALIENT exhibits a preference for complex scene as it
is under light reduction with the difference that the H
indices increase with the compression rate. Although, JPEG
compression is lossy and may alter the shape of the edges
delimiting the potential salient region in an image, the impact
on the information content is lower than the one caused
by Gaussian blur. Indeed, the share of simple images is
constantly low: H below 30% up to 98%. At 98% the share
of simple images in the top twenty increase dramatically
to 65% as the images lose a huge part of their information
content due to the compression, which produce wide uniform
regions (see the Figure 1 for an example).

6) MSER: Figure 10 shows the trait indices for MSER
[19]. Due to sensitivity of MSER to uniform light reduction
and Gaussian blur, it has not been possible to compute the
trait indices for the lowest rankings at light reductions of
more than 60% and for the last three steps of blurring as
the number of scenes with repeatability equal to 0 exceed
the length of the lowest rankings at those transformation
amounts.
The trait indices draw a very clear picture of the MSER’s
biases. The very high values of G and H of the top ranking
indices and the relatively low values obtained for the lowest
twenty rankings, lead to the conclusion that MSER performs
significantly better on simple and human-made dominated
scenes for every transformation type and amount. Finally,
the outdoor scenes populate mainly the lowest rankings built
under light reduction and JPEG compression transformations
while F for blurring has low and balanced values between
the top and lowest rankings.
7) SALIENT: The results for uniform light reduction (Figure 11.a) show a strong preference of SALIENT [21] for
complex scenes as can be inferred by the low values of
the index H in the top twenties contrary to high values in
the lowest rankings. This can be explained considering that
uniform light reduction does not alter the shape of the edges

8) SFOP: Under JPEG compression and Gaussian blur,
the bias of SFOP [22] is towards simple scenes representing
non-outdoor scenes. The kind of objects favored are human
made under JPEG compression, while for blurring no clear
preference can be inferred, due to closeness of the values of
G indices between the top and lowest rankings. The relative
measures of those biases are reflected by the G and H
indices reported in the Figures 12.b and 12.c: H assumes
high percentage values in the top rankings and low values
in the lowest rankings; the indices G for the top rankings
of JPEG compression are constantly above 70% whereas the
related value registered for the lowest rankings exceeds 55%
only at 10% of compression rate. The indices obtained for

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

Fig. 11: Top and lowest trait indices of SALIENT in percentage for different amounts of light reduction (a,d), JPEG
compression (b,e) and blurring (c,f).

8

Fig. 13: Top and lowest trait indices of SURF in percentage for
different amounts of light reduction (a,d), JPEG compression
(b,e) and blurring (c,f).

that in the lowest rankings and the performance are worse with
outdoor scenes than with non-outdoor scenes. Finally, the G
indices do not express a true bias, neither for human-made nor
for natural elements. (Figure 13.d).
Under blurring SURF best performs on simple scenes whereas
it performs poorer on complex scenes. The values of F and
G for the top and lowest rankings are fairly close. F values
for both rankings groups are very low (except for 0.5σ which
reaches 60% for the lowest ranking) while G’s values fluctuate
around 50%.
VI. C ONCLUSIONS
Fig. 12: Top and lowest trait indices of SFOP in percentage for
different amounts of light reduction (a,d), JPEG compression
(b,e) and blurring (c,f).

uniform light reduction reveal that SFOP performs lowest
on outdoor scenes as can be seen by the lowest ranking F
values, which mostly fluctuate between 50% and 60%.
9) SURF: The performance of SURF is particularly affected by uniform light transformation and, because of that, it
has not been possible to compute the trait indices at 65% and
further brightness reductions (Figure 13.a). The effect of this
transformation is to focus the biases of SURF towards human
made objects (G greater than or equal to 65%). Although the
available F indices for lowest rankings are extremely low
(normally within 15%), only a weak bias towards outdoor
scenes can be claimed as the highest values for F indices in
the top twenties are only 60% between 10% and 60% of light
reduction. The percentage of simple scenes in the top rankings
fluctuate between 50% and 85% which, unfortunately, is
reached in a region where the indices are not available thus,
a comparison is not possible.
JPEG compression produces more predictable biases on
SURF: H’s values are significantly higher in the top rankings

For several state-of-the-art feature detectors, the dependency
of the repeatability from input scene type has been investigated
utilizing a large database composed of images from a wide
variety of human-classified scenes under three different types
and amounts of image transformations. Although the utilized
human-based classification method includes just three independently assigned labels, it is enough to prove that the feature
detectors tend to score their highest and lower repeatability
scores with particular types of scenes. The detector preferences
for a particular category of scene are pronounced and stable
across the type and amount of image transformation for some
detectors, such as MSER and EBR. Some detectors’ bias are
influenced more than others by the amount of transformation
and the top-trait indices of SFOP under light changes are
a good example: G and H reach a peak at 98% of light
reduction. In a few cases the indices show very similar values
between top end lower rankings. This allows to conclude that
biases of a detector are not sensitive to a particular image
change. For example, the trait indices of SALIENT for JPEG
compression are between 40% and 60% for most of JPEG
compression rate.
A significant number of local image feature detector have
been assessed in this work, however the proposed framework
is general and can be utilized for assessing any arbitrary
set of detectors. A designer who needs to maximize the
performance of a vision system starting from the choice of

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 20XX

the better possible local feature detector could take advantage
from the proposed framework. Indeed, the framework could
be utilized for identifying the detectors which perform better
with the type of scene most common in the application before
any task-oriented evaluation (e.g. [25], [26]) thus, such a
selection would be carried out on a smaller set of local feature
detectors. For example, for an application which deals mainly
with indoor scenes, the detectors should be short-listed are
HESAFF, HESLAP, HARAFF and HASAFF which have been
proven to achieve their highest repeatability rate with nonoutdoor scenes. On the other hand, if an application is intended
for working in an outdoor environment, EBR should be one of
the considered local feature detectors, especially under light
reduction transformation.
In brief, the framework proposed permits to characterize the
feature detector against the scene content and, at the same
time, represent a useful tool for facilitating the design of those
visual application which utilize a local feature detector stage.
ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers
for their helpful comments and suggestions. This work was
supported by the EPSRC grant number EP/K004638/1 (RoBoSAS).
R EFERENCES
[1] P. Tissainayagam and D. Suter, “Assessing the performance of corner
detectors for point feature tracking applications,” Image and Vision
Computing, vol. 22, no. 8, pp. 663–679, 2004.
[2] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas,
F. Schaffalitzky, T. Kadir, and L. Van Gool, “A comparison of affine
region detectors,” International Journal of Computer Vision, vol. 65,
no. 1-2, pp. 43–72, 2005.
[3] K.
Mikolajczyk,
“Oxford
Data
Set,
http://www.robots.ox.ac.uk/ vgg/research/affine/.”
[4] F. Fraundorfer and H. Bischof, “A novel performance evaluation method
of local detectors on non-planar scenes,” in Computer Vision and Pattern Recognition-Workshops, 2005. CVPR Workshops. IEEE Computer
Society Conference on. IEEE, 2005, pp. 33–33.
[5] S. Ehsan, N. Kanwal, A. Clark, and K. McDonald-Maier, “Improved
repeatability measures for evaluating performance of feature detectors,”
Electronics Letters, vol. 46, no. 14, pp. 998–1000, 2010.
[6] S. Ehsan, A. F. Clark, B. Ferrarini, and K. McDonald-Maier,
“JPEG, Blur and Uniform Light Changes Image Database,
http://vase.essex.ac.uk/datasets/index.html.”
[7] B. Ferrarini, S. Ehsan, N. U. Rehman, and K. D. McDonald-Maier,
“Performance characterization of image feature detectors in relation to
the scene content utilizing a large image database,” in Systems, Signals
and Image Processing (IWSSIP), 2015 International Conference on.
IEEE, 2015, pp. 117–120.
[8] C. Schmid, R. Mohr, and C. Bauckhage, “Evaluation of interest point
detectors,” International Journal of Computer Vision, vol. 37, no. 2, pp.
151–172, 2000.
[9] K. Mikolajczyk and C. Schmid, “Scale & affine invariant interest point
detectors,” International Journal of Computer Vision, vol. 60, no. 1, pp.
63–86, 2004.
[10] B. Ferrarini, S. Ehsan, N. U. Rehman, and K. D. McDonald-Maier,
“Performance comparison of image feature detectors utilizing a large
number of scenes,” Journal of Electronic Imaging, vol. 25, no. 1, pp.
010 501–010 501, 2016.
[11] S. Ehsan, A. F. Clark, A. Leonardis, A. Khaliq, M. Fasli, K. D.
McDonald-Maier et al., “A generic framework for assessing the performance bounds of image feature detectors,” Remote Sensing, vol. 8,
no. 11, p. 928, 2016.
[12] F. Fraundorfer and H. Bischof, “Evaluation of local detectors on nonplanar scenes,” in In Proc. 28th workshop of the Austrian Association
for Pattern Recognition, 2004.

9

[13] T. Dickscheid, F. Schindler, and W. Forstner, “Coding images with local
features,” International Journal of Computer Vision, vol. 94, no. 2, pp.
154–174, 2011.
[14] S. Ehsan, A. F. Clark, and K. D. McDonald-Maier, “Rapid online
analysis of local feature detectors and their complementarity,” Sensors,
vol. 13, no. 8, pp. 10 876–10 907, 2013.
[15] H. Aanaes, A. L. Dahl, and K. S. Pedersen, “Interesting interest points,”
International Journal of Computer Vision, vol. 97, no. 1, pp. 18–35,
2012.
[16] T. Tuytelaars and K. Mikolajczyk, “Local invariant feature detectors:
a survey,” Foundations and Trends in Computer Graphics and Vision,
vol. 3, no. 3, pp. 177–280, 2008.
[17] T. Tuytelaars and L. Van Gool, “Content-based image retrieval based on
local affinely invariant regions,” in Visual Information and Information
Systems, 1999.
[18] K. Mikolajczyk and C. Schmid, “An affine invariant interest point
detector,” in Computer Vision ECCV 2002. Springer, 2002, pp. 128–
142.
[19] J. Matas, O. Chum, M. Urban, and T. Pajdla, “Robust wide-baseline
stereo from maximally stable extremal regions,” Image and Vision
Computing, vol. 22, no. 10, pp. 761–767, 2004.
[20] T. Tuytelaars and L. Van Gool, “Matching widely separated views based
on affine invariant regions,” International Journal of Computer Vision,
vol. 59, no. 1, pp. 61–85, 2004.
[21] T. Kadir, A. Zisserman, and M. Brady, “An affine invariant salient region
detector,” in ECCV, 2004, pp. 228–241.
[22] W. Förstner, T. Dickscheid, and F. Schindler, “Detecting interpretable
and accurate scale-invariant keypoints,” in IEEE International Conference on Computer Vision, 2009.
[23] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust
features (SURF),” Computer Vision and Image Understanding, vol. 110,
no. 3, pp. 346–359, 2008.
[24] D. G. Lowe, “Object recognition from local scale-invariant features,” in
IEEE international conference on Computer Vision, 1999.
[25] M. C. Shin, D. Goldgof, and K. W. Bowyer, “An objective comparison
methodology of edge detection algorithms using a structure from motion
task,” in Computer Vision and Pattern Recognition, 1998. Proceedings.
1998 IEEE Computer Society Conference on. IEEE, 1998, pp. 190–195.
[26] M. C. Shin, D. Goldgof, and K. W. Bowyer, “Comparison of edge
detectors using an object recognition task,” in cvpr. IEEE, 1999, p.
1360.

