1

Hydra: an Ensemble of Convolutional Neural
Networks for Geospatial Land Classification

arXiv:1802.03518v1 [] 10 Feb 2018

Rodrigo Minetto, Maurı́cio Pamplona Segundo, Sudeep Sarkar
Abstract—We describe in this paper Hydra, an ensemble of convolutional neural networks (CNN) for geospatial land classification.
The idea behind Hydra is to create an initial CNN that is coarsely optimized but provides a good starting pointing for further
optimization, which will serve as the Hydra’s body. Then, the obtained weights are fine tuned multiple times to form an ensemble of
CNNs that represent the Hydra’s heads. By doing so, we were able to reduce the training time while maintaining the classification
performance of the ensemble. We created ensembles using two state-of-the-art CNN architectures, ResNet and DenseNet, to
participate in the Functional Map of the World challenge. With this approach, we finished the competition in third place. We also applied
the proposed framework to the NWPU-RESISC45 database and achieved the best reported performance so far. Code and CNN
models are available at https://github.com/maups/hydra-fmow.
Index Terms—Geospatial land classification, remote sensing image classification, functional map of world, ensemble learning, on-line
data augmentation, convolutional neural network.

F

1

I NTRODUCTION

Land use is a critical piece of information for a wide
range of applications, from humanitarian to military purposes. For this reason, automatic land use classification
from satellite images has been drawing increasing attention
from academia, industry and government agencies. One of
the most recent initiatives in this area is the Functional
Map of the World (FMOW) challenge [1] sponsored by the
Intelligence Advanced Research Projects Activity (IARPA),
an organization within the Office of the USA Director of
National Intelligence.
The FMOW challenge consists of creating automatic
solutions to classify a specific given location as one of the
62 target classes (e.g. airport, flooded road, nuclear power
plant and so on) or as none of them (false detections).
FMOW images vary in quality and are distributed over
more than 100,000 globe locations, which leads to high
intraclass variations and considerable interclass confusion.
This, added to traditional satellite imaging problems like
viewpoint, weather, shadow and scale variations, makes this
classification problem a lot harder than previous land use
datasets, such as UC Merced Land Use Dataset [2], WHURS19 [3] and NWPU-RESISC45 [4]. Finally, the FMOW
challenge also limits time and computational resources for
training and testing to minimize the disparity among participants’ solutions.
To cope with the difficulties and restrictions of the
FMOW challenge, as our main contribution, in this work
•
•
•
•

R. Minetto is with Universidade Tecnológica Federal do Paraná (UTFPR),
Brazil. E-mail: rodrigo.minetto@gmail.com
M. P. Segundo is with Universidade Federal da Bahia (UFBA), Brazil.
E-mail: mauriciops@ufba.br
S. Sarkar is with Department of Computer Science and Engineering, University of South Florida (USF), Tampa, FL, USA. E-mail: sarkar@usf.edu
The research in this paper was conducted while the authors were at the
Computer Vision and Pattern Recognition Group, USF.

we present a framework that creates ensembles of Convolutional Neural Networks (CNN) for land use classification in satellite images, which we called Hydra. The idea
behind Hydra is to create an initial CNN that is coarsely
optimized but provides a good starting pointing for further
optimization, which will serve as the Hydra’s body. Then,
the obtained weights are fine tuned multiple times to form
an ensemble of CNNs that represent the Hydra’s heads. The
Hydra framework tackles one of the most common problem
in multiclass classification, which is the existence of several
local minima that prioritize some classes over others and the
eventual absence of a global minimum within the classifier
search space. The ensemble ends up expanding this space
by combining multiple classifiers that converged to local
minima and reaches a better global approximation. Figure 1
illustrates this process, where a black line represents the
body optimization part and the red lines represent the heads
reaching different local minima. To stimulate convergence to
different end points, we exploit different strategies, such as
using online data augmentation, variations in the size of the
region of interest, and different image formats released by
FMOW.

2

T HE FMOW C HALLENGE

The main goal of the FMOW challenge was to encourage
researchers and machine learning experts to design automatic classification solutions for land use interpretation
in satellite images. The competition was hosted by TopCoder1 , where participants had one submission every three
hours to keep track of their ongoing performance. Details
regarding dataset, scoring, and restrictions are given in
Sections 2.1, 2.2, and 2.3, respectively.
1. https://www.topcoder.com/

2

(a)

(b)

(c)

Fig. 1. Illustration of the optimization process in the Hydra framework.
The black line represents the coarse parameter optimization that forms
the Hydra’s body. The red lines represent the Hydra’s heads, which are
obtained after fine tuning the body parameters multiple times seeking to
reach different local minima.

2.1

Dataset

The FMOW dataset, as detailed by Christie et al. [1], contains
almost half of a million images split into training, evaluation
and testing subsets, as presented in Table 1. It was publicly
released2 in two image formats: JPEG (194 GB) and TIFF (2.8
TB). For both formats, high-resolution pan-sharpened [5]
and multi-spectral images were available, although the latter one varies with the format. While TIFF images provided
the original 4/8-band multi-spectral information, the color
space was compressed in JPG images to fit their 3-channel
limitation.

Fig. 2. Different challenges in FMOW satellite images: (a) shadow and
viewpoint variations, (b) arbitrary weather conditions, (c) inaccurate
annotations and (d) scale variations.

TABLE 1
Table of contents of the FMOW dataset.

Training
Evaluation
Testing
Total

# of images

# of boxes

# of distinct boxes

363,572
53,041
53,473
470,086

363,572
63,422
76,988
503,982

83,412
12,006
16,948
112,366

Pan-sharpened image dimensions range from 293 × 230
to 16184 × 16288 pixels, and multi-spectral from 74 × 58
to 4051 × 4077. Each image has one or more annotated
boxes that correspond to regions of interest. Some of these
regions may appear in several images taken in different
time periods, adding a temporal element to the problem,
as shown in Figures 2(a) and 2(b). Traditional problems
arising from that include, but are not limited to, variations
in shadows, satellite viewpoint and weather conditions. Figures 2(c) and 2(d) illustrate other recurrent problems in the
FMOW dataset: inaccurate annotations and large variations
in region size.
Each region is an instance of one of the 62 classes
or a false detection. All classes considered in the FMOW
challenge are illustrated in Figure 3. Those classes have an
unbalanced distribution, as shown in Figure 4. Images in
the training subset contain a single region that is never a
2. https://www.iarpa.gov/challenges/fmow.html

(d)

false detection. The evaluation subset has images with one
or two regions, with at least one not being a false detection.
Testing images may have several regions but their labels are
unknown (see Figure 2(d)). From 53,473 images in the test
subset, 82.60% have a single region, 4.96% have two regions,
5.66% have three regions and 6.78% have four regions or
more, with a maximum number of fourteen regions per
image.
The FMOW dataset also provides different metadata
along with each region, such as bounding box and image
size, ground sample distance, region visibility, cloud cover,
timestamps, wavelengths, scan direction, sun elevation,
Universal Transverse Mercator (UTM) zone, and so on.
2.2

Scoring

Submissions consisted of a list with one label prediction
— among the m possible classes (m = 63, i.e. 62 land use
classes plus false detection) — for each distinct region in the
test set. When the same region appeared in several images,
like the ones in Figures 2(a) and 2(b), the competitors
were asked to merge all these information into a single
label outcome. The quantitative criteria used by the FMOW
challenge to rank its competitors was a weighted average of
the F -measure [6] for each class.

3

Airport
(w: 0.6)

Airport hangar
(w: 1.0)

Airport terminal
(w: 1.0)

Amusement
park (w: 1.0)

Aquaculture
(w: 1.0)

Archaeological
site (w: 1.0)

Barn
(w: 1.0)

Border checkpoint
(w: 1.4)

Burial site
(w: 1.0)

Car dealership
(w: 1.0)

Construction
site (w: 1.4)

Crop field
(w: 0.6)

Dam
(w: 1.0)

Debris or rubble
(w: 0.6)

Educational institution (w: 1.4)

Electric substation
(w: 1.0)

Factory or power
plant (w: 1.4)

Fire station
(w: 1.4)

Flooded road
(w: 0.6)

Fountain
(w: 1.0)

Gas station
(w: 1.4)

Golf course
(w: 1.0)

Ground transp.
station (w: 1.0)

Helipad
(w: 1.0)

Hospital
(w: 1.0)

Impoverished settlement (w: 1.0)

Interchange
(w: 1.0)

Lake or pond
(w: 1.0)

Lighthouse
(w: 1.0)

Military facility
(w: 0.6)

Multi-unit residen- Nuclear powerplant
tial (w: 1.0)
(w: 0.6)

Office building
(w: 1.0)

Oil or gas facility
(w: 1.0)

Park
(w: 1.0)

Parking lot or
garage (w: 1.0)

Place of worship
(w: 1.0)

Police station
(w: 1.4)

Port
(w: 1.0)

Prison
(w: 1.0)

Race track
(w: 1.0)

Railway bridge
(w: 1.0)

Recreational
facility (w: 1.0)

Road bridge
(w: 1.4)

Runway
(w: 1.0)

Shipyard
(w: 1.0)

Shopping mall
(w: 1.0)

Single-unit residential (w: 0.6)

Smokestack
(w: 1.4)

Solar farm
(w: 0.6)

Space facility
(w: 1.0)

Stadium
(w: 1.0)

Storage tank
(w: 1.0)

Surface mine
(w: 1.0)

Swimming pool
(w: 1.0)

Toll booth
(w: 1.0)

Tower
(w: 1.4)

Tunnel opening
(w: 0.6)

Waste disposal
(w: 1.0)

Water treatment
facility (w: 1.0)

Wind farm
(w: 0.6)

Zoo
(w: 1.0)

False detection
(w: 0.0)

Fig. 3. List of classes of the FMOW challenge. An image example, a label and an associated weight are presented for each class. The least
challenging classes have a lower weight (0.6) while the most difficult ones have a higher weight (1.4). These weights impact the final classification
score.

4

Fig. 4. The class histogram distribution for the training (gray bars) and evaluation sets (black bars) by using the 3-band pansharpened
JPEG/RGB set. Note the highly unbalanced class distribution.

class 2

class m

class 1

C=

Ground-truth label

Let P = he1 , e2 , . . . , en i be the predicted labels provided
by a competitor and G = hg1 , g2 , . . . , gn i the ground-truth
labels for n distinct test regions. From P and G, one can
derive a confusion matrix C :
Predicted label

···

class 1

C1,1

C1,2

...

C1,m

class 2
..
.

C2,1
..
.

C2,2
..
.

...
..
.

C2,m
..
.

class m-1

Cm−1,1

Cm−1,2

...

Cm−1,m

class m

Cm,1

Cm,2

...

Cm,m

where Ci,i is the number of correct classifications for the ith class and Ci,j the number of images from the i-th class
that were mistakenly classified as the j -th class. With these
values we compute the precision Pi and recall Ri scores for
the i-th class as follows:

Pi =

tpi
tpi + f pi

Ri =

tpi
tpi + f ni

Pm
Fi × w i
Pm
F̄ = i=1
i=1 wi

where wi is the weight of i-th class. The weights for each
class are provided in Figure 3. It is worth noting that the
weight for false detections is 0, but it does not mean it will
not affect F̄ . If a region is misclassified as a false detection,
its class will have an additional false negative. If a false
detection is misclassified as another class, one more false
positive will be taken into account.
2.3 Hardware and time restrictions
The competition ran from September 21st to December 31st,
2017. The provisional standings were used to select the top
10 competitors as finalists. For the final round, competitors
were asked to provide a dockerized version3 of their solutions, and the final standings were determined based on
the results for a separate sequestered dataset (i.e. none of
the competitors had access to these images). The dockerized
solution had to cope with the following limitations:
•

•

where:

tpi = Ci,i

f pi =

m
X

Cj,i

j=1,j6=i

f ni =

m
X

•

Ci,j

All models required by the solution should be generated during training using raw data only (i.e. preprocessed data was not allowed) within 7 days.
The prediction for all test images should be performed in no more than 24 hours. Prebuilt models
should be provided to enable testing before training.
Hardware was restricted to two types of Linux AWS
instances: m4.10xlarge and g3.16xlarge.

j=1,j6=i

are respectively the number of true positives, false positives
and false negatives for the i-th class. The F -measure for each
class is then computed as the harmonic mean of precision
and recall:

Fi =

(1)

2 × Pi × R i
Pi + R i

The final score F̄ is the weighted sum of the F -measure
for all classes:

Instances of the type m4.10xlarge have 40 Intel Xeon E52686 v4 (Broadwell) processors and 160GB of memory and
were recommended for CPU-based systems. Instances of the
type g3.16xlarge have 4 NVIDIA Tesla M60 GPUs, 64 similar
processors and 488GB of memory and were recommended
for GPU-based systems. However, as can be noticed, a
g3.16xlarge instance was best suited for both CPU-based and
GPU-based systems.
3. https://www.docker.com/

5

2.4

Baseline
4

The organizers released a baseline classification code in
October 14th, 2017, and updated that solution in November
17th of the same year. The most recent and accurate version
of the baseline consists in extracting discriminant features
from images and metadata using deep neural networks.
To this end, they concatenated image features obtained
by a CNN (DenseNet-161 [7]) to a preprocessed vector of
metadata (mean subtraction) and fed this to a multilayer
perceptron (MLP) with 2 hidden layers. A softmax loss is
then used by the Adam optimizer [8] during training. They
used both training and validation subsets for training, and
no augmentation was performed. More details can be found
in the work of Christie et al. [1]. The classifiers employed in
our Hydra framework are variations of this baseline code.

3

on I MAGE N ET [15] as a starting point. Then, the obtained
weights serve as starting point for several copies of these
CNNs, which are further optimized to form the heads of the
Hydra. During testing, an image is fed to all Hydra’s heads
and their outputs are fused to generate a single decision, as
illustrated in Figure 6.
FMoW dataset

...

T HE H YDRA F RAMEWORK

Ensembles of learning algorithms have been effectively used
to improve the classification performance in many computer vision problems [9], [10], [11]. The reasons for that,
as pointed out by Dietterich [12] are: the training phase
could not provide sufficient data to build a single best
classifier; the optimization algorithm fails to converge to the
global minimum, but an ensemble using distinct starting
points could better approximate the optimal result; or the
space being searched may not contain the optimal global
minimum, but an ensemble may expand this space for a
better approximation.
Formally,
a
supervised
machine
learning
algorithm receives a list of training samples
{(x1 , y1 ), (x2 , y2 ), . . . , (xm , ym )} drawn from some
unknown function y = f (x) — where x is the feature
vector and y the class — and searches for a function h
through a space of functions H, also known as hypotheses,
that best matches y . As described by Dietterich [12],
an ensemble of learning algorithms build a set of
hypotheses {h1 , h2 , . . . , hk } ∈ H with respective weights
{w1 , w2 , . . . , wk }, and then provides a classification decision
ȳ through hypotheses fusion. Different fusion approaches
Pk
may be exploited, such as weighted sum ȳ = i=1 wi hi (x),
majority voting, and so on.
Hydra is a framework to create ensembles of CNNs. As
pointed out by Sharkey [13], a neural network ensemble
can be build by: varying the initial weights; varying the
network architecture; and varying the training set. Varying
the initial weights, however, requires a full optimization for
each starting point and thus consumes too much computational resources. For the same reason, varying the network
architecture demands caution when it is not possible to
share learned information between different architectures.
Unlike previous possibilities, varying the training set can be
done in any point of the optimization process for one or
multiple network architectures, and, depending on how its
done, it may not impact the final training cost.
Our framework uses two state-of-the-art CNN architectures, ResNet [14] and DenseNet [7], as shown in Figures 5 and 6. The training process depicted in Figure 5 consists in first creating the body of the Hydra by coarsely optimizing the chosen architectures using weights pretrained
4. https://github.com/fmow/baseline

ResNet1

ResNet

FMoW
Weights

ResNet2
ResNet3

...
ImageNet
Weights

ResNetm
DenseNet1

DenseNet

FMoW
Weights

DenseNet2
DenseNet3

...

Hydra’s body

DenseNetn

Hydra’s heads
Fig. 5. Hydra’s training flowchart: two CNNs, ResNet and DenseNet,
are optimized starting from the ImageNet weights to create the Hydra’s
body. The obtained weights are then used as starting point for several
copies of these CNNs, which are further optimized to form the heads of
the Hydra.

ResNet1

O UTPUT
Shipyard

ResNet2

I NPUT I MAGE

ResNet3

...
ResNetm

Fusion
DenseNet1

...

DenseNet2
DenseNet3

...

63 classes

DenseNetn
Fig. 6. Hydra’s inference flowchart: an input image is fed to all heads of
the Hydra, and each head outputs the probability of each class being the
correct one. A majority voting is then used to determine the final label,
considering that each head votes for its most probable label.

6

A key point on ensemble learning, as observed by
Krogh et al. [16] and Xandra et al. [17], is that the hypotheses should be as accurate and as diverse as possible. In Hydra, such properties were prompted by applying different geometric transformations over the training
samples, like zoom, shift, reflection and rotations. More
details regarding the proposed framework are presented in
Sections 3.1, 3.2 and 3.3.
3.1

Input

ResNet
or
DenseNet

I MAGE

CNN

x` = F` (x`−1 ) + x`−1

(2)

where x`−1 and x` are the input and output of layer `, and
x0 is the input image. ResNets are built by stacking several
of these units, creating bypasses for gradients that improve
the back-propagation optimization in very deep networks.
As a consequence, ResNets may end up generating redundant layers. As a different solution to improve the flow of
gradients in deep networks, Huang et al. [7] proposed to
connect each layer to all preceding layers, which was named
a DenseNet unit:

x` = F` ([x0 , x1 , . . . , x`−1 ])

(3)

where [x0 , x1 , . . . , x`−1 ] is the concatenation of the ` previous layer outputs. DenseNets also stack several units,
but they use transition layers to limit the number of connections. In the end, DenseNets have less parameters than
ResNets and compensate this facts through feature reuse.
This difference in behavior may be one of the causes for
their complementarity.
Both networks are initialized with weights from ImageNet [15] and trained for six epochs using the Adam optimization algorithm [8] with a learning rate of 10−4 . Fully
connected layers with ImageNet weights are discarded, and
three new layers with 4096 neurons are trained from scratch
using a 50% dropout rate, as illustrated in Figure 7. Any
given image metadata that is useful for classification (e.g.
ground sample distance, sun angle) is concatenated to the
input of the first fully connected layer.
The obtained CNN weights from the previous step will
then be used as starting point to multiple copies of the
chosen networks, which will be trained for five more epochs
using progressive drops in the learning rate as follows: a
learning rate of 10−4 is used for the first epoch, 10−5 for the
following three epochs, and 10−6 for the last epoch.
During training, we added different class weights to
the loss function for different heads seeking to enhance
the complementarity between them. Four different weighting schemes were used: unweighted classes, FMOW class
weights as presented in Figure 3 (false detection weight was
set to 1), a frequency-based weighting using the balanced
heuristic [18], and a manual adjustment of the latter.

Metadata

FC

FC

Softmax

...

...

...

...

...

CNN architectures

Both CNN architectures used in this work, ResNet (ResNet50) [14] and DenseNet (DenseNet-161) [7], were chosen for
achieving state-of-the-art results in different classification
problems while being complementary to each other. Let
F` (·) a non-linear transformation — a composite function
of convolution, relu, pooling, batch normalization and so on
— for layer `. As described by He et al. [14], a ResNet unit
can be defined as:

FC

...

Fig. 7. The convolutional neural network architecture used by the Hydra
framework.

3.2

On-line data augmentation

The process of artificially augmenting the image samples in
machine learning by using class-preserving transformations
is meant to reduce the overfitting in imbalanced classification problems or to improve the generalization ability in the
absence of enough data [19], [20], [21], [22], [23]. The breakthroughs achieved by many state-of-the-art algorithms that
benefit from data augmentation has lead many authors [24],
[25], [26], [27] to study the effectiveness of this technique.
Different techniques can be considered, such as geometric (e.g. reflection, scaling, warping, rotation and translation)
and photometric (e.g. brightness and saturation enhancements, noise perturbation, edge enhancement and color
palette changing) transformations. However, as pointed out
by Ratner et al. [28], selecting transformations and tuning
their parameters can be a tricky and time-consuming task.
A careless augmentation may produce unrealistic or meaningless images that will negatively affect the performance.
The process of data augmentation can be conducted offline, as illustrated in Figure 8. In this case, training images
undergo geometric and/or photometric transformations before the training. Although this approach avoids repeating
the same transformation several times, it also considerably
increases the size of the training set. For instance, if we
only consider basic rotations (90, 180 and 270 degrees) and
reflections, the FMOW training set will have more than 2.8
million samples. This would not only increase the amount of
required storage space, but the time consumed by a training
epoch as well.
In our framework we apply an on-line augmentation,
as illustrated in Figure 9. To this end, different random
transformations were applied to the training samples every
epoch. This way, the number of transformations seen during
training increase with the number of epochs. In addition,
different heads of the Hydra experience different versions
of the training set and hardly converge to the same local
minima.
On-line data augmentations were carried out by the
ImageDataGenerator function from Keras5 . We used random flips in vertical and horizontal directions, random
zooming and random shifts over different image crops. We
did not use any photometric distortion to avoid creating
5. https://keras.io/

7

Training samples (n regions)

Training samples (n regions)

...

...

CNN-Training

Off-line data augmentation

Randomly transformed samples for the 1st epoch (n regions)

...

CNN-Training
Image samples for all epochs (8n regions)

CNN-Training

...

Fig. 8. Illustration of an off-line data augmentation process. In this
example considering basic rotations (90, 180 and 270 degrees) and
reflections only, each image is augmented to a set of 8 images before
the training starts. After that, the augmented training set is static for all
CNN epochs.

meaningless training samples. Three image crop styles were
considered, as illustrated in Figure 10: the original bounding
box that was provided as a metadata (Figure 10(a)), and
an expanded bounding box extracted from both JPG pansharpened images (Figure 10(b)) and JPG multi-spectral
images (Figure 10(c)). As multi-spectral images have low
resolution, we discarded samples with less than 96 pixels
of width or height. Table 2 presents the configuration of
each head in terms of network architecture, augmentation
technique and class weighting.

Randomly transformed samples (n regions)
for the 2nd, 3rd and following epochs
Fig. 9. Illustration of an on-line data augmentation process. In this
example considering basic rotations (90, 180 and 270 degrees) and
reflections only, each image is randomly transformed into a new image
before each training epoch starts.

(a)

(c)

Fig. 10. Different image crops used for training: (a) original bounding
box and (b) expanded bounding box extracted from JPG pan-sharpened
images, and (c) expanded bounding box extracted from JPG multispectral images.

4
3.3

(b)

E XPERIMENTAL RESULTS

Fusion and decision

To determine the most appropriate label for an input region
according to the models generated by Hydra, we have to
combine the results from each head. Each head produces
a vector of score values that indicate how well the input
region fits each class. As some regions may appear in
more than one image, their vectors are summed so that
each region ends up with only one vector of scores. The
softmax function is then applied to convert these scores to
probabilities, and the most probable class is selected for each
head. A majority voting is the used to select the final label.
However, if the number of votes is below or equal to half of
the head count, the region is considered a false detection.

4.1

FMOW dataset

In this section we report and compare the Hydra performance in the FMOW dataset. First, to visualize the benefits
of using the Hydra framework, we show the accuracy of
the trained CNNs (body and heads) over the epochs in
Figure 11. We also show the ensemble accuracy when the
heads are available. In this experiment, we use the entire
training subset and false detections of the evaluation subset
for training, and the remaining images of the evaluation
subset for accuracy computation. As can be observed, the
body of the Hydra provides a good starting point for its

8

TABLE 2
Hydra’s heads description. For each head are given the network
architecture, image crop style, augmentation technique and class
weighting method. Three different image crop styles were considered:
ORIG-PAN (Figure 10(a)), EXT-PAN (Figure 10(b)) and EXT-MULTI
(Figure 10(c)). Augmentation techniques included random flips in
vertical and horizontal directions (Flip), random zooming (Zoom) and
random shifts (Shift). Class weighting methods were: no weighting,
FMOW weights (Figure 3) and frequency-based weighting using the
balanced heuristic [18] before (Frequency #2) and after (Frequency #1)
a manual adjustment.
Head
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
#11
#12

CNN
DenseNet
DenseNet
DenseNet
DenseNet
DenseNet
DenseNet
DenseNet
DenseNet
ResNet
ResNet
ResNet
ResNet

Crop
EXT-PAN
ORIG-PAN
EXT-MULTI
EXT-PAN
EXT-PAN
EXT-MULTI
EXT-PAN
ORIG-PAN
EXT-PAN
EXT-MULTI
ORIG-PAN
EXT-MULTI

Augment
Flip
Flip
Flip
Zoom
Shift
Shift
Flip
Flip
Flip
Flip
Flip
Flip

Class weighting
Unweighted
Frequency #1
Frequency #1
Frequency #2
Unweighted
FMOW weights
Frequency #2
Frequency #2
Unweighted
Frequency #1
Frequency #1
Frequency #2

heads, which have slightly different accuracy in the end
of the training process. When the heads are combined, the
obtained accuracy is considerably higher than the accuracy
of the best performing head.

We then reran the training process using all images from
the training and evaluation subsets. The TopCoder submission system was used to obtain the F̄ -measure (Equation 1)
for the testing subset, as it has no ground truth available
so far. The obtained F̄ -measure is shown in Table 3 along
with results reported by Christie et al. [1] and results for
our best individual classifiers. As previously reported by
Christie et al. [1], using metadata for classification purposes
is better than just using the image information. In Table 3
we show that on-line data augmentation has also a big
impact, increasing the accuracy of their respective versions
without augmentation in more than 2%. Still, our ensemble
can improve the performance even more, with a gap of at
least 3% to all individual classifiers.
TABLE 3
F̄ -measure for the FMOW testing subset. Evaluated classifiers may
use metadata information (M ) and/or on-line data augmentation (O).
F̄ -measure values for classifiers marked with † were reported by
Christie et al. [1].

F̄ -measure
Hydra (DenseNet + ResNet)M,O

0.781

DenseNetM,O

0.749

ResNetM,O

0.737

†
LSTMM

0.734

†
DenseNetM

0.722

ResNetM

0.712

LSTM

†

0.688

DenseNet†

Fig. 11. Evaluation of the ensemble accuracy. The first six 6 epochs
show the accuracy of the Hydra’s body using DenseNet and Resnet.
The following epochs show the accuracy for each head separately as
well as for the fusion of all heads (ensemble).

The confusion matrix for the best ensemble result in
Figure 11 (epoch 11) is presented on Figure 12. The hardest
classes for Hydra were: shipyard (mainly confused with
port), hospital (confused with educational institution),
multi unit residential (confused with single unit residential), police station (confused with educational institution),
and office building (confused with fire station, police station, etc). We also had high confusion between nuclear powerplant and port, between prison and educational institution, between space facility and border checkpoint, and
between stadium and recreational facility. These, however,
are natural mistakes that could easily be repeated by human
experts. Most of these classes look alike in aerial images (e.g.
shipyard and port) or have similar features that can trick
an automatic classifier (e.g. both stadium and recreational
facility have a sport court). In Figure 13 we show examples
of mislabeled regions that cannot be easily identified, even
by a human.

0.679

With this performance, we managed to finish the competition among the top ten competitors in the provisional
scoreboard (5th place). We then submitted a dockerized
version of Hydra, which was evaluated by the FMOW
organizers using a sequestered dataset. As we show in
Table 4, the Hydra ensemble was very stable, with nearly
the same performance in the testing subset and sequestered
dataset. Thanks to this, our final ranking was third place,
with accuracy 1% lower than the best result achieved by
another competitor.
TABLE 4
Final ranking: F̄ -measure for the top five participants of the FMOW
challenge using the available testing subset (Open) and the
sequestered dataset (Sequestered).

Rank

Handle

1

pfr

0.7934

0.7917

2

jianminsun

0.7940

0.7886

usf bulls (Hydra) 0.7811

0.7817

3

Open Sequestered

4

zaq1xsw2tktk

0.7814

0.7739

5

prittm

0.7681

0.7670

9

Fig. 12. Confusion matrix for the Hydra ensemble after 11 training epochs (Figure 11). Each row represents the ground truth label, while the
column shows the label obtained by the Hydra. Large values outside of the main diagonal indicate that the corresponding classes are hard to be
discriminated.

4.2

NWPU-RESISC45 dataset

We also reported the Hydra performance in the NWPURESISC45 dataset, which was recently compiled by
Cheng et al. [4]. This dataset has 45 classes, with 700 images
per class, each one with resolution of 256 × 256 pixels,
totaling 31,500 images. The intersection of classes between
FMOW and NWPU-RESISC45 is reasonable, with some
land uses being overcategorized in NWPU-RESISC45 (e.g.
basketball court, baseball diamond and tennis court are
all labeled as recreational facility in the FMOW dataset) and
some classes not representing land use, such as objects (e.g.
airplane and ship) and vegetation (e.g. desert, forest and
wetland). This dataset does not provide satellite metadata.
We followed the experimental setup proposed by
Cheng et al. [4] and randomly chose 10% or 20% of the
samples in each class for training. The remaining samples
were used for testing. We repeated this experiment five
times for each configuration, and reported the average accuracy and standard deviation. As can be shown in Table 5,
Cheng et al. [4] compared six classifiers based on handcrafted features and three different CNN architectures. In

their experiments, the best performance was achieved by
a VGG16 network [10] that was trained using ImageNet
weights as a starting point.
The configuration of the Hydra was slightly different
in this experiment. We used 4 DenseNets and 4 ResNets,
which were trained for 16 epochs (8 for the body and 8 for
the heads). Each head for each architecture has a different
augmentation: 1) no augmentation; 2) random vertical and
horizontal flips; 3) random zoomimg; and 4) random shifts.
The accuracy results for the NWPU-RESISC45 dataset are
shown in Table 5. As NWPU-RESISC45 is less challenging
than FMOW, the results are higher and the improvements
are lower. The on-line data augmentation increased ResNet
and DenseNet accuracies in about 1.0% and 1.6%, respectively. Individual classifiers for both architectures outperformed all previous results reported by Cheng et al. [4], and
the Hydra ensemble was able to achieve the highest accuracy. The improvement on accuracy, however, was slightly
greater than 1% when compared to individual CNNs. It is
worth noting that we outperformed the best accuracy results
reported by Cheng et al. [4] by more than 4%.

10

5

(a) Shipyard mislabeled as port

(b) Hospital mislabeled as educational institution.

(c) Multi-unit residential misla- (d) Office building mislabeled as
beled as single-unit residential
gas station
Fig. 13. Regions from the evaluation subset that were mislabeled by our
Hydra ensemble.

TABLE 5
Accuracy for the NWPU-RESISC45 dataset using training/testing splits
of 10%/90% and 20%/80%. Average accuracy and standard deviation
were reported by using five different random splits. Evaluated
classifiers may use on-line data augmentation (O) and/or ImageNet
weights (I ). Accuracy values for classifiers marked with † were
reported by Cheng et al. [4].

Hydra (DenseNet + ResNet)O,I
DenseNetO,I
ResNetO,I
VGG16I †
AlexNetI †
GoogleNetI †
AlexNet†
VGG16†
GoogleNet†
BoVW†
LLC†
BoVW + SPM†
Color histograms†
LBP†
GIST†

Accuracy
10%/90% 20%/80%
92.44 ± 0.34 94.51 ± 0.21
91.06 ± 0.61 93.33 ± 0.55
89.24 ± 0.75 91.96 ± 0.71
87.15 ± 0.45 90.36 ± 0.18
81.22 ± 0.19 85.16 ± 0.18
82.57 ± 0.12 86.02 ± 0.18
76.69 ± 0.21 79.85 ± 0.13
76.47 ± 0.18 79.79 ± 0.15
76.19 ± 0.38 78.48 ± 0.26
41.72 ± 0.21 44.97 ± 0.28
38.81 ± 0.23 40.03 ± 0.34
27.83 ± 0.61 32.96 ± 0.47
24.84 ± 0.22 27.52 ± 0.14
19.20 ± 0.41 21.74 ± 0.18
15.90 ± 0.23 17.88 ± 0.22

D ISCUSSION

There is an extensive literature on algorithms for geospatial
land image processing — often called remote sensing image
processing. This broad and active field of research has many
branches, such as semantic segmentation [29], [30], [31],
[32], [33], target location [34] and region classification [35],
[36], [37], [38], [39]. An exhaustive literature review was
carried out by Cheng and Han [40] and covered most of the
literature works based on handcrafted features. However, as
can be seen in Table 5, handcrafted approaches are no longer
competitive due to the recent advances on deep learning.
The FMOW challenge is focused on the classification
branch and currently constitutes the most difficult task of
its kind (63 classes, 470086 images). Other smaller datasets,
such as UC Merced Land Use Dataset [2] (UCMerced, 21
classes, 2100 images), WHU-RS19 [3] (19 classes, 950 image),
Brazilian Coffe Scenes [41] (BCS, 2 class, 2876 images)
and Aerial Image Dataset et al. [42] (AID, 30 classes, 10000
images), were used in the literature but not necessarily
represent a challenging task (i.e. reported results are reaching a nearly perfect score). Even NWPU-RESISC45 [4] (45
classes, 31500 images), the most recent dataset other than
FMOW, is not as challenging. We were able to achieve nearly
95% accuracy using only 20% of its images for training. In
comparison, the training/testing split for FMOW is about
85%/15%, but the top performance achieved by a competitor was still below 80%.
Anyway, there is no doubt that deep CNNs are becoming more and more popular for land use classification.
Nogueira et al. [38] compared different state-of-the-art CNN
architectures, such as AlexNet [19], GoogleNet [22] and
VGG [10], and evaluated different training setups, such as
using ImageNet weights as starting point and Support Vector Machines (SVM) for classification. They reported results
for UCMerced, WHU-RS19 and BCS datasets, which were
very successful when compared to many other approaches
based on handcrafted features. Chaib et al. [39] improved
regular VGG networks [10] by fusing the output of the last
two fully connected layers. The work was validated using
UCMerced, WHU-RS19 and AID. We also have CNN-based
results reported by Christie et al. [1] and Cheng et al. [4]
for NWPU-RESISC45 and FMOW datasets, respectively,
as shown in Tables 3 and 5. Thus, CNN classifiers were
definitely the best choice for our Hydra framework. After
experimenting different architectures, we ended up using
DenseNets and ResNets. The main reasons are that they
perform better than previously mentioned architectures,
have a reasonable level of complementarity, and achieve
similar classification results.
In terms of computational resources consumption, Hydra has some disadvantages when compared to individual
classifiers. If N heads are used, its training and inference
costs tend to be N/2 and N times more expensive, respectively. On the other hand, when compared to other ensemble
techniques with the same number of classifiers, Hydra’s
training is approximately two times faster because half of
the epochs are carried out for the Hydra’s body only.
Overall, the Hydra framework was a good fit to the
FMOW restrictions described in Section 2.3. The time limit
for testing was large enough to allow using many classi-

11

fiers, making the training time to generate them the main
problem. Hydra succeeded in increasing the number of classifiers for inference without losing complementarity, a key
requirement for ensembles. As a consequence, we finished
the competition in the third place, with accuracy 1% lower
than the best result achieved by another competitor.

6

C ONCLUSIONS

Automatic and robust classification of aerial scenes is
critical for decision-making by governments and intelligence/military agencies in order to support fast disaster
response. This research problem, although extensively addressed by many authors, is far from being solved, as
evidenced by the final results of the FMOW challenge.
The main contribution of this work was the development
of a framework that automatically creates ensembles of
CNNs to perform land use classification in satellite images. We called this framework Hydra due to its training
procedure: the Hydra’s body is a CNN that is coarsely
optimized and then fine tuned multiple times to form the
Hydra’s heads. The resulting ensemble achieved the fifth
best performance for the open testing subset of the FMOW
challenge, qualifying our solution to the second phase of
the competition. Then, the same ensemble was applied to
a sequestered testing set that was used to officially rank
the competitors, and it obtained the third best performance.
Given that there were more than 50 active participants, including teams and individuals from academia and industry,
and most of them probably were experimenting different
CNN-based setups, we believe that Hydra is well aligned
with the most recent state-of-the-art developments.

[9]

[10]
[11]

[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]

[22]

ACKNOWLEDGMENTS
Part of the equipments used in this project are supported by
a grant (CNS-1513126) from the National Science Foundation (NSF) of USA. The Titan Xp used for this research was
donated by the NVIDIA Corporation.

[23]

[24]

R EFERENCES

[25]

[1]

[26]

[2]

[3]
[4]
[5]

[6]
[7]

[8]

G. Christie, N. Fendley, J. Wilson, and R. Mukherjee, “Functional
map of the world,” arXiv preprint arXiv:1711.07846, 2017.
Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions for land-use classification,” in Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information
Systems, ser. GIS ’10. ACM, 2010, pp. 270–279.
D. Dai and W. Yang, “Satellite image classification via two-layer
sparse coding with biased image representation,” IEEE Geoscience
and Remote Sensing Letters, vol. 8, no. 1, pp. 173–176, 2011.
G. Cheng, J. Han, and X. Lu, “Remote sensing image scene
classification: Benchmark and state of the art,” Proceedings of the
IEEE, vol. 105, no. 10, pp. 1865–1883, Oct 2017.
H. Li, L. Jing, Y. Tang, Q. Liu, H. Ding, Z. Sun, and Y. Chen,
“Assessment of pan-sharpening methods applied to worldview-2
image fusion,” in IEEE International Geoscience and Remote Sensing
Symposium (IGARSS), 2015, pp. 3302–3305.
D. M. W. Powers, “Evaluation: From precision, recall and fmeasure to roc., informedness, markedness & correlation,” Journal
of Machine Learning Technologies, vol. 2, no. 1, pp. 37–63, 2011.
G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger,
“Densely connected convolutional networks,” in IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4700–
4708.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014.

[27]

[28]

[29]

[30]
[31]
[32]

H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua, “A convolutional
neural network cascade for face detection,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2015, pp. 5325–
5334.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” pp. 1–14, 2015.
C. Ding and D. Tao, “Trunk-branch ensemble convolutional neural
networks for video-based face recognition,” IEEE Transactions on
Pattern Analysis and Machine Intelligence (TPAMI), vol. PP, no. 99,
pp. 1–11, 2017.
T. G. Dietterich, Ensemble Methods in Machine Learning. Berlin,
Heidelberg: Springer Berlin Heidelberg, 2000, pp. 1–15.
A. J. Sharkey, Ed., Combining Artificial Neural Nets: Ensemble and
Modular Multi-Net Systems, 1st ed. Secaucus, NJ, USA: SpringerVerlag New York, Inc., 1999.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning
for image recognition,” in IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei,
“ImageNet: A Large-Scale Hierarchical Image Database,” in IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
A. Krogh and J. Vedelsby, “Neural network ensembles, cross validation and active learning,” in International Conference on Neural
Information Processing Systems (NIPS), 1994, pp. 231–238.
A. Chandra and X. Yao, “Evolving hybrid ensembles of learning
machines for better generalisation,” Neurocomputing, vol. 69, no. 7,
pp. 686–700, 2006.
G. King and L. Zeng, “Logistic regression in rare events data,”
Political Analysis, vol. 9, pp. 137–163, 2001.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM,
vol. 60, no. 6, pp. 84–90, May 2017.
A. G. Howard, “Some improvements on deep convolutional neural network based image classification,” CoRR, vol. abs/1312.5402,
2013. [Online]. Available: http://arxiv.org/abs/1312.5402
K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling
in deep convolutional networks for visual recognition,” IEEE
Transactions on Pattern Analysis and Machine Intelligence (TPAMI),
vol. 37, no. 9, pp. 1904–1916, 2015.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with
convolutions,” in IEEE Computer Vision and Pattern Recognition
(CVPR), 2015, pp. 1–9.
E. Ahmed, M. Jones, and T. K. Marks, “An improved deep learning
architecture for person re-identification,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3908–
3916.
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return
of the devil in the details: Delving deep into convolutional nets.”
in British Machine Vision Conference (BMVC). BMVA Press, 2014.
T. DeVries and G. W. Taylor, “Dataset Augmentation in Feature
Space,” ArXiv e-prints, 2017.
L. Perez and J. Wang, “The Effectiveness of Data Augmentation
in Image Classification using Deep Learning,” ArXiv e-prints, Dec.
2017.
S. C. Wong, A. Gatt, V. Stamatescu, and M. D. McDonnell, “Understanding data augmentation for classification: When to warp?” in
International Conference on Digital Image Computing: Techniques and
Applications, 2016, pp. 1–6.
A. J. Ratner, H. R. Ehrenberg, J. D. Zeshan Hussain, and C. Re,
“Learning to compose domain-specific transformations for data
augmentation,” in International Conference on Neural Information
Processing Systems (NIPS), 2017, pp. 1–11.
Q. Shi, B. Du, and L. Zhang, “Spatial coherence-based batchmode active learning for remote sensing image classification,”
IEEE Transactions on Image Processing (TIP), vol. 24, no. 7, pp. 2037–
2050, 2015.
H. Lee and H. Kwon, “Going deeper with contextual cnn for
hyperspectral image classification,” IEEE Transactions on Image
Processing (TIP), vol. 26, no. 10, pp. 4843–4855, 2017.
H. Wu and S. Prasad, “Semi-supervised deep learning using
pseudo labels for hyperspectral image classification,” IEEE Transactions on Image Processing (TIP), vol. 27, no. 3, pp. 1259–1270, 2018.
J. Zhao, Y. Zhong, H. Shu, and L. Zhang, “High-resolution image
classification integrating spectral-spatial-location cues by conditional random fields,” IEEE Transactions on Image Processing (TIP),
vol. 25, no. 9, pp. 4033–4045, 2016.

12

[33] X. Xu, W. Li, Q. Ran, Q. Du, L. Gao, and B. Zhang, “Multisource
remote sensing data classification based on convolutional neural
network,” IEEE Transactions on Geoscience and Remote Sensing,
vol. PP, no. 99, pp. 1–13, 2017.
[34] Z. Zou and Z. Shi, “Random access memories: A new paradigm for
target detection in high resolution aerial remote sensing images,”
IEEE Transactions on Image Processing (TIP), vol. 27, no. 3, pp. 1100–
1111, 2018.
[35] Y. Xia, L. Zhang, Z. Liu, L. Nie, and X. Li, “Weakly supervised
multimodal kernel for categorizing aerial photographs,” IEEE
Transactions on Image Processing (TIP), vol. 26, no. 8, pp. 3748–3758,
2017.
[36] E. Li, J. Xia, P. Du, C. Lin, and A. Samat, “Integrating multilayer
features of convolutional neural networks for remote sensing
scene classification,” IEEE Transactions on Geoscience and Remote
Sensing, vol. 55, no. 10, pp. 5653–5665, 2017.
[37] X. Lu, X. Zheng, and Y. Yuan, “Remote sensing scene classification
by unsupervised representation learning,” IEEE Transactions on
Geoscience and Remote Sensing, vol. 55, no. 9, pp. 5148–5157, 2017.
[38] K. Nogueira, O. A. Penatti, and J. A. dos Santos, “Towards better
exploiting convolutional neural networks for remote sensing scene
classification,” Pattern Recognition - Elsevier, vol. 61, pp. 539–556,
2017.
[39] S. Chaib, H. Liu, Y. Gu, and H. Yao, “Deep feature fusion for
vhr remote sensing scene classification,” IEEE Transactions on
Geoscience and Remote Sensing, vol. 55, no. 8, pp. 4775–4784, 2017.
[40] G. Cheng and J. Han, “A survey on object detection in optical
remote sensing images,” ISPRS Journal of Photogrammetry and
Remote Sensing - Elsevier, vol. 117, pp. 11–28, 2016.
[41] O. A. B. Penatti, K. Nogueira, and J. A. dos Santos, “Do deep
features generalize from everyday objects to remote sensing and
aerial scenes domains?” in IEEE Conference on Computer Vision and
Pattern Recognition Workshops (CVPRW), 2015, pp. 44–51.
[42] G. S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, L. Zhang, and
X. Lu, “Aid: A benchmark data set for performance evaluation
of aerial scene classification,” IEEE Transactions on Geoscience and
Remote Sensing, vol. 55, no. 7, pp. 3965–3981, 2017.

