1

MARTA GANs: Unsupervised Representation
Learning for Remote Sensing Image Classification

As satellite imaging techniques improve, an ever-growing
number of high-resolution satellite images provided by special
satellite sensors have become available. It is urgent to be
able to interpret these massive image repositories in automatic
and accurate ways. In recent decades, scene classification has
become a hot topic and is now a fundamental method for landresource management and urban planning applications. Compared with other images, remote sensing images have several
special features. For example, even in the same category, the
objects we are interested in usually have different sizes, colors
and angles. Moreover, other materials around the target area
cause high intra-class variance and low inter-class variance.
Therefore, learning robust and discriminative representations
from remotely sensed images is difficult.
Previously, the bag of visual words (BoVW) [1] method
was frequently adopted for remote sensing scene classification.
BoVW includes the following three steps: feature detection,
feature description, and codebook generation. To overcome
the problems of the orderless bag-of-features image representation, the spatial pyramid matching (SPM) model [2] was
This work was supported in part by the National Natural Science Foundation
of China under Grant No.41501485 and No.61331017. (Corresponding author:
Kun Fu.)
The authors are with the Key Laboratory of Technology in Geo-spatial
Information Processing and Application System, Chinese Academy of
Sciences, Beijing, 100190, China (e-mail: lindaoyu15@mails.ucas.ac.cn;
fukun@mail.ie.ac.cn;
primular@163.com;
gluanxu@mail.ie.ac.cn;
sunxian@mail.ie.ac.cn).

Fake

0』

I. I NTRODUCTION

Rm ol
um
。』-HH

Index Terms—Unsupervised representation learning, generative adversarial networks, scene classification.

Unlabeled
cde

m 旧

Abstract—With the development of deep learning, supervised
learning has frequently been adopted to classify remotely sensed
images using convolutional networks (CNNs). However, due to
the limited amount of labeled data available, supervised learning
is often difficult to carry out. Therefore, we proposed an unsupervised model called multiple-layer feature-matching generative
adversarial networks (MARTA GANs) to learn a representation
using only unlabeled data. MARTA GANs consists of both a
generative model G and a discriminative model D. We treat D
as a feature extractor. To fit the complex properties of remote
sensing data, we use a fusion layer to merge the mid-level and
global features. G can produce numerous images that are similar
to the training data; therefore, D can learn better representations
of remotely sensed images using the training data provided by G.
The classification results on two widely used remote sensing image
databases show that the proposed method significantly improves
the classification performance compared with other state-of-theart methods.

』句〉E。℃cm」UZωu
」m
-DE

arXiv:1612.08879v3 [] 21 Nov 2017

Daoyu Lin, Kun Fu, Yang Wang, Guangluan Xu, and Xian Sun

nud

r3

Discriminator

(J)

Generator

Feature Match Loss

Figure 1: Overview of the proposed approach. The discriminator (2) learns to make classifications between real and
synthesized images, while the generator (1) learns to fool the
discriminator.
proposed, which works by partitioning the image into increasingly fine sub-regions and computing histograms of local
features found inside each sub-region. The above-mentioned
methods have comprised the state of the art for several years
in the remote sensing community [3], but they are based on
hand-crafted features, which are difficult, time-consuming, and
require domain expertise to produce.
Deep learning algorithms can learn high-level semantic
features automatically rather than requiring handcrafted features. Some approaches [4], [5] based on convolutional neural
networks (CNNs) [6] have achieved success in remote sensing
scene classification, but those methods usually require an
enormous amount of labeled training data or are fine-tuned
from pre-trained CNNs.
Several unsupervised representation learning algorithms
have been based on the autoencoder [7], [8], which receives
corrupted data as input and is trained to predict the original,
uncorrupted input. Although training the autoencoder requires
only unlabeled data, input reconstruction may not be the
ideal metric for learning a general-purpose representation. The
concept of Generative Adversarial Networks (GANs) [9] is one
of the most exciting unsupervised algorithm ideas to appear in
recent years; its purpose is to learn a generative distribution
of data through a two-player minimax game. In subsequent
work, a deep convolutional GAN (DCGAN) [10] achieved a
high level of performance on image synthesis tasks, showing
that its latent representation space captures important variation
factors.
GANs is a promising unsupervised learning method, yet
thus far, it has rarely been applied in the remote sensing field.
Due to the tremendous volume of remote sensing images,
it would be prohibitively time-consuming and expensive to
label all the data. To tackle this issue, GANs would be the
excellent choice because it is an unsupervised learning method

2

3

3

conv 4×4
Features

16 conv 4×4
32 conv 4×4

16

64
32

896 × 4 × 4 = 14336

conv 4×4
conv 4×4
128
256

Multi-feature Layer

conv 4×4
512

64

128

+ 256 +

512 = 896
Flatten

128
256

256

4

512

8

64

128
128

16

256

32

100
4 4

8

32
16

8

project and reshape
deconv 4×4

16

deconv 4×4

32

16

32

64

64

8

4
4

Full

4

Identity

2×2 Max Pooling

4×4 Max Pooling

64
128

deconv 4×4

256

128
256

deconv 4×4
deconv 4×4
deconv 4×4

(a) generator

(b) discriminator

Figure 2: Network architectures of a generator and a discriminator: (a) a MARTA GANs generator is used for the UC-Merced
Land-use dataset. The input is a 100-dimensional uniform distribution pz (z) and the output is a 256 × 256-pixel RGB image;
(b) a MARTA GANs discriminator is used for the UC-Merced Land-use dataset. The discriminator is treated as a feature
extractor to extract features from the multi-feature layer.
in which the required quantities of training data would be
provided by its generator. Therefore, in this paper, we propose a multiple-layer feature-matching generative adversarial
networks (MARTA GANs) model to learn the representation
of remote sensing images using unlabeled data.
Although based on DCGAN, our approach is rather different
in the following aspects. 1) DCGAN can, at most, produce
images with a 64 × 64 resolution, while our approach can
produce remote sensing images with a resolution of 256 × 256
by adding two deconvolutional layers in the generator; 2) To
avoid the problem of such deconvolutional layers producing
checkerboard artifacts, the kernel sizes of our networks are
4 × 4, while those in DCGAN are 5 × 5; 3) We propose
a multi-feature layer to aggregate the mid- and high-level
information; 4) We combine both the perceptual loss and
feature matching loss to produce more accurate fake images.
Based on the improvements above, our method can realize
the better representation of remote sensing images among all
methods. Fig. 1 shows the overall model.
The contributions of this paper are the following:
1) To our knowledge, this is the first time that GANs have
been applied to classify unsupervised remote sensing
images.
2) The results of experiments on the UC-Merced Landuse and Brazilian Coffee Scenes datasets showed that
the proposed algorithm outperforms state-of-the-art unsupervised algorithms in terms of overall classification
accuracy.
3) We propose a multi-feature layer by combining perceptual
loss and loss of feature matching to learn better image
representations.
II. M ETHOD
A GAN is most straightforward to apply when the involved
models are both multilayer perceptrons; however, to apply
a GAN to remote sensing images, we used CNNs for both
the generator and discriminator in this work. The generator
network directly produces samples x = G(z; θg ) with parameters θg and z, where z obeys a prior noise distribution
pz (z). Its adversary, the discriminator network, attempts to
distinguish between samples drawn from the training data and

samples created by the generator. The discriminator emits a
probability value denoted by D(x; θd ) with parameters θd ,
indicating the probability that x is a real training example
rather than a fake sample drawn from the generator. During
the classification task, the discriminative model D is regarded
as the feature extractor. Then, additional training data so that
the discriminator can learn a better representation is provided
by the generative model G.
A. Training the discriminator
When training the discriminator, the weights of the generator are fixed. The goals of training the discriminator D(x) are
as follows:
1) Maximize D(x) for every image from the real training
examples.
2) Minimize D(x) for every image from the fake samples
drawn from the generator.
Therefore, the objective function of training discriminator
is to maximize:
Ex∼pdata (x) log D(x) + Ez∼pz (z) [log(1 − D(G(z)))] .

(1)

B. Training the generator
When training the generator, the weights of the discriminator are fixed. The goal of training the generator G(z) is
to produce samples that fool D. The output of the generator
is an image that can be used as the input for the discriminator. Therefore, the generator wants to maximize D(G(z))
(or equivalently, minimize 1 − D(G(z))) because D is a
probability estimate that ranges only between 0 and 1. We call
this concept perceptual loss; it encourages the reconstructed
image to be similar to the samples drawn from the training
set by minimizing the perceptual loss.
`perceptual = Ez∼pz (z) [log(1 − D(G(z)))].

(2)

In summary, the discriminator D is shown an image produced from the generator G and adjusts its parameters to
make its output, D(G(Z)), larger. But G(Z) will train itself to
produce images that fool D into thinking they are real. It does
this by getting the gradient of D with respect to each sample
it produces. In other words, the G is trying to minimize the

ry images produced by a generator trained onJOURNAL
UC-Merced
using the `f ianl
(Eqn. 4) objective. Generator
OF LATEX CLASS FILES,
VOL. 14, NO. 8, AUGUST 2015
ecedented resolution (256 × 256) Remote Sensing Images.

(a) real images

3

(b) fake images

ry images produced by a generator trained onFigure
UC-Merced
using the
`f ianlproduced
(Eqn. 4) by
objective.
Generator
3: Exemplary
images
a generator
trained on UC-Merced using the `f ianl (Eqn. 4)
Figure 3:(256
Part× of
exemplary
images.
(a) Ten
images resolution
from UC-Merced
data Remote
set. (b) Sensing
Exemplary
images produced by
ecedented resolution
256)
Remote Sensing
Images.
sampled
withrandom
unprecedented
(256 × 256)
Images.
generator trained on UC-Merced using the `f inal (Eqn. 5) objective.

t distinguishoutput
between
them.
B. Brazilian
Coffee Scenes
We set the kernel sizes to 4 × 4 and the stride to 2 in
while
D is trying to maximize
it; consequently,
it is a
visualize theminimax
image global
This dataset [1] is a composition
of scenes
taken byand
SPOT
all the
convolutional
deconvolutional layers, because the
game representations
that is defined as follows:
RTA GANs features for the UC-Merced sensor in 2005 over four counties
in
the
State
of
Minas
deconvolutional
layers
can
avoid uneven overlap when the
min max V (D, G) =Ex∼pdata (x) log D(x)+
D image scenes of
compute features G
for all
Gerais, Brazil: Arceburgo, Guaranésia,
Guaxupé
and Monte
kernel size
is divisible
by the stride [11]. In the generator, all
n use the t-SNE algorithm [15] to embed
Ez∼pz (z)Santo.
[log(1The
− D(G(z)))]
.
(3) of layers
whole image
set
each county
wasactivation
partitioned
use ReLU
except for the output layer, which
al image features in 2-D space. We show into multiple tiles of 64 × 64 pixels.
Fortanh
this function.
dataset, itWe
wasuse LeakyReLU activation in the
uses the
make colors
the images
generated
by generator
similar
ng points with To
different
and shapes
considered
only more
the green,
red, and
near-infrared
discriminator
forbands,
all thewhich
convolutional layers; the slope of the
to the
real images,
wevisutrain the
generator
to match
the leak was
eir actual scene
categories.
The final
are the
most useful
and representative
ones
for
discriminating
set to 0.2. We used batch normalization in both the
expected
values
of the features
in the multi-feature
of ofgenerator
e shown in Fig.
7. This
observation
show vegetation
areas. Thelayer
creation
the dataset
as on
theperformed
discriminator,
andUC-Merced
the decay factor
Figure 4: Exemplary
images produced
byand
a is
generator
trained
using was
the `0.9.
f ianl (Eqn. 4)
the discriminator.
Letting
f (x) denote
activations
onatthe
multied from matri-feature
layers are
high-level
follows:
tiles
with
least
85%
of
coffee
pixels
were
assigned
sampled with unprecedented resolution (256 × 256) Remote Sensing Images.
feature layer
of the discriminator,
of feature
n abstract semantic
information,
because tothe
theloss
coffee
class; matching
tiles with less than 10% of coffee
III.pixels
E XPERIMENTS
the in
generator
is defined as follows:
are still veryfor
close
2-D space.
were assigned to the non-coffee class, it has 1438 tiles of
To 8verify
the
effectiveness
proposed
method,
and
1438
uncoffee.
Fig.
shows
some
examples ofB.the
even
canftiles
not of
distinguish
between
them.
Brazilian
Coffee
Sceneswe
2
some approaches
have been developed
for(x) fcoffee
`f eature_match
= ||Ex∼pdata
(x) −human
Ez∼p
(G(z))||
2.
trained
MARTA
GANs
on
two
datasets:
the
UC
Merced
Land
z (z)
by a generator
trained
on
this global
dataset.representations
ne classification and most of them have produced
In addition,
we visualize
image
dataset
[1]dataset
is a composition
of sc
(4) theUse
dataset [12]
and the Brazilian This
Coffee
Scenes
[4].
he UC-Merced
dataset.
Therefore,
a
lot
encoded
via
MARTA
GANs
features
for
the
UC-Merced
sensor
in
2005
over
four
counties
in
Therefore, our final object (the Table
combination
of Eqn. 2 accurary
and We(%)
carried
out experiments
III: Classification
of representation
ex-on both datasets using a 5-fold
an be compared
with
our
results
in
this
dataset.
Here,
we
compute
features
for
all
image
scenes
of
Gerais,
Brazil:
Arceburgo,
Guaranésia,
G
Eqn. 4) for training the generatortracted
is to minimize
Eqn.
5 MARTA
.
cross-validation
protocol
and a regularized linear L2-SVM as
by DCGANs
and
GANS on the Coffee
Scenes
e report the overall accuracies for all these the dataset, and then use the t-SNE algorithm [15] to embed Santo. The whole image set
1 of each cou
a classifier. We implemented MARTA GANs in TensorLayer ,
dataset. Best result in bold.
`f inal
= `perceptual
.
(5) features in 2-D space. We show into multiple tiles of 64 × 64 pixels. For
ds, as they appear in the
original
papers, + `the
high-dimensional
image
f eature_matching
a deep learning and reinforcement learning library extended
curacy of our best MARTA GANs method. these
2-D embedding
shapes considered only the green, red, and near-in
Architectures
Design points with different colors andAccuracy
from Google TensorFlow [13]. We
scaled the input image to
hod guarantees a better performance gain corresponding
to
their
actual
scene
categories.
The
final
visu- are the most useful and representative one
DCGANs
Without data augmentation
85.36
C. Network architectures
the range of [-1, 1] before
training. All the models were trained
ces.
alization results With
are data
shown
in Fig. 7. This observation85.01
show vegetation areas. The creation of the dat
augmentation
by SGD with a batch 87.69
size of 64, and we used the Adam
MARTA
GANsextracted
Without
data augmentation
The details of the generator and
discriminator
in MARTA
that
features
from
matri-feature layers are high-level
follows: tiles with at least 85% of coffee
optimizer
with only
a learning
With data augmentation,
`perceptual
87.73rate of 0.0002 and a momentum
are as follows:
abstract
semantic
information,
because
to the coffee class; tiles with less than 1
ion accuracy GANs
(%) of reference
and proposed features that contain
With data augmentation,
88.36
f inal
term use
β `of
0.5.
TheBest
generator
drawnarefrom
thosenumbers
close classes
still avery close 1in 2-D space.
were assigned to the non-coffee class,
C-Merced dataset.
result intakes
bold.100 random
uniform distribution as input. Then,
the result
is some
reshaped
In
recent
years,
approaches
have
been
developed
for
Table III shows the results obtained with the proposed coffee and 1438 tiles of uncoffee. Fig. 8 sh
od
Accuracy
into a four-dimensional
tensor. We
usedsensing
six deconvolutional
A. UCand
Merced
remote
scenenotable
classification
mostthedataset
ofUC-Merced
them have produced by a generator trained on this d
techniques.
The most
difference
w.r.t.
layers in our
own spatial
upsampling
and
F [16]
92.38generator to learn itsbeen
This
dataset
consists
verified
on
the
UC-Merced
dataset.
Therefore,
aoflotimages of 21 land-use classes
case is the data augmentation doesn’t improve the accuracy
[17]
Table III: Classification accurary (%) o
upsample94.30
the 4 × 4 feature maps of
to 256
×available
256 remote
sensing
(100
256
×
256-pixel
images
each class). Some of the
data
can
be
compared
with
our
results
in
conspicuously. In fact, it’s just a 2-class problem, andthis
it for
SC [18]
90.26
tracted by DCGANs and MARTA GANS o
images.
Fig.
2a
shows
a
visualization
of
the
generator.
images
from
this
dataset
are
shown
Fig. 3a. We used
19]
89.10
paper.
In
table
II
we
report
the
overall
accuracies
for
all
these
has enough data to train
network. In general, results dataset.inBest
result in bold.
For the88.20
discriminator, the firstcomparable
layer takesmethods,
input images,
I-based [20]
a moderate
augmentation
in this dataset via flipping
asthan
they with
appear
in thedata
original
papers,
are
significantly
worse
UC-Merced,
despite
the
TA GANs(proposed)
including95.00
both real and synthesized
images.
We
use
convoimages
horizontally
and
verticallyArchitectures
and rotating them
togethervs.
with
the accuracy
of ourThis
bestisMARTA
GANs
method.
2-class
21-class
problem.
a rather
challenging
Design by 90
lutions in our discriminator which
it to learn itsguarantees
own
to caused
increase
the effective
training
set
size.
Training
Theallows
proposed
a better
performance
gain
dataset,
due to amethod
large intra-class degrees
variability
by different
Without data augmentation
spatial downsampling. As shownw.r.t.
in Fig.
by performing takes approximately 4 hours on a DCGANs
single NVIDIAWith
GTX
to all2b,references.
data1080
augmentation
4 × 4 max pooling, 2 × 2 max pooling and the identity GPU.
MARTA GANs Without data augmentation
With data augmentation, `perc
function separately in the last three convolutional layers, we
Toofevaluate
theand
quality
of the representations learned
by the
Table II: Classification accuracy (%)
reference
proposed
With data augmentation, use `
can produce feature maps that have the same spatial size, 4×4. multi-feature layer, we trained on the UC-Merced
data and
methods on the UC-Merced dataset. Best result in bold.
Then, we concatenate the 4 × 4 feature maps through channel extracted the features from different multi-feature layers. To
Table III shows the results obtained
dimension in the multi-feature layer. Finally, theMethod
multi-feature improveAccuracy
the clarity of the expression,
we use
denote difference w
1 to notable
techniques.
The fmost
layer is flattened and fed into a single sigmoidHMFF
output.
[16] The
92.38from the last convolutional layer, f2 to denote
the features
case is the data augmentation doesn’t im
[17] used
94.30
multi-feature layer includes two functions: 1) theVLAT
features
features combined
from the lastconspicuously.
two convolutional
layers’
In fact,
it’s just a 2-cla
UFL-SC
[18]
90.26
for classification are extracted from the flatted multi-feature features, and so on. Based on the results shown
in
Fig.
4,the network.
PSR [19]
89.10
has
enough
data
to
train
layer; 2) when training the generator, we use feature
matching
MCMI-based
[20] we found 88.20
that f3 achieved the highest
accuracy.
These
results
are significantly worse than with UC-M
MARTA
GANs(proposed)
95.00
loss (Eqn. 4) to evaluate the similarities of the features
between
2-class vs. 21-class problem. This is a
1
http://tensorlayer.readthedocs.io/en/latest/
the fake and real images in the flatted multi-feature layer.
dataset, due to a large intra-class variabilit

4

Table I: Classification accuracy (%) in the form of the means ± standard deviation bars of DCGAN and MARTA GAN for
every class.. The class labels are as follows: 1 = Mobile home park, 2 = Beach, 3 = Tennis courts, 4 = Airplane, 5 = Dense
residential, 6 = Harbor, 7 = Buildings, 8= Forest, 9 = Intersection, 10 = River, 11 = Sparse residential, 12 = Runway, 13 =
Parking lot, 14 = Baseball diamond, 15 = Agricultural, 16 = Storage tanks, 17 = Chaparral, 18 = Golf course, 19 = Freeway,
20 = Medium residential, and 21 = Overpass.
1
85 ± 5.0
95 ± 3.5
12
89 ± 4.2
94 ± 4.2

2
94 ± 2.2
100 ± 0.0
13
88 ± 2.7
98 ± 2.7

3
89 ± 4.2
96 ± 4.2
14
95 ± 3.5
100 ± 0.0

4
95 ± 3.5
100 ± 0.0
15
78 ± 4.5
85 ± 5.0

1.0

1.0

0.8

0.8

0.8

0.6
0.4
0.2
0.0
0

0.6
0.4
0.2

50

100
Epoch

(a)

150

200

0.0
0

Average accuracy

1.0

0.8

Average accuracy

1.0

Average accuracy

Average accuracy

Class
DCGAN
MARTA GAN
Class
DCGAN
MARTA GAN

0.6
0.4
0.2

50

100
Epoch

(b)

150

200

0.0
0

5
82 ± 2.7
89 ± 4.2
16
93 ± 2.7
100 ± 0.0

6
91 ± 2.2
99 ± 2.2
17
88 ± 2.7
93 ± 2.7

7
78 ± 2.7
86 ± 6.5
18
97 ± 2.7
100 ± 0.0

8
83 ± 2.7
97 ± 2.7
19
77 ± 2.7
87 ± 5.7

9
88 ± 2.7
98 ± 2.7
20
95 ± 5.0
97 ± 5.5

10
90 ± 0.0
94 ± 2.2
21
89 ± 4.2
95 ± 5.0

11
79 ± 2.2
89 ± 2.2

0.6
0.4
0.2

50

100
Epoch

(c)

150

200

0.0
0

50

100
Epoch

150

200

(d)

Figure 4: The performance comparison uses different features.
(a) f1 ; (b) f2 ; (c) f3 ; (d) f4 . The red curves: training with
`f inal and with data augmentation; Cyan curves: training with
`perceptual and with data augmentation; Yellow curves: training with `f inal and without data augmentation; Green curves:
training with `perceptual and without data augmentation.
can be explained by two reasons. First, f3 has the same
high-level information as f1 and f2 , but it has more midlevel information compared with f1 and f2 . However, f4 has
too much low-level information, which leads to the "curse
of dimensionality." Therefore, the features extracted from the
last three convolutional layers in the discriminator resulted in
the highest accuracy. As shown in Fig. 4, data augmentation
is an effective way to reduce overfitting when training a
large deep network. Augmentation generates more training
image samples by rotating and flipping patches from original
images. We also evaluated the performance between two types
of loss: `perceptual (Eqn. 2) and `f inal (Eqn. 5) and found
that using `f inal achieved the best performance. Synthesized
remote sensing images when using `f inal are shown in Fig. 3b.
Fig. 5 depicts the confusion matrix of classification results
for the two GAN architectures, DCGAN and MARTA GAN.
DCGAN and MARTA GAN reached an overall accuracy of
87.76±0.64% and 94.86±0.80%, respectively. MARTA GAN
is approximately 7% better because it used the multi-feature
layer to merge the mid-level and global features. To improve
the comparison, the accuracy classification performances of
the methods for each class are shown in Table I. Compared
to DCGAN, MARTA GAN achieves 100.00% accuracy in
some scene categories (e.g., Beach, Airplane, etc.). Moreover,
MARTA GAN also achieves higher accuracy in some very
close classes, such as dense residential, building, medium
residential, sparse residential.
In addition, we visualized the global image representations encoded via MARTA GANs features of the UC-Merced
dataset. We computed the features for all the scenes of the
dataset and then used the t-SNE algorithm to embed the highdimensional features in 2-D space. The final results are shown
in Fig. 6. This visualization shows that features extracted from
the multi-feature layer contain abstract semantic information

(a)

(b)

Figure 5: Confusion matrix of (a):DCGAN, (b):MARTA GAN.
The class labels are same as Table I.

Figure 6: 2-D feature visualization of image global representations of the UC-Merced dataset. The class labels are same
as Table I.
because those close classes are also very close in 2-D space.
Compared with the results of other tested methods, the
method proposed in this work achieves the highest classification accuracy among the unsupervised methods. As shown
in Table II, our method outperforms the SCMF [14] (a sparse
coding based multiple-feature fusion method) by 3.82%. When
the classification accuracy of our method is compared with
LRFF [15] (an improved unsupervised feature learning algorithm based on spectral clustering), our method outperforms
LRFF by more than 4%. While some of the supervised
methods [4], [5] achieved an accuracy above 99%, these
methods are fine-tuned from pre-trained models, which are
usually trained with a large amount of labeled data (such as
ImageNet). Compared with those methods, our unsupervised
method requires fewer parameters.

y images produced by a generator trained onJOURNAL
UC-Merced
using the ` ianl 14,
(Eqn.
4) objective. Generator
OF LATEX CLASS FILES,f VOL.
NO. 8, AUGUST 2015
cedented resolution (256 × 256) Remote Sensing Images.

(a) real images

5

(b) fake images

7: Exemplary
images
a generator
trained on UC-Merced using the `f ianl (Eqn. 4)
y images produced by a generator trained onFigure
UC-Merced
using the
`f ianlproduced
(Eqn. 4)byobjective.
Generator
Figure 7:(256
Parts
of exemplary
images:
(a) ten
images from
the Brazilian
Coffee
ScenesSensing
dataset;Images.
(b) exemplary images
sampled
withrandom
unprecedented
resolution
(256 × 256)
Remote
cedented resolution
× 256)
Remote Sensing
Images.
produced by a generator trained on the Brazilian Coffee Scenes dataset using the `f inal (Eqn. 5) objective.

Table
II: Overall
classification
ish Agarwal, Paul
Barham,
Eugene Brevdo,
Zhifeng accuracy (%) of reference and
Greg S Corrado,proposed
Andy Davis,methods
Jeffrey Dean,
onMatthieu
the UC-Merced dataset and Coffee
nsorflow: Large-scale machine learning on heterogeScenes
dataset.
Our
result
is in bold.
ystems,” arXiv preprint arXiv:1603.04467, 2016.
n Newsam, “Bag-of-visual-words
and spatial exten- Description
DataSet
Method
Parameters Accuracy
classification,” in Proceedings SCMF
of the [14]
18th SIGSPA- Unsupervised
91.03±0.48
conference on advances in geographic
information Unsupervised
UFL-SC [15]
90.26±1.51
OverFeatL + Caffe [4]
Supervised
205M
99.43±0.27
10, pp. 270–279. UC-Merced
GoogLeNet data
[5] using Supervised
5M
99.47±0.50
Maaten and Geoffrey Hinton, “Visualizing
GANs
94.86±0.80
Machine Learning Research, MARTA
vol. 9, no.
Nov, pp. Unsupervised 2.8M

R EFERENCES

[1] J. Sivic, A. Zisserman et al., “Video google: A text retrieval approach
to object matching in videos.”
[2] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features:
Spatial pyramid matching for recognizing natural scene categories,” in
Computer vision and pattern recognition, 2006 IEEE computer society
conference on, vol. 2. IEEE, 2006, pp. 2169–2178.
[3] M. Lienou, H. Maitre, and M. Datcu, “Semantic annotation of satellite
images using latent dirichlet allocation,” IEEE Geoscience and Remote
BIC [4]
Unsupervised
87.03±1.07
Sensing Letters, vol. 7, no. 1, pp. 28–32, 2010.
OverFeatL +OverFeatS [4]Supervised
289M
83.04±2.00
[4] O. A.byPenatti,
K. Nogueira,
and on
J. A.UC-Merced
dos Santos, “Do
deepthe
features
ang, Gui Song Xia,Coffee
and Gang CaffeNet
Liu, “A [5]
hierarchical Supervised
Figure 8: Exemplary
images
produced
a generator
trained
using
`f ianl
60M
94.45±1.20
generalize
from
everyday
objects
to
remote
sensing
and
aerial
scenes
e feature fusion for high-resolution
satellite
scene
MARTA GANs
Unsupervised
0.18Munprecedented
89.86±0.98 resolution (256 × 256) Remote Sensing Images.
sampled
with
domains?” in Proceedings of the IEEE Conference on Computer Vision
cture Notes in Computer Science (including subseries
and Pattern Recognition Workshops, 2015, pp. 44–51.
Artificial Intelligence and Lecture Notes in Bioinfor[5] K. Nogueira, O. A. Penatti, and J. A. dos Santos, “Towards better
LNCS, pp. 324–333,
2013.
B. Brazilian Coffee Scenes dataset
exploiting convolutional neural networks for remote sensing scene
vid Picard, and Philippe Henri Gosselin, “Evaluation
[13] Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng
classification,” Pattern Recognition, vol. 61, pp. 539–556, 2017.
ual features for land-use
classification,”
To evaluate
the Proceedings
generalization power
of our
model,
we
Chen, Craig
Citro,
Greg S Corrado,
Andy Davis, Jeffrey Dean, Matthieu
[6] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
kshop on Content-Based Multimedia Indexing, 2014.
Devin,
et
al.,
“Tensorflow:
Large-scale
machine learning on heterogealso
performed
experiments
using the Brazilian Coffee Scenes
with deep convolutional neural networks,” in Advances in neural inforang, X Huang, L
Zhang,
and H Sun,
“Unsupervised
neous distributed systems,” arXiv preprint arXiv:1603.04467, 2016.
mation processing systems, 2012, pp. 1097–1105.
datasetof[4],
which is a Patches
composition
ofYang
scenes
taken Newsam,
by the “Bag-of-visual-words
Via Spectral Clustering
Multidimensional
[14] Yi
and Shawn
and spatial extenP. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol,
ed Scene Classification,”
Selected in
Topics
Applied red, and
SPOT sensor
thein green,
bands. in[7]Proceedings
sionsnear-infrared
for land-use classification,”
of the 18th SIGSPA“Stacked denoising autoencoders: Learning useful representations in a
s and Remote Sensing, IEEE Journal of, vol. 8, no.
TIAL internationalscenes.
conference
in geographic information
This dataset has 2,876 multispectral high-resolution
It on advances
deep network with a local denoising criterion,” Journal of Machine
2015.
systems. ACM, 2010, pp. 270–279.
Learning Research, vol. 11, no. Dec, pp. 3371–3408, 2010.
includes
tiles of for
coffee
and [15]
1,438Laurens
tiles of
YingLi Tian, “Pyramid
of 1,438
spatial relatons
scenevannon-coffee
der Maaten with
and Geoffrey Hinton, “Visualizing data using
[8] A. Makhzani and B. Frey, “K-sparse autoencoders,” arXiv preprint
sification,” IEEE
Transactions
on resolution.
Geoscience and
a 64
× 64-pixel
Fig. 7a shows
t-sne,” some
Journalexamples
of MachineofLearning Research, vol. 9, no. Nov, pp.
arXiv:1312.5663, 2013.
ol. 53, no. 4, pp. 1947–1957, apr 2015.
2579–2605, 2008.
thisJunsong
dataset.
We did
not use
data augmentation
on this dataset
[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
dong Jiang, and
Yuan,
“Learning
LBP
[16] Wen Shao, Wen Yang, Gui Song Xia, and Gang Liu, “A hierarchical
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in
because
it contains
sufficient
train the
mizing the conditional
mutual
information,”
Patterndata to scheme
of network.
multiple feature fusion for high-resolution satellite scene
Advances in neural information processing systems, 2014, pp. 2672–
8, no. 10, pp. 3180–3190,
2015.
Table II shows the results obtained
with theLecture
proposed
categorization,”
Notes in Computer Science (including subseries
2680.
Lecture
Notes
in
Artificial
Intelligence
and Lecture Notes in Bioinformethod. In general, the results are significantly worse than [10] A.
Radford, L. Metz, and S. Chintala, “Unsupervised Representation
matics), vol. 7963 LNCS, pp. 324–333, 2013.
Learning
with Deep Convolutional Generative Adversarial Networks,”
those on the UC-Merced dataset, [17]
despite
reducing
the
classifiRomain Negrel, David Picard, and Philippe Henri Gosselin, “Evaluation
arXiv, pp. 1–15, 2015. [Online]. Available: http://arxiv.org/abs/1511.
cation from a 21-class to a 2-class problem.
Brazilian
of second-order
visualCoffee
features for land-use classification,” Proceedings
06434
- International
Workshop
on Content-Based Multimedia Indexing, 2014.
Scenes is a challenging dataset because
of the high
intra-class
[11] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard
[18] F Hu, G Xia, Z Wang, X Huang, L Zhang, and H Sun, “Unsupervised
artifacts,” Distill, 2016. [Online]. Available: http://distill.pub/2016/
variability caused by different crop management
Feature Learningtechniques,
Via Spectral Clustering of Multidimensional Patches
deconv-checkerboard
different plant ages and spectral distortions
shadows.
for Remotelyand
Sensed
Scene Classification,” Selected Topics in Applied
[12] Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions
Earth
Observations
and
Remote
Sensing,
IEEE Journal of, vol. 8, no.
Nevertheless, our results are better than that of BIC [4].
for land-use classification,” in Proceedings of the 18th SIGSPATIAL in5, pp. 2015–2030, 2015.
ternational conference on advances in geographic information systems.
[19] Shizhi Chen and YingLi Tian, “Pyramid of spatial relatons for sceneACM, 2010, pp. 270–279.
level land use classification,” IEEE Transactions on Geoscience and
IV. CONCLUSION
[13] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Remote Sensing, vol. 53, no. 4, pp. 1947–1957, apr 2015.
Corrado, A. Davis, J. Dean, M. Devin et al., “Tensorflow: Large-scale
This paper introduced a representation
learning algorithm
[20] Jianfeng Ren, Xudong Jiang, and Junsong Yuan, “Learning LBP
machine learning on heterogeneous distributed systems,” arXiv preprint
called MARTA GANs. In contrast structure
to previous
approaches
by maximizing
the conditional mutual information,” Pattern
arXiv:1603.04467, 2016.
Recognition,
vol. 48, no.unsu10, pp. 3180–3190, 2015.
that require supervision, MARTA GANs
is completely
[14] G. Sheng, W. Yang, T. Xu, and H. Sun, “High-resolution satellite scene
classification using a sparse coding based multiple feature combination,”
pervised; it can learn interpretable representations even from
International journal of remote sensing, vol. 33, no. 8, pp. 2395–2412,
challenging remote sensing datasets. In addition, MARTA
2012.
GANs introduces a new multiple-feature-matching layer that [15] F. Hu, G.-S. Xia, Z. Wang, X. Huang, L. Zhang, and H. Sun, “Unsupervised feature learning via spectral clustering of multidimensional patches
learns multi-scale spatial information for high-resolution refor remotely sensed scene classification,” IEEE Journal of Selected
mote sensing. Other possible future extensions to the work
Topics in Applied Earth Observations and Remote Sensing, vol. 8, no. 5,
described in this paper include: producing high-quality sam2015.

ples of remote sensing images using the generator and classifying remote sensing images in a semi-supervised manner to
improve classification accuracy.

(Eqn. 4)

