1

SCH-GAN: Semi-supervised Cross-modal Hashing
by Generative Adversarial Network

arXiv:1802.02488v1 [cs.CV] 7 Feb 2018

Jian Zhang, Yuxin Peng and Mingkuan Yuan

Abstract—Cross-modal hashing aims to map heterogeneous
multimedia data into a common Hamming space, which can
realize fast and flexible retrieval across different modalities.
Supervised cross-modal hashing methods have achieved considerable progress by incorporating semantic side information.
However, they mainly have two limitations: (1) Heavily rely on
large-scale labeled cross-modal training data which are labor
intensive and hard to obtain, since multiple modalities are
involved. (2) Ignore the rich information contained in the large
amount of unlabeled data across different modalities, especially
the margin examples from another modality that are easily to be
incorrectly retrieved, which can help to model the correlations
between different modalities. To address these problems, in this
paper we propose a novel Semi-supervised Cross-Modal Hashing
approach by Generative Adversarial Network (SCH-GAN). We
aim to take advantage of GAN’s ability for modeling data
distributions, so that SCH-GAN can model the distribution across
different modalities, and select informative margin examples from
large amount of unlabeled data to promote cross-modal hashing
learning in an adversarial way. The main contributions can
be summarized as follows: (1) We propose a novel generative
adversarial network for cross-modal hashing. In our proposed
SCH-GAN, the generative model tries to select margin examples
of one modality from unlabeled data when giving a query of
another modality (e.g. giving a text query to retrieve images and
vice versa). While the discriminative model tries to distinguish the
selected examples and true positive examples of the query. These
two models play a minimax game so that the generative model
can promote the hashing performance of discriminative model.
(2) We propose a reinforcement learning based algorithm to
drive the training of proposed SCH-GAN. The generative model
takes the correlation score predicted by discriminative model as
a reward, and tries to select the examples close to the margin to
promote discriminative model by maximizing the margin between
positive and negative data. Extensive experiments compared with
8 state-of-the-art methods on 3 widely-used datasets verify the
effectiveness of our proposed approach.
Index Terms—Cross-modal hashing, generative adversarial
network, semi-supervised.

I. I NTRODUCTION

W

ITH the fast development of Internet and multimedia
technologies, heterogeneous multimedia data including
image, video, text and audio, has been growing very fast
and enriching people’s life. To make better use of such rich
multimedia data, it is an important application to retrieve multimedia contents that users have interests in. Thus multimedia
retrieval has attracted much attention over the past decades.
This work was supported by National Natural Science Foundation of China
under Grants 61771025 and 61532005.
The authors are with the Institute of Computer Science and Technology,
Peking University, Beijing 100871, China. Corresponding author: Yuxin Peng
(e-mail: pengyuxin@pku.edu.cn).

Text
The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

The word Panthera is
probably of Oriental origin
and retraceable to the
Ancient Greek word panther,
the Latin word panthera, the
Old French word pantere,
most likely meaning ...

Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

Although dogs are the most
closely related canids to gray
wolves
(the
sequence
divergence between gray
wolves and dogs is only
1.8%, as opposed to over 4%
between gray wolves...

Hashing Tables
Bucket

Indexed cross-modal data
This reclassification
implications

has
for
conservation. If there are two
separate species, each will be
less abundant (particularly
the rarer) and could be more
endangered than .

0001...

Hashing
Functions

...

This reclassification
implications

0110...
Bucket
...

0011...

has
for
conservation. If there are two
separate species, each will be
less abundant (particularly
the rarer) and could be more
endangered than .

...

Indexed cross-modaldata
...
Although dogs are the most
closely related canids to gray
wolves
(the
sequence
divergence between gray
wolves and dogs is only
1.8%, as opposed to over 4%
between gray wolves...

...

Image

Fig. 1. An example of cross-modal hashing with image and text, which
projects data of different modalities into a common hamming space and
performs fast retrieval.

However, with the explosive increase of multimedia data on
the Internet, efficient retrieval from large scale databases has
become an urgent need and a big challenge. To tackle this
problem, many hashing methods [1]–[8] have been proposed
to perform efficient yet effective retrieval. Generally speaking,
hashing methods aim to transfer high dimensional feature into
short binary codes so that similar data can have similar binary
codes. Hashing methods have two major advantages when
applied in multimedia retrieval: (1) Binary codes enable fast
Hamming distance computation based on bit operations which
can be efficiently implemented. (2) Binary codes take much
less storage compared with original high dimensional feature.
A large amount of hashing methods are designed for single
modality retrieval [1]–[8], that is to say, data can only be
retrieved by an query of the same modality, such as image
retrieval [3] and video retrieval [9]. However, in real world
applications, multimedia data is usually presented with different modalities. For example, an image is usually associated
with a textual description, and both of them describe the same
semantic. In this case, an increasing need of users is to retrieve
across different modalities, such as using an image to retrieve
relevant textual descriptions. Such retrieval paradigm is called
cross-modal hashing. It is more useful and flexible than single
modality retrieval because users can retrieve whatever they
want by submitting whatever they have [10].
The key challenge of cross-modal hashing is the “heterogeneity gap”, which means the distribution and representation
of different modalities are inconsistent, and makes it hard to
directly measure the similarity between different modalities.
To bridge this gap, many cross-modal hashing methods [11]–
[24] have been proposed. Generally speaking, cross-modal

2

hashing methods can be divided into traditional methods and
deep learning based methods. Traditional methods can be
further divided into unsupervised methods and supervised
methods based on whether semantic information is involved.
The basic idea of unsupervised cross-modal hashing methods
is to project data from different modalities into a common
space where correlations between them are maximized, which
is similar to Canonical Correlation Analysis (CCA) [25].
Representative unsupervised cross-modal hashing methods including Cross-view hashing (CVH) [11], Inter-Media Hashing
(IMH) [12], Predictable Dual-View Hashing (PDH) [13] and
Collective Matrix Factorization Hashing (CMFH) [14]. While
supervised cross-modal hashing methods try to learn hash
functions to preserve the semantic correlations provided by
labels. Representative supervised cross-modal hashing methods include Co-Regularized Hashing (CRH) [15], Heterogeneous Translated Hashing (HTH) [16], Semantic Correlation
Maximization (SCM) [17], Quantized Correlation Hashing
(QCH) [18] and Semantics-Preserving Hashing (SePH) [19].
Recently, inspired by the successful applications of deep
learning on image classification [26] and object recognition [27], some researches try to bridge the “heterogeneity
gap” by deep learning technique. Representative deep learning
based methods include Deep and Bidirectional Representation
Learning Model (DBRLM) [20], Deep Visual-semantic Hashing (DVH) [23] and Cross-Media Neural Network Hashing
(CMNNH) [22].
Among the above cross-modal hashing methods, supervised
methods typically achieve better retrieval accuracy due to
the utilization of semantic information. However, supervised
methods are very labor intensive to obtain large scale labeled
training data. It is even harder to label cross-modal hashing
training data, since multiple modalities are involved. Thus
it is of significant importance to improve retrieval accuracy
by fully exploiting unlabeled data which is very convenient
to get. Traditional semi-supervised learning methods [28]
can effectively exploit distribution of unlabeled data to help
supervised training. However, little efforts have been done for
semi-supervised cross-modal hashing. The key challenge of
semi-supervised cross-modal hashing is to exploit informative
unlabeled data to promote hashing learning. With the recent
progress of generative adversarial network (GAN) [29]–[32],
which has been applied in many computer vision problems,
such as image synthesis [30], video prediction [31] and object
detection [32]. GAN has shown its promising ability to model
the data distributions. Inspired by this ability, in this paper
we propose a novel semi-supervised cross-modal hashing
approach by generative adversarial network (SCH-GAN). We
aim to design a generative model to fit the relevance distribution of unlabeled data near the margins between different
modalities, so that it can select informative unlabeled examples
close to the margin to fool the discriminative model. We also
design a discriminative model to distinguish the selected data
from generative model and the true positive data, so that it
can better estimate the cross-modal ranking. These two models
play a minimax game to iteratively optimize each other and
boost cross-modal hashing accuracy. The main contributions
of this paper can be summarized as follows:

A Generative adversarial network for cross-modal hashing (SCH-GAN) is proposed to fully exploit unlabeled
data to improve hashing performance. In our proposed
SCH-GAN, the generative model learns to fit the relevance distribution of unlabeled data, and tries to select
margin examples from unlabeled data of one modality
giving a query of another modality. While the discriminative model learns to judge the relevance between query
and selected examples by the guidance of labeled data.
Generative model tries to select the margin examples
that are easily to be retrieved incorrectly to fool the
discriminative model, while the latter tries to distinguish
those selected examples from true positive data. Thus
these two models act as two players in a minimax game,
and each of them improves itself to “beat” each other.
The finally learned hashing functions from discriminative
model can better reflect both semantic information of
labeled data and data distributions of unlabeled data.
• Reinforcement learning based loss function is proposed
to train the generative model and discriminative model.
The generative model takes the correlation score predicted by discriminative model as a reward, and tries
to select the examples close to the margin to promote
discriminative model to maximize the margin between
positive and negative data. The discriminative model
utilizes a cross-modal triplet ranking loss to learn the
ranking information provided by semantic labels, and it
also acts like a critic that tries to distinguish the selected
examples from generative model and true positive data.
Extensive experiments on the widely-used Wikipedia, NUSWIDE and MIRFlickr datasets demonstrate that our proposed
SCH-GAN outperforms 8 state-of-the-art methods, which verify the effectiveness of SCH-GAN approach.
The rest of this paper is organized as follows. We briefly
review the related works in Section II. In Section III, our
proposed SCH-GAN approach is presented in detail. Then
Section IV reports the experimental results as well as analyses.
Finally, Section V concludes this paper.
•

II. R ELATED W ORKS
In this section, we briefly review some related works from
two aspects: cross-modal hashing and generative adversarial
network.
A. Cross-modal Hashing
Hashing methods for single modality retrieval have been
extensively studied in the past decades [1]–[8], and crossmodal hashing methods are receiving increasing attention in
recent years. Generally speaking, most cross-modal hashing
methods project data of different modalities into a common
Hamming space to perform fast retrieval, and it can be divided
into traditional methods and deep learning based methods.
Traditional methods can be further divided into unsupervised
and supervised methods based whether semantic information
is involved. We will briefly review some representative works
of cross-modal hashing methods.

3

Unsupervised cross-modal hashing methods have the similar idea with Canonical Correlation Analysis (CCA) [25],
which maps heterogeneous data into a common Hamming
space to maximize the correlation. Inter-Media Hashing
(IMH) [12] is proposed to learn a common Hamming space to
preserve both inter-media and intra-media consistency. Kumar
et al. propose Cross-view Hashing (CVH) [11], which extends
image hashing method Spectral Hashing (SH) [7] to consider
both intra-view and inter-view similarities with a generalized
eigenvalue formulation. Rastegari et al. propose Predictable
Dual-View Hashing (PDH) [13] for two modalities, which
proposes a objective function to preserve the predictability of
pre-generated binary codes, and optimize the objective function by an iterative method based on block coordinate descent.
Ding et al. propose Collective Matrix Factorization Hashing
(CMFH) [14], which learns unified hash codes from different
modalities of one instance by collective matrix factorization
with a latent factor model. Latent Semantic Sparse Hashing
(LSSH) [33] is proposed to use sparse coding and matrix
factorization to learn separate semantic features for images
and text, and then map them into a joint abstract space to
reduce semantic difference. Wang et al. propose Semantic
Topic Multimodal Hashing (STMH) [34], which models texts
as semantic topics while images as latent semantic concepts,
and maps the learned semantic features for different modalities
into a common semantic space, finally generates hash codes
by predicting whether topics or concepts are available in the
original data. Unsupervised methods try to learn cross-modal
hashing functions from data distributions, thus they achieve
limited accuracy on retrieving semantically similar data. To
improve retrieval accuracy, some supervised methods are then
proposed.
Supervised cross-modal hashing methods leverage semantic information obtained from labels of training data, which
achieve better retrieval accuracy than unsupervised methods.
Cross-modality Similarity Sensitive Hashing (CMSSH) [35]
is proposed to model hashing learning as a classification
problem, and can be learned in a boosting manner. Zhen
et al. propose Co-Regularized Hashing (CRH) [15], which
learns hash functions of each bit sequentially so that the
bias introduced by each hash functions can be minimized.
Hu et al. propose Iterative Multi-view Hashing (IMVH) [36],
which tries to learn hashing functions by preserving both
within-view similarity and between-view correlations. Heterogeneous Translated Hashing (HTH) [16] is proposed to
learn different Hamming spaces for different modalities, and
then learn translators to align these spaces to perform crossmodal retrieval. Zhang et al. propose Semantic Correlation
Maximization (SCM) [17], which constructs semantic similarity matrix based on labels and learns hashing functions to
preserve the constructed matrix. Wu et al. propose Quantized
Correlation Hashing (QCH) [18] to simultaneously optimize
cross-modal correlation and quantization error, which is also
considered in many single modality hashing methods. Lin
et al. propose Semantics-Preserving Hashing (SePH) [19],
which is a two-step supervised hashing method, it firstly
transforms the given semantic matrix of training data into a
probability distribution and approximates it with learned hash

codes in Hamming space via minimizing the KL-divergence.
Supervised cross-modal hashing methods often achieve better
results than unsupervised methods because of utilization of
semantic information.
Deep learning based methods are inspired by recent advance of deep neural networks, which have been applied in
many computer vision problems, such as image classification [26] and object recognition [27]. Zhuang et al. propose
Cross-Media Neural Network Hashing (CMNNH) [22], which
learns hashing functions by preserving intra-modal discriminative capability and inter-modal pairwise correspondence. Wang
et al. propose Deep Multimodal Hashing with Orthogonal Regularization (DMHOR) [21], [37], which learns hashing functions by preserving intra-modal and inter-modal correlation,
as well as reducing redundant information between hash bits.
Cao et al. propose Cross Autoencoder Hashing (CAH) [24],
which maximizes the feature correlation of bimodal data and
the semantic correlation provided by similarity labels, and
CAH is based on deep autoencoder structure. Deep Visualsemantic Hashing (DVH) [23] is proposed, which is an endto-end framework that integrates both feature learning and
hashing function learning.
Supervised methods, especially supervised deep learning
based methods have achieved promising results. However,
supervised methods rely on large amount of labeled training
data which are labor intensive to obtain. It is even harder
to label cross-modal hashing training data, since multiple
modalities are involved. Traditional semi-supervised learning
methods [28] can exploit unlabeled data effectively to help
supervised training. Little efforts have been done for semisupervised cross-modal hashing learning. In this paper, we
attempt to fully exploit the unlabeled data to promote crossmodal hashing learning.
B. Generative Adversarial Network
Generative Adversarial Network (GAN) [29] is first proposed to estimate generative model by an adversarial process.
GAN consists of two models: a generative model G that
captures the data distributions, and a discriminative model
D that estimates the probability that a sample comes from
real data rather than G. These two models are trained in a
adversarial way so that they compete with each other, and both
of them can learn better representations of the data. Inspired
by the ability of modeling data distributions of GAN, many
works have attempted to apply GAN in many computer vision
problems. Most popular one is image synthesis. Radford et
al. propose Deep Convolutional GAN (DCGAN) [38], which
adopts convolutional decoder with batch normalization and
achieves better image synthesis results. Mirza et al. proposed
Conditional GAN (CGAN) [39], which provides side information for both generative and discriminative model to control
the generated data. Inspired by CGAN, many works extend
its idea to image synthesis problem, Reed et al. propose textconditional GAN [40] which can generate images conditioned
by textual descriptions. Odena et al. propose auxiliary classifier GAN (AC-GAN) [41] that generates images conditioned
by class labels. Besides image synthesis, GAN is also applied
to video prediction [31] and object detection [32].

4

Generative Model G
Image Representation Learning

Images

Hashing Layers

Generated pairs
Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

HI

Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

Texts
The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

The word Panthera is
probably of Oriental origin
and retraceable to the
Ancient Greek word panther,
the Latin word panthera, the
Old French word pantere,
most likely meaning ...

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

The word Panthera is
probably of Oriental origin
and retraceable to the
Ancient Greek word panther,
the Latin word panthera, the
Old French word pantere,
most likely meaning ...

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

HT

Text Feature Extraction

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

Hashing Layers
Minimax Game

Generated pairs

Image Representation Learning

True pairs

Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

Hashing Layers

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

HI
Early technological progress
owed much to the firm of
Broadwood. John Broadwood
joined with another Scot,
Robert
Stodart,
and
a
Dutchman, Americus Backers
...

Ranking
Loss

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

The oldest remains of a tigerlike cat, called Panthera
palaeosinensis, have been
found in China and Java. This
species lived about 2 million
years ago....

HT

·

Text Feature Extraction

Relevance
Score

Hashing Layers

Discriminative Model D
Labeled data

Class1

Class2

Unlabeled data
Class1

Class2

Fig. 2. The overall framework of our proposed semi-supervised cross-modal hashing approach by generative adversarial network (SCH-GAN), which consists
a generative model and a discriminative model. The generative model attempts to select informative unlabeled data to form a pair with a given labeled query
to fool discriminative model, while the discriminative model tries to distinguish if a pair is generated or a true pair. Those two models act as two players to
play a minimax game to optimize each other, and promote cross-modal hashing performance.

Inspired by the ability of GAN to model data distributions,
in this paper we propose a novel semi-supervised cross-modal
hashing by generative adversarial network (SCH-GAN). It
aims design a generative model to learn the distributions of
different modalities, and a discriminative model to maintain
the semantic ranking information, these two models play a
minimax game to iteratively optimize each other and boost
cross-modal hashing accuracy.
III. T HE P ROPOSED A PPROACH
The overall framework of our proposed approach is demonstrated in Figure 2, which consists of a generative model and a
discriminative model. The generative model receives the input
of both labeled and unlabeled data of different modalities, and
given a query of labeled data, it attempts to select informative
unlabeled data around the error margins of another modality
to form a pair of data. While the discriminative model receives
both the generated pairs and true pairs as input, and tries to
distinguish them so that it can better discriminate those margin
examples. These two models are trained by playing a minimax
game, and the finally trained discriminative model can be used
to perform cross-modal retrieval.

A. Notation
Firstly, the formal definition of cross-modal hashing and
some notations used in this paper are introduced. The two
modalities involved in this paper are denoted as I for image
and T for text. The multimodal dataset is denoted as D =
{I, T }, I ∈ RI , T ∈ RT , which is further split into a retrieval
database Ddb and a query set Dq . The retrieval database Ddb
is also the training set, which consists of two parts, namely
L
U
U
U
L
labeled data Ddb
and unlabeled data Ddb
= {Idb
, Tdb
}. Ddb
L
L
L
L
L n
is defined as Ddb = {Idb , Tdb }, where Idb = {ip }p=1 and
L
L
n
Tdb
= {tL
p }p=1 , n is the number of labeled data. Ddb is also
associated with corresponding class labels, which are denoted
U
U
U
U
as {cIp }ni and {cTp }ni . Ddb
is defined as Ddb
= {Idb
, Tdb
},
U
U m
U
U m
where Idb = {ip }p=1 and Tdb = {tp }p=1 , m is the number
of unlabeled data and m >> n. The query set Dq is defined
as Dq = {Iq , Tq }, where Iq = {ip }tp=1 and Tq = {tp }tp=1 .
The goal of cross-modal hashing is to learn two mapping
functions HI : RI → RH and HT : RT → RH , so that
semantically similar data of different modalities are close in
the common Hamming space. Given a query of any modality,
by the learned hashing functions, the semantically similar data

5

of another modality can be retrieved by the fast Hamming
distance measurement.
L
It is noted that supervised methods use labeled data Ddb
to
train hashing functions, however labeled data are labor intensive and hard to obtain, while unlabeled data are convenient to
acquire. Thus, in this paper we attempt to further exploit the
U
large amount unlabeled data Ddb
to promote hashing learning.
B. Network structure
As shown in Figure 2, the proposed SCH-GAN consists
of a generative model and a discriminative model. We will
introduce the detailed network structures of them in this
subsection.
The generative model is a two-pathway architecture, which
receives both image and text as inputs. Each pathway consists
of two parts: feature representation layers and hashing layers.
For image pathway, we adopt widely-used convolutional neural networks (CNN) to represent each image, and it is noted
that we keep the parameters of the CNN fixed during the
training phase, since our focus is hashing function learning.
While for text pathway, we use bag-of-words (BoW) features
to represent textual descriptions. The structure of hash layers
is the same in the two pathways, and it consists of two fullyconnected layers. The first fully-connected layer serves as an
intermediate layer that maps modality specific feature into a
common space. The second fully-connected layer serves as
hashing functions, which further map the intermediate feature
into hash codes:
h(x) = sigmoid(W T f (x) + v)

(1)

where f (x) is the intermediate feature extracted from last
layer, W denotes the weights in the hash code learning layer,
and v is the bias parameter.The dimension of last fullyconnected layer is set to be the same as the hash code length
q. Through the hash code learning layer, intermediate features
f (x) are mapped into [0, 1]q . Since hash codes h(x) ∈ [0, 1]q
are continuous real values, we apply a thresholding function
g(x) = sgn(x − 0.5) to obtain binary codes:
bk (x) = g(h(x)) = sgn(hk (x) − 0.5),

k = 1, 2, · · · , q (2)

However, binary codes are hard to directly optimize, thus we
relax binary codes b(x) with continuous real valued hash codes
h(x) in the rest of this paper. Through the hashing layers, the
features of different modalities are mapped into the Hamming
space with same dimensions so that the similarity between different modalities can be measured by fast Hamming distance
calculation. The input of generative model consists of both
labeled and unlabeled data, and the goal of generative model
is to select informative unlabeled data of another modality that
lies around margins when given a query of one modality. This
goal is achieved by the adversarial training algorithm, which
will be introduced in the following subsections.
The discriminative model is also a two-pathway structure,
whose detailed settings are exactly the same as the generative
model. The input of discriminative model is the generated (selected) relevant pairs by generative model, and the true relevant
pairs sampled from labeled data. The goal of discriminative

model is to distinguish whether an input pair is generated or
a true pair.
C. Objective function
Firstly, we give the formal definitions of the objectives of
proposed generative model and discriminative model.
U
U
• Generative model: pθ (i |qt , r) and pθ (t |qi , r), which
try to select relevant data of one modality from unlabeled
data when given a query of another modality. For example, given a text query qt , the generative model tries to
U
select relevant image iU from Idb
. The goal of generative
model is to approximate the true relevance distribution
across different modalities ptrue (iU |qt , r).
• Discriminative model: fφ (i, qt ) and fφ (t, qi ), which try
to predict the relevance score of the query and candidate
data pair. The inputs of discriminative model consist
of true pairs sampled by semantic labels, as well as
generated pairs from generative model. The goal of
discriminative model is to distinguish the relevant data
(true pairs) and non-relevant data (generated pairs) for a
query qi as accurate as possible.
Given above definitions, the generative model and discriminative model act as two players that play a minimax game:
Given a query, the generative model tries to select margin data
that is likely to be retrieved incorrectly to fool the discriminative model, while the discriminative model tries to distinguish
between true relevant data sampled from ground-truth and the
selected ones generated by its adversarial generative model.
Inspired by the GAN [29]–[32], the adversarial process is
defined as follows:
n
X
V(G, D) = min max
(Ei∼ptrue (iL |qj ,r) [log(D(iL |qtj ))]
θ

φ

t

j=1

+ Ei∼pθ (iU |qj ,r) [log(1 − D(iU |qtj ))])
t

(3)
Above equation is for text query image task, and the image
query text task is similarly defined as:
V(G, D) = min max
θ

φ

n
X

(Et∼ptrue (tL |qj ,r) [log(D(tL |qij ))]
i

j=1

+ Et∼pθ (tU |qj ,r) [log(1 − D(tU |qij ))])
i

(4)
These two equations are symmetric, thus in the following parts
we use the text query image task as an example to present the
detailed objective function. In above equations, the generative
model G is denoted as pθ (iU |qt , r), which is defined as a
softmax function:
exp(−khT (qt ) − hI (iU )k2 )
(5)
pθ (iU |qt , r) = P
U 2
iU exp(−khT (qt ) − hI (i )k )
where hi (∗) and ht (∗) denote the hashing functions of image
pathway and text pathway respectively. Intuitively, equation 5
calculates the probability that we select an image according to
a given text query. It is based on the distance between image
and text, smaller distance leads to larger probability. While for
the true data distributions ptrue (iL |qtj , r), we sample relevant

6

images iL of query qt based on labels, such that the generative
model can preserve the semantic distribution of labeled data.
The discriminative model D predicts the probability of
selected image iU being relevant to text query qt , and D is
defined as the sigmoid function of relevance score:
exp(fφ (iU , qt ))
1 + exp(fφ (iU , qt ))
exp(fφ (iL , qt ))
D(iL |qt ) = sigmoid(fφ (iL , qt )) =
1 + exp(fφ (iL , qt ))

D(iU |qt ) = sigmoid(fφ (iU , qt )) =

(6)
U

L

The relevance score of fφ (i , qt ) and fφ (i , qt ) are defined
by triplet ranking loss as follows:
fφ (iU , qt ) = max(0, mi + khT (qt ) − hI (i+ )k2
− khT (qt ) − hI (iU )k2 )
fφ (iL , qt ) = max(0, mi + khT (qt ) − hI (iL )k2
− khT (qt ) − hI (i− )k2 )

θ∗ = arg min

n
X

θ

(Ei∼ptrue (iU |qj ,r) [log(sigmoid(fφ∗ (iL , qtj )))]
t

j=1

+ Ei∼pθ (iU |qj ,r) [log(1 − sigmoid(fφ∗ (iU , qtj )))])

(7)

t

= arg min

(8)

where i+ is a semantically similar image with text query qt
according to labels, i− is a semantically dissimilar image
sampled from labeled data, iU is the selected image by
generative model, and mi is a margin parameter which is set
to be 1 in our proposed approach. Above formulation means
that we want the distance between true relevant pair (qt , i+ )
smaller than that of generated pair (qt , iU ) by a margin mi , so
that the discriminative model can draw a clear distinguish line
between the true and generated pairs. Similarly, we also want
to keep the ranking based relations between labeled data.
From above objective definitions, we can observe that the
generative model and discriminative model can be learned
by iteratively maximizing and minimizing the same object
function. The discriminative model tries to draw a margin
between generated (selected) data and positive data, while the
generative mode tries to select data near the margin to fool
the discriminative model.
D. Optimization
By the objective function defined in equation 3, the overall
training flow of proposed approach is shown in Figure 3. We
keep the parameters of generative model fixed while training
the discriminative model and vise versa. We’ll introduce the
optimization algorithm of these two models separately.
1) Optimizing discriminative model: As shown in Figure 3,
when updating the parameters of discriminative model, we
keep the generative model fixed. Firstly we use the generative
model of previous iteration to select text-image and imagetext pairs when given text and image as queries respectively,
and we further sample true text-image and image-text pairs
from labeled data. Then the discriminative model tries to
maximize the log-likelihood of correctly distinguishing the true
and generated relevant pairs. When fixing generative model,
based on equation 6, equation 3 can be rewritten as:
n
X
∗
φ = arg max
(Ei∼ptrue (iU |qj ,r) [log(sigmoid(fφ (iL , qtj )))]
φ

where pθ∗ is the generative model obtained in previous
iteration. According to equations 7 and 8, equation 9 is
differentiable with respect to φ, thus it can be solved by
stochastic gradient descent algorithm.
2) Optimizing generative model: As demonstrated in Figure 3, the discriminative model is fixed when training the
generative model. On the contrary, the generative model tries
to minimize equation 3 and fits the true relevance distribution.
When the discriminative model is fixed, the generative model
can be trained by minimizing equation 3:

t

j=1

+ Ei∼pθ∗ (iU |qj ,r) [log(1 − sigmoid(fφ (iU , qtj )))])
t

(9)

n
X

θ

Ei∼pθ (iU |qj ,r) [log(1 −
t

= arg max

j=1
n
X

θ

exp(fφ (iU , qt ))
)]
1 + exp(fφ (iU , qt ))

Ei∼pθ (iU |qj ,r) [log(1 + exp(fφ (iU , qt )))]
t

j=1

(10)
where fφ∗ is the generative model trained in previous iteration.
Different from the generative model of traditional GAN, which
generates new data from continuous noise vector and can
be optimized via stochastic gradient descent. The generative
model of the proposed SCH-GAN selects data from unlabeled
data. Since the selective strategy is discrete, it can not be
optimized by stochastic gradient descent. We use policy gradient based reinforcement learning to update the parameters
of generative model, which is derived as follows:
∇θ Ei∼pθ (iU |qj ,r) [log(1 + exp(fφ (iU , qtj )))]
t

=
=

m
X
k=1
m
X

j
j
U
∇θ pθ (iU
k |qt , r)log(1 + exp(fφ (ik , qt )))

j
j
U j
U
pθ (iU
k |qt , r)∇θ logpθ (ik |qt , r)log(1 + exp(fφ (ik , qt )))

k=1

= Ei∼pθ (iU |qj ,r) [∇θ logpθ (iU |qtj , r)log(1 + exp(fφ (iU , qtj )))]
t

m

1 X
j
j
U
∇θ logpθ (iU
'
k |qt , r)log(1 + exp(fφ (ik , qt )))
m
k=1

(11)
where k denotes the k-th image selected by current generative
model when given a text query qtj . From the perspective
of deep reinforcement learning, iU
k is the action taken by
j
policy logpθ (iU
|q
,
r)
according
to
the environment qtk , and
k t
j
U
log(1 + exp(fφ (ik , qt ))) acts as the reward for corresponding action, which will encourage the generative model to
select data close to margins. As illustrated in Figure 3, the
rewards are calculated by the fixed discriminative network.
We summarize the training process of proposed SCH-GAN
in algorithm 1. It is noted that to simplify the descriptions,
we take text query image task as an example in the training
algorithm pseudo code.

7

IV. E XPERIMENTS

Orange color means parameters are fixed

Blue color means parameters are updating
Training flow of discriminative model

Text Query
Image Query

Unlabled data

Generated Text-Image Pair

G

Labled data

Generated Image-Text Pair

True Text-Image Pair

D

Loss

True Image-Text Pair

Generated Text-Image Pair
Generated Image-Text Pair

Text Query
Image Query

G

In this section, we demonstrate the experimental results
of our proposed SCH-GAN approach. We first introduce the
datasets, evaluation metrics, implementation details and comparison methods used in our experiments. Then we compare
the proposed SCH-GAN with 8 state-of-the-art methods and
analyze the results. Finally, we further conduct several baseline
experiments to investigate the performance of generative and
discriminative model.

D

Unlabled data
Reward
Training flow of generative model

Fig. 3. The training flow of the generative model and discriminative model.
Best viewed in color.

Algorithm 1 Training algorithm of proposed SCH-GAN
Input: The generative model pθ (i|qt , r), the discriminative
L
U
model fφ (i, qt ), training data Ddb
and Ddb
1: Randomly initialize the parameters of pθ (i|qt , r) and
fφ (i, qt )
2: repeat
3:
for d-step do
4:
Generate m text-image pairs by pθ∗ (iU |qtj , r) given
text query qtj
L
5:
Sampled m true text-image pairs from Ddb
based on
labels
6:
Train discriminative model fφ (i, qt ) by equation 9
7:
end for
8:
for g-step do
9:
Generate m text-image pairs by pθ (iU |qtj , r) given
text query qt
j
10:
Calculate reward by log(1 + exp(fφ∗ (iU
k , qt )))
11:
Update parameters of generative model pθ (iU |qtj , r)
by equation 11
12:
end for
13: until SCH-GAN converges
Output: Optimized generative model pθ∗ (i|qt , r) and discriminative model fφ∗ (i, qt )

E. Cross-modal retrieval by learned discriminative model
It is noted that the design idea of our SCH-GAN is that
the generative model tries to fit the distribution near the
decision boundary, thus it is not suitable to perform crossmodal retrieval. However, the discriminative model is promoted greatly by the generative model since it can better
distinguish the margin examples. Thus after the proposed
SCH-GAN is trained, cross-modal retrieval can be performed
by the discriminative model. More specifically, given a query
of any modality (e.g. text or image), it can be first encoded into
binary hash code by equation 2. Then cross-modal retrieval
is performed by fast Hamming distance computation between
query and each data in the database.

A. Dataset
We evaluate the proposed approach and compared methods
on 3 widely-used datasets: Wikepedia [42], NUSWIDE [43]
and MIRFLICKR [44]. We’ll briefly introduce three datasets.
• Wikipedia dataset [42] is a widely-used dataset for
cross-modal retrieval, which is collected from “featured
articles” in Wikipedia with 10 most populated categories. This dataset consists of 2866 image/text pairs, of
which the images are represented by 4096 deep features
extracted from 19-layer VGGNet [45], and texts are
represented by 1000 dimensional BoW (Bag of Words)
features. Following [19], Wikipedia dataset is split into
a training set of 2173 pairs and a test set of 693 pairs.
Since Wikipedia daaset is small, the training set is also
used as the retrieval database, while the test set works as
query.
• NUSWIDE dataset [43] contains 269498 images associated with 81 concepts as ground truth, each image also
has corresponding tags. Following [19], we select the 10
most common concepts and the corresponding 186557
images. We take 1% data of NUSWIDE dataset as the
query set, and the rest as the retrieval database. As for
the supervised methods, we further randomly sampled
5000 images as training set, which is similar to real
world applications where only a fraction of the database
are labeled. We also represent each image by 4096 deep
features extracted from 19-layer VGGNet, and each texts
by 1000 dimensional BoW.
• MIRFlickr dataset [44] contains 25000 images collected
from Flickr, each image is also associated with textual
tags and labeled with one or more of 24 provided semantic labels. Following [19], we take 5% of the dataset as
the query set and the remaining as the retrieval database.
Similar wih NUSWIDE dataset, we also randomly sample
5000 images to form the supervised training set. Similarly, we represent each image by 4096 deep features
extracted from 19-layer VGGNet, and each texts by 1000
dimensional BoW.
B. Retrieval tasks and evaluation metrics
We perform cross-modal retrieval on the 3 datasets with two
kinds of retrieval tasks:
• Image query text: Using image as query to retrieve
semantically similar texts from retrieval database, we
denote it as image→text.

8

TABLE I
T HE MAP SCORES OF TWO RETRIEVAL TASKS ON W IKIPEDIA DATASET WITH DIFFERENT LENGTH OF HASH CODES .
Methods
CVH [11]
PDH [13]
CMFH [14]
CCQ [46]
CMSSH [35]
SCM orth [17]
SCM seq [17]
SePH [19]
DCMH [50]
SCH-GAN (Ours)

16
0.193
0.483
0.439
0.463
0.160
0.229
0.396
0.515
0.475
0.525

image→text
32
64
0.161
0.144
0.483
0.494
0.496
0.473
0.471
0.470
0.159
0.157
0.192
0.171
0.459
0.462
0.518
0.533
0.508
0.507
0.530
0.551

128
0.134
0.497
0.461
0.456
0.156
0.161
0.442
0.538
0.503
0.546

16
0.297
0.842
0.484
0.744
0.206
0.238
0.442
0.748
0.819
0.860

text→image
32
64
0.225
0.187
0.842
0.838
0.548
0.573
0.788
0.785
0.208
0.206
0.171
0.145
0.557
0.538
0.781
0.792
0.828
0.788
0.876
0.889

128
0.167
0.851
0.568
0.741
0.205
0.131
0.510
0.805
0.720
0.888

TABLE II
T HE MAP SCORES OF TWO RETRIEVAL TASKS ON NUSWIDE DATASET WITH DIFFERENT LENGTH OF HASH CODES .
Methods
CVH [11]
PDH [13]
CMFH [14]
CCQ [46]
CMSSH [35]
SCM orth [17]
SCM seq [17]
SePH [19]
DCMH [50]
SCH-GAN (Ours)

16
0.458
0.475
0.517
0.504
0.512
0.389
0.517
0.701
0.631
0.713

image→text
32
64
0.432
0.410
0.484
0.480
0.550
0.547
0.505
0.506
0.470
0.479
0.376
0.368
0.514
0.518
0.712
0.719
0.653
0.653
0.726
0.734

128
0.392
0.490
0.520
0.505
0.466
0.360
0.518
0.726
0.671
0.748

16
0.474
0.489
0.439
0.499
0.519
0.388
0.518
0.642
0.702
0.741

text→image
32
64
0.445
0.419
0.512
0.507
0.416
0.377
0.496
0.492
0.498
0.456
0.372
0.360
0.510
0.517
0.653
0.657
0.695
0.694
0.743
0.771

128
0.398
0.517
0.349
0.488
0.488
0.353
0.518
0.662
0.693
0.779

TABLE III
T HE MAP SCORES OF TWO RETRIEVAL TASKS ON MIRF LICKR DATASET WITH DIFFERENT LENGTH OF HASH CODES .
Methods
CVH [11]
PDH [13]
CMFH [14]
CCQ [46]
CMSSH [35]
SCM orth [17]
SCM seq [17]
SePH [19]
DCMH [50]
SCH-GAN (Ours)

16
0.602
0.623
0.659
0.637
0.611
0.585
0.636
0.704
0.721
0.739

image→text
32
64
0.587
0.578
0.624
0.621
0.660
0.663
0.639
0.639
0.602
0.599
0.576
0.570
0.640
0.641
0.711
0.716
0.729
0.735
0.747
0.755

Text query image: Using text as query to retrieve semantically similar images from retrieval database, we denote
it as text→image.
We utilize Hamming ranking to evaluate the proposed
SCH-GAN approach and compared state-of-the-art methods.
Hamming ranking gives the ranking list of a given query based
on the Hamming distance, where ideal semantic neighbors are
expected to be returned on the top of the ranking list. The
retrieval results are evaluated based on whether the returned
data and the query share the same semantic labels. We use
three evaluation metrics to measure the retrieval effectiveness:
Mean Average Precision (MAP), precision recall curve (PRcurve) and precision at top k returned results (topK-precision),
which are defined as follows:
• Mean Average Precision (MAP): MAP is the mean value
of average precisions (AP) of all queries, and AP is
defined as:
n
1 X k
× relk
(12)
AP =
R
Rk
•

k=1

128
0.572
0.626
0.653
0.638
0.591
0.566
0.643
0.711
0.731
0.769

16
0.607
0.627
0.611
0.628
0.612
0.585
0.661
0.699
0.764
0.775

text→image
32
64
0.591
0.581
0.628
0.628
0.606
0.575
0.628
0.622
0.604
0.592
0.584
0.574
0.664
0.668
0.705
0.711
0.771
0.774
0.790
0.798

128
0.574
0.629
0.563
0.618
0.585
0.568
0.670
0.710
0.760
0.799

where n is the size of database, R is the number of
relevant images in database, Rk is the number of relevant
images in the top k returns, and relk = 1 if the image
ranked at k-th position is relevant and 0 otherwise.
• Precision recall curve (PR-curve): The precision at certain level of recall of the retrieved ranking list, which
is frequently used to measure the information retrieval
performance.
• Precision at top k returned results (topK-precision): The
precision with respect to different numbers of retrieved
samples.
Those three metrics can evaluate the proposed approach and
compared methods objectively and comprehensively.
C. Comparison methods
We compare with 8 state-of-the-art methods to verify the effectiveness of our proposed approach, including unsupervised
methods CVH [11], PDH [13], CMFH [14] and CCQ [46],

9

(a) The top−k results of image−>text on the Wikipedia dataset

(b) The top−k results of image−>text on the NUSWIDE dataset

(c) The top−k results of image−>text on the MIRFlickr dataset

0.9

0.6

0.9

0.55

0.85
0.8

0.5

0.8
0.45

0.75

0.35

Precision

Precision

Precision

SCH−GAN(Ours)
DCMH
SePH
SCM(seq)
SCM(orth)
CMSSH
CCQ
CMFH
PDH
CVH

0.7

0.4
0.6

0.3

0.7
0.65

0.5

0.25

0.6
0.2

0.4
0.55

0.15
0.1

0

100

200

300

400

500
Top K

600

700

800

900

0.3

1000

0

(a) The top−k results of text−>image on the Wikipedia dataset

100

200

300

400

500
Top K

600

700

800

900

0.5

1000

0

400

600

800

1000

Top K

(b) The top−k results of text−>image on the NUSWIDE dataset

1

200

(c) The top−k results of text−>image on the MIRFlickr dataset

0.9

1
0.95

0.9
0.8

0.9

0.8
0.7

SCH−GAN(Ours)
DCMH
SePH
SCM(seq)
SCM(orth)
CMSSH
CCQ
CMFH
PDH
CVH

0.85

0.7

0.6
0.5

Precision

Precision

Precision

0.8
0.6

0.75
0.7

0.4

0.5

0.65

0.3

0.6

0.4
0.2
0.1

0.55

0

100

200

300

400

500
Top K

600

700

800

900

1000

0.3

0

100

200

300

400

500
Top K

600

700

800

900

1000

0.5

0

200

400

600

800

1000

Top K

Fig. 4. The topK-precision curves on three datasets with 64bit hash codes. The first row demonstrates the result of image query text task, while the second
row shows the result of text query image task. Left, middle and right columns demonstrate Wikipedia, NUSWIDE and MIRFlickr datasets respectively.

supervised methods CMSSH [35], SCM [17] and SePH [19],
and deep learning based methods DCMH [50]. We’ll briefly
introduce those compared methods.
• CVH [11] extends Spectral Hashing (SH) [7] to considers
both intra-view and inter-view similarities with a generalized eigenvalue formulation.
• PDH [13] tries to preserve the predictability of pregenerated binary codes, and optimize the objective function by an iterative method based on block coordinate
descent algorithm.
• CMFH [14] learns unified hash codes from different
modalities of one instance by collective matrix factorization with a latent factor model.
• CCQ [46] jointly learns the correlation-maximal mappings that transform different modalities into isomorphic
latent space, and learns composite quantizers that convert
the isomorphic latent features into compact binary codes.
• CMSSH [35] models hashing learning as a classification
problem, and it is learned in a boosting manner.
• SCM [17] constructs semantic similarity matrix based
on labels and learns hashing functions to preserve the
constructed matrix.
• SePH [19] is a two-step supervised hashing methods, it
firstly transforms the given semantic matrix of training
data into a probability distribution and approximates it
with learned hash codes in Hamming space via minimizing the KL-divergence.
• DCMH [50] is an end-to-end deep learning based method,
which performs feature learning and hashing function
learning simultaneously.
It is noted that for a fair comparison between traditional
methods and deep learning based methods, we uniformly use

the same deep features as the input of traditional methods.
Specifically, we use the 4096 deep features extracted from 19layer VGGNet pre-trained on ImageNet for images, and 1000
dimensional BoW for texts. While for deep learning based
methods, we use the same 19-layer VGGNet as their base
network for image pathway, while keep the same settings for
the text pathway.

D. Implementation details
We implement the proposed SCH-GAN in Figure 2 by
tensorflow1 , which is a widely-used open source software
library for numerical computation using data flow graphs. The
implementation details are described as follows:
1) Data processing: For the image pathway of our proposed SCH-GAN, we use the same 19-layer VGGNet as the
base network for image representation learning, and for the
text pathway, we use the same 1000 dimensional BoW for
text representation. It is noted that we keep the parameters of
VGGNet fixed since we focus on the adversarial training of
cross-modal hashing learning.
2) Details of network: We introduce the hashing layers in
Figure 2 in detail. The hashing layers consist of an intermediate layer and a hashing layer. The intermediate layer is a
fully-connected layer, whose dimension is set to be 4096 in
the experiments. While the hashing layer is a fully-connect
layer whose dimension is set the same as the hash code length,
we also use sigmoid activation for hashing layer to force the
output in the range of [0, 1].
1 https://www.tensorflow.org

10

(a) The PR curves of image−>text on the Wikipedia dataset

(b) The PR curves of image−>text on the NUSWIDE dataset

(c) The PR curves of image−>text on the MIRFlickr dataset

0.9

0.6

0.9

0.55

0.85
0.8

0.5

0.8
0.45

0.75

0.35

Precision

Precision

Precision

SCH−GAN(Ours)
DCMH
SePH
SCM(seq)
SCM(orth)
CMSSH
CCQ
CMFH
PDH
CVH

0.7

0.4
0.6

0.3

0.7
0.65

0.5

0.25

0.6
0.2

0.4
0.55

0.15
0.1

0

0.1

0.2

0.3

0.4

0.5
Recall

0.6

0.7

0.8

0.9

0.3

1

0

(a) The PR curves of text−>image on the Wikipedia dataset

0.1

0.2

0.3

0.4

0.5
Recall

0.6

0.7

0.8

0.9

0.5

1

0

0.4

0.6

0.8

1

Recall

(b) The PR curves of text−>image on the NUSWIDE dataset

1

0.2

(c) The PR curves of text−>image on the MIRFlickr dataset

0.9

1
0.95

0.9
0.8

0.9

0.8
0.7

SCH−GAN(Ours)
DCMH
SePH
SCM(seq)
SCM(orth)
CMSSH
CCQ
CMFH
PDH
CVH

0.85

0.7

0.6
0.5

Precision

Precision

Precision

0.8
0.6

0.75
0.7

0.4

0.5

0.65

0.3

0.6

0.4
0.2
0.1

0.55

0

0.1

0.2

0.3

0.4

0.5
Recall

0.6

0.7

0.8

0.9

1

0.3

0

0.1

0.2

0.3

0.4

0.5
Recall

0.6

0.7

0.8

0.9

1

0.5

0

0.2

0.4

0.6

0.8

1

Recall

Fig. 5. The precision-recall curves on three datasets with 64bit hash codes. The first row demonstrates the result of image query text task, while the second
row shows the result of text query image task. Left, middle and right columns demonstrate Wikipedia, NUSWIDE and MIRFlickr datasets respectively.
TABLE IV
C OMPARISON BETWEEN PROPOSED APPROACH SCH-GAN AND BASELINE APPROACH D IS .

Wikipedia
NUSWIDE
MIRFlickr

Dis
SCH-GAN
Dis
SCH-GAN
Dis
SCH-GAN

16
0.508
0.525
0.611
0.713
0.627
0.739

image→text
32
64
0.494
0.510
0.530
0.551
0.659
0.673
0.726
0.734
0.713
0.719
0.747
0.755

3) Training details: Here we introduce some details of the
training flow demonstrated in algorithm 1. The proposed SCHGAN is trained in a mini-batch way. The mini-batch size is set
to be 64 for both image and text pathway, and m in algorithm 1
is set to be 20, namely we generate 20 pairs for each query.
The proposed SCH-GAN is trained iteratively, specifically for
each d-step and g-step, we train 1 epoch for the discriminative
and generative model respectively. The learning rate of our
proposed network are initialized as 0.01, and it is decreased
by a factor of 10 each two epochs.
For the compared methods, all the implementations are
provided by their authors, and we follow the best settings in
their papers to conduct the experiments.
E. Comparison with state-of-the-art methods
The MAP scores of two retrieval tasks on Wikepedia,
NUSWIDE and MIRFlickr datasets are shown in Tables I, II
and III respectively. From the result tables, we can observe that
our proposed SCH-GAN approach achieves the best retrieval
accuracy compared with state-of-the-art methods on three
datasets. More specifically, the result tables are partitioned into
three categories, namely unsupervised, supervised and deep

128
0.510
0.546
0.646
0.748
0.717
0.769

16
0.859
0.860
0.654
0.741
0.667
0.775

text→image
32
64
0.847
0.873
0.876
0.889
0.691
0.705
0.743
0.771
0.743
0.753
0.790
0.798

128
0.858
0.888
0.694
0.779
0.751
0.799

learning based methods. Compared with these three categories,
from the result tables we can observe that: (1) Our proposed
SCH-GAN outperforms the unsupervised methods. For example, on NUSWIDE dataset compared with best unsupervised
methods CCQ [46], our proposed SCH-GAN improves the
average MAP score from 0.505 to 0.730 on image query text
task, and improves the average MAP score from 0.490 to 0.758
on text query image task. Similar trends can be observed on
Wikipedia and MIRFlickr datasets from Tables I and III. It
is because unsupervised methods only learn hashing functions
from data distributions, which achieve limited accuracy. (2)
Compared with supervised methods, our proposed SCH-GAN
achieves the best results. For example, compared with best
supervised method SePH [19] on NUSWIDE dataset, our
SCH-GAN improves average MAP scores from 0.715 to 0.730
on image query text task, and improves from 0.654 to 0.758 on
text query image task. Similar results can be observed on both
Wikipedia and MIRFlickr datasets. It is because our proposed
SCH-GAN fully exploits the unlabeled data to promote the
hash learning. (3) Our proposed SCH-GAN also outperforms
deep learning based methods, which improves average MAP
scores from 0.652 (DCMH [50]) to 0.730 on image query

11

The MAP scores of discriminative model on the NUSWIDE dataset.
image query text
text query image
0.74

The MAP scores of generative model on the NUSWIDE dataset.
image query text
0.44
text query image
0.42
0.40

0.70

MAP

MAP

0.72

0.38
0.36

0.68

0.34
0.32

0.66
0

2

4

Epoch

6

8

10

0.30
0

2

4

Epoch

6

8

10

Fig. 6. The retrieval accuracy of discriminative and generative model with respect to training iterations on NUSWIDE dataset with 64bit code length.

text task, and from 0.696 to 0.758 on text query image task.
It demonstrates that the generative model can promote the
discriminative model by challenging it with hard examples
around margins.
Figures 4 and 5 show the topK-precision and precisionrecall curves on the three datasets with 64bit code length. We
can observe that on both image query text and text query image
tasks, our proposed SCH-GAN achieves the best accuracy,
which further demonstrates the effectiveness of our proposed
approach.
F. Baseline experiments
We conduct two baseline experiments to demonstrate the
performance of generative and discriminative model. Firstly,
we investigate the retrieval performance of generative and
discriminative model to give more insight of adversarial
training. Secondly, we compare proposed SCH-GAN with a
baseline approach without adversarial training to verify its
effectiveness.
1) Performance of adversarial training: We demonstrate
the retrieval accuracy of generative and discriminative models
during the training process. The result is shown in Figure 6, we
can observe that during the adversarial training, the accuracy
of discriminative model is increasing after the generative
model updated. It means that the generative model selects
more informative examples for the discriminative model to
promote its accuracy.
2) Comparison with baseline approach: In our proposed
approach, the discriminative model can be trained solely
without generative model by using the triplet ranking loss in
equation 8. This is considered as a simple supervised baseline
approach without using adversarial training, we denote this
baseline approach as Dis. Compare SCH-GAN with Dis, we
can verify the effectiveness of adversarial training. The results
are shown in Table IV, and we can observe that the proposed
SCH-GAN constantly outperforms baseline method Dis on all
the three datasets. It demonstrates that the generative model
selects informative margin examples to promote the training

of discriminative model, thus promotes the accuracy of crossmodal hashing.
V. C ONCLUSION
In this paper we have proposed a novel semi-supervised
cross-modal hashing approach based generative adversarial
network (SCH-GAN). Firstly, we propose a novel generative
adversarial network to model cross-modal hashing. In our proposed SCH-GAN, the generative model tries to select margin
examples of another modality from unlabeled data when giving
a query of one modality (e.g. giving a text query to retrieve
images and vice versa). While the discriminative model tries to
predict the correlation between query and selected examples of
generative model. These two models play a minimax game to
iteratively optimize each other in an adversarial way. Secondly,
we propose a reinforcement learning based algorithm to drive
the training of proposed SCH-GAN. The generative model
takes the correlation score predicted by discriminative model
as a reward, and tries to select the examples close to the margin
to promote discriminative model by maximizing the margin
between positive and negative data. Experiments compared
with 8 state-of-the-art methods on 3 widely-used datasets
verify the effectiveness of our proposed approach.
In the future works, on one hand, we attempt to extend
current framework to an unsupervised scenario which is more
generalized. On the other hand, we will extend current approach to other modalities such as video and audio so that it
can exploit complex correlations between multiple modalities.
R EFERENCES
[1] A. Gionis, P. Indyk, and R. Motwani, “Similarity search in high
dimensions via hashing,” in VLDB, vol. 99, 1999, pp. 518–529.
[2] W. Liu, J. Wang, S. Kumar, and S.-F. Chang, “Hashing with graphs,” in
Proceedings of the 28th international conference on machine learning
(ICML-11), 2011, Conference Proceedings, pp. 1–8.
[3] J. Wang, W. Liu, S. Kumar, and S. F. Chang, “Learning to hash for
indexing big data-a survey,” Proceedings of the IEEE, vol. 104, no. 1,
pp. 34–57, 2016.

12

[4] G. Irie, Z. Li, X.-M. Wu, and S.-F. Chang, “Locally linear hashing for
extracting non-linear manifolds,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2014, pp. 2115–2122.
[5] H. Lai, Y. Pan, Y. Liu, and S. Yan, “Simultaneous feature learning and
hash coding with deep neural networks,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2015, pp.
3270–3278.
[6] W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang, “Supervised
hashing with kernels,” in Computer Vision and Pattern Recognition
(CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 2074–2081.
[7] Y. Weiss, A. Torralba, and R. Fergus, “Spectral hashing,” in Advances
in neural information processing systems, 2009, pp. 1753–1760.
[8] J. Wang, S. Kumar, and S.-F. Chang, “Semi-supervised hashing for
scalable image retrieval,” in Computer Vision and Pattern Recognition
(CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 3424–3431.
[9] G. Ye, D. Liu, J. Wang, and S.-F. Chang, “Large-scale video hashing via
structure learning,” in Proceedings of the IEEE International Conference
on Computer Vision, 2013, pp. 2272–2279.
[10] Y. Yang, D. Xu, F. Nie, J. Luo, and Y. Zhuang, “Ranking with
local regression and global alignment for cross media retrieval,” in
Proceedings of the 17th ACM international conference on Multimedia.
ACM, 2009, pp. 175–184.
[11] S. Kumar and R. Udupa, “Learning hash functions for cross-view
similarity search,” in IJCAI proceedings-international joint conference
on artificial intelligence, vol. 22, 2011, p. 1360.
[12] J. Song, Y. Yang, Y. Yang, Z. Huang, and H. T. Shen, “Inter-media
hashing for large-scale retrieval from heterogeneous data sources,” in
Proceedings of the 2013 ACM SIGMOD International Conference on
Management of Data. ACM, 2013, pp. 785–796.
[13] M. Rastegari, J. Choi, S. Fakhraei, D. Hal, and L. Davis, “Predictable
dual-view hashing,” in Proceedings of the 30th International Conference
on Machine Learning (ICML-13), 2013, pp. 1328–1336.
[14] G. Ding, Y. Guo, J. Zhou, and Y. Gao, “Large-scale cross-modality
search via collective matrix factorization hashing,” IEEE Transactions
on Image Processing, vol. 25, no. 11, pp. 5427–5440, 2016.
[15] Y. Zhen and D.-Y. Yeung, “Co-regularized hashing for multimodal data,”
in Advances in neural information processing systems, 2012, pp. 1376–
1384.
[16] Y. Wei, Y. Song, Y. Zhen, B. Liu, and Q. Yang, “Scalable heterogeneous
translated hashing,” in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM,
2014, pp. 791–800.
[17] D. Zhang and W.-J. Li, “Large-scale supervised multimodal hashing with
semantic correlation maximization,” in AAAI, vol. 1, 2014, p. 7.
[18] B. Wu, Q. Yang, W.-S. Zheng, Y. Wang, and J. Wang, “Quantized
correlation hashing for fast cross-modal search,” in IJCAI, 2015, pp.
3946–3952.
[19] Z. Lin, G. Ding, M. Hu, and J. Wang, “Semantics-preserving hashing
for cross-view retrieval,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2015, pp. 3864–3872.
[20] Y. He, S. Xiang, C. Kang, J. Wang, and C. Pan, “Cross-modal retrieval
via deep and bidirectional representation learning,” IEEE Transactions
on Multimedia, vol. 18, no. 7, pp. 1363–1377, 2016.
[21] D. Wang, P. Cui, M. Ou, and W. Zhu, “Learning compact hash codes
for multimodal representations using orthogonal deep structure,” IEEE
Transactions on Multimedia, vol. 17, no. 9, pp. 1404–1416, 2015.
[22] Y. Zhuang, Z. Yu, W. Wang, F. Wu, S. Tang, and J. Shao, “Crossmedia hashing with neural networks,” in Proceedings of the 22nd ACM
international conference on Multimedia. ACM, 2014, pp. 901–904.
[23] Y. Cao, M. Long, J. Wang, Q. Yang, and P. S. Yuy, “Deep visualsemantic hashing for cross-modal retrieval,” in Proceedings of the ACM
SIGKDD International Conference on Knowledge Discovery and Data
Mining, 2016, p. 1445.
[24] Y. Cao, M. Long, J. Wang, and H. Zhu, “Correlation autoencoder hashing
for supervised cross-modal search,” in Proceedings of the 2016 ACM
on International Conference on Multimedia Retrieval. ACM, 2016, pp.
197–204.
[25] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical correlation
analysis: An overview with application to learning methods,” Neural
computation, vol. 16, no. 12, pp. 2639–2664, 2004.
[26] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[27] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
no. 7553, pp. 436–444, 2015.
[28] X. Zhu, “Semi-supervised learning literature survey,” 2005.

[29] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in
Advances in neural information processing systems, 2014, pp. 2672–
2680.
[30] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and
P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in Advances in Neural
Information Processing Systems, 2016, pp. 2172–2180.
[31] C. Finn, I. Goodfellow, and S. Levine, “Unsupervised learning for
physical interaction through video prediction,” in Advances in Neural
Information Processing Systems, 2016, pp. 64–72.
[32] J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan, “Perceptual generative adversarial networks for small object detection,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition,
2017, pp. 1222–1230.
[33] J. Zhou, G. Ding, and Y. Guo, “Latent semantic sparse hashing for crossmodal similarity search,” in Proceedings of the 37th international ACM
SIGIR conference on Research & development in information retrieval.
ACM, 2014, pp. 415–424.
[34] D. Wang, X. Gao, X. Wang, and L. He, “Semantic topic multimodal
hashing for cross-media retrieval,” in IJCAI, 2015, pp. 3890–3896.
[35] M. M. Bronstein, A. M. Bronstein, F. Michel, and N. Paragios, “Data
fusion through cross-modality metric learning using similarity-sensitive
hashing,” in Computer Vision and Pattern Recognition (CVPR), 2010
IEEE Conference on. IEEE, 2010, pp. 3594–3601.
[36] Y. Hu, Z. Jin, H. Ren, D. Cai, and X. He, “Iterative multi-view hashing
for cross media indexing,” in Proceedings of the 22nd ACM international
conference on Multimedia. ACM, 2014, pp. 527–536.
[37] D. Wang, P. Cui, M. Ou, and W. Zhu, “Deep multimodal hashing with
orthogonal regularization,” in IJCAI, 2015, pp. 2291–2297.
[38] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation
learning with deep convolutional generative adversarial networks,” arXiv
preprint arXiv:1511.06434, 2015.
[39] M. Mirza and S. Osindero, “Conditional generative adversarial nets,”
arXiv preprint arXiv:1411.1784, 2014.
[40] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee,
“Generative adversarial text to image synthesis,” in Proceedings of the
33rd International Conference on International Conference on Machine
Learning-Volume 48, 2016, pp. 1060–1069.
[41] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with
auxiliary classifier gans,” arXiv preprint arXiv:1610.09585, 2016.
[42] J. C. Pereira, E. Coviello, G. Doyle, N. Rasiwasia, G. R. Lanckriet,
R. Levy, and N. Vasconcelos, “On the role of correlation and abstraction
in cross-modal multimedia retrieval,” IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 36, no. 3, pp. 521–535, 2014.
[43] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “Nus-wide:
a real-world web image database from national university of singapore,”
in Proceedings of the ACM international conference on image and video
retrieval. ACM, 2009, p. 48.
[44] M. J. Huiskes and M. S. Lew, “The mir flickr retrieval evaluation,” in
Proceedings of the 1st ACM international conference on Multimedia
information retrieval. ACM, 2008, pp. 39–43.
[45] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[46] M. Long, Y. Cao, J. Wang, and P. S. Yu, “Composite correlation quantization for efficient multimodal retrieval,” in Proceedings of the 39th
International ACM SIGIR conference on Research and Development in
Information Retrieval. ACM, 2016, pp. 579–588.
[47] Y. Zhen and D.-Y. Yeung, “A probabilistic model for multimodal
hash function learning,” in Proceedings of the 18th ACM SIGKDD
international conference on Knowledge discovery and data mining.
ACM, 2012, pp. 940–948.
[48] W. Wang, B. C. Ooi, X. Yang, D. Zhang, and Y. Zhuang, “Effective
multi-modal retrieval based on stacked auto-encoders,” Proceedings of
the VLDB Endowment, vol. 7, no. 8, pp. 649–660, 2014.
[49] W. Wang, X. Yang, B. C. Ooi, D. Zhang, and Y. Zhuang, “Effective
deep learning-based multi-modal retrieval,” The VLDB Journal, vol. 25,
no. 1, pp. 79–101, 2016.
[50] Q.-Y. Jiang and W.-J. Li, “Deep cross-modal hashing,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition,
2017, pp. 3232–3240.

