Unsupervised Generative Adversarial Cross-modal Hashing
Jian Zhang, Yuxin Peng∗ , and Mingkuan Yuan

arXiv:1712.00358v1 [cs.CV] 1 Dec 2017

Institute of Computer Science and Technology, Peking University
Beijing 100871, China
pengyuxin@pku.edu.cn

Abstract
Cross-modal hashing aims to map heterogeneous multimedia data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Unsupervised cross-modal hashing is more flexible and applicable than supervised methods, since no intensive labeling
work is involved. However, existing unsupervised methods
learn hashing functions by preserving inter and intra correlations, while ignoring the underlying manifold structure
across different modalities, which is extremely helpful to capture meaningful nearest neighbors of different modalities for
cross-modal retrieval. To address the above problem, in this
paper we propose an Unsupervised Generative Adversarial
Cross-modal Hashing approach (UGACH), which makes full
use of GAN’s ability for unsupervised representation learning
to exploit the underlying manifold structure of cross-modal
data. The main contributions can be summarized as follows:
(1) We propose a generative adversarial network to model
cross-modal hashing in an unsupervised fashion. In the proposed UGACH, given a data of one modality, the generative
model tries to fit the distribution over the manifold structure,
and select informative data of another modality to challenge
the discriminative model. The discriminative model learns to
distinguish the generated data and the true positive data sampled from correlation graph to achieve better retrieval accuracy. These two models are trained in an adversarial way to
improve each other and promote hashing function learning.
(2) We propose a correlation graph based approach to capture the underlying manifold structure across different modalities, so that data of different modalities but within the same
manifold can have smaller Hamming distance and promote
retrieval accuracy. Extensive experiments compared with 6
state-of-the-art methods on 2 widely-used datasets verify the
effectiveness of our proposed approach.

Introduction
Multimedia retrieval has become an important application
over the past decades, which can retrieve multimedia contents that users have interests in. However, it is a big challenge to retrieve multimedia data efficiently from large scale
databases, due to the explosive growth of multimedia information. To address this issue, there are many hashing methods (Wang et al. 2016; Gionis, Indyk, and Motwani 1999;
∗

Corresponding author.
Copyright c 2018, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

Zhang and Peng 2017) proposed to accomplish efficient retrieval. The goal of hashing methods is to map high dimensional representations in the original space to short binary
codes in the Hamming space. By using these binary hash
codes, faster Hamming distance computation can be applied
based on bit operations that can be implemented efficiently.
Moreover, binary codes take much less storage compared
with original high dimensional representations.
There are large numbers of hashing methods applied to
single modality retrieval (Wang et al. 2016), by which users
can only retrieve data by a query with the same modality,
such as text retrieval (Baeza-Yates and Ribeiro-Neto 1999)
and image retrieval (Wang et al. 2016). Nevertheless, single
modality retrieval can not meet users’ increasing demands,
due to the different modalities of multimedia data. For example, by single modality retrieval, it is impracticable to
search an image by using a textual sentence that describes
the semantic content of the image. Therefore, cross-modal
hashing has been proposed to meet this kind of retrieval demands in large scale cross-modal databases. Owing to the effectiveness and flexibility of cross-modal hashing, users can
submit whatever they have to retrieve whatever they want
(Peng, Huang, and Zhao 2017; Peng et al. 2017).
“Heterogeneous gap” is the key challenge of cross-modal
hashing, which means the similarity of between different
modalities cannot be measured directly. Consequently, some
cross-modal hashing methods (Kumar and Udupa 2011;
Rastegari et al. 2013; Ding et al. 2016; Zhang and Li 2014;
Zhuang et al. 2014) have been proposed to bridge this gap.
Existing cross-modal hashing methods can be categorized
into traditional methods and Deep Neural Networks (DNN)
based methods. Moreover, traditional methods can be divided into unsupervised methods and supervised methods
by whether semantic information is leveraged.
Unsupervised cross-modal hashing methods usually
project data from different modalities into a common
Hamming space to maximize their correlations, which
hold the similar idea with Canonical Correlation Analysis (CCA) (Hardoon, Szedmak, and Shawe-Taylor 2004).
Song et al. propose Inter-Media Hashing (IMH) (Song et al.
2013) to establish a common Hamming space by preserving
inter-media and intra-media consistency. Cross-view Hashing (CVH) (Kumar and Udupa 2011) is proposed to consider
both intra-view and inter-view similarities, which is an ex-

tension of image hashing method named Spectral Hashing
(SH) (Weiss, Torralba, and Fergus 2009). Predictable Dualview Hashing (PDH) (Rastegari et al. 2013) designs an objective function to keep the predictability of pre-generated
binary codes. Ding et al. propose Collective Matrix Factorization Hashing (CMFH) (Ding et al. 2016) to learn unified
hash codes by collective matrix factorization. Composite
Correlation Quantization (CCQ) (Long et al. 2016) jointly
learns the correlation-maximal mappings that transform different modalities into an isomorphic latent space, and learns
composite quantizers that convert the isomorphic latent features into compact binary codes.
Supervised cross-modal hashing methods utilize labeled
semantic information to learn hashing functions. Bronstein
et al. propose Cross-Modality Similarity Sensitive Hashing (CMSSH) (Bronstein et al. 2010) to model hashing
learning by a classification paradigm with a boosting manner. Wei et al. propose Heterogeneous Translated Hashing
(HTH) (Wei et al. 2014), which learns translators to align
separate Hamming spaces of different modalities to perform
cross-modal hashing. Semantic Correlation Maximization
(SCM) (Zhang and Li 2014) is proposed to learn hashing
functions by constructing and preserving the semantic similarity matrix. Semantics-Preserving Hashing (SePH) (Lin
et al. 2015) transforms the semantic matrix into a probability distribution and minimizes the KL-divergence in order
to approximate the distribution with learned hash codes in
Hamming space.
DNN based methods are inspired by the successful
applications of deep learning, such as image classification (Krizhevsky, Sutskever, and Hinton 2012). Cross-Media
Neural Network Hashing (CMNNH) (Zhuang et al. 2014) is
proposed to learn cross-modal hashing functions by preserving intra-modal discriminative ability and inter-modal pairwise correlation. Cross Autoencoder Hashing (CAH) (Cao
et al. 2016b) is based on deep autoencoder structure to maximize the feature correlation and the semantic correlation between different modalities. Cao et al. propose Deep Visualsemantic Hashing (DVH) (Cao et al. 2016a) as an end-to-end
framework that combines both representation learning and
hashing function learning. Jiang et al. propose Deep Crossmodal Hashing (DCMH) (Jiang and Li 2017), which performs feature learning and hashing function learning simultaneously.
Compared with unsupervised paradigm, supervised methods use labeled semantic information that requires massive labor to collect, resulting in a high labor cost in real
world applications. On the contrary, unsupervised crossmodal hashing methods can leverage unlabeled data to realize efficient cross-modal retrieval, which is more flexible
and applicable in real world applications. However, most unsupervised methods learn hashing functions by preserving
inter and intra correlations, while ignoring the underlying
manifold structure across different modalities, which is extremely helpful to capture meaningful nearest neighbors of
different modalities. To address this problem, in this paper,
we exploit correlation information from underlying manifold structure of unlabeled data across different modalities
to enhance cross-modal hashing learning.

Inspired by recent progress of Generative Adversarial
Network (GAN) (Goodfellow et al. 2014; Reed et al. 2016;
Zhao and Gao 2017; Wang et al. 2017), which has shown
its ability to model the data distribution in an unsupervised
fashion. In this paper, we propose an unsupervised generative adversarial cross-modal hashing (UGACH) approach.
We design a graph-based unsupervised correlation method
to capture the underlying manifold structure across different
modalities, and a generative adversarial network to learn the
manifold structure and further enhance the performance by
an adversarial boosting paradigm. The main contributions of
this paper can be summarized as follows:
• We propose a generative adversarial network to model
cross-modal hashing in an unsupervised fashion. In the
proposed UGACH, given the data of any modality, the
generative model tries to fit the distribution over the manifold structure, and selects informative data of another
modality to challenge the discriminative model. While the
discriminative model learns to distinguish the generated
data and the true positive data sampled from correlation
graph to achieve better retrieval accuracy.
• We propose a correlation graph based learning approach
to capture the underlying manifold structure across different modalities, so that data of different modalities but
within the same manifold can have smaller Hamming distance and promote retrieval accuracy. We also integrate
the proposed correlation graph into proposed generative
adversarial network to provide manifold correlation guidance to promote the cross-modal retrieval accuracy.
Extensive experiments compared with 6 state-of-the-art
methods on 2 widely-used datasets verify the effectiveness
of our proposed approach.
The rest of this paper is organized as follows. In “The Proposed Approach” section, we present our UGACH approach
in detail. The experimental results and analyses are reported
in “Experiment” section. Finally, we conclude this paper in
“Conclusion” section.

The Proposed Approach
Figure 1 presents the overview of our proposed approach,
which consists of three parts, namely feature extraction, generative model G and discriminative model D. The feature
extraction part employs image feature and text feature extraction to represent unlabeled data of different modalities
as original features. The detailed implementation of this part
will be described at “Experiment” section. Given a data of
one modality, G attempts to select informative data from another modality to generate a pair of data and send them to
D. In D, we construct a correlation graph, which can capture the manifold structure among the original features. D
receives the generated pairs as inputs, and also samples positive data from constructed graph to form a true manifold
pair. Then D tries to distinguish the manifold and generated
pairs in order to get better discriminate ability. These two
models play a minimax game to boost each other, and the
finally trained D can be used as cross-modal hashing model.
We denote the cross-modal dataset as D = {I, T }, where
I represents image modality and T represents text modal-

Correlation Graph

Images

Feature Extraction
gor geou s j ohnny
act or p
j ohnnyd epp depp
om g goodl ook ing
al bandr i

boy peop le m an
guy cel ebr it y
f r ie nd m ovi es
act or t r ibu te
j akegyl l enhaal
heat hl edger
god fa t her m ati l da

Image Feature Extraction
ar t f il m vint age
nahp ro gr ay s all y
act re s act or
s al lyg ra y
br i ti s hf il m act or

s t re et bw f es ti val
edi nbur gh act or
per f or m er 2 b
m onoch rom i a
hou rof t hedi am ondl
i ght
hum anb odygal l ery
bw
cel ebr it y dave tv
cool hil l s i nger
act or ku wai t
bas har aziz
kuw ai ty 7ar akat
al ja s m i al s hat ti
al ra i alj as s m i
har akat w oor t hy

por t r ait m an
cel ebr it y
s ungl as s es
decem ber fu ji
bok ehf l as hact or
j ell o
col um bi auni ver s it
y te ac hers co l ege

Manifold Pairs

Texts
port ra it ma n cel ebrit y
le ic a f 1 0 ac tor
gorgeous johnny ac tor
sungl asses dece mbe r
oscar squirrel
p johnnyde pp de pp
fuj i bokeh fla sh ac tor y ak ush i ma c rusa de r
squi rre ls smil ey a ctor
m5 ce
short
lmdave
fort ias
o mg go o dl o ok in g
le brifi ty
t v pcool
je ll o
ac ademyawa rds
o mo
fh earve
al bandri
l s inge
acntor
col umbia uni ve rs it y k in gdhil
boy people man guy squa rlet t ba s
ja ywe stitnoct
lux az iz
bashar
te ac he rsc olle
st rege
et bw festikuwa
val
ce le bri ty friend
tekuwa
ngugaiden
it y 7ara ka t
edi nburgh ac tor
movi es act or tribute
p erforme r al
2bja smi al sha tt i a lrai
art fil m vinta ge
ja kegyl le nhaa l
monoc hromiaal ja ssmi ha rak at
nahpro gray sal ly
heat hledger
h
ac tress ac tor sal lygra yh ou roft he di amo n dl igwoorthy
g od fat he rm ati ld a
t
bri ti shfi lm act or
huma nbodygal le rybw

Hashing Layers

Hashing Layers

HT

HI

Reward Loss

os car sq uir r el
s qui r re l s s mi l ey
act or
academ yaw ar ds
s quar l et t bas

Text Feature Extraction

Generated Pairs

port ra it ma n cel ebrit y
sungl asses dece mbe r
fuj i bo k eh fla sh ac to r
je ll o
col umbia uni ve rs it y
te ac he rsc olle ge

Generative
Probability

boy people man guy
ce le bri ty frien d
movi es act or tribute
ja kegyl le nhaa l
heat hledger
godfat he rm ati lda

ce le bri ty dave t v cool
hil l s inge r ac tor
kuwa it bashar az iz
kuwa it y 7ara ka t
al ja smi al sha tt i a lrai
al ja ssmi ha rak at
woorthy

port ra it ma n cel ebrit y
sungl asses dece mbe r
fuj i bokeh fla sh ac tor
je ll o
col umbia uni ve rs it y
te ac he rsc o lle g e

boy people man guy
ce le bri ty friend
movi es act or tribute
ja kegyl le nhaa l
heat hledger
godfat he rm ati lda

Generative Model G

le ic a f 10 ac tor
yakushi ma c rusa de r
m5 short fi lm fort ias p
kingdomofhea ve n
ja ywe st noct lux
te ngugaiden

Hashing Layers

Hashing Layers

HT

HI

Ranking Loss
Reward

ce le bri ty dave t v cool
hil l s inge r ac tor
k uwa it b ash ar az iz
kuwa it y 7ara ka t
al ja smi al sha tt i a lrai
al ja ssmi ha rakat
wo orth y

Relevance
Score

Discriminative Model D

Figure 1: The overall framework of our proposed unsupervised generative adversarial cross-modal hashing approach (UGACH),
which consists of feature extraction part, generative model G and discriminative model D.
ity. In this paper, D = {I, T } is further split into a retrieval database Ddb = {Idb , Tdb } and a query set Dq =
{Iq , Tq }. In the retrieval database Ddb , Idb = {ip }np=1 ,
Tdb = {tp }np=1 and n is the number of data pairs. The retrieval database Ddb is also the training set. In the query
set Dq , Iq = {ip }tp=1 , Tq = {tp }tp=1 and t is the number of data pairs. The aim is to learn cross-modal hashing
functions to generate hash codes for data C I = HI (I) and
C T = HT (T ), so that different modal data that has similar
semantic are close in the common Hamming space. In addition, we denote the hash code length as l. By the generated
hash codes, we can retrieve the relevant data by a query of
any modality from database of another modality efficiently.

Generative Model
The network of generative model has a two-pathway architecture, which receives the original features of both images
and texts as inputs. Each pathway consists of a common representation layer and a hashing layer, whose implementations are two fully-connected layers. The first layer can convert the modality specific features to common representations, which make the instances of different modalities measurable in a common space. The representation produced by
this layer can be denoted as follows:
φc (x) = tanh(Wc x + bc )

(1)

where x denotes the original features of images or texts, Wc
is the weight parameter of the common representation layer,
and bc is the bias parameter.

The hashing layer can map the common representations
into binary hash codes, so that the similarity between different modalities can be measured by fast Hamming distance
calculation. The continuous real values of hash code is defined as:
h(x) = sigmoid(Wh φc (x) + bh )
(2)
where Wh is the weight parameter and bh is the bias parameter. Then we can get the binary codes by a thresholding
function:
b(x) = sgn(hk (x) − 0.5), k = 1, 2, · · · , l
(3)
where l denotes the hash code length. Considering that it is
hard to optimize binary codes directly, we use relaxed continuous real valued hash codes h(x) in the training process.
Given a data of one modality, the goal of generative model
G is to fit the distribution over the manifold structure and select informative data of another modality to challenge the
discriminative model. The generative probability of G is
pθ (xU |q), which is the foundation to select relevant instance
of one modality from unpaired data when given a query of
another modality. For example, given a image query qi , the
generative model tries to select relevant text tU from Tdb .
The generative probability pθ (xU |q) is defined as a softmax function:
exp(−kh(q) − h(xU )k2 )
(4)
pθ (xU |q) = P
U 2
xU exp(−kh(q) − h(x )k )
Given a query, we can use equation (4) to calculate the probability of each candidate, which indicates the possibility of
becoming a relevant sample.

Discriminative Model
The network structure of discriminative model is same as the
generative model. The input of this network is the generated
pairs by generative model, and the manifold pairs provided
by a correlation graph. The goal of discriminative model is
to distinguish whether an input pair is generated or from the
correlation graph.
First of all, we introduce the correlation graph in the discriminative model. We propose a correlation graph to guide
the training of discriminative model. The correlation graph
can capture the underlying manifold structure across different modalities, so that data of different modalities but within
the same manifold can have small Hamming distance and
promote retrieval accuracy.
Specifically, we first construct undirected graphs
Graphi = (V, Wi ) and Grapht = (V, Wt ) for image and
text modality respectively, where V denotes the vertices
and Wi and Wt are the similarity matrix. Wi is defined as
follows:

1 : xp ∈ N Nk (xq )
w(p, q) =
(5)
0:
otherwise
where N Nk (xq ) denotes the k-nearest neighbors of xq in
the training set, similarly we can define Wt . Then we sample the data of the true distributions based on the constructed
graph. For real pair ptrue (x|q j ) provided by the dataset, we
select xk as the true relevant instance of given query q j
and associate them as manifold pair pmanif old (xk |q j ), when
w(k, j) = 1. It is noted that pairwise information naturally
exists in cross-modal data, thus if the corresponding text tk
is within the same manifold with text query q j , the paired
image ik is also in the same manifold with q j and vice versa.
By this definition, we intend to utilize the underlying data
manifold of different modality to guide the training of discriminative model. Intuitively, we want the data of different
modality but within the same manifold to have small Hamming distance (e.g. small Hamming distance between text
query q j and image ik ).
After receiving the generated and manifold pairs, discriminative model predicts a relevance score between each pair
as the judgment result. So the relevance score between instance x and its query q is defined as fφ (x, q). The goal of
discriminative model is to distinguish the true relevant data
(manifold pairs) and non-relevant data (generated pairs) for
a query q accurately.
The relevance score of fφ (xG , q) is defined by triplet
ranking loss as follows:

Then discriminative model D uses the relevance score to
produce predicted probability of instance x by a sigmoid
function:
D(x|q) = sigmoid(fφ (x, q)) =

M

Adversarial Learning
Given the definitions of the generative and discriminative
models, we can conduct a minimax game for training them.
Given a query, the generative model attempts to generate a
pair which is close to the manifold pair to fool the discriminative model. The discriminative model tries to distinguish
between manifold pair sampled from the correlation graph
and the generated pair, which forms an adversarial process
against the generative model. Inspired by the GAN (Goodfellow et al. 2014), this adversarial process can be defined:

V(G, D) = min max
θ

fφ (x , q) = max(0, m + kh(q) − h(x )k
G

2

− kh(q) − h(x )k )

where xM is a manifold paired instance with query q selected from the correlation graph, xG is the selected instance
by generative model, and m is a margin parameter which
is set to be 1 in our proposed approach. The above equation means that we want the distance between manifold pair
(q, xM ) smaller than that of generated pair (q, xG ) by a margin m, so that the discriminative model can draw a clear distinguishing line between the manifold and generated pairs.

(Ex∼ptrue (xM |qj ) [log(D(xM |q j ))]

j=1

(8)
The generative and discriminative models can be learned
iteratively by maximizing and minimizing the above object function. As general training process, the discriminative
model tries to maximize equation (8), while the generative
model attempts to minimize equation (8) and fit the distribution over the manifold structure. The learning process of the
discriminative model is fixed when the generative model can
be trained as follows:

θ

(6)

φ

n
X

+ Ex∼pθ (xG |qj ) [log(1 − D(xG |q j ))])

n
X

θ = arg min

2

(7)

The generative model tries to select informative data to
challenge the discriminative model, which limits its capability to perform cross-modal retrieval. By contrast, the
discriminative model is suitable for retrieving data across
different modalities, after being promoted greatly by the
generative model. Therefore after the proposed UGACH is
trained, we use the discriminative model to perform crossmodal retrieval via produced hash codes.

∗

G

exp(fφ (x, q))
1 + exp(fφ (x, q))

(Ex∼p

true (x

G |q j ) [log(sigmoid(fφ∗ (x

M

+ Ex∼p

G j [log(1
θ (x |q )

j

, q )))]
(9)

j=1
G

j

− sigmoid(fφ∗ (x , q )))])

where fφ∗ denotes the discriminative model at previous iteration. The traditional GAN uses continuous noise vector
to generate new data and is trained by stochastic gradient
descent algorithm. By contrast, the generative model of our
proposed UGACH selects data from unlabeled data to generate pairs and can not be optimized continuously due to the
discrete selective strategy. We utilize reinforcement learning
based parameters update policy to train the generative model

Compared Methods

as follows:
G

j

∇θ Ex∼pθ (xG |qj ) [log(1 + exp(fφ (x , q )))]
=
=

m
X
k=1
m
X

j
G
j
∇θ pθ (xG
k |q )log(1 + exp(fφ (xk , q )))

j
G j
G
j
pθ (xU
k |q )∇θ logpθ (xk |q )log(1 + exp(fφ (xk , q )))

k=1

= Ex∼pθ (xG |qj ) [∇θ logpθ (xG |q j )log(1 + exp(fφ (xG , q j )))]
'

m
1 X
j
G
j
∇θ logpθ (xG
k |q )log(1 + exp(fφ (xk , q )))
m
k=1

(10)

where k denotes the k-th instance selected by generative
model according to a query q j . From the perspective of reinforcement learning, according to the environment q k , xG
k
j
is the action taken by policy logpθ (xG
k |q ), and log(1 +
j
exp(fφ (xG
k , q ))) acts as the reward, which encourages the
generative model to select data close to the distribution over
manifold structure. Finally the trained discriminative model
can be used to generate binary codes for any input data of
any modality, and cross modal retrieval can be performed
by fast Hamming distance computation between query and
each data in the database.

Experiments
In this section, we present the experimental results of our
proposed UGACH approach. We first introduce the datasets,
evaluation metrics and implementation details. Then we
compare and analyze the results of UGACH with 6 stateof-the-art methods and 2 baseline methods.

Dataset
In the experiments, we conduct cross-modal hashing on 2
widely-used datasets: NUS-WIDE (Chua et al. 2009) and
MIRFLICKR (Huiskes and Lew 2008).
• NUS-WIDE dataset (Chua et al. 2009) is a relatively
large-scale image/tag dataset with 269498 images. Each
image has corresponding textual tags, which are regarded
as the text modality in our experiments. NUS-WIDE
dataset has 81 categories, but there are overlaps among
the categories. Following (Lin et al. 2015), we select the
10 largest categories and the corresponding 186557 images. We take 1% data of NUS-WIDE dataset as the query
set, and the rest as the retrieval database. We randomly
selected 5000 images as training set for the supervised
methods. We represent each image by 4096 dimensional
deep features extracted from 19-layer VGGNet, and each
text by 1000 dimensional BoW.
• MIRFlickr dataset (Huiskes and Lew 2008) has 25000
images collected from Flickr, which has 24 categories.
Each image is also associated with text tags. Following (Lin et al. 2015), we take 5% of the dataset as the
query set and the remaining as the retrieval database. We
also randomly select 5000 images as training set for supervised methods. Similarly, we represent each image by
4096 dimensional deep features extracted from 19-layer
VGGNet, and each texts by 1000 dimensional BoW.

In order to verify the effectiveness of our proposed approach,
there are 4 unsupervised methods and 2 supervised methods
compared in the experiment, including unsupervised methods CVH (Kumar and Udupa 2011), PDH (Rastegari et al.
2013), CMFH (Ding et al. 2016) and CCQ (Long et al.
2016), and supervised methods CMSSH (Bronstein et al.
2010) and SCM (Zhang and Li 2014). Besides state-of-theart methods, we also compare our UGACH approach with 2
baseline methods to verify the effectiveness of our contributions.
• Baseline: We design a baseline method without the correlation graph and adversarial training, we denote this
method as Baseline. It is implemented by training the
discriminative model alone with a triplet ranking loss in
equation (6), where the positive data is only the paired
data provided by cross-modal datasets.
• Baseline-GAN: We add the adversarial training to Baseline, which means that we further promote discriminative
model in Baseline by adversarial training defined in equation (8).
Comparing Baseline-GAN with Baseline, we can verify the
effectiveness of our proposed generative adversarial network
for cross-modal hashing, and comparing our final approach
UGACH with Baseline-GAN, we can verify the effectiveness of proposed correlation graph.

Retrieval Tasks and Evaluation Metrics
In the experiments, two retrieval tasks are performed: retrieving text by image query (image→text) and retrieving
images by text query (text→image). Specifically, we first
obtain the hash codes for the images and texts in the query
and retrieval database with our UGACH approach and all
the compared methods. Then we take one of the images as
query, compute the Hamming distance with all text in retrieval database, and evaluate the ranking list by 3 evaluation
metrics to measure the retrieval effectiveness: Mean Average
Precision (MAP), precision recall curve (PR-curve) and precision at top k returned results (topK-precision), which are
defined as follows:
• The MAP scores are computed as the mean of average
precision (AP) for all queries, and AP is computed as:
n

AP =

1 X k
× relk
R
Rk

(11)

k=1

where n is the size of database, R is the number of relevant images in database, Rk is the number of relevant
images in the top k returns, and relk = 1 if the image
ranked at k-th position is relevant and 0 otherwise.
• Precision recall curve (PR-curve): The precision at certain
level of recall of the retrieved ranking list, which is widely
used to measure the information retrieval performance.
• Precision at top k returned results (topK-precision): The
precision with respect to different numbers of retrieved
samples from the ranking list.

Table 1: The MAP scores of two retrieval tasks on NUS-WIDE dataset with different lengths of hash codes.
image→text
text→image
Methods
16
32
64
128
16
32
64
128
CVH (Kumar and Udupa 2011) 0.458 0.432 0.410 0.392 0.474 0.445 0.419 0.398
PDH (Rastegari et al. 2013)
0.475 0.484 0.480 0.490 0.489 0.512 0.507 0.517
CMFH (Ding et al. 2016)
0.517 0.550 0.547 0.520 0.439 0.416 0.377 0.349
CCQ (Long et al. 2016)
0.504 0.505 0.506 0.505 0.499 0.496 0.492 0.488
CMSSH (Bronstein et al. 2010) 0.512 0.470 0.479 0.466 0.519 0.498 0.456 0.488
SCM orth (Zhang and Li 2014) 0.389 0.376 0.368 0.360 0.388 0.372 0.360 0.353
SCM seq (Zhang and Li 2014) 0.517 0.514 0.518 0.518 0.518 0.510 0.517 0.518
Baseline
0.540 0.537 0.573 0.598 0.554 0.555 0.583 0.608
Baseline-GAN
0.575 0.594 0.602 0.623 0.580 0.609 0.617 0.629
UGACH (Ours)
0.613 0.623 0.628 0.631 0.603 0.614 0.640 0.641
Table 2: The MAP scores of two retrieval tasks on MIRFlickr dataset with different lengths of hash codes.
image→text
text→image
Methods
16
32
64
128
16
32
64
128
CVH (Kumar and Udupa 2011) 0.602 0.587 0.578 0.572 0.607 0.591 0.581 0.574
0.623 0.624 0.621 0.626 0.627 0.628 0.628 0.629
PDH (Rastegari et al. 2013)
CMFH (Ding et al. 2016)
0.659 0.660 0.663 0.653 0.611 0.606 0.575 0.563
0.637 0.639 0.639 0.638 0.628 0.628 0.622 0.618
CCQ (Long et al. 2016)
CMSSH (Bronstein et al. 2010) 0.611 0.602 0.599 0.591 0.612 0.604 0.592 0.585
SCM orth (Zhang and Li 2014) 0.585 0.576 0.570 0.566 0.585 0.584 0.574 0.568
SCM seq (Zhang and Li 2014) 0.636 0.640 0.641 0.643 0.661 0.664 0.668 0.670
Baseline
0.619 0.631 0.633 0.646 0.625 0.635 0.634 0.649
Baseline-GAN
0.630 0.643 0.651 0.664 0.660 0.657 0.670 0.688
UGACH (Ours)
0.685 0.693 0.704 0.702 0.673 0.676 0.686 0.690
It should be noted that the MAP score is computed for all the
retrieval results with 4 different lengths of hash codes, while
PR-curve and topK-precision are evaluated on 128 bit hash
codes.

Implementation Details
In this section, we present the implementation details of our
UGACH in the experiments. We take 4096 dimensional feature extracted from 19-layer VGGNet for images, and use
the 1000 dimensional BoW feature for texts. We implement
the proposed UGACH by tensorflow1 . The dimension of
common representation layer is set to be 4096, while the
hashing layer’s dimension is set to be the same as hash code
length.
Moreover, we train the proposed UGACH in a mini-batch
way and set the batch size as 64 for discriminative and generative models. We train the proposed UGACH iteratively.
After the discriminative model is trained in 1 epoch, the generative model respectively will be trained in 1 epoch. The
learning rate of UGACH is decreased by a factor of 10 each
two epochs, while it is initialized as 0.01.
For the compared methods, we apply the implementations
provided by their authors, and follow their best settings to
preform the experiments. And it is noted that for a fair comparison between different methods, we use the same image
and text features for all compared methods.
1

https://www.tensorflow.org

Experiment Results
Tables 1 and 2 show the MAP scores of our UGACH and the
compared methods on NUS-WIDE and MIRFlickr datasets.
Compared with state-of-the-art methods, it can be seen
that our proposed UGACH approach achieves the best retrieval accuracy on all 2 datasets. For convenience, we categorize these result tables into three parts: unsupervised compared methods, supervised compared methods and baseline
methods. On NUS-WIDE dataset, our proposed UGACH
keeps the best average MAP score of 0.624 on image→text
and 0.625 on text→image tasks. Compared with the best unsupervised methods CCQ (Long et al. 2016), our UGACH
achieves an inspiring accuracy improvement from 0.505 to
0.624 on image→text task, and improves the average MAP
score from 0.494 to 0.625 on text→image task. Even compared with supervised methods SCM seq (Zhang and Li
2014), our UGACH also improves average MAP scores from
0.517 to 0.624 on image→text task, and from 0.516 to 0.625
on text→image task. We can observe the similar trends on
MIRFlickr dataset from Tables 2.
Figures 2 and 3 show the topK-precision and precisionrecall curves on the two datasets with 128 bit code length.
We can observe that on both image→text and text→image
tasks, UGACH achieves the best accuracy among all compared unsupervised methods. And UGACH even achieves
better retrieval accuracy than compared supervised methods
on most of the evaluation metrics, which further demonstrates the effectiveness of our proposed approach.

(a) Image−>text task on NUS−WIDE dataset

(b) Image−>text task on MIRFlickr dataset

0.9

(c) Text−>image task on NUS−WIDE dataset

0.9

(d) Text−>image task on MIRFlickr dataset

0.9

0.9

0.85

0.85

0.8

0.8
0.8

0.8

0.7

0.7

0.7

Precision

0.6

0.75
Precision

Precision

Precision

0.75
0.6

0.65
0.5
0.6

0.6

0.4

0.4
0.55

0.3

0.7
0.65

0.5

0

200

400

600
Top K

800

0.5

1000

UGACH(Ours)

0.55

0

200

400

SCM_seq

600
Top K

800

0.3

1000

SCM_orth

0

200

400

CMSSH

600
Top K

800

0.5

1000

CCQ

0

200

CMFH

400

600
Top K

PDH

800

1000

CVH

Figure 2: The topK-precision curves with 128 bit hash codes. The left two figures present the result of image→text task on
NUS-WIDE and MIRFlickr datasets, while the right two figures show the result of text→image task.
(a) Image−>text task on NUS−WIDE dataset

(b) Image−>text task on MIRFlickr dataset

0.9

(c) Text−>image task on NUS−WIDE dataset

0.9

(d) Text−>image task on MIRFlickr dataset

0.9

0.9

0.85

0.85

0.8

0.8
0.8

0.8

0.7

0.7

0.7

Precision

0.6

0.75
Precision

Precision

Precision

0.75
0.6

0.65
0.5
0.6

0.6

0.4

0.4
0.55

0.3

0.7
0.65

0.5

0

0.2

0.4

0.6

0.8

1

0.5

0.55

0

Recall
UGACH(Ours)

0.2

0.4

0.6

0.8

1

0.3

0

Recall
SCM_seq

SCM_orth

0.2

0.4

0.6

0.8

0.5

1

0

0.2

0.4

Recall
CMSSH

CCQ

0.6

0.8

1

Recall
CMFH

PDH

CVH

Figure 3: The precision-recall curves with 128 bit hash codes. The left two figures present the result of image→text task on
NUS-WIDE and MIRFlickr datasets, while the right two figures show the result of text→image task.
Compared with 2 baseline methods on NUS-WIDE
dataset, we can observe that Baseline-GAN has an improvement of 0.037 and 0.034 on two tasks, which demonstrates the effectiveness of our proposed generative adversarial network for cross-modal hashing. Compared our proposed UGACH with Baseline-GAN, we can observe an improvement of 0.025 and 0.016 on two tasks, which demonstrates the effectiveness of our proposed correlation graph.
Similar trends can be also observed on MIRFlickr dataset.

manifold can have smaller Hamming distance and promote
retrieval accuracy. Experiments compared with 6 state-ofthe-art methods on 2 widely-used datasets verify the effectiveness of our proposed approach.
The future works lie in two aspects. Firstly, we will focus
on extending our approach to support retrieval across multiple modalities, such as cross-modal retrieval across image,
text, video and audio. Secondly, we attempt to extend current framework to other scenarios such as image caption to
verify its versatility.

Conclusion
In this paper, we have proposed an Unsupervised Generative Adversarial Cross-modal Hashing approach (UGACH),
which intends to make full use of GAN’s ability of unsupervised representation learning to exploit the underlying manifold structure of cross-modal data. On one hand, we propose
a generative adversarial network to model cross-modal hashing in an unsupervised fashion. In the proposed UGACH, the
generative model tries to fit the distribution over the manifold structure, and select informative data of another modality to challenge the discriminative model. While the discriminative model learns to preserve traditional inter correlation,
and the manifold correlations provided by generative model
to achieve better retrieval accuracy. Those two models are
trained in an adversarial way to improve each other and
achieve better retrieval accuracy. On the other hand, we propose a graph based correlation learning approach to capture
the underlying manifold structure across different modalities, so that data of different modalities but within the same

Acknowledgments
This work was supported by National Natural Science Foundation of China under Grant 61771025 and Grant 61532005.

References
[Baeza-Yates and Ribeiro-Neto 1999] Baeza-Yates, R., and
Ribeiro-Neto, B. 1999. Modern information retrieval, volume 463. ACM press New York.
[Bronstein et al. 2010] Bronstein, M. M.; Bronstein, A. M.;
Michel, F.; and Paragios, N. 2010. Data fusion through
cross-modality metric learning using similarity-sensitive
hashing. In Computer Vision and Pattern Recognition
(CVPR), 2010 IEEE Conference on, 3594–3601. IEEE.
[Cao et al. 2016a] Cao, Y.; Long, M.; Wang, J.; Yang, Q.; and
Yuy, P. S. 2016a. Deep visual-semantic hashing for crossmodal retrieval. In Proceedings of the ACM SIGKDD In-

ternational Conference on Knowledge Discovery and Data
Mining, 1445.
[Cao et al. 2016b] Cao, Y.; Long, M.; Wang, J.; and Zhu,
H. 2016b. Correlation autoencoder hashing for supervised
cross-modal search. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, 197–204.
ACM.
[Chua et al. 2009] Chua, T.-S.; Tang, J.; Hong, R.; Li, H.;
Luo, Z.; and Zheng, Y. 2009. Nus-wide: a real-world web
image database from national university of singapore. In
Proceedings of the ACM international conference on image
and video retrieval, 48. ACM.
[Ding et al. 2016] Ding, G.; Guo, Y.; Zhou, J.; and Gao, Y.
2016. Large-scale cross-modality search via collective matrix factorization hashing. IEEE Transactions on Image Processing 25(11):5427–5440.
[Gionis, Indyk, and Motwani 1999] Gionis, A.; Indyk, P.;
and Motwani, R. 1999. Similarity search in high dimensions via hashing. In VLDB, volume 99, 518–529.
[Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.;
Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville,
A.; and Bengio, Y. 2014. Generative adversarial nets. In
Advances in neural information processing systems, 2672–
2680.
[Hardoon, Szedmak, and Shawe-Taylor 2004] Hardoon,
D. R.; Szedmak, S.; and Shawe-Taylor, J. 2004. Canonical
correlation analysis: An overview with application to
learning methods. Neural computation 16(12):2639–2664.
[Huiskes and Lew 2008] Huiskes, M. J., and Lew, M. S.
2008. The mir flickr retrieval evaluation. In Proceedings
of the 1st ACM international conference on Multimedia information retrieval, 39–43. ACM.
[Jiang and Li 2017] Jiang, Q.-Y., and Li, W.-J. 2017. Deep
cross-modal hashing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3232–
3240.
[Krizhevsky, Sutskever, and Hinton 2012] Krizhevsky, A.;
Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances
in neural information processing systems, 1097–1105.
[Kumar and Udupa 2011] Kumar, S., and Udupa, R. 2011.
Learning hash functions for cross-view similarity search. In
IJCAI proceedings-international joint conference on artificial intelligence, volume 22, 1360.
[Lin et al. 2015] Lin, Z.; Ding, G.; Hu, M.; and Wang, J.
2015. Semantics-preserving hashing for cross-view retrieval. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 3864–3872.
[Long et al. 2016] Long, M.; Cao, Y.; Wang, J.; and Yu, P. S.
2016. Composite correlation quantization for efficient multimodal retrieval. In Proceedings of the 39th International
ACM SIGIR conference on Research and Development in Information Retrieval, 579–588. ACM.
[Peng et al. 2017] Peng, Y.; Zhu, W.; Zhao, Y.; Xu, C.;
Huang, Q.; Lu, H.; Zheng, Q.; Huang, T.; and Gao, W. 2017.

Cross-media analysis and reasoning: advances and directions. Frontiers of Information Technology & Electronic Engineering 18(1):44–57.
[Peng, Huang, and Zhao 2017] Peng, Y.; Huang, X.; and
Zhao, Y. 2017. An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges. IEEE
Transactions on Circuits and Systems for Video Technology
PP(99):1–14.
[Rastegari et al. 2013] Rastegari, M.; Choi, J.; Fakhraei, S.;
Hal, D.; and Davis, L. 2013. Predictable dual-view hashing. In Proceedings of the 30th International Conference on
Machine Learning (ICML-13), 1328–1336.
[Reed et al. 2016] Reed, S.; Akata, Z.; Yan, X.; Logeswaran,
L.; Schiele, B.; and Lee, H. 2016. Generative adversarial
text to image synthesis. In Proceedings of the 33rd International Conference on International Conference on Machine
Learning-Volume 48, 1060–1069.
[Song et al. 2013] Song, J.; Yang, Y.; Yang, Y.; Huang, Z.;
and Shen, H. T. 2013. Inter-media hashing for large-scale
retrieval from heterogeneous data sources. In Proceedings of
the 2013 ACM SIGMOD International Conference on Management of Data, 785–796. ACM.
[Wang et al. 2016] Wang, J.; Liu, W.; Kumar, S.; and Chang,
S. F. 2016. Learning to hash for indexing big data-a survey.
Proceedings of the IEEE 104(1):34–57.
[Wang et al. 2017] Wang, J.; Yu, L.; Zhang, W.; Gong, Y.;
Xu, Y.; Wang, B.; Zhang, P.; and Zhang, D. 2017. Irgan:
A minimax game for unifying generative and discriminative
information retrieval models. In Proceedings of the 40th
International ACM SIGIR Conference on Research and Development in Information Retrieval, 515–524.
[Wei et al. 2014] Wei, Y.; Song, Y.; Zhen, Y.; Liu, B.; and
Yang, Q. 2014. Scalable heterogeneous translated hashing.
In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 791–800.
ACM.
[Weiss, Torralba, and Fergus 2009] Weiss, Y.; Torralba, A.;
and Fergus, R. 2009. Spectral hashing. In Advances in
neural information processing systems, 1753–1760.
[Zhang and Li 2014] Zhang, D., and Li, W.-J. 2014. Largescale supervised multimodal hashing with semantic correlation maximization. In AAAI, volume 1, 7.
[Zhang and Peng 2017] Zhang, J., and Peng, Y. 2017. Ssdh:
Semi-supervised deep hashing for large scale image retrieval. IEEE Transactions on Circuits and Systems for Video
Technology PP(99):1–14.
[Zhao and Gao 2017] Zhao, Xin, D. G. G. Y. H. J., and Gao,
Y. 2017. Tuch: Turning cross-view hashing into single-view
hashing via generative adversarial nets. In Proceedings of
the Twenty-Sixth International Joint Conference on Artificial
Intelligence, IJCAI-17, 3511–3517.
[Zhuang et al. 2014] Zhuang, Y.; Yu, Z.; Wang, W.; Wu, F.;
Tang, S.; and Shao, J. 2014. Cross-media hashing with neural networks. In Proceedings of the 22nd ACM international
conference on Multimedia, 901–904. ACM.

