1

Compact Hash Code Learning with
Binary Deep Neural Network

arXiv:1712.02956v2 [] 6 Feb 2018

Thanh-Toan Do, Dang-Khoa Le Tan, Tuan Hoang, Ngai-Man Cheung

Abstract—In this work, we firstly propose deep network models
and learning algorithms for learning binary hash codes given
image representations under both unsupervised and supervised
manners. Then, by leveraging the powerful capacity of convolutional neural networks, we propose an end-to-end architecture
which jointly learns to extract visual features and produce
binary hash codes. Our novel network designs constrain one
hidden layer to directly output the binary codes. This addresses
a challenging issue in some previous works: optimizing nonsmooth objective functions due to binarization. Additionally, we
incorporate independence and balance properties in the direct
and strict forms into the learning schemes. Furthermore, we also
include similarity preserving property in our objective functions.
Our resulting optimizations involving these binary, independence,
and balance constraints are difficult to solve. We propose to
attack them with alternating optimization and careful relaxation.
Experimental results on the benchmark datasets show that our
proposed methods compare favorably with the state of the art.

I. I NTRODUCTION
We are interested in learning binary hash codes for visual
search problems. Given an input image treated as a query, the
visual search systems search for visually similar images in a
database. In the state-of-the-art image retrieval systems [1],
[2], [3], [4], images are represented as high-dimensional feature vectors which later can be searched via classical distance
such as the Euclidean or Cosin distance. However, when
the database is scaled up, there are two main requirements
for retrieval systems, i.e., efficient storage and fast search.
Among solutions, binary hashing is an attractive approach for
achieving those requirements [5], [6], [7], [8] due to its fast
computation and efficient storage.
Briefly, in binary hashing problem, each original high
dimensional vector x ∈ RD is mapped into a very compact
binary vector b ∈ {−1, 1}L , where L  D.
Many hashing methods have been proposed in the literature.
They can be divided into two categories: data-independence
methods and data-dependence methods. The former ones [9],
[10], [11], [12] rely on random projections to construct hash
functions. The representative methods in this category are
Locality Sensitive Hashing (LSH) [9] and its kernelized or
discriminative extensions [10], [11]. The latter ones use the
available training data to learn the hash functions in unsupervised [13], [14], [15], [16], [17] or (semi-)supervised [18],
[19], [20], [21], [22], [23], [24] manners. The prepresentative
Thanh-Toan Do is with The University of Adelaide, Australia. Dang-Khoa
Le Tan, Tuan Hoang, Ngai-Man Cheung are with Singapore University of
Technology and Design, Singapore.
Email: thanhtoan do@adelaide.edu.au; {letandang khoa, nguyenanhtuan hoang, ngaiman cheung}@sutd.edu.sg.

unsupervised hashing methods, e.g. Spectral Hashing [13],
Iterative Quantization (ITQ) [14], K-means Hashing [15],
Spherical Hashing [16], try to learn binary codes which
preserve the distance similarity between samples. The prepresentative supervised hashing methods, e.g., ITQ-CCA [14],
Binary Reconstructive Embedding [23], Kernel Supervised
Hashing [19], Two-step Hashing [22], Supervised Discrete
Hashing [24], try to learn binary codes which preserve the
label similarity between samples. The detailed review of
data-independent and data-dependent hashing methods can be
found in the recent surveys [5], [6], [7], [8].
One difficult problem in hashing is to deal with the binary
constraint on the codes. Specifically, the outputs of the hash
functions have to be binary. In general, this binary constraint
leads to an NP-hard mixed-integer optimization problem. To
handle this difficulty, most aforementioned methods relax the
constraint during the learning of hash functions. With this
relaxation, the continuous codes are learned first. Then, the
codes are binarized (e.g., by thresholding). This relaxation
greatly simplifies the original binary constrained problem.
However, the solution can be suboptimal, i.e., the binary codes
resulting from thresholded continuous codes could be inferior
to those that are obtained by directly including the binary
constraint in the learning.
Furthermore, a good hashing method should produce binary
codes with the following properties [13]: (i) similarity preserving, i.e., (dis)similar inputs should likely have (dis)similar
binary codes; (ii) independence, i.e., different bits in the binary
codes are independent to each other so that no redundant
information is captured; (iii) bit balance, i.e., each bit has a
50% chance of being 1 or −1. Note that direct incorporation
of the independent and balance properties can complicate
the learning. Previous works have used some relaxation or
approximation to overcome the difficulties [14], [20], but there
may be some performance degradation.
Recently, the deep learning has given great attention
to the computer vision community due to its superiority
in many vision tasks such as classification, detection,
segmentation [25], [26], [27]. Inspired by the success
of deep learning in different vision tasks, recently, some
researchers have used deep learning for joint learning image
representations and binary hash codes in an end-to-end deep
learning-based supervised hashing framework [28], [29],
[30], [31]. However, learning binary codes in deep networks
is challenging. This is because one has to deal with the
binary constraint on the hash codes, i.e., the final network
outputs must be binary. A naive solution is to adopt the sign
activation layer to produce binary codes. However, due to the

2

non-smoothness of the sign function, it causes the vanishing
gradient problem when training the network with the standard
back propagation [32].
Contributions: In this work, firstly, we propose a novel
deep network model and a learning algorithm for unsupervised hashing. In order to achieve binary codes, instead of
involving the sgn or step function as in recent works [33],
[34], our proposed network design constrains one layer to
directly output the binary codes (therefore, the network is
named as Binary Deep Neural Network). In addition, we
propose to directly incorporate the independence and balance
properties. Furthermore, we include the similarity preserving
in our objective function. The resulting optimization with these
binary and direct constraints is NP-hard. We propose to attack
this challenging problem with alternating optimization and
careful relaxation. Secondly, in order to enhance the discriminative power of the binary codes, we extend our method
to supervised hashing by leveraging the label information so
that the binary codes preserve the semantic similarity between
samples. Finally, to demonstrate the flexibility of our proposed
method and to leverage the powerful capacity of convolution
deep neural networks, we adapt our optimization strategy and
the proposed supervised hashing model to an end-to-end deep
hashing network framework. The solid experiments on various
benchmark datasets show the improvements of the proposed
methods over state-of-the-art hashing methods.
A preliminary version of this work has been reported
in [35]. In this work, we present substantial extension to our
previous work. In particular, the main extension is that we
propose the end-to-end binary deep neural network framework
which jointly learns the image features and the binary codes.
The experimental results show that the proposed end-to-end
hashing framework significantly boosts the retrieval accuracy.
Other minor extensions are we conduct more experiments
(e.g., new experiments on SUN397 dataset [36], comparison
to state-of-the-art end-to-end hashing methods) to evaluate the
effectiveness of the proposed methods.
The remaining of this paper is organized as follows.
Section II presents related works. Section III presents and
evaluates the proposed unsupervised hashing method. Section IV presents and evaluates the proposed supervised hashing
method. Section V presents and evaluates the proposed endto-end deep hashing network. Section VI concludes the paper.
II. R ELATED WORK
Our work is inspired by the recent successful hashing
methods which define hash functions as a neural network [37],
[33], [38], [34], [28], [29], [30]. We propose an improved
design to address their limitations. In Semantic Hashing [37],
the model is formed by a stack of Restricted Boltzmann
Machine, and a pretraining step is required. Additionally, this
model does not consider the independence and balance of
the codes. In Binary Autoencoder [34], a linear autoencoder
is used as hash functions. As this model only uses one
hidden layer, it may not well capture the information of
inputs. Extending [34] with multiple, nonlinear layers is not

straight-forward because of the binary constraint. They also
do not consider the independence and balance of codes. In
Deep Hashing [33], [38], a deep neural network is used as
a hash function. However, this model does not fully take
into account the similarity preserving. They also apply some
relaxation in arriving the independence and balance of codes
and this may degrade the performance. In the end-to-end deep
hashing [29], [30], the independence and balance of codes are
not considered.
In order to handle the binary constraint, Semantic Hashing [37] first solves the relaxed problem by discarding the
constraint and then thresholds the solved continuous solution.
In Deep Hashing (DH) [33], [38], the output of the last
layer, H(n) , is binarized by the sgn function. They include
a term in the objective function
to reduce this binarization

loss: sgn(H(n) ) − H(n) . Solving the objective function
of DH [33], [38] is difficult because the sgn function is
non-differentiable. The authors in [33], [38] overcome this
difficulty by assuming that the sgn function is differentiable
everywhere. In Binary Autoencoder (BA) [34], the outputs of
the hidden layer are passed into a step function to binarize the
codes. Incorporating the step function in the learning leads
to a non-smooth objective function, hence the optimization is
NP-complete. To handle this difficulty, the authors [34] use
binary SVMs to learn the model parameters in the case when
there is only a single hidden layer.
Joint learning image representations and binary hash codes
in an end-to-end deep learning-based supervised hashing
framework [28], [29], [30] has shown considerable boost in
retrieval accuracy. By joint optimization, the produced hash
codes are more sufficient to preserve the semantic similarity
between images. In those works, the network architectures
often consist of a feature extraction sub-network and a subsequent hashing layer to produce hash codes. Ideally, the
hashing layer should adopt a sign activation function to output
exactly binary codes. However, due to the vanishing gradient
difficulty of the sign function, an approximation procedure
must be employed. For example, sign can be approximated
by a tanh-like function y = tanh(βx), where β is a free
parameter controlling the trade off between the smoothness
and the binary quantization loss [29]. However, it is nontrivial to determine an optimal β. A small β causes large
binary quantization loss while a big β makes the output of
the function close to the binary values, but the gradient of the
function almost vanishes, making back-propagation infeasible.
The problem still remains when the logistic-like functions [28],
[30] are used.
III. U NSUPERVISED H ASHING WITH B INARY D EEP
N EURAL N ETWORK (UH-BDNN)
A. Formulation of UH-BDNN
For easy following, we first summarize the notations in
Table I. In our work, the hash functions are defined by a
deep neural network. In our proposed architecture, we use
different activation functions in different layers. Specifically,
we use the sigmoid function as the activation function for
layers 2, · · · , n − 2, and the identity function as the activation

3

codes. The second term is a regularization that tends to decrease the magnitude of the weights, and this helps to prevent
overfitting. Note that in our proposed design, we constrain the
network to directly output the binary codes at one layer, which
avoids the difficulty of the sgn / step function which is nondifferentiability. On the other hand, our formulation with (1)
under the binary constraint (2) is very difficult to solve. It is a
mixed-integer problem which is NP-hard. In order to attack the
problem, we propose to introduce an auxiliary variable B and
use alternating optimization. Consequently, we reformulate the
objective function (1) under constraint (2) as the following

TABLE I
N OTATIONS AND THEIR CORRESPONDING MEANINGS .
Notation
X

Meaning
D×m : set of m training samples;
X = {xi }m
i=1 ∈ R
each column of X corresponds to one sample
L×m : binary code of X
B = {bi }m
i=1 ∈ {−1, +1}
Number of required bits to encode a sample
Number of layers (including input and output layers)
Number of units in layer l
Activation function of layer l
W(l) ∈ Rsl+1 ×sl : weight matrix connecting layer l + 1
and layer l
c(l) ∈ Rsl+1 :bias vector for units in layer l + 1
H(l) = f (l) W(l−1) H(l−1) + c(l−1) 11×m :
output values of layer l; convention: H(1) = X
Matrix has a rows, b columns and all elements equal to 1

B
L
n
sl
f (l)
W(l)
c(l)
H(l)
1a×b

min J

W,c,B

=

1
X − W(n−1) B − c(n−1) 11×m
2m
+

Layer 1
(input layer)

Layer 2

Layer 𝑛 − 2

𝑾(1)

Layer 𝑛 − 1
Output 𝑯(𝑛−1) ∈
−1,1 and is used as
binary code

𝑾(𝑛−2)

Layer 𝑛
(reconstruction layer)
Output 𝐇 (n) ≈ 𝑿

𝑾(𝑛−1)

W,c



1
X − W(n−1) H(n−1) + c(n−1) 11×m
2m
+

n−1
λ1 X
W(l)
2 l=1

2

2

s.t. H(n−1) ∈ {−1, 1}L×m

(3)

(4)

B ∈ {−1, 1}L×m .

function for layer n − 1 and layer n. Our idea is to learn
the network such that the output values of the penultimate
layer (layer n − 1) can be used as the binary codes. We
introduce constraints in the learning algorithm such that the
output values at the layer n − 1 have the following desirable
properties: (i) belonging to {−1, 1}; (ii) similarity preserving;
(iii) independence and (iv) balancing. Fig. 1 illustrates our
network for the case D = 4, L = 2.
Let us start with first two properties of the codes, i.e.,
belonging to {−1, 1} and similarity preserving. To achieve
the binary codes having these two properties, we propose to
optimize the following constrained objective function
=

2

s.t. B = H(n−1) ,

Fig. 1. The illustration of our UH-BDNN (D = 4, L = 2). In our proposed
network design, the outputs of layer n − 1 are constrained to {−1, 1} and are
used as the binary codes. During training, these codes are used to reconstruct
the input samples at the final layer.

min J

n−1
λ1 X
W(l)
2 l=1

2

(1)

(2)

The constraint (2) is to ensure the first property. As the
activation function for the last layer is the identity function,
the term W(n−1) H(n−1) + c(n−1) 11×m is the output of
the last layer. The first term of (1) makes sure that the
binary code gives a good reconstruction of X. It is worth
noting that the reconstruction criterion has been used as an
indirect approach for preserving the similarity in state-ofthe-art unsupervised hashing methods [14], [34], [37], i.e.,
it encourages (dis)similar inputs map to (dis)similar binary

(5)

The benefit of introducing the auxiliary variable B is that
we can decompose the difficult constrained optimization problem (1) into two sub-optimization problems. Consequently, we
are able to iteratively solve the optimization problem by using
alternating optimization with respect to (W, c) and B while
holding the other fixed. Inspired from the quadratic penalty
method [39], we relax the equality constraint (4) by converting
it into a penalty term. We achieve the following constrained
objective function
min J

=

W,c,B

1
X − W(n−1) B − c(n−1) 11×m
2m
+

n−1
λ1 X
W(l)
2 l=1

2

+

2

λ2
H(n−1) − B
2m

2

(6)

s.t. B ∈ {−1, 1}L×m ,

(7)

in which, the third term in (6) measures the (equality) constraint violation. By setting the penalty parameter λ2 sufficiently large, we penalize the constraint violation severely,
thereby forcing the minimizer of the penalty function (6) closer
to the feasible region of the original constrained function (3).
We now consider the two remaining properties of the codes,
i.e., independence and balance. Unlike previous works which
use some relaxation or approximation on the independence and
balance properties [14], [33], [20], we propose to encode these
properties strictly and directly based on the binary outputs of
our layer n − 1. Specifically, we encode the independence
and balance properties of the codes by introducing the fourth
and the fifth terms respectively in the following constrained
objective function
min J

W,c,B

=

1
X − W(n−1) B − c(n−1) 11×m
2m

+

n−1
λ1 X
W(l)
2 l=1

+

λ3
2

2

+

λ2
H(n−1) − B
2m

1 (n−1) (n−1) T
H
(H
) −I
m
s.t. B ∈ {−1, 1}L×m .

2

+

2

2

λ4
H(n−1) 1m×1
2m

2

(8)
(9)

4

The objective function (8) under constraint (9) is our final
formulation. Before discussing how to solve it, let us present
the differences between our work and the recent deep learning
based hashing models Deep Hashing [33], Binary Autoencoder [34], and end-to-end hashing methods [28], [29], [30].
The main important difference between our model and other
deep learning-based hashing methods Deep Hashing [33],
Binary Autoencoder [34], end-to-end hashing [28], [29], [30]
is the way to achieve the binary codes. Instead of involving
the sgn, step function as in [33], [34] or the relaxation of sgn
as sigmoid [28], [30], tanh [29], we constrain the network to
directly output the binary codes at one layer. Other differences
to the most related works Deep Hashing [33] and Binary
Autoencoder [34] are presented as follows.
a) Comparison to Deep Hashing (DH) [33], [38]: the
deep model of DH is learned by the following objective
function
α1  (n) (n) T 
tr H (H )
W,c
2m

n−1
n−1 
2
2
2
α3 X
α2 X
W(l) (W(l) )T − I +
W(l) + c(l)
+
2 l=1
2 l=1
min J

=

1
sgn(H(n) ) − H(n)
2

2

−

The DH’s model does not have the reconstruction layer. The
authors apply the sgn function to the outputs at the top layer
of the network to obtain the binary codes. The first term aims
to minimize quantization loss by applying the sgn function to
the outputs at the top layer. The balancing and the independent
properties are presented in the second and the third terms. It
is worth noting that minimizing DH’s objective function is
difficult due to the non-differentiability of the sgn function.
The authors work around this difficulty by assuming that the
sgn function is differentiable everywhere.
Contrary to DH, we propose a different model design. In
particular, our model encourages the similarity preserving by
obtaining the reconstruction layer in the network.For the balancing property, they maximize tr H(n) (H(n) )T . According
to [20], maximizing this term is only an approximation in
arriving the balancing property. In our objective function,
the balancing property is directly enforced on the codes
2
by the term H(n−1) 1m×1 . For the independent property,
DH uses a relaxed orthogonality constraint on the network
2
weights W, i.e., W(l) (W(l) )T − I . On the contrary, we
(once again) directly constrain the code independence using
2
1
(n−1)
(H(n−1) )T − I . Incorporating the direct conmH
straints can lead to better performance.
b) Comparison to Binary Autoencoder (BA) [34]: the
differences between our model and BA are quite clear. Firstly,
BA, as described in [34], is a shallow linear autoencoder
network with one hidden layer. Secondly, the BA’s hash
function is a linear transformation of the input followed by the
step function to obtain the binary codes. In BA, by treating
the encoder layer as binary classifiers, they use binary SVMs
to learn the weights of the linear transformation. On the
contrary, our hash function is defined by multiple, hierarchical
layers of nonlinear and linear transformations. It is not clear
if the binary SVMs approach in BA can be used to learn the
weights in our deep architecture with multiple layers. Instead,

we use alternating optimization to derive a backpropagation
algorithm to learn the weights in all layers. Additionally,
another difference is that our model ensures the independence
and balance of the binary codes while BA does not. Note
that independence and balance properties may not be easily
incorporated in their framework, as these would complicate
their objective function and the optimization problem may
become very difficult to solve.
B. Optimization
In order to solve (8) under constraint (9), we propose to use
alternating optimization over (W, c) and B.
1) (W, c) step: When fixing B, the problem becomes the
unconstrained optimization. We use L-BFGS [40] optimizer
with backpropagation for solving. The gradients of the objective function J (8) w.r.t. different parameters are computed as
follows.
At l = n − 1, we have
−1
∂J
(X − W(n−1) B − c(n−1) 11×m )BT + λ1 W(n−1)
=
m
∂W(n−1)
(10)

∂J
−1 
(n−1)
(n−1)
(X
−
W
B)1
−
mc
=
(11)
m×1
m
∂c(n−1)

For other layers, let us define

∆(l)




λ2  (n−1)
H
−B
m


2λ3
1 (n−1) (n−1) T
+
H
(H
) − I H(n−1)
m
m

0
λ4  (n−1)
H
1m×m
f (n−1) (Z(n−1) ) (12)
+
m


0
= (W(l) )T ∆(l+1)
f (l) (Z(l) ), ∀l = n − 2, · · · , 2
(13)

∆(n−1)

=

where Z(l) = W(l−1) H(l−1) + c(l−1) 11×m , l = 2, · · · , n;
denotes the Hadamard product.

Then, ∀l = n − 2, · · · , 1, we have
∂J
= ∆(l+1) (H(l) )T + λ1 W(l)
∂W(l)

(14)

∂J
= ∆(l+1) 1m×1
∂c(l)

(15)

2) B step: When fixing (W, c), we can rewrite problem (8)
as
min J
B

=

X − W(n−1) B − c(n−1) 11×m
+λ2 H(n−1) − B

2

2

s.t. B ∈ {−1, 1}L×m .

(16)
(17)

We adaptively use the recent method discrete cyclic coordinate
descent [24] to iteratively solve B, i.e., row by row. The
advantage of this method is that if we fix L − 1 rows of
B and only solve for the remaining row, we can achieve a
closed-form solution for that row.
Let V = X−c(n−1) 11×m ; Q = (W(n−1) )T V+λ2 H(n−1) .
For k = 1, · · · L, let wk be k th column of W(n−1) ; W1 be
matrix W(n−1) excluding wk ; qk be k th column of QT ; bTk
be k th row of B; B1 be matrix of B excluding bTk . We have
closed-form for bTk as
T
T
bT
k = sgn(q − wk W1 B1 ).

(18)

5

Algorithm 1 Unsupervised Hashing with Binary Deep Neural
Network (UH-BDNN)
Input:
D×m : training data; L: code length; T : maximum
X = {xi }m
i=1 ∈ R
iteration number; n: number of layers; {sl }n
l=2 : number of units of layers
2 → n (note: sn−1 = L, sn = D); λ1 , λ2 , λ3 , λ4 .
Output:
Parameters {W(l) , c(l) }n−1
l=1
1: Initialize B(0) ∈ {−1, 1}L×m using ITQ [14]
(l) n−2
2: Initialize {c(l) }n−1
l=1 = 0sl+1 ×1 . Initialize {W }l=1 by getting the
top sl+1 eigenvectors from the covariance matrix of H(l) . Initialize
W(n−1) = ID×L
3: Fix B(0) , compute (W, c)(0) with (W, c) step using initialized
{W(l) , c(l) }n−1
l=1 (line 2) as starting point for L-BFGS.
4: for t = 1 → T do
5:
Fix (W, c)(t−1) , compute B(t) with B step
6:
Fix B(t) , compute (W, c)(t) with (W, c) step using (W, c)(t−1)
as starting point for L-BFGS.
7: end for
8: Return (W, c)(T )

The proposed UH-BDNN method is summarized in Algorithm 1. In the Algorithm 1, B(t) and (W, c)(t) are values
of B and {W(l) , c(l) }n−1
l=1 at iteration t, respectively.
C. Evaluation of Unsupervised Hashing with Binary Deep
Neural Network (UH-BDNN)
This section evaluates the proposed UH-BDNN and compares it with the following state-of-the-art unsupervised hashing methods: Spectral Hashing (SH) [13], Iterative Quantization (ITQ) [14], Binary Autoencoder (BA) [34], Spherical
Hashing (SPH) [16], K-means Hashing (KMH) [15]. For
all compared methods, we use the implementations and the
suggested parameters provided by the authors.
1) Dataset, evaluation protocol, and implementation notes:
a) Dataset: CIFAR10 [41] dataset consists of 60,000
images of 10 classes. The training set (also used as the
database for the retrieval) contains 50,000 images. The query
set contains 10,000 images. Each image is represented by a
800-dimensional feature vector extracted by PCA from 4096dimensional CNN feature produced by the AlexNet [25].
MNIST [42] dataset consists of 70,000 handwritten digit
images of 10 classes. The training set (also used as database
for retrieval) contains 60,000 images. The query set contains
10,000 images. Each image is represented by a 784 dimensional gray-scale feature vector by using its intensity.
SIFT1M [43] dataset contains 128 dimensional SIFT vectors [44]. There are 1M vectors used as database for retrieval;
100K vectors for training (separated from retrieval database)
and 10K vectors for query.
b) Evaluation protocol: We follow the standard setting in
unsupervised hashing methods [14], [16], [15], [34] which use
Euclidean nearest neighbors as the ground truths for queries.
The number of ground truths are set as in [34], i.e., for
CIFAR10 and MNIST datasets, for each query, we use 50 its
Euclidean nearest neighbors as the ground truths; for large
scale dataset SIFT1M, for each query, we use 10, 000 its
Euclidean nearest neighbors as the ground truths. We use the

following evaluation metrics which have been used in the
state of the art [14], [34], [33] to measure the performance
of methods. 1) mean Average Precision (mAP); 2) precision
of Hamming radius 2 (precision@2) which measures precision
on retrieved images having Hamming distance to query ≤ 2
(if no images satisfy, we report zero precision). Note that as
computing mAP is slow on large dataset SIFT1M, we consider
top 10, 000 returned neighbors when computing mAP.
c) Implementation notes: In our deep model, we use n =
5 layers. The parameters λ1 , λ2 , λ3 and λ4 are empirically
set by cross validation as 10−5 , 5 × 10−2 , 10−2 and 10−6 ,
respectively. The max iteration number T is empirically set to
10. The number of units in hidden layers 2, 3, 4 are empirically
set as [90 → 20 → 8], [90 → 30 → 16], [100 → 40 → 24]
and [120 → 50 → 32] for the code length 8, 16, 24 and 32
bits, respectively.
2) Retrieval results: Fig. 2 and Table II show comparative
mAP and precision of Hamming radius 2 (precision@2) of
methods, respectively.
We find the following observations are consistent for all
three datasets. In term of mAP, the proposed UH-BDNN
is comparable to or outperforms other methods at all code
lengths. The improvement is more clear at high code length,
i.e., L = 24, 32. The best competitor to UH-BDNN is
binary autoencoder (BA) [34] which is the current stateof-the-art unsupervised hashing method. Compare to BA,
at high code length, UH-BDNN consistently outperforms
BA on all datasets. In term of precision@2, UH-BDNN
is comparable to other methods at low code lengths, i.e.,
L = 8, 16. At L = 24, 32, UH-BDNN significantly achieve
better performance than other methods. Specifically, the
improvements of UH-BDNN over the best competitor
BA [34] are clearly observed at L = 32 on the MNIST and
SIFT1M datasets.
Comparison with Deep Hashing (DH) [33], [38] As
the implementation of DH is not available, we set up the
experiments on CIFAR10 and MNIST similar to [33] to make
a fair comparison. For each dataset, we randomly sample 1,000
images (i.e., 100 images per class) as query set; the remaining
images are used as training and database set. Follow [33],
for CIFAR10, each image is represented by 512-D GIST
descriptor [45].
Follow Deep Hashing [33], the ground truths of queries
are based on their class labels1 . Similar to [33], we report
comparative results in term of mAP and the precision of
Hamming radius r = 2. The results are presented in the
Table III. It is clearly showed that the proposed UH-BDNN
outperforms DH [33] at all code lengths, in both mAP and
precision of Hamming radius.
IV. S UPERVISED H ASHING WITH B INARY D EEP N EURAL
N ETWORK (SH-BDNN)
In order to enhance the discriminative power of the binary codes, we extend UH-BDNN to supervised hashing by
1 It is worth noting that in the evaluation of unsupervised hashing, instead
of using class label as ground truths, most state-of-the-art methods [14], [16],
[15], [34] use Euclidean nearest neighbors as ground truths for queries.

6

mAP

8

25
UH−BDNN
BA
ITQ
SH
SPH
KMH

20

mAP

10

6

15

25
UH−BDNN
BA
ITQ
SH
SPH
KMH

20

mAP

12

10

15

UH−BDNN
BA
ITQ
SH
SPH
KMH

10

4
5

2
0
8

16

24

0
8

32

5

16

24

0
8

32

16

24

32

number bits (L)

number bits (L)

number bits (L)

(a) CIFAR10

(b) MNIST

(c) SIFT1M

Fig. 2. mAP comparison between UH-BDNN and state-of-the-art unsupervised hashing methods on CIFAR10, MNIST, and SIFT1M.
TABLE II
P RECISION AT H AMMING DISTANCE r = 2 COMPARISON BETWEEN UH-BDNN AND STATE - OF - THE - ART UNSUPERVISED HASHING METHODS ON
CIFAR10, MNIST, AND SIFT1M.

L
UH-BDNN
BA[34]
ITQ[14]
SH[13]
SPH[16]
KMH[15]

8
0.55
0.55
0.54
0.39
0.43
0.53

CIFAR10
16
24
5.79
22.14
5.65
20.23
5.05
18.82
4.23
14.60
3.45
13.47
5.49
19.55

32
18.35
17.00
17.76
15.22
13.67
15.90

8
0.53
0.51
0.51
0.43
0.44
0.50

MNIST
16
24
6.80
29.38
6.44
27.65
5.87
23.92
6.50
27.08
5.02
22.24
6.36
25.68

32
38.50
35.29
36.35
36.69
30.80
36.24

8
4.80
3.85
3.19
4.67
4.25
3.74

SIFT1M
16
24
25.20
62.20
23.19
61.35
14.07
35.80
24.82
60.25
20.98
47.09
20.74
48.86

32
80.55
77.15
58.69
72.40
66.42
76.04

TABLE III
C OMPARISON WITH D EEP H ASHING (DH) [33].
CIFAR10
precision@2
32
16
32
16.62
23.33
15.77
18.52
24.97
18.85

mAP
L
DH [33]
UH-BDNN

16
16.17
17.83

leveraging the label information. There are several approaches
proposed to leverage the label information, leading to different
criteria on binary codes. In [18], [46], binary codes are
learned such that they minimize Hamming distances between
samples belonging to the same class, while maximizing the
Hamming distance between samples belonging to different
classes. In [24], the binary codes are learned such that they
are optimal for linear classification.
In this work, in order to exploit the label information,
we follow the approach proposed in Kernel-based Supervised
Hashing (KSH) [19]. The benefit of this approach is that it
directly encourages the Hamming distances between binary
codes of within-class samples equal to 0, and the Hamming
distances between binary codes of between-class samples
equal to L. In the other words, it tries to perfectly preserve
the semantic similarity. To achieve this goal, it enforces the
Hamming distances between learned binary codes to be highly
correlated with the pre-computed pairwise label matrix.
In general, the network structure of SH-BDNN is similar
to UH-BDNN, excepting that the last layer preserving the
reconstruction of UH-BDNN is removed. The layer n − 1 in
UH-BDNN becomes the last layer in SH-BDNN. All desirable
properties, i.e., semantic similarity preserving, independence,
and balance, in SH-BDNN are constrained on the outputs of
its last layer.

MNIST
precision@2
32
16
32
44.97
66.10
73.29
47.21
69.13
75.26

mAP
16
43.14
45.38

A. Formulation of SH-BDNN
We define the pairwise label matrix S as


1
−1

Sij =

if xi and xj are same class
if xi and xj are not same class

(19)

In order to achieve the semantic similarity preserving property, we learn the binary codes such that the Hamming
distance between learned binary codes highly correlates
with the matrix S, i.e., we want to minimize the quantity
2
1
(n) T
) H(n) − S . In addition, to achieve the indepenL (H
dence and balance properties of codes, we want to minimize
2
2
1
the quantities m
H(n) (H(n) )T − I and H(n) 1m×1 , respectively.
Follow the same reformulation and relaxation as UH-BDNN
(Sec. III-A), we solve the following constrained optimization
which ensures the binary constraint, the semantic similarity
preserving, the independence, and the balance properties of
codes.
min J

W,c,B

=

1
2m

1
(H(n) )T H(n) − S
L

2
λ2
λ3
H(n) − B +
2m
2
2
λ4
(n)
+
H 1m×1
2m

+

s.t. B ∈ {−1, 1}L×m

2

+

n−1
λ1 X
W(l)
2 l=1

2

1 (n) (n) T
H (H ) − I
m

2

(20)
(21)

7

Algorithm 2 Supervised Hashing with Binary Deep Neural
Network (SH-BDNN)
Input:
D×m : labeled training data; L: code length; T :
X = {xi }m
i=1 ∈ R
maximum iteration number; n: number of layers; {sl }n
l=2 : number of
units of layers 2 → n (note: sn = L); λ1 , λ2 , λ3 , λ4 .
Output:
Parameters {W(l) , c(l) }n−1
l=1

It is easy to see that the optimal solution for (26) under
constraint (27) is B = sgn(H(n) ).
The proposed SH-BDNN method is summarized in Algorithm 2. In Algorithm 2, B(t) and (W, c)(t) are values of B
and {W(l) , c(l) }n−1
l=1 at iteration t, respectively.

1: Compute pairwise label matrix S using (19).
2: Initialize B(0) ∈ {−1, 1}L×m using ITQ [14]
(l) n−1
3: Initialize {c(l) }n−1
l=1 = 0sl+1 ×1 . Initialize {W }l=1 by getting the
top sl+1 eigenvectors from the covariance matrix of H(l) .
4: Fix B(0) , compute (W, c)(0) with (W, c) step using initialized
{W(l) , c(l) }n−1
l=1 (line 3) as starting point for L-BFGS.
5: for t = 1 → T do
6:
Fix (W, c)(t−1) , compute B(t) with B step
7:
Fix B(t) , compute (W, c)(t) with (W, c) step using (W, c)(t−1)
as starting point for L-BFGS.
8: end for
9: Return (W, c)(T )

C. Evaluation of Supervised Hashing with Binary Deep Neural Network

(20) under constraint (21) is our formulation for supervised
hashing. The main difference in formulation between UHBDNN (8) and SH-BDNN (20) is that the reconstruction
term preserving the neighbor similarity in UH-BDNN (8) is
replaced by the term preserving the label similarity in SHBDNN (20).

a) Dataset: We evaluate and compare methods on
CIFAR-10 and MNIST, and SUN397 datasets. The descriptions of the first two datasets are presented in section III-C1.
SUN397 [36] dataset contains about 108K images from 397
scene categories. We use a subset of this dataset including
42 largest categories in which each category contains more
than 500 images. This results about 35K images in total.
The query set contains 4,200 images (100 images per class)
randomly sampled from the dataset. The rest images are used
as the training set and also the database set. Each image is
represented by a 800-dimensional feature vector extracted by
PCA from 4096-dimensional CNN feature produced by the
AlexNet [25].
b) Evaluation protocol: Follow the literature [24], [14],
[19], we report the retrieval results in two metrics: 1) mean
Average Precision (mAP) and 2) precision of Hamming radius
2 (precision@2).
c) Implementation notes: The network configuration is
same as UH-BDNN except the final layer is removed. The
values of parameters λ1 , λ2 , λ3 , and λ4 are empirically set
using cross validation as 10−3 , 5, 1, and 10−4 , respectively.
The max iteration number T is empirically set to 5.
Follow the settings in ITQ-CCA [14], SDH [24], all training
samples are used in the learning for these two methods.
For SH-BDNN, KSH [19] and BRE [23] where the label
information is leveraged by the pairwise label matrix, we
randomly select 3, 000 training samples from each class and
use them for learning. The ground truths of queries are defined
by the class labels from the datasets.
2) Retrieval results: Fig. 3 and Table IV show comparative
results between the proposed SH-BDNN and other supervised
hashing methods on CIFAR10, MNIST, and SUN397 datasets.
On the CIFAR10 dataset, Fig. 3(a) and Table IV clearly
show that the proposed SH-BDNN outperforms all compared
methods by a fair margin at all code lengths in both mAP and
precision@2. The best competitor to SH-BDNN on this dataset
is CCA-ITQ [14]. The more improvements of SH-BDNN over
CCA-ITQ are observed at high code lengths, i.e., SH-BDNN
outperforms CCA-ITQ about 4% at L = 24 and L = 32.

B. Optimization
In order to solve (20) under constraint (21), we use alternating optimization, which comprises two steps over (W, c)
and B.
1) (W, c) step: When fixing B, (20) becomes unconstrained optimization. We used L-BFGS [40] optimizer with
backpropagation for solving. The gradients of objective function J (20) w.r.t. different parameters are computed as follows.
Let us define
∆(n)

where V =
∆(l)


=



 λ 
1
2
H(n) V + VT +
H(n) − B
mL
m


2λ3
1 (n) (n) T
H (H ) − I H(n)
+
m
m

0
λ4  (n)
+
H 1m×m
f (n) (Z(n) )
m

1
(H(n) )T H(n)
L

(22)

− S.



= (W(l) )T ∆(l+1)

0

f (l) (Z(l) ), ∀l = n − 1, · · · , 2

(23)

where Z(l) = W(l−1) H(l−1) + c(l−1) 11×m , l = 2, · · · , n;
denotes the Hadamard product.

Then ∀l = n − 1, · · · , 1, we have
∂J
= ∆(l+1) (H(l) )T + λ1 W(l)
∂W(l)

(24)

∂J
= ∆(l+1) 1m×1
∂c(l)

(25)

2) B step: When fixing (W, c), we can rewrite problem (20) as
min J = H(n) − B

2

B

s.t. B ∈ {−1, 1}L×m

(26)
(27)

This section evaluates our proposed SH-BDNN and compares it to the following state-of-the-art supervised hashing
methods including Supervised Discrete Hashing (SDH) [24],
ITQ-CCA [14], Kernel-based Supervised Hashing (KSH) [19],
Binary Reconstructive Embedding (BRE) [23]. For all compared methods, we use the implementations and the suggested
parameters provided by the authors.
1) Dataset, evaluation protocol, and implementation notes:

8

SH−BDNN
SDH
ITQ−CCA
KSH
BRE

50

40

mAP

mAP

60

100

70

90

60

80

50

mAP

70

70

20
8

SH−BDNN
SDH
ITQ−CCA
KSH
BRE

50

16

24

32

40
8

number bits (L)

40
30

60
30

SH-BDNN
SDH
ITQ-CCA
KSH
BRE

16

24

20
10
8

32

(a) CIFAR10

16

24

32

number bits (L)

number bits (L)
(b) MNIST

(c) SUN397

Fig. 3. mAP comparison between SH-BDNN and state-of-the-art supervised hashing methods on CIFAR10, MNIST and SUN397 datasets.
TABLE IV
P RECISION AT H AMMING DISTANCE r = 2 COMPARISON BETWEEN SH-BDNN AND STATE - OF - THE - ART SUPERVISED HASHING METHODS ON CIFAR10,
MNIST, AND SUN397 DATASETS .

L
SH-BDNN
SDH[24]
ITQ-CCA[14]
KSH[19]
BRE[23]

8
54.12
31.60
49.14
44.81
23.84

CIFAR10
16
24
67.32
69.36
62.23
67.65
65.68
67.47
64.08
67.01
41.11
47.98

32
69.62
67.63
67.19
65.76
44.89

8
84.26
36.49
54.35
68.07
37.67

DR Layer
Binary Optimizer

AlexNet

Fig. 4. The illustration of the end-to-end hashing framework (E2E-BDNN).

On the MNIST dataset, Fig. 3(b) and Table IV show that the
proposed SH-BDNN significant outperforms the current stateof-the-art SDH [24] at low code length, i.e., L = 8. When
L increases, SH-BDNN and SDH [24] achieve similar performance. SH-BDNN significantly outperforms other methods,
i.e., KSH [19], ITQ-CCA [14], BRE [23], on both mAP and
precision@2.
On the SUN397 dataset, the proposed SH-BDNN outperforms other competitors at all code lengths in terms of both
mAP and precision@2. The best competitor to SH-BDNN
on this dataset is SDH [24]. At high code lengths (e.g.,
L = 24, 32), SH-BDNN achieves more improvements over
SDH.
V. S UPERVISED H ASHING WITH E ND - TO -E ND B INARY
D EEP N EURAL N ETWORK (E2E-BDNN)
Even though the proposed SH-BDNN can significantly
enhance the discriminative power of the binary codes, similar
to other hashing methods, its capability is partially depended
on the discriminative power of image features. The recent endto-end deep learning-based supervised hashing methods [28],

MNIST
16
24
94.67
94.69
93.00
93.98
79.99
84.12
90.79
92.86
69.80
83.24

32
95.51
94.43
84.57
92.41
84.61

8
15.52
13.89
13.22
12.64
9.26

SUN397
16
24
41.98
52.53
40.39
49.54
37.53
50.07
40.67
49.29
26.95
38.36

32
56.82
53.25
53.12
46.45
40.36

[29], [30] have shown that joint learning image representations
and binary hash codes in an end-to-end fashion can boost the
retrieval accuracy. Therefore, in this section, we propose to
extend the proposed SH-BDNN to an end-to-end framework.
Specifically, we integrate the convolutional neural network
(CNN) with our supervised hashing network (SH-BDNN) into
a unified end-to-end deep architecture, namely End-to-End
Binary Deep Neural Network (E2E-BDNN), which can jointly
learn visual features and binary representations of images.
In the following, we first introduce our proposed network
architecture. We then describe the training process. Finally,
we present experiments on various benchmark datasets.
A. Network architecture
Fig. 4 illustrates the overall architecture of the end-toend binary deep neural network – E2E-BDNN. In details,
the network consists of three main components: (i) a feature
extractor, (ii) a dimensional reduction layer, and (iii) a binary
optimizer component. We utilize AlexNet [25] as the feature
extractor component of the E2E-BDNN. In our configuration,
we remove the last layer of AlexNet, namely the softmax
layer, and consider its last fully connected layer (fc7) as the
image representation.
The dimensional reduction component (the DR layer) involves a fully connected layer for reducing the high dimensional image representations outputted by the feature extractor
component into lower dimensional representations. We use the
identity function as the activation function for this DR layer.
The reduced representations are then used as inputs for the
following binary optimizer component.
The binary optimizer component of E2E-BDNN is similar
to SH-BDNN. It means that we also constrain the output
codes of E2E-BDNN to be binary. These codes also have
desired properties such as semantic similarity preserving,

9

Algorithm 3 End-to-End Binary Deep Neural Network (E2EBDNN) Learning
Input:
X = {xi }n
i=1 : labeled training images; m: minibatch size; L: code
length; K, T : maximum iteration. λ1 , λ2 , λ3 , λ4 : hyperparameters.
Output:
Network parameters W
(0)

1: Initialize the network W(0)
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:

Initialize B(0) ∈ {−1, 1}L×n via ITQ [14]
for k = 1 → K do
for t = 1 → T do
A minibatch X(t) is sampled from X
Compute the corresponding similarity matrix S(t)
From B(k−1) , sample B(t) corresponding to X(t)
(k)
Fix B(t) , optimize W(t) via SGD
end for
(k)
Update B(k) by W(T )
end for
(K)
Return W(T )

WP CA = [v1T v2T · · · vkT ]
cP CA = −WP CA m

independence, and balance. By using the same design as SHBDNN for the last component of E2E-BDNN, it allows us
to observe the advantages of the end-to-end architecture over
SH-BDNN.
The training data for the E2E-BDNN is labelled images,
which contrasts to SH-BDNN which uses visual features such
as GIST [45], SIFT [44] or deep features from convolutional
deep networks. Given the input labeled images, we aim to
learn binary codes with aforementioned desired properties, i.e.,
semantic similarity preserving, independence, and balance.
In order to achieve these properties on the codes, we use
the similar objective function as SH-BDNN. However, it is
important to mention that in SH-BDNN, by its non end-to-end
architecture, we can feed the whole training set to the network
at a time for training. This does not hold for E2E-BDNN.
Due to the memory consuming of the end-to-end architecture,
we can only feed a minibatch of images to the network at
a time for training. Technically, let H be the output of the
last fully connected layer of E2E-BDNN for a minibatch of
size m; S be similarity matrix defined over the minibatch
(using equation (19)); B is an auxiliary variable. Similar to
SH-BDNN, we train the network to minimize the following
constrained loss function
min J

W,B

=

λ1
2m

1 T
H H−S
L

2

λ2
λ3
kH − Bk2 +
2m
2
λ4
2
kH1m×1 k
+
2m

+

L×m

s.t. B ∈ {−1, 1}

1
HHT − I
m

At beginning (line 1 in the Algorithm 3), we initialize
(0)
the network parameter W(0) as follows. For the feature
extractor component, we initialize it by the pretrained AlexNet
model [25]. The dimensional reduction (DR) layer is initialized
by applying PCA on the AlexNet features, i.e. the outputs of
fc7 layer, of the training set. Specifically, let WP CA and
cP CA be the weights and bias of the DR layer, respectively,
we initialize them as follows,

where v1 , v2 , . . . , vk are the top eigenvectors corresponding
to the largest eigenvalues extracted from the covariance matrix
of training features; m is the mean of the training features.
The binary optimizer component is initialized by training
SH-BDNN using AlexNet features (of training set) as inputs.
We then initialize the binary code matrix of the whole dataset
B(0) ∈ {−1, 1}L×n via ITQ [14] (line 2 in the Algorithm 3).
Here, AlexNet features are used as training inputs for ITQ.
In each iteration t of the Algorithm 3, we only sample a
minibatch X(t) from the training set to feed to the network
(line 5 in the Algorithm 3). So after T iterations, we exhaustively sample all training data. In each iteration t, we
first create the similarity matrix S(t) (using equation (19))
corresponding to X(t) as well as the B(t) matrix (line 6
and line 7 in the Algorithm 3). Since B(t) has been already
computed, we can fix that variable and update the network
(k)
parameter W(t) by standard backpropagation with Stochastic
Gradient Descent (SGD) (line 8 in the Algorithm 3). After
T iterations, since the network was exhaustively learned from
the whole training set, we update B(k) = sgn(F) (line 10
in the Algorithm 3), where F is the outputs of the last fully
connected layer for all training samples. We then repeat the
optimization procedure until it reaches a criterion, i.e., after
K iterations.
Implementation details The proposed E2E-BDNN is implemented in MATLAB with MatConvNet library [47]. All
experiments are conducted on a workstation machine with a
GPU Titan X. Regarding the hyperparameters of the loss function, we empirically set λ1 = 10−1 , λ2 = 10−2 , λ3 = 10−2
and λ4 = 10−3 . The learning rate is set to 10−4 and the weight
decay is set to 5 × 10−4 . The minibatch size is 256.

2

C. Evaluation of End-to-End Binary Deep Neural Network
(E2E-BDNN)
(28)
(29)

B. Training
The training procedure of E2E-BDNN is presented in the
Algorithm 3. In the Algorithm 3, X(t) and B(t) ∈ {−1, 1}L×m
are a minibatch sampled from the training set at iteration t and
its corresponding binary codes, respectively. B(k) is the binary
(k)
codes of the whole training set X at iteration k; W(t) is the
network weight when learning up to iterations t and k.

Since we have already compare SH-DBNN to other supervised hashing methods in Section IV-C, in this experiment
we focus to compare E2E-BDNN with SH-BDNN. We also
compare the proposed E2E-BDNN to other end-to-end hashing
methods [29], [28], [30], [31].
a) Comparison between SH-BDNN and E2E-BDNN:
Table V presents comparative mAP between SH-BDNN and
E2E-BDNN. The results shows that E2E-BDNN consistently
improves over SH-BDNN at all code lengths on all datasets.
The large improvements of E2E-BDNN over SH-BDNN are
observed on the CIFAR10 and MNIST datasets, especially at

10

M AP COMPARISON BETWEEN

L
SH-BDNN
E2E-BDNN

8
57.15
64.83

TABLE V
SH-BDNN AND E2E-BDNN ON CIFAR10, MNIST, AND SUN397 DATASETS .

CIFAR10
16
24
66.04
68.81
71.02
72.37

32
69.66
73.56

8
84.65
88.82

low code lengths, i.e., on CIFAR10, E2E-BDNN outperforms
SH-BDNN 7.7% and 5% at L = 8 and L = 16, respectively;
on MNIST, E2E-BDNN outperforms SH-BDNN 4.2% and
3.8% at L = 8 and L = 16, respectively. On SUN397 dataset,
the improvements of E2E-BDNN over SH-BDNN are more
clear at high code length, i.e., E2E-BDNN outperforms SHBDNN 2.5% and 2.7% at L = 24 and L = 32, respectively.
The improvements of E2E-BDNN over SH-BDNN confirm
the effectiveness of the proposed end-to-end architecture for
learning discriminative binary codes.
b) Comparison between E2E-BDNN and other end-toend supervised hashing methods: We also compare our proposed deep networks SH-BDNN and E2E-BDNN with other
end-to-end supervised hashing architectures, i.e., Hashing with
Deep Neural Network (DNNH) [30], Deep Hashing Network
(DHN) [48], Deep Quantization Network (DQN) [31], Deep
Semantic Ranking Hashing (DSRH) [28], and Deep Regularized Similarity Comparison Hashing (DRSCH) [29]. In
those works, the authors propose the frameworks in which
the image features and hash codes are simultaneously learned
by combining CNN layers and a binary quantization layer
into a large model. However, their binary mapping layer only
applies a simple operation, e.g., an approximation of sgn
function (logistic [28], [30], tanh [29]), l1 norm approximation of binary constraints [48]. Our SH-BDNN and E2EBDNN advances over those works in the way to map the
image features to the binary codes. Furthermore, our learned
codes ensure good properties, i.e. independence and balance,
while [30], [29], [31] do not consider such properties, and [28]
only considers the balance of codes. It is worth noting that,
in [30], [28], [29], [48], different settings are used when
evaluating. For fair comparison, follow those works, we setup
two different experimental settings as follows
•

•

Setting 1: according to [30], [31], [48], we randomly
sample 100 images per class to form 1K testing images.
The remaining 59K images are used as database images.
Furthermore, 500 images per class are sampled from the
database to form 5K training images.
Setting 2: following [28], [29], we randomly sample
1K images per class to form 10K testing images. The
remaining 50K images are served as the training set. In
the test phase, each test image is searched through the
test set by the leave-one-out procedure.

Table VI shows the comparative mAP between our methods
and DNNH [30], DHN [48], DQN [31] on the CIFAR10
dataset with the setting 1. Interestingly, even with non end-toend approach, our SH-BDNN outperforms DNNH and DQN
at all code lengths. This confirms the effectiveness of the
proposed approach for dealing with binary constrains and the
having of desire properties such as independence and balance

MNIST
16
24
94.24
94.80
98.03
98.26

32
95.25
98.21

8
33.06
34.15

SUN397
16
24
47.13
57.02
48.21
59.51

32
61.89
64.58

TABLE VI
M AP COMPARISON BETWEEN E2E-BDNN, SH-BDNN AND
DQN[31] ON CIFAR10 (S ETTING 1).

L
SH-E2E
SH-BDNN
DNNH[30]
DQN[31]
DHN[48]

24
60.02
57.30
56.60
55.80
59.40

32
61.35
58.66
55.80
56.40
60.30

DNNH[30],

48
63.59
60.08
58.10
58.00
62.10

TABLE VII
M AP COMPARISON BETWEEN E2E-BDNN, SH-BDNN AND END - TO - END
HASHING METHODS DSRH[28], DRSCH[29] ON CIFAR10 (S ETTING 2).

L
SH-E2E
SH-BDNN
DSRH [28]
DRSCH [29]

24
67.16
65.21
61.08
62.19

32
68.72
66.22
61.74
62.87

48
69.23
66.53
61.77
63.05

on the produced codes. With the end-to-end approach, it helps
to boost the performance of the SH-BDNN. The proposed
E2E-BDNN outperforms all compared methods, DNNH [30],
DHN [48], DQN [31]. It is worth noting that in Lai et
al. [30], increasing the code length does not necessary boost
the retrieval accuracy, i.e., they report a mAP 55.80 at L = 32,
while a higher mAP, i.e., 56.60 is reported at L = 24. Different
from [30], both SH-BDNN and E2E-BDNN improve mAP
when increasing the code length.
Table VII presents the comparative mAP between the
proposed SH-BDNN, E2E-BDNN and the competitors
DSRH [28], DRSCH [29] on the CIFAR10 dataset with the setting 2. The results clearly show that the proposed E2E-BDNN
significantly outperforms the DSRH [28] and DRSCH [29] at
all code lengths. Compare to the best competitor DRSCH [29],
the improvements of E2E-BDNN over DRSCH are from 5%
to 6% at different code lengths. Furthermore, we can see that
even with non end-to-end approach, the proposed SH-BDNN
also outperforms DSRH [28] and DRSCH [29].
VI. C ONCLUSION
In this paper, we propose three deep hashing neural networks for learning compact binary presentations. Firstly, we
introduce UH-BDNN and SH-BDNN for unsupervised and
supervised hashing respectively. Our network designs constrain to directly produce binary codes at one layer. The
designs also ensure good properties for codes: similarity
preserving, independence and balance. Together with designs,
we also propose alternating optimization schemes which allow
to effectively deal with binary constraints on the codes. We
then propose to extend the SH-BDNN to an end-to-end binary
deep neural network (E2E-BDNN) framework which jointly

11

learns the image representations and the binary codes. The
solid experimental results on various benchmark datasets show
that the proposed methods compare favorably or outperform
state-of-the-art hashing methods.
R EFERENCES
[1] R. Arandjelovic and A. Zisserman, “All about VLAD,” in CVPR, 2013.
[2] H. Jégou, M. Douze, C. Schmid, and P. Pérez, “Aggregating local
descriptors into a compact image representation,” in CVPR, 2010.
[3] T.-T. Do and N.-M. Cheung, “Embedding based on function approximation for large scale image search,” TPAMI, 2017.
[4] H. Jégou and A. Zisserman, “Triangulation embedding and democratic
aggregation for image search,” in CVPR, 2014.
[5] K. Grauman and R. Fergus, “Learning binary hash codes for large-scale
image search,” Machine Learning for Computer Vision, 2013.
[6] J. Wang, H. T. Shen, J. Song, and J. Ji, “Hashing for similarity search:
A survey,” CoRR, 2014.
[7] J. Wang, T. Zhang, J. Song, N. Sebe, and H. T. Shen, “A survey on
learning to hash,” TPAMI, 2017.
[8] J. Wang, W. Liu, S. Kumar, and S. Chang, “Learning to hash for indexing
big data - A survey,” Proceedings of the IEEE, pp. 34–57, 2016.
[9] A. Gionis, P. Indyk, and R. Motwani, “Similarity search in high
dimensions via hashing,” in VLDB, 1999.
[10] B. Kulis and K. Grauman, “Kernelized locality-sensitive hashing for
scalable image search,” in ICCV, 2009.
[11] M. Raginsky and S. Lazebnik, “Locality-sensitive binary codes from
shift-invariant kernels,” in NIPS, 2009.
[12] B. Kulis, P. Jain, and K. Grauman, “Fast similarity search for learned
metrics,” TPAMI, pp. 2143–2157, 2009.
[13] Y. Weiss, A. Torralba, and R. Fergus, “Spectral hashing,” in NIPS, 2008.
[14] Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin, “Iterative quantization: A procrustean approach to learning binary codes for large-scale
image retrieval,” TPAMI, pp. 2916–2929, 2013.
[15] K. He, F. Wen, and J. Sun, “K-means hashing: An affinity-preserving
quantization method for learning binary compact codes,” in CVPR, 2013.
[16] J.-P. Heo, Y. Lee, J. He, S.-F. Chang, and S.-e. Yoon, “Spherical
hashing,” in CVPR, 2012.
[17] W. Kong and W.-J. Li, “Isotropic hashing,” in NIPS, 2012.
[18] C. Strecha, A. M. Bronstein, M. M. Bronstein, and P. Fua, “LDAhash:
Improved matching with smaller descriptors,” TPAMI, pp. 66–78, 2012.
[19] W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang, “Supervised
hashing with kernels,” in CVPR, 2012.
[20] J. Wang, S. Kumar, and S. Chang, “Semi-supervised hashing for largescale search,” TPAMI, pp. 2393–2406, 2012.
[21] M. Norouzi, D. J. Fleet, and R. Salakhutdinov, “Hamming distance
metric learning,” in NIPS, 2012.
[22] G. Lin, C. Shen, D. Suter, and A. van den Hengel, “A general two-step
approach to learning-based hashing,” in ICCV, 2013.
[23] B. Kulis and T. Darrell, “Learning to hash with binary reconstructive
embeddings,” in NIPS, 2009.
[24] F. Shen, C. Shen, W. Liu, and H. Tao Shen, “Supervised discrete
hashing,” in CVPR, 2015.
[25] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in NIPS, 2012.
[26] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “CNN features
off-the-shelf: An astounding baseline for recognition,” in CVPRW, 2014.
[27] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in CVPR, 2015.
[28] F. Zhao, Y. Huang, L. Wang, and T. Tan, “Deep semantic ranking based
hashing for multi-label image retrieval,” in CVPR, 2015.
[29] R. Zhang, L. Lin, R. Zhang, W. Zuo, and L. Zhang, “Bit-scalable
deep hashing with regularized similarity learning for image retrieval
and person re-identification,” TIP, pp. 4766–4779, 2015.
[30] H. Lai, Y. Pan, Y. Liu, and S. Yan, “Simultaneous feature learning and
hash coding with deep neural networks,” in CVPR, 2015.
[31] Y. Cao, M. Long, J. Wang, H. Zhu, and Q. Wen, “Deep quantization
network for efficient image retrieval,” in AAAI, 2016.
[32] G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learning algorithm
for deep belief nets,” Neural Computation, pp. 1527–1554, 2006.
[33] V. Erin Liong, J. Lu, G. Wang, P. Moulin, and J. Zhou, “Deep hashing
for compact binary codes learning,” in CVPR, 2015.
[34] M. A. Carreira-Perpinan and R. Raziperchikolaei, “Hashing with binary
autoencoders,” in CVPR, 2015.

[35] T.-T. Do, A.-D. Doan, and N.-M. Cheung, “Learning to hash with binary
deep neural network,” in ECCV, 2016.
[36] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “SUN
database: Large-scale scene recognition from abbey to zoo,” in CVPR,
2010.
[37] R. Salakhutdinov and G. E. Hinton, “Semantic hashing,” Int. J. Approx.
Reasoning, pp. 969–978, 2009.
[38] J. Lu, V. E. Liong, and J. Zhou, “Deep hashing for scalable image
search,” TIP, 2017.
[39] J. Nocedal and S. J. Wright, Numerical Optimization, Chapter 17,
2nd ed. World Scientific, 2006.
[40] D. C. Liu and J. Nocedal, “On the limited memory bfgs method for large
scale optimization,” Mathematical Programming, vol. 45, pp. 503–528,
1989.
[41] A. Krizhevsky, “Learning multiple layers of features from tiny images,”
University of Toronto, Tech. Rep., 2009.
[42] Y. Lecun and C. Cortes, “The MNIST database of handwritten digits,”
http://yann.lecun.com/exdb/mnist/.
[43] H. Jégou, M. Douze, and C. Schmid, “Product quantization for nearest
neighbor search,” TPAMI, pp. 117–128, 2011.
[44] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
IJCV, pp. 91–110, 2004.
[45] A. Oliva and A. Torralba, “Modeling the shape of the scene: A holistic
representation of the spatial envelope,” IJCV, pp. 145–175, 2001.
[46] V. A. Nguyen, J. Lu, and M. N. Do, “Supervised discriminative hashing
for compact binary codes,” in ACM Multimedia, 2014.
[47] A. Vedaldi and K. Lenc, “Matconvnet: Convolutional neural networks
for matlab,” in ACM Multimedia, 2015.
[48] H. Zhu, M. Long, J. Wang, and Y. Cao, “Deep hashing network for
efficient similarity retrieval,” in AAAI, 2016.

