SUPERVISED HASHING WITH END-TO-END BINARY DEEP NEURAL NETWORK
Dang-Khoa Le Tan?

arXiv:1711.08901v1 [] 24 Nov 2017

?

Thanh-Toan Do†

Ngai-Man Cheung?

Singapore University of Technology and Design (SUTD)
†
The University of Adelaide

ABSTRACT
Image hashing is a popular technique applied to large scale
content-based visual retrieval due to its compact and efficient
binary codes. Our work proposes a new end-to-end deep
network architecture for supervised hashing which directly
learns binary codes from input images and maintains good
properties over binary codes such as similarity preservation,
independence, and balancing. Furthermore, we also propose
a new learning scheme that can cope with the binary constrained loss function. The proposed algorithm not only is
scalable for learning over large-scale datasets but also outperforms state-of-the-art supervised hashing methods, which
are illustrated throughout extensive experiments from various
image retrieval benchmarks.
Index Terms— deep neural network, learning to hash,
optimization, image retrieval
1. INTRODUCTION AND RELATED WORKS
Content-based image retrieval (CBIR) is one of the interesting problems in computer vision community. There are enormous applications of CBIR. For example, CBIR is a common
approach of image captioning problem [1]. In reality, people use systems such as Google Images, Flickr to search visually similar images. State-of-the-art image search systems
involve a major procedure which maps an input image into a
visual representation. The output vector is searched through
a database via Euclidean distance-based comparison or quatization scheme in order to retrieve relevant images [2, 3, 4].
With the exponential growth of image data, it is neccessary to reduce memory storage and searching time in CBIR
systems. An interesting approach for achieving these requirements is binary hashing [5, 6, 7, 8, 9, 10]. Technically, instead
of producing a real vector, the hashing approach maps an input into a compact binary code via data-independent or dataindependent algorithms. Consequently, the produced binary
codes dramatically reduce the storage. In addition, thanks to
the advance of hardware computing, we are able to compute
the Hamming distance between the binary codes with only
one cycle clock via the POPCNT instruction.
Data-dependent approaches utilize training dataset to
learn hashing models, and thus they usually outperform data-

independent approaches. Along with the spectacular rise of
deep learning, recent data-dependent methods try to construct
end-to-end models which are able to simultaneously learn
image representations and binary codes [5, 6, 7, 9]. Thanks
to joint optimization, the binary codes are sufficient to retain
the label information and increase the discrimination between
samples of different labels. However, since the hashing network has to produce binary codes, the loss function usually
involves binary constraints which are usually represented
by a non-differentiable sign function [11, 12]. One of the
workarounds is to use approximated functions of sign. For
example, [13] used a piece-wise threshold function in place of
sign and relaxed the binary constraints to range constraints.
Although the proposed function is differentiable, these functions may still cause the vanishing gradient problem when
training via gradient descent [14]. On the other hand, [15]
resolved the binary constraints by assuming that the absolute
function and l1 regularization are differentiable everywhere.
However, it could be some degradation in performance due to
the assumption.
In [9], the authors proposed a deep hashing neural networks for supervised learning (SH-BDNN). They used the
idea of the penalty method [16] for dealing with the binary
constraints, i.e., minimizing the binary quantization loss.
Their model optimizes binary codes from image features via
L-BFGS. In SH-BDNN, because the loss function is defined
on the labels of the whole training dataset, it is not scalable
for training on large-scale datasets. It is also worth noting
that the work in [9] is not an end-to-end model.
Our specific contributions are: (i) We propose a novel
end-to-end deep neural network (SH-E2E) for supervised
hashing which integrates three components of a visual search
system, i.e., a feature extraction component, a dimension
reduction component, and a binary learning component. (ii)
We also introduce a learning scheme which not only is able
to cope with binary constraints but also is scalable for largescale training. (iii) The solid experiments on various image
retrieval benchmarks show the method significantly outperforms other recent supervised hashing methods.
The paper is organized as follows. Section 2 presents
the proposed end-to-end hashing network and the learning
scheme. Section 3 describes the experiments and analyzes
the results. Section 4 concludes the paper.

2. METHODOLOGY

DR Layer

2.1. Binary constrained loss function

Binary Optimizer

{xi }ni=1

Let X =
be a training set of n images; W be set
of network parameters; f (W, x) be the network output for
given input x. Our goal is to design a network and a learning
algorithm such that f (W, x) is binary codes. Technically,
let F be the output of a minibatch of size m, we want them
to be binary, namely F ∈ {−1, 1}L×m . Another important
property is similarity preserving, i.e., the distances between
in-class samples tend to be 0, that is to say that the term
2
1 T
is minimized where:
LF F − S

Sij =

−1
1

if xi and xj are not the same class
if xi and xj are the same class

(1)

From definitions mentioned, we define the loss function:
min =
W

α 1 T
F F−S
2 L

2

s.t F ∈ {−1, 1}L×m

(2)

Hyperparameter α controls the similarity preseving property.
Because of the binary constraints under Equation 2, the loss
function becomes a mixed-integer programming problem
which is NP-hard. Inspired from [9], we relax the mixedinteger problem by introducing an auxiliary variable. Let
B ∈ {−1, 1}L×m , the loss function is formulated:
min =

W,B

α 1 T
F F−S
2 L

2

+

s.t B ∈ {−1, 1}

β
2
kF − Bk
2

(3)

L×m

The second term plays a role as a measure of constraint violation. If we set β sufficiently, we can strongly force the output
to be binary which helps us easy to reach the feasible solution.
Additionally, by introducing B, we can optimize Equation 3
by alternating method (Section 2.3). Finally, we formulate
the independence and balance properties by directly attaching these terms into the objective function:

AlexNet

Fig. 1. Overview of the proposed end-to-end hashing framework.
cludes three major components: (i) a learning feature component which extracts visual image representation, (ii) a dimension reduction layer, and (iii) the hashing component which
produces binary codes.
In CBIR systems, the feature extractor can be constructed
from hand-crafted features [8] or learnable features such as
convolutional neural networks [4]. In order to build an endto-end system which can optimize all components, we choose
the convolutional deep networks as the feature extractor. It is
flexible that we can choose any convolutional deep networks
such as VGG [17], AlexNet [18] or ResNet [19]. To make fair
experiments, we utilize AlexNet as the feature extractor of the
network for which is consistent with other compared hashing methods. In our configuration, we remove the last layer
(i.e. softmax layer) of AlexNet, and consider its last fully connected layer (f c7) as the image representation. Thus, the output of the feature extractor component is a 1024-dimensional
real-valued vector.
The dimension reduction component (the DR layer) involves a fully connected layer initialized by PCA from the
outputs of AlexNet, i.e., the outputs of f c7 layer from the
training set. Specifically, let WP CA and bP CA be the weights
and bias of the layer, respectively:
T
WP CA = [v1T v2T · · · vm
]

2

α 1 T
β
2
F F − S + kF − Bk
2 L
2
θ
γ
2
2
+ FFT − I + kF1k
2
2
s.t B ∈ {−1, 1}L×m

bP CA = −WP CA E[X]

min =

W,B

(4)

Where θ, γ are hyperparameters of the network, 1m×1
is a column vector with size m. Since two new terms in
Equation 4 do not depend on B, the optimization procedure
is equivalent to the one in Equation 3.

where v1 , v2 , . . . , vm are eigenvectors extracted from the
covariance matrix Cov[X] and E[X] is the mean of the training set X.
The last component, namely hashing optimizer, is constructed from several fully connected layers. The output of
this component has the same length as the length of required
binary output.
2.3. Training

2.2. Network architecture
Figure 1 illustrates the overall architecture of the proposed
end-to-end deep hashing network. The proposed network in-

The training procedure is demonstrated in the Algorithm 1.
In the Algorithm 1, X(t) and B(t) ∈ {−1, 1}L×m are a minibatch and corresponding binary codes which are sampled

Algorithm 1 End-to-End Supervised Hashing Deep Network
Learning
Input:
X = {xi }n
i=1 : training images; L: code length; K, T : maximum iteration.
Output:
Network parameters W
(0)
1: Initialize the network W(0)
(0)
2: Initialize B ∈ {−1, 1}L×n via ITQ [20]
3: for k = 1 → K do
4:
for t = 1 → T do
5:
A minibatch X(t) is sampled from X
6:
Compute the corresponding similarity matrix S(t)
7:
From B(k−1) , sample B(t) corresponding to X(t)
(k)
8:
Fix B(t) , optimize W(t) via SGD
9:
end for
(k)
10:
Update B(k) by W(T )
11: end for
K
12: Return WT

Table 1. The number of units in the binary optimizer.
L Layer 1 Layer 2 Layer 3
8
90
20
8
16
90
30
16
24
100
40
24
32
120
50
32
48
140
80
48

3. EXPERIMENTS
3.1. Datasets

from training set at iteration t, respectively. B(k) is the binary codes of training set X at iteration k, while W(t) is the
network weight when learning up to iteration t. Firstly, the
pretrained AlexNet model is chosen to initialize the feature
extractor component. In the next step, we initialize B by
ITQ [20] which makes the training convergence faster. The
DR layer is generated from PCA as mentioned in Section 2.2
and the remaining full connected layers are initialized randomly. We apply the alternating approach for constrained
optimization [16] to minimize the loss function (4). In each
iteration t of the Algorithm 1, we only sample a minibatch
(X(t) ) of the training set to feed to the network. Therefore,
through T iterations, we exhaustively sample all training data.
In each iteration t, we first create the similarity matrix S(t)
(Equation 1) corresponding to X(t) as well as the B(t) matrix. Since B(t) has been already computed, we can fix that
(k)

variable and update the network parameter W(t) by standard
back-propagation with Stochastic Gradient Descent (SGD).
After T iterations, since the network was exhaustively
learned from the whole training set, we update B(k) =
(k)
sign(F(W(T ) , X)) and start the optimization procedure again
until it reaches a criterion, i.e., after K iterations.
Compare to SH-BDNN, our proposed method has several
interesting advantages. Thanks to the SGD, SH-E2E can apply for large scale datasets which has a vast amount of training data because the network builds the similarity matrix from
an image batch of 256 samples. On the other hand, the SHBDNN has to use the whole training samples in each iteration. In terms of memory consumption, our method uses less
memory than SH-BDNN since SH-BDNN uses all network
parameters for regularization, we instead apply weight decay
which actually does not require computing all weights at the
time. Consequently, SH-E2E scales up the model capacity
since we can design the model with a large number of layers.

MNIST [21] comprises 70k grayscale image of size 28 × 28
which are divided into 2 parts: 60k training images and 10k
testing images. The dataset has 10 labels which are handwriteen of 10 digits.
Cifar10 [22] includes 60k 30 × 30 RGB images categorized
into 10 classes where 6k images are divided into 5k training
images and 1k testing images. For end-to-end networks comparisons, we follows 2 settings:
• Setting 1: according to [13], we randomly select 100
samples per class to form 1k testing images. The rest
59k images are database images. The training set are
sampled 5k from the database (500 images per class).
• Setting 2: following [11], for test set, we sample 1k images for each class (10k images). In the test phase, each
test image is searched through the test set by the leaveone-out procedure. The rest 59k images are served as
the training set.
SUN397 [23] is a large-scale dataset which contains 108754
images categorized into 397 classes. Following the setting
from [10], we select 42 categories which have more than 500
images. From the subset with 35k images constructed, we
randomly sample 100 images as test samples and therefore we
get 4200 images to from a testing set. The remaining images
are considered as training samples.
3.2. Implementation details
The proposed method is implemented in MATLAB with
MatConvNet library [24] as a deep learning framework. All
experiments are conducted in workstation machine (Intel (R)
Xeon(R) CPU E5-2650 2.20GHz) with a GPU Titan X.
The last component includes 3 fully connected layer where
sigmoid is used as the activation function. The number
of units in the binary optimizer are empirically selected as
described in Table 1.
Regarding the hyperparameters of the loss function, we
finetune α in range 10−3 to 10−1 , β = 10−2 , γ = 10−2 and
θ = 10−3 . For training the network, we select learning rate
lr = 10−4 and weights decay to be 5 ∗ 10−4 . The size of

100

80

90

70

60

mAP

70

50
40

50

SH−E2E
SH−BDNN
ITQ−CCA
KSH
BRE

30

40
30
8

60

60

mAP

mAP

80

70

16

24

number of bits (L)

32

20
8

16

24

number of bits (L)

(a) MNIST

(b) Cifar10

32

50

40

30

20
8

16

24

32

number of bits (L)
(c) SUN397

Fig. 2. mAP comparison between SH-E2E and other state-of-the-art methods on MNIST, Cifar10, SUN397.
minibatch is 256 for all experiments. Other settings in the
4n
algorithm are set as following: K = 5, T = d 256
e where n is
the number of training samples.
3.3. Compare with other supervised hashing methods
We compare the proposed method with other supervised hashing methods, i.e., SH-BDNN [9], ITQ-CCA [20], KSH [25],
BRE [26]. In order to make fair comparisons, we use the
pre-trained network AlexNet to extract features from fully
connected layer f c7 then use PCA to map 4096-dimensional
features to lower dimensional space, i.e., a 800-dimensional
space. Therefore, the input of other supervised hashing methods is 800-dimensional vectors corresponding to input image.
On MNIST dataset, SH-E2E has gets an impressive results. At code length 8, it gets higher mAP than SH-BDNN
4.17%. Interestingly, our proposed method achieves nearly
100% at higher code length, i.e, its mAP are 98.03%, 98.26%
and 98.21% at code length 16, 24, 32, respectively.
The similar results can be observed on the Cifar10 dataset.
As shown in Figure 2(b), SH-E2E outperforms other supervised hashing methods with a large margin. Especially, at
code length L = 8, our method gets higher mAP than SHBDNN 7.68%. In higher code length, it achieves higher than
SH-BDNN 4.15% mAP. On dataset SUN397, our method has
better results than other methods. According to the results, the
longer the code length is, the better SH-E2E achieves.
3.4. Compare with other end-to-end hashing networks
We also compare our proposed deep network with other
supervised hashing architectures, i.e., Lai et al. [13] and
DRSCH network [11] along with SH-BDNN. Compared to
our method, the main differences is that our method not only
learns the binary codes but also retains hashing properties,
i.e, independent, balancing. Besides, the proposed learning
scheme also affects the performance. It is noted that the
results of [13, 11] are directly cited from the corresponding
papers.

Table 2. mAP comparison between SH-E2E, SH-BDNN and
Lai et al.[13] on Cifar10 (Setting 1).
L
24
32
48
SH-E2E
58.13 60.29 61.53
SH-BDNN
57.30 58.66 60.08
Lai et al.[13] 56.60 55.80 58.10

Table 3. mAP comparison between SH-E2E, SH-BDNN and
[11] on Cifar10 (Setting 2).
L
24
32
48
SH-E2E
67.16 68.72 69.23
SH-BDNN
65.21 66.22 66.53
DRSCH [11] 62.19 62.87 63.05

From Table 2 and 3, our proposed network clearly outperforms other end-to-end networks and SH-BDNN as well. On
the setting of [13], we achieve better mAP at all code lengths
(3.15% on average) than Lai et al. method [13] because of
the superiority of the proposed network and learning scheme.
In Table 3, we again achieve better optimal solutions than
the others. It is worth to note that our method outperforms
SH-BDNN since we are able to train 59k images on SH-E2E
whereas SH-BDNN can not.

4. CONCLUSIONS
In this paper, we introduced a new deep network architecture which efficiently learns compact binary codes of images.
The new network comprises three components, i.e., feature
extractor, dimension reduction and hashing learner. In addition, we also propose a new loss function which can maintain binary code properties and a new learning scheme which
can cope with binary constraints. The experimental results
clearly show the advance of the proposed architecture compare to other state-of-the-art supervised hashing methods.

5. REFERENCES
[1] Raffaella Bernardi, Ruket Cakici, Desmond Elliott,
Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank
Keller, Adrian Muscat, and Barbara Plank, “Automatic
description generation from images: A survey of models, datasets, and evaluation measures.,” JAIR, vol. 55,
pp. 409–442, 2016.
[2] Filip Radenović, Giorgos Tolias, and Ondřej Chum,
“Cnn image retrieval learns from bow: Unsupervised
fine-tuning with hard examples,” in ECCV. Springer,
2016, pp. 3–20.
[3] Thanh-Toan Do and Ngai-Man Cheung, “Embedding
based on function approximation for large scale image
search,” TPAMI, 2017.
[4] Giorgos Tolias, Ronan Sicre, and Hervé Jégou, “Particular object retrieval with integral max-pooling of cnn
activations,” 2016.
[5] Kevin Lin, Jiwen Lu, Chu-Song Chen, and Jie Zhou,
“Learning compact binary descriptors with unsupervised deep neural networks,” in CVPR, 2016, pp. 1183–
1192.
[6] Bohan Zhuang, Guosheng Lin, Chunhua Shen, and Ian
Reid, “Fast training of triplet-based deep binary embedding networks,” in CVPR, 2016, pp. 5955–5964.
[7] Jie Feng, Svebor Karaman, and Shih-Fu Chang, “Deep
image set hashing,” in WACV. IEEE, 2017, pp. 1241–
1250.

[13] Hanjiang Lai, Yan Pan, Ye Liu, and Shuicheng Yan, “Simultaneous feature learning and hash coding with deep
neural networks,” in CVPR, 2015.
[14] Geoffrey E Hinton, Simon Osindero, and Yee-Whye
Teh, “A fast learning algorithm for deep belief nets,”
Neural computation, vol. 18, no. 7, pp. 1527–1554,
2006.
[15] Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin
Chen, “Deep supervised hashing for fast image retrieval,” in CVPR, 2016, pp. 2064–2072.
[16] Jorge Nocedal and Stephen J. Wright, Numerical Optimization, Chapter 17, World Scientific, 2nd edition,
2006.
[17] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”
CoRR, vol. abs/1409.1556, 2014.
[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Imagenet classification with deep convolutional
neural networks,” in NIPS, 2012, pp. 1097–1105.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun, “Deep residual learning for image recognition,” in
CVPR, 2016, pp. 770–778.
[20] Yunchao Gong and Svetlana Lazebnik, “Iterative quantization: A procrustean approach to learning binary
codes,” in CVPR, 2011.
[21] Yann Lecun and Corinna Cortes, “The MNIST database
of handwritten digits,” http://yann.lecun.com/
exdb/mnist/.

[8] Thanh-Toan Do, Dang-Khoa Le Tan, Trung T. Pham,
and Ngai-Man Cheung, “Simultaneous feature aggregating and hashing for large-scale image search,” in
CVPR, July 2017.

[22] Alex Krizhevsky, “Learning multiple layers of features
from tiny images,” Tech. Rep., University of Toronto,
2009.

[9] Thanh-Toan Do, Anh-Dzung Doan, and Ngai-Man Cheung, “Learning to hash with binary deep neural network,” in ECCV. Springer, 2016, pp. 219–234.

[23] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude
Oliva, and Antonio Torralba, “Sun database: Largescale scene recognition from abbey to zoo,” in CVPR.
IEEE, 2010, pp. 3485–3492.

[10] Thanh-Toan Do, Anh-Dzung Doan, Duc-Thanh
Nguyen, and Ngai-Man Cheung, “Binary hashing with
semidefinite relaxation and augmented lagrangian,” in
ECCV. Springer, 2016, pp. 802–817.
[11] Ruimao Zhang, Liang Lin, Rui Zhang, Wangmeng Zuo,
and Lei Zhang, “Bit-scalable deep hashing with regularized similarity learning for image retrieval and person
re-identification,” TIP, pp. 4766–4779, 2015.
[12] Venice Erin Liong, Jiwen Lu, Gang Wang, Pierre
Moulin, and Jie Zhou, “Deep hashing for compact binary codes learning,” in CVPR, 2015.

[24] A. Vedaldi and K. Lenc, “Matconvnet – convolutional
neural networks for matlab,” in ACM Multimedia, 2015.
[25] Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, and
Shih-Fu Chang, “Supervised hashing with kernels,” in
CVPR. IEEE, 2012, pp. 2074–2081.
[26] Brian Kulis and Trevor Darrell, “Learning to hash with
binary reconstructive embeddings,” in NIPS, 2009, pp.
1042–1050.

