Learning Deep Similarity Models with Focus Ranking for Fabric Image Retrieval
Daiguo Denga , Ruomei Wanga , Hefeng Wub,∗, Huayong Hea , Qi Lic , Xiaonan Luod
a Sun

Yat-sen University, Guangzhou 510006, China
University of Foreign Studies, Guangzhou 510006, China
c Western Kentucky University, Bowling Green, KY, 42101, USA
d Guilin University of Electronic Technology, Guilin 541004, China

b Guangdong

arXiv:1712.10211v1 [] 29 Dec 2017

Abstract
Fabric image retrieval is beneficial to many applications including clothing searching, online shopping and cloth modeling. Learning
pairwise image similarity is of great importance to an image retrieval task. With the resurgence of Convolutional Neural Networks
(CNNs), recent works have achieved significant progresses via deep representation learning with metric embedding, which drives
similar examples close to each other in a feature space, and dissimilar ones apart from each other. In this paper, we propose a
novel embedding method termed focus ranking that can be easily unified into a CNN for jointly learning image representations and
metrics in the context of fine-grained fabric image retrieval. Focus ranking aims to rank similar examples higher than all dissimilar
ones by penalizing ranking disorders via the minimization of the overall cost attributed to similar samples being ranked below
dissimilar ones. At the training stage, training samples are organized into focus ranking units for efficient optimization. We build
a large-scale fabric image retrieval dataset (FIRD) with about 25,000 images of 4,300 fabrics, and test the proposed model on the
FIRD dataset. Experimental results show the superiority of the proposed model over existing metric embedding models.
Keywords: convolutional neural network, fabric image retrieval, metric embedding, focus ranking
2010 MSC: 00-01, 99-00
1. Introduction
Fabric image retrieval, as a special case of generic image
retrieval, benefits a wide range of applications, e.g., large-scale
clothing searching, online shopping and cloth modeling. Given
a query fabric image, a retrieval algorithm is expected to output
fabric images that are identical or similar to the query image. It
remains a challenging task due to severe variations in illuminations, orientations, scales and wrinkles among fabric images.
Traditional methods for image retrieval mainly involve two
critical components: i) design a robust and discriminative image representation, and ii) determine an effective distance or
similarity metric for a given image representation. Image representations used in a traditional method are usually hand-crafted,
e.g., SIFT [1], GIST [2, 3], Bag of Words (BoW) [4], Fisher
Vector (FV) [5, 6], and VLAD [7]. Although achieving reasonable successes in image retrieval, these methods depend heavily on feature engineering. More seriously, the two components
are designed or learned separately, leading to a sub-optimal solution.
Recently, methods were proposed to learn an image representation and a distance or similarity metric jointly [8, 9] based
on Convolutional Neural Networks (CNN) [10, 11], which can
be used seamlessly in image retrieval. Specifically, these methods trained a CNN with metric learning embedding. Two simple yet effective metric learning embedding methods are pair
∗ Corresponding

author.
Email address: wuhefeng@gmail.com (Hefeng Wu)

Preprint submitted to Image and Vision Computing

[12, 13] and triplet embedding [14, 15]. These two embedding
methods are optimized to pull samples of different labels apart
from each other and push samples with the same labels close to
each other. The most important advantage of these discriminative models is that they can jointly learn an image representation and semantically meaningful metric, which is more robust
against intra-class variations and inter-class confusions.
An image retrieval system aims to find out samples with the
same labels against a great many negative ones. But pair and
triplet embedding methods model a metric with no more than
one negative image as reference, which are extremely rough approximation to a real setting. In this paper, we propose a novel
embedding method named focus ranking, which can easily be
unified into a CNN for a joint optimization. In particular, the
proposed model aims to rank a sample with the same label (i.e.,
the matched sample) top over all negative ones. Hence, we penalize any ranking disorder by minimizing the overall cost of
the matched samples ranked below any negative ones. At the
training stage, we organize training samples into focus ranking
units, each of which consists of a probe sample, a matched sample, and a reference set, for an efficient optimization. It learns
to rank the matched sample top over all the negative ones in the
reference set.
To the best of our knowledge, there are not fabric image retrieval datasets publicly available. We build a large-scale fabric
image retrieval datasets (FIRD). It contains 4,300 fabrics, each
of which has five to ten instances. We divide the FIRD dataset
into two partitions, via randomly selecting half of the fabrics
as the training set and the rest as the test set. We randomly
January 1, 2018

select 2/5 images as queries for each fabric in the test set and
the rest form the retrieval set. Given a query image, we aim to
find out the ones of the same fabric at the retrieval set. Extensive experiments on FIRD demonstrate that the proposed model
outperforms existing ones by a large margin.
In summary, this paper makes four main contributions to the
community.

[16] hierarchically quantized local descriptors to a hierarchical vocabulary, which improves not only the retrieval efficiency
but also its quality. Jegou et al. [17] presented a contextual
dissimilarity measure for accurate and efficient image search,
while Fraundorfer et al. [18] proposed a binning scheme for
fast hard drive based image search. Jegou et al. [19, 20] developed a more precise representation by integrating hamming
embedding and weak geometric consistency within the inverted
file. Some works also aggregated a set of local descriptors into
global ones for large scale image retrieval, e.g., Fisher Vector (FV) [5, 6], and Vector of Locally Aggregated Descriptors
(VLAD) [7]. Perronnin et al. [21] represented an image as a
Fisher Vector, and compressed Fisher Vectors using the binarization technique to reduce their memory footprint and speed
up the retrieval. VLAD can jointly optimize dimensionality reduction and indexing algorithms, and thus produce a more compact representation for image retrieval. Although achieving significant progress in visual retrieval, these works depended heavily on hand-crafted features, which were not always optimized
for particular tasks.

• We propose a novel focus ranking embedding method,
which aims to rank the matched sample top over any negative ones. This method can be easily unified into a CNN
to learn a deep similarity model by optimizing the image
representation and metric jointly. Compared to existing
embedding methods, it can learn more robust and discriminative representation to reduce intra-class variations
and enhance inter-class discrimination.
• We apply focus ranking unit generation to model training,
which highlights the focus by the negative-positive ratio
in the process of training. It can also help increase the
diversity of training samples, which is greatly important
to learning.

2.2. Deep representation learning
Recently, deep representation learning has been success• We build a large-scale fabric image retrieval dataset (FIRD),
fully applied to various computer vision areas, such as image
which contains 4,300 fabrics and about 25,000 images for
classification [10, 22, 23], object detection [24, 25, 26], pixeltraining and test. To the best of our knowledge, it is the
wise image labeling [27, 28] and human centric analysis [29,
first large-scale dataset for this task.
30]. We review some recent works that applied deep learning
• Extensive experiments demonstrate the superiority of the
to image retrieval [31, 32, 8, 33, 34, 35]. Babenko et al. [31]
proposed method over existing metric embedding methextracted global features for image retrieval from the fully conods. We also carefully evaluate and discuss key componected layer of CNN pre-trained with an image classification
nents of our model that improve the performance.
dataset [36], and demonstrated fine-tuning the network with
annotated target images could boost the retrieval performance.
The rest of this paper is organized as follows. Related works
Their succeeding work [32] simply aggregated the average and
are reviewed in Section 2. The proposed model is presented in
max pooling of the last convolutional layer as image repreSection 3, and the FIRD dataset is described in Section 4. Secsentation and achieved better results than using fully convotion 5 presents an extensive evaluation and comparison between
lutional layer. Perronnin and Larlus [34] proposed a hybrid
the proposed method and several state-of-the-art methods, in
image retrieval architecture that combined the strengths of suaddition to the component analysis of the proposed method.
pervised deep representation and unsupervised Fisher Vectors
Section 6 concludes the paper.
[5, 6]. However, these methods extracted deep features using
the CNN trained with a classification objective function, which
merely focused on feature learning and ignored metric learning.
2. Related Works
In this section, we review three topics most related to this
work according to its objective and the technical components.
These three topics are traditional descriptors, deep representation learning and distance learning.

2.3. Distance learning
Distance metric learning (DML) plays an important role in
many computer vision tasks such as face recognition, person
re-identification and image retrieval. Xing et al. [37] proposed
to learn a distance metric from given examples of similar pairs
and demonstrated the learned distance metric can improve clustering performance significantly. Zheng et al. [38] formulated
distance learning as a probabilistic relative distance comparison model to maximize the likelihood in which the examples
from a matched pair have smaller distance than those from a
mismatched pair. Mignon and Jurie [39] proposed a new distance learning method with sparse pairwise constraints. The
success of deep learning also inspired recent works that applied
neural network models to address the distance learning problem
[40, 41, 42].

2.1. Traditional descriptors
Traditional image retrieval systems mainly focus on designing discriminative hand-crafted image representations for matching two images. Scale-Invariant Feature Transform (SIFT) [1]
was used widely in image retrieval, due to its distinct invariance to image translation, scaling, and rotation. Many works
applied Bag of Features (BOF) using SIFT features for large
scale image retrieval, such as [4]. They first quantized local
feature vectors into visual words, and represented the image by
the frequency histogram of visual words. Nister and Stewenius
2

the scenario that has K categories of fabrics, in which the jth category has nj images. Suppose there are N training images X = {xi |i = 1, 2, 3, ..., N }, so the training set is T =
{(xi , yi )|i = 1, 2, 3, ..., N }, where xi is the i-th training image
and yi ∈ {1, 2, 3, ..., K} denotes the fabric category the image
xi belongs to. Given an image x, there exist some images with
the same label, which are the positive ones and denoted by X + .
All the others are considered to be negative ones, denoted by
X − . A CNN is applied to model the transformation f (·) from
a raw image to its feature vector.
Given an image x, it is expected that the distances between
f (x) and f (x+ ) should be smaller than those between f (x) and
f (x− ), for ∀x+ ∈ X + and ∀x− ∈ X − , i.e.,

The following works are related to our work in spirit of deep
metric embedding. Hu et al. [43] presented a new discriminative deep metric learning (DDML) method for face verification
in the wild. Bell and Bala [41] learned deep metric for visual
search in interior design using contrastive embedding. In [44],
Rippel et al. proposed to maintain an explicit model of the distributions of different classes in representation space and employ it to achieve local discrimination in DML by penalizing
class distribution overlap. Wang et al. [8] used a multi-scale
CNN trained using triplet embedding for learning fine-grained
image similarity directly from the image pixels. Similar ideas
were employed in [14] and [45] for face verification and hash
learning, respectively.
Two common embedding methods are closely related to our
focus ranking embedding method, i.e., pair [12, 13] and triplet
[14, 15] embedding. The goal of all the three embedding methods are to push the samples with the same labels close to each
other and pull the samples of different labels apart from each
other. However, they are quite different in training and optimization. The pair method builds its energy function on a training set composed of image pairs and optimize to assign small
distance to similar pairs and large distance to dissimilar pairs.
The training set of the triplet method is arranged into image
triplets (a probe image, a matched one and a reference one). The
resulting optimization aims to make the distance of the matched
image to the probe image smaller than that of the negative reference one. However, these two methods do not address the challenging fine-grained fabric image retrieval problem well. The
proposed focus ranking method arranges the training set into focus ranking units (see Section 3.3), with much larger number of
reference images in contrast to the matched one. Then our optimization penalizes any ranking disorder that negative images in
the reference set rank over the matched one. Extensive experimental validation demonstrates that the deep similarity model
optimized with our focus ranking embedding method outperforms previous models favorably.

D(f (x), f (x+ )) < D(f (x), f (x− )), ∀x+ ∈ X + , ∀x− ∈ X − ,
(1)
where D(·, ·) denotes the distance between two feature vectors. In this paper, we apply L2 -norm to measure the distance,
i.e., D(f (x), f (y)) = kf (x) − f (y)k22 . Intuitively, we penalize the ranking disorder cases that are D(f (x), f (x+ )) >
D(f (x), f (x− )). Formally, we apply a 0-1 function to measure
the penalty, so the ranking loss of x with respective to X + and
X − can be expressed as
loss(x|X + , X − ) =

X

X

x+ ∈X +

x− ∈X −

1(D(f (x), f (x− ))−D(f (x), f (x+ )) < 0),
(2)

where 1(·) is the indictor function whose value is 1 if the expression is true, and 0 otherwise. The loss function can be formulated
P by summing up the ranking loss over the training set:
L = x∈X loss(x|X + , X − ).
Optimizing the above loss function can minimize the ranking disorder cases and benefit for top ranking the positive samples. However, the 0-1 loss function is non-differential in optimization, so it is difficult to be jointly optimized with the CNN.
One common solution for this problem is to apply a differential upper-bound function of the 0-1 function. Inspired by
[46], we replace the 1(x < 0) by the logistic function σ(x) =
log2 (1 + 2−x ). The loss function can then be expressed as
X X
X
L=
σ(D(f (x), f (x− ))−D(f (x), f (x+ ))).

3. Deep Similarity Model with Focus Ranking
In this section, we describe the proposed deep focus ranking
model in detail. In the training stage, we first arrange training
images into focus ranking units, each of which contains one
probe image, one image of the same fabric (i.e., positive image), and a number of images of different fabrics (i.e., negative images). Then, images in each focus ranking unit are fed
into a CNN to derive features, based on which the “probe-topositive” and “probe-to-negative” distances will be computed,
respectively. Recall that the probe-to-positive distance is expected to be constantly smaller than the probe-to-negative distance. Thus, we penalize ranking disorders in all units. For
each pair of input images, the trained CNN will be applied to
extract their features separately and then compute their distance
directly in the test stage.

x∈X x+ ∈X + x− ∈X −

(3)
It is worth noting that the proposed model successfully integrates feature extraction and metric learning. In a traditional
method, features are usually hand-crafed, and the selection of a
metric is done separately from feature extraction. In this work,
we take advantage of the CNN for feature learning and optimize
the two components jointly. We will introduce the detailed network architecture and the optimization in the following subsections.
3.2. Network architecture
We build our deep focus ranking model on the popular CNN
architecture Alex-Net [10]. The first seven layers of Alex-Net
are used as the early layers of our network architecture, and then
we add auxiliary structure to fulfill similarity evaluation. Our
network architecture is illustrated in Fig. 1. Specifically, the

3.1. Deep metric formulation
We first present some notations that would be used throughout this article. Without loss of generalization, we consider
3

fc

3

fc

conv
96

conv
384

Stride 4

27

227 Max-pooling
3x3 kernel, stride 2
Normalization

Input image

Layer1

Kernel 5x5x96
Stride 1

256

13

Kernel 3x3x256
Stride 1

13
13

13

27
55

conv
384

13

55

227

conv

conv

256

Kernel
11x11x3

Max-pooling
3x3 kernel, stride 2

Layer3

1

Loss layer

Kernel 3x3x384 Kernel 3x3x384
Stride 1
Stride 1

Max-pooling
3x3 kernel, stride 2
Normalization

Layer2

13

Layer4

Layer5

4096

Layer6

4096

Layer7

Figure 1: Illustration of our network architecture.

mentioned σ(x) = log2 (1 + 2−x ), we first obtain

first five of the seven layers are convolutional layers and the remaining two are fully connected layers. The first convolutional
layer takes an image of size 227 × 227 × 3 as input and has 96
kernels of size 11 × 11 × 3 with a stride of 4 pixels, resulting
in 96 feature maps of size 55 × 55. The second convolutional
layer has 256 kernels of size 5 × 5 × 96 and outputs 256 feature
maps of size 27 × 27. The output feature maps of the first two
convolutional layers are max-pooled (3 × 3 kernel with stride
2) and normalized before being fed into the next convolutional
layer. The third, fourth, and fifth convolutional layers have 384
kernels of size 3 × 3 × 256, 384 kernels of size 3 × 3 × 384, and
256 kernels of size 3 × 3 × 384, respectively. The output feature
maps of the third and fourth convolutional layers are not intervened by any pooling or normalization, while the output of the
fifth convolutional layer is max-pooled (3 × 3 kernel with stride
2) and results in 256 feature maps of size 6 × 6. The sixth and
seventh fully-connected layers have 4096 neurons each. Moreover, the outputs of all the seven layers are immediately filtered
by a rectified linear unit (ReLU) before any pooling or normalization operation. Finally, we add an auxiliary layer to calculate
the loss function (3).

"
#

X X
−δ(x, x+ , x− )
∂D(f (x), f (x− ))
∂L
∂D(f (x), f (x+ ))


=
−



∂f (x)
1 + δ(x, x+ , x− )
∂f (x)
∂f (x)

x∈X x− ∈Ux




"
#


+
−
−
X X
 ∂L
−δ(x, x , x )
∂D(f (x), f (x ))
∂D(f (x), f (x+ ))
=
−
∂f (x+ )
1 + δ(x, x+ , x− )
∂f (x+ )
∂f (x+ )


x∈X x− ∈Ux



"
#



X X

∂L
−δ(x, x+ , x− )
∂D(f (x), f (x− ))
∂D(f (x), f (x+ ))


=
−

 ∂f (x− )
1 + δ(x, x+ , x− )
∂f (x− )
∂f (x− )
x∈X −
x

∈Ux

where
δ(x, x+ , x− ) = 2−[D(f (x),f (x

−

))−D(f (x),f (x+ ))]

.

(5)

Remembering that D(·, ·) is the L2 norm, then we can obtain the derivatives as follows:

X X −δ(x, x+ , x− ) 


 ∂L = 2
f (x+ ) − f (x− )


+
−

∂f (x)
1 + δ(x, x , x )

x∈X x− ∈Ux




X X −δ(x, x+ , x− ) 
 ∂L

=
2
f (x) − f (x+ )
+)
+ , x− )
∂f
(x
1
+
δ(x,
x


x∈X x− ∈Ux



X X −δ(x, x+ , x− ) 
 ∂L




=
2
f (x− ) − f (x)
 ∂f (x− )
+
−

1 + δ(x, x , x )
−

3.3. Optimization with focus ranking unit generation
We optimize the proposed model using stochastic gradient
descent (SGD) algorithm with momentum [47]. It need to feed
the whole dataset into the GPU memory if directly using loss
function (3), which is almost intractable for large scale optimization. To tackle this problem, similar to [14, 30], we organize the training data into ranking units to optimize the model.
Specifically, for a probe image x, we take one positive image
x+ ∈ X + and a small subset of X − , i.e., Ux ⊂ X − , into consideration. Note that x+ , randomly sampled from X + , is the
positive match and Ux randomly sampled from X − , is the reference set, for probe image x. So a focus ranking unit is defined
to contain a probe image x, a positive match image x+ and the
reference set Ux . Then the loss can be expressed as
X X
L=
σ(D(f (x), f (x− )) − D(f (x), f (x+ ))). (4)

x∈X x ∈Ux

(6)
We employ the unit sampling generation to optimize the
model for the following serval reasons. First, taking a random subset in each iteration and applying sufficient iterations
is approximately equivalent to using the whole reference set.
Second, it is more efficient and practical to take a subset for
training, especially for large-scale learning. Third, training the
model with a random subset introduces a high degree of diversity, which is of great importance to learning.
4. Fabric Image Retrieval Dataset (FIRD)
To the best of our knowledge, there are not public datasets
for the fabric image retrieval problem. Since fabric image retrieval benefits a wide range of application, e.g., large-scale
clothing searching and online shopping, we construct a largescale fabric image retrieval dataset (FIRD). Specifically, we collect 4,300 fabrics, and take different photos for each fabric. In

x∈X x− ∈Ux

The gradients of the loss with respective to the feature vector can be computed according to the chain rule. With the afore4

Figure 2: Samples from the FIRD dataset under different photographing conditions.

(the ith fabric labelled i). We randomly select half of the 4,300
fabrics as training set and the rest as test set. So both training
and test sets contain 2,150 categories of fabric. During testing,
we randomly select 2/5 images of each category in the test set
as query images, and arrange the rest images as a retrieval set.
Given a query image, we aim to find the ones with the same
label.

order to imitate real application scenarios, we vary the collecting conditions from the following perspectives.
Photographing device: Without loss of generalization, we use
cell phones for collecting images as most of clothing searching
and online shopping applications are used in cell phones. We
use varied cell phones, from Android to iOS, from low- to highend, to increase the diversity of collected images.
Photographing circumstance: We take photos of a fabric in
varied circumstances, including different scenes (indoor and
outdoor), time (day and night), and illumination conditions.
Photographing view: We change viewing conditions as follows: (1) we rotate the fabric to collect images with different
angles; (2) we take photos in near and far views to collect images with different scales; (3) we wrinkle the fabric to collect
fabric images with various degrees of wrinkle.
Figure 2 illustrates some samples from the FIRD dataset.
It is a challenging dataset due to the complex variations in illumination, rotation, scale and wrinkle. All the 4,300 fabrics
are incorporated in the dataset and, for each fabric, 5–10 images are collected under different above-mentioned conditions.
The images of the same fabric are annotated by the same label

5. Experiments
In this section, we will present experimental comparions between the proposal method and existing metric learning methods. We will also present intra-model comparisons to analyze
the contribution of focus ranking unit generation.
5.1. Evaluation protocols
We utilize the following measures to give an extensive evaluation and analysis of the tested methods’ performance.
Recall@K. One of the critical measures for evaluating the
retrieval performance is the recall at particular ranks, i.e., Recall@K [42, 48]. In this measure, given a query image, the
5

images with top K smallest distances to it are retrieved, and
Recall@K is the ratio of retrieved relevant images over the total
relevant images. The Recall@K on the dataset is then computed
by averaging the scores of all query images.
mAP. Another commonly used measure is the mean average precision (mAP) [49]. For each query image, a precision/recall curve can be obtained, and its average precision (the
area under the curve) is computed. The mAP is obtained as the
mean over all the queries.
F1 and NMI scores. To further demonstrate the superiority of the proposed model, we also apply F1 and NMI scores
[50] to evaluate the performance. The Normalized Mutual Information (NMI) score is calculated by conducting clustering
experiments on the test set with the learned features.

5.3. Comparison results
Note that the original feature dimensions of all models are
4,096, and they are too large for an efficient retrieval system.
Therefore, we follow Babenko et al. [31] to apply the Principal
Component Analysis (PCA) [52] to reduce the dimension to
smaller ones (i.e., 512, 64, 16, 8), and perform comparisons on
various feature dimensions.
We first perform comparisons on the FIRD test set. For
each fabric, we randomly select 2/5 images of each category
as queries and use the rest as retrieval ones, making up a retrieval set of about 7,500 images. The comparison results of
Recall@K are presented in Figure 3. It can be observed that all
the other tested models perform well beyond the Alex baseline
one. In addition, Figure 3(a) reveals that, as the feature dimension is reduced from 4,096 to 16, the recall rates of the four
metric embedding models are nearly fixed but the Alex baseline and Softmax models suffer from a severe drop, suggesting
that models with metric embedding can generate more compact
and discriminative features. Among the four metric embedding
models, our model can still outperform the other three by a large
margin. For example, with the feature dimension of 4096, although the pair, Magnet and triplet models can achieve the high
recall rates of 83.9%, 85.6% and 88.5% respectively, our model
further improves them to 93.9%. Figure 3(b) reports the Recall@K scores when changing the number of retrieved images
K from 1 to 32 and fixing the the feature dimension as 256.
Since more relevant images can be retrieved when K increases,
so the scores of all the tested methods will also monotonously
increase. The figure shows that our method has a superior performance over the other methods at each value of K.
Figure 4 shows the comparison results measured in mAP
with respect to the feature dimension. It reveals again the other
tested models all perform better than the Alex baseline one. It
should be noted that the Softmax model with fine-tuning on
the FIRD training set has significant performance gain from the
Alex one. The evaluation results in mAP further demonstrate
the superiority of our model. It suggests that our model can
perform much better than the existing ones in fabric image retrieval systems.
We also present the comparison results of F1 and NMI scores
in Figure 5. The F1 score takes both accuracy and recall into
consideration, and the results depicted in Figure 5(a) give further approval in the effectiveness of our model. We also conduct clustering experiments on the FIRD test set using the features extracted by all the tested models, and Figure 5(b) reports the comparison results measured in NMI. Our model performs much better than all the others, suggesting that it can also
achieve much better clustering quality.
In summary, observed from the quantitative performance of
all the tested models in different measures, the effectiveness of
the proposed method is sufficiently verified. We also observe
that the pair model’s performance is the least sensitive to feature dimension changes, while the triplet model’s performance
ranks second in most measures. The Softmax model exhibits
impressive performance improvement from the Alex baseline
model and can outperform the pair model in some measures,

5.2. Compared methods and implementation details
We compare our model with five methods, i.e., Alex baseline, Softmax, pair, triplet and Magnet. Among them, the Alex
baseline method applies the Alex-net model [10], which is trained
on the ImageNet classification dataset [36] using the Caffe framework [51], and extracts the features of the first fully connected
layer (i.e., fc6), so that the best performance can be ensured,
as suggested in [31]. The Softmax method uses the Alex-net
model pre-trained on ImageNet and then fine-tunes it on our
FIRD training set. Afterwards, the feature extracted on the fc6
layer is used for evaluation. We also note that, for fair comparison, all the other tested methods are based on the Alex-net
model and use the feature from the fc6 layer.
The Magnet model applies the Magnet loss [44]. It is a
recent deep metric embedding method and sculpts the feature
representation by identifying and handling intra-class variation
and inter-class similarity. We use its publicly available TensorFlow code1 and the Alex-net tensorflow model2 pre-trained
on ImageNet to train on our FIRD dataset. We initialize the
learning rate as 0.01, with momentum of 0.9, weight decay of
0.0005, and batch size of 80 (each batch randomly selects 16
classes and 5 images for each class).
The two common deep metric embedding methods (i.e.,
pair and triplet) and the proposed method are implemented in
the Caffe framework. These models are initialized with the parameters of the Alex-net model pre-trained on ImageNet and
fine-tuned on our training set using the SGD method with backpropagation. For pair and triplet models, we initialize the learning rate as 0.001, with momentum of 0.9, weight decay of 0.0005,
and batch sizes of 80 and 120 respectively. For our focus ranking model, we fix the positive and negative ratio as 1:32 in the
focus ranking unit, which can well approximate the proposed
model.
During training, the images are resized to 256 × 256. In
addition, the images are randomly cropped and randomly mirrored horizontally for data augmentation. During inference, all
images are resized to 256 × 256 and cropped 227 × 227 at the
image center for feature extraction. All experiments were tested
on a single NVIDIA Tesla K40 GPU.
1 https://github.com/pumpikano/tf-magnet-loss
2 http://www.cs.toronto.edu/˜guerzhoy/tf

alexnet/

6

100

87

92

Recall@K (%)

Recall@K (%)

95

79
Alex

Softmax

71

84
Alex

Softmax

76

Pair

Pair

magnet

63

magnet

68

Triplet

Triplet

Ours

Ours

55

60

8

16

64

512

4096

1

2

4

Feature Dimension

8

16

32

Top K

(a)

(b)

Figure 3: Comparison on the FIRD dataset using the Recall@K measure. (a) Recall@K vs. feature dimension with K=16. (b) Recall@K vs. K with the feature
dimension of 256. Best viewed in color.
75

mAP score (%)

65

55
Alex

45

Softmax
Pair
magnet

35

Triplet
Ours
25
8

16

64

512

4096

Feature Dimension

42

95

37

91

NMI score (%)

F1 Score (%)

Figure 4: Comparison on the FIRD dataset using the mAP measure. Best viewed in color.

32
Alex
27

Softmax

87

Alex
83

Softmax

Pair

Pair

magnet

22

magnet

79

Triplet

Triplet

Ours
17

Ours

75

8

16

64

512

4096

8

Feature Dimension

16

64

512

4096

Feature Dimension

(a)

(b)

Figure 5: Comparison on the FIRD dataset with F1 and NMI scores. (a) F1 score vs. feature dimension. (b) NMI score vs. feature dimension. Best viewed in color.

which shows that fine-tuning on the target dataset can boost the
baseline model’s performance. For the Magnet model, it does
not work quite well in our experiments. We should note that it is
implemented in TensorFlow, different from the others, and better explicit models of distributions of classes and more careful
fine-tuning may boost its performance further.
To give a qualitative analysis of our method, Figure 6 presents
the top 8 retrieved images of some random query examples and
Figure 7 presents those of some failure cases. We find that, in
most cases, our model can successfully retrieve the images be-

longing to the same fabric, despite of the complex variations in
illumination, rotation, scale, and wrinkle. Moreover, most images of the same fabric rank top, and other retrieved ones also
have very similar textural pattern with the query ones, which is
also very essential for real-world systems. On the other hand,
for the failure examples, most retrieved images come from the
fabrics with subtle differences to the query.
In order to better mimic the real-world scenarios, we further
do experiments with a much larger dataset. We collect 100,000
fabrics and take one image for each fabric under the varied pho7

Figure 6: Retrieval results of query images randomly selected from the FIRD dataset. The first column are the query images, and the rest are the retrieved ones,
arranged according to the distance to the corresponding query images. The retrieved images of the same class to the query image are marked in red boxes.

Figure 7: Retrieval results of failure query images selected from the FIRD dataset. The first column are the query images, and the rest are the retrieved ones,
arranged according to the distance to the corresponding query images.

8

95

81

83

Recall@K (%)

Recall@K (%)

93

69
Alex
57

Softmax

71
Alex
Softmax

59

Pair

Pair
magnet

45

magnet

47

Triplet

Triplet

Ours

Ours

33

35

8

16

64

512

4096

1

2

4

Feature Dimension

8

16

32

Top K

(a)

(b)

95

98

92

93

Recall@K (%)

Recall@K (%)

Figure 8: Comparison on the FIRD dataset with additional 100,000 fabric images. (a) Recall@K vs. feature dimension with K=16. (b) Recall@K vs. K with the
feature dimension of 256. Best viewed in color.

89
Ours (1:1)
Ours (2:1)

86

88
Ours (1:1)
Ours (2:1)

83

Ours (4:1)

Ours (4:1)
Ours (8:1)

83

Ours (8:1)

78

Ours (16:1)

Ours (16:1)
Ours (32:1)

Ours (32:1)
80

73
8

16

64

512

4096

1

2

4

Feature Dimension

8

16

32

Top K

(a)

(b)

Figure 9: Analysis of focus ranking unit generation. (a) Recall@K vs. feature dimension with K=16. (b) Recall@K vs. K with the feature dimension of 256. Best
viewed in color.

tographing conditions, making up a dataset of 100,000 fabric
images. The original retrieval set of about 7,500 images and
these 100,000 images are mixed to construct a new retrieval
set. The retrieval results using the new retrieval set are presented in Figure 8 with the Recall@K measure. It shows that
our model again suppresses all other methods by an even larger
margin, suggesting our model is more suitable for real-world
large scale fabric retrieval systems.

increasing the ratio further can hardly obtain more improvement on the recall rate. These experimental results show that
increasing the ratio may boost the performance by quite a large
margin, but it becomes near saturation when the ratio reaches
32:1. It also suggests that a ratio of 32:1 can well approximate
our ranking model.

5.4. Analysis of focus ranking unit generation

In this paper, we propose a novel focus ranking embedding
method, which aims to rank similar examples top over all dissimilar ones. The focus ranking method can easily be unified
into a deep CNN to jointly learn an image representation and a
metric. At the training stage, we apply focus ranking unit generation to approximate the proposed model for efficient training, and at the inference stage, a test image can be directly fed
into the learnt CNN to extract the features in one shot. Extensive experimental results suggest that our deep focus ranking
model shows superior performance over existing metric embedding methods.

6. Conclusion

In this work, we generate focus ranking units by sampling
to approximate our ranking model in the process of model optimization, and the negative-positive ratio is a very critical parameter. In this section, we analyze the ranking unit generation algorithm by evaluating the performance when training the
model with the negative-positive ratio of 1:1, 2:1, 4:1, 8:1, 16:1
and 32:1 on the FIRD dataset. Figure 9 presents the quantitative
evaluation results measured in Recall@K. It shows that there
are obvious improvement consistently when increasing the ratio from 1:1 to 16:1. However, the improvement of recall rate
when increasing the ratio from 16:1 to 32:1 is negligible, and
9

Acknowledgment

[20] H. Jégou, M. Douze, C. Schmid, Improving bag-of-features for large scale
image search, International Journal of Computer Vision 87 (3) (2010)
316–336.
[21] F. Perronnin, Y. Liu, J. Sánchez, H. Poirier, Large-scale image retrieval
with compressed fisher vectors, in: Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2010, pp. 3384–3391.
[22] J. Zhu, S. Liao, Z. Lei, S. Z. Li, Multi-label convolutional neural network
based pedestrian attribute classification, Image and Vision Computing 58
(2017) 224–229.
[23] Z. Wang, T. Chen, G. Li, R. Xu, L. Lin, Multi-label image recognition
by recurrently discovering attentional regions, in: Proceedings of IEEE
International Conference on Computer Vision (ICCV), 2017.
[24] R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for
accurate object detection and semantic segmentation, in: Proceedings of
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2014, pp. 580–587.
[25] R. Girshick, Fast R-CNN, in: Proceedings of IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.
[26] T. Chen, L. Lin, W. Zuo, X. Luo, L. Zhang, Learning a wavelet-like autoencoder to accelerate deep neural networks, in: Proceedings of AAAI
Conference on Artificial Intelligence, 2018.
[27] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2015, pp. 3431–3440.
[28] T. Chen, L. Lin, L. Liu, X. Luo, X. Li, DISC: deep image saliency
computing via progressive representation learning, IEEE Transactions on
Neural Networks and Learning Systems 27 (6) (2016) 1135–1149.
[29] L. Lin, K. Wang, W. Zuo, M. Wang, J. Luo, L. Zhang, A deep structured
model with radius-margin bound for 3D human activity recognition, International Journal of Computer Vision 118 (2) (2016) 256–273.
[30] S.-Z. Chen, C.-C. Guo, J.-H. Lai, Deep ranking for person reidentification via joint representation learning, IEEE Transactions on Image Processing 25 (5) (2016) 2353–2367.
[31] A. Babenko, A. Slesarev, A. Chigorin, V. Lempitsky, Neural codes for
image retrieval, in: Proceedings of European Conference on Computer
Vision (ECCV), 2014, pp. 584–599.
[32] A. Babenko, V. Lempitsky, Aggregating local deep features for image
retrieval, in: Proceedings of IEEE International Conference on Computer
Vision (ICCV), 2015, pp. 1269–1277.
[33] M. Paulin, M. Douze, Z. Harchaoui, J. Mairal, F. Perronin, C. Schmid, Local convolutional features with unsupervised training for image retrieval,
in: Proceedings of IEEE International Conference on Computer Vision
(ICCV), 2015, pp. 91–99.
[34] F. Perronnin, D. Larlus, Fisher vectors meet neural networks: A hybrid
classification architecture, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3743–3752.
[35] N. Jammalamadaka, A. Zisserman, C. V. Jawahar, Human pose search
using deep networks, Image and Vision Computing 59 (2017) 31–43.
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang,
A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual
recognition challenge, International Journal of Computer Vision 115 (3)
(2015) 211–252.
[37] E. P. Xing, A. Y. Ng, M. I. Jordan, S. J. Russell, Distance metric learning
with application to clustering with side-information, in: Proceedings of
Advances in Neural Information Processing Systems (NIPS), 2002, pp.
505–512.
[38] W. Zheng, S. Gong, T. Xiang, Person re-identification by probabilistic relative distance comparison, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 649–656.
[39] A. Mignon, F. Jurie, PCCA: A new approach for distance learning from
sparse pairwise constraints, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 2666–2672.
[40] H. Liu, B. Ma, L. Qin, J. Pang, C. Zhang, Q. Huang, Set-label modeling and deep metric learning on person re-identification, Neurocomputing
151 (2015) 1283–1292.
[41] S. Bell, K. Bala, Learning visual similarity for product design with convolutional neural networks, ACM Transactions on Graphics 34 (4) (2015)
98.
[42] H. O. Song, Y. Xiang, S. Jegelka, S. Savarese, Deep metric learning via
lifted structured feature embedding, in: Proceedings of IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4004–

This work was supported in part by the National Natural
Science Foundation of China (61379112, 61402120, 61672547,
61472455, 61370186). The authors would like to thank Shenzhen Micro Vision Technology Co., Ltd. for providing the
large-scale dataset of 100,000 fabric images for evaluation.
References
[1] D. G. Lowe, Distinctive image features from scale-invariant keypoints,
International Journal of Computer Vision 60 (2) (2004) 91–110.
[2] A. Oliva, A. Torralba, Modeling the shape of the scene: A holistic representation of the spatial envelope, International Journal of Computer Vision 42 (3) (2001) 145–175.
[3] M. Douze, H. Jégou, H. Sandhawalia, L. Amsaleg, C. Schmid, Evaluation
of GIST descriptors for web-scale image search, in: Proceedings of the
ACM International Conference on Image and Video Retrieval, 2009, pp.
19:1–8.
[4] J. Sivic, A. Zisserman, Video google: A text retrieval approach to object
matching in videos, in: Proceedings of IEEE International Conference on
Computer Vision (ICCV), 2003, pp. 1470–1477.
[5] F. Perronnin, C. Dance, Fisher kernels on visual vocabularies for image
categorization, in: Proceedings of IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2007, pp. 1–8.
[6] F. Perronnin, J. Sánchez, T. Mensink, Improving the fisher kernel for
large-scale image classification, in: Proceedings of European Conference
on Computer Vision (ECCV), 2010, pp. 143–156.
[7] H. Jégou, M. Douze, C. Schmid, P. Pérez, Aggregating local descriptors
into a compact image representation, in: Proceedings of IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 3304–
3311.
[8] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen,
Y. Wu, Learning fine-grained image similarity with deep ranking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1386–1393.
[9] E. Hoffer, N. Ailon, Deep metric learning using triplet network, in: Proceedings of International Workshop on Similarity-Based Pattern Recognition, 2015, pp. 84–92.
[10] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with
deep convolutional neural networks, in: Proceedings of Advances in Neural Information Processing Systems (NIPS), 2012, pp. 1097–1105.
[11] M. D. Zeiler, R. Fergus, Visualizing and understanding convolutional
networks, in: Proceedings of European Conference on Computer Vision
(ECCV), 2014, pp. 818–833.
[12] R. Hadsell, S. Chopra, Y. LeCun, Dimensionality reduction by learning
an invariant mapping, in: Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), Vol. 2, 2006, pp. 1735–1742.
[13] S. Chopra, R. Hadsell, Y. LeCun, Learning a similarity metric discriminatively, with application to face verification, in: Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1,
IEEE, 2005, pp. 539–546.
[14] F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A unified embedding for
face recognition and clustering, in: Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815–823.
[15] K. Q. Weinberger, L. K. Saul, Distance metric learning for large margin
nearest neighbor classification, Journal of Machine Learning Research 10
(2009) 207–244.
[16] D. Nister, H. Stewenius, Scalable recognition with a vocabulary tree, in:
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006, pp. 2161–2168.
[17] H. Jegou, H. Harzallah, C. Schmid, A contextual dissimilarity measure for
accurate and efficient image search, in: Proceedings of IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1–8.
[18] F. Fraundorfer, H. Stewénius, D. Nistér, A binning scheme for fast hard
drive based image search, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1–6.
[19] H. Jegou, M. Douze, C. Schmid, Hamming embedding and weak geometric consistency for large scale image search, in: Proceedings of European
Conference on Computer Vision (ECCV), 2008, pp. 304–317.

10

4012.
[43] J. Hu, J. Lu, Y.-P. Tan, Discriminative deep metric learning for face verification in the wild, in: Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2014, pp. 1875–1882.
[44] O. Rippel, M. Paluri, P. Dollár, L. D. Bourdev, Metric learning with adaptive density discrimination, Proceedings of International Conference on
Learning Representations (ICLR) (2016) 1–15.
[45] R. Zhang, L. Lin, R. Zhang, W. Zuo, L. Zhang, Bit-scalable deep hashing with regularized similarity learning for image retrieval and person
re-identification, IEEE Transactions on Image Processing 24 (12) (2015)
4766–4779.
[46] H. Yun, P. Raman, S. Vishwanathan, Ranking via robust binary classification, in: Proceedings of Advances in Neural Information Processing
Systems (NIPS), 2014, pp. 2582–2590.
[47] L. Bottou, Stochastic gradient descent tricks, in: Neural Networks: Tricks
of the Trade, 2012, pp. 421–436.
[48] H. Jegou, M. Douze, C. Schmid, Product quantization for nearest neighbor search, IEEE Transactions on Pattern Analysis and Machine Intelligence 33 (1) (2011) 117–128.
[49] J. Philbin, O. Chum, M. Isard, J. Sivic, A. Zisserman, Object retrieval
with large vocabularies and fast spatial matching, in: Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2007,
pp. 1–8.
[50] C. D. Manning, P. Raghavan, H. Schütze, Introduction to information retrieval, Cambridge University Press, Cambridge, 2008.
[51] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in: Proceedings of the ACM International Conference on
Multimedia, 2014, pp. 675–678.
[52] I. Jolliffe, Principal component analysis, Springer-Verlag, New York,
2002.

11

