arXiv:1711.04047v1 [cs.CV] 11 Nov 2017

DeepKSPD: Learning Kernel-matrix-based
SPD Representation for Fine-grained Image
Recognition
Melih Engin, Lei Wang∗, Luping Zhou
School of Computing and Information Technology
University of Wollongong
Wollongong, NSW 2500, Australia
Xinwang Liu
School of Computer
National University of Defense Technology
Changsha, Hunan 410073, China

Abstract
Being symmetric positive-definite (SPD), covariance matrix has
traditionally been used to represent a set of local descriptors in visual
recognition. Recent study shows that kernel matrix can give considerably better representation by modelling the nonlinearity in the local
descriptor set. Nevertheless, neither the descriptors nor the kernel matrix is deeply learned. Worse, they are considered separately, hindering
the pursuit of an optimal SPD representation. This work proposes a
deep network that jointly learns local descriptors, kernel-matrix-based
SPD representation, and the classifier via an end-to-end training process. We derive the derivatives for the mapping from a local descriptor
set to the SPD representation to carry out backpropagation. Also, we
exploit the Daleckiǐ-Kreǐn formula in operator theory to give a concise
and unified result on differentiating SPD matrix functions, including
∗

Corresponding author (leiw@uow.edu.au)

1

the matrix logarithm to handle the Riemannian geometry of kernel
matrix. Experiments not only show the superiority of kernel-matrixbased SPD representation with deep local descriptors, but also verify
the advantage of the proposed deep network in pursuing better SPD
representations for fine-grained image recognition tasks.

1

Introduction

To deal with image variations, modern visual recognition usually models the
appearance of an image by a set of local descriptors. They evolve from
early filter bank responses, through traditional local invariant features, to
the activation feature maps of recent deep convolutional neural networks
(CNNs). During the course, how to represent a set of local descriptors
to obtain a global image representation has always been a central issue.
Among the best known methods in this line of research are the bag-of-features
(BoF) model [21], sparse coding [22], vector of locally aggregated descriptors
(VLAD) [12], and Fisher vector encoding [20]. In recent years, representing a
set of descriptors with a covariance matrix has attracted increasing attention.
It characterizes the pairwise correlation of descriptor components presented
in a set, and is generally called SPD representation since covariance matrix is
symmetric positive-definite. This representation is robust to noisy descriptors and independent of the cardinality of a descriptor set. Also, it does
not need a large number of images to generate common bases for encoding,
and can therefore be individually applied to single images. In the past few
years, covariance-matrix-based SPD representation has been employed in a
variety of visual recognition tasks including the recognition of texture, face
and object [11], the classification of image set [24], and so on.
A recent progress on SPD representation is to model the nonlinear information in a set of descriptors. As reported in [23], directly using a kernel
matrix to represent a descriptor set demonstrates its superiority. Given a
set of d-dimensional descriptors, a d × d kernel matrix is computed with a
predefined kernel function, where each entry is the kernel value between the
realization of two descriptor components in this set. This method effectively
models the nonlinear correlation among these descriptor components. The
kernel function can be flexibly chosen to extract various nonlinear relationship, and the covariance is just a special case using a linear kernel. The
resulting kernel-matrix-based SPD representation maintains the same size as
2

its covariance-matrix-based counterpart, but produces considerable improvement on recognition performance.
Nevertheless, this kernel-matrix-based SPD representation [23] is only
developed upon traditional local descriptors like the pixel intensities or the
Gabor filter responses of a textured or facial image. Its potential with deep
local descriptors on image recognition has not been explored in the literature and therefore remains unclear. Another more critical issue, which is the
main focus of this work, is that the local descriptors and the kernel matrix in
the existing SPD representation are detached. In other words, they cannot
effectively negotiate with each other to obtain an optimal representation for
the ultimate goal of classification. To address these two issues, this work
completely builds the kernel-matrix-based SPD representation built upon
deep local descriptors and benchmarks it against the state-of-the-art image
recognition methods. More importantly, we develop a deep network called
DeepKSPD to jointly learn the deep local descriptors, the kernel-matrixbased SPD representation, and the classifier. This is achieved by an end-toend training process between input images and class labels. The presence
of kernel matrix computation in the proposed deep network complicates the
backpropagation process. Also, to make the resulting SPD representation
better work with the classifier, a matrix logarithm function is usually required to map the kernel matrix from Riemannian geometry to Euclidean
geometry. In this work, we derive all the matrix derivatives involved in the
mapping from a local descriptor set to the kernel-matrix-based SPD representation to fulfill the backpropagation algorithm for the proposed deep network.
Also, by exploiting the Daleckiǐ-Kreǐn formula in operator theory [6, 2], we
provide a concise and unified result on the derivative of the functions on
SPD matrices, in which matrix logarithm is a special case. Together, these
produce a backpropagation algorithm that could deal with a deep network
with various kernel-matrix-based SPD representations.
Experimental study is conducted on multiple benchmark datasets, especially on fine-grained image recognition, to demonstrate the efficacy of the
proposed DeepKSPD framework. First, in contrast to the existing kernelmatrix-based representation built upon traditional local descriptors, we demonstrate the superiority of the kernel-matrix-based SPD representation built
upon deep local descriptors. On top of that, we further demonstrate the
advantage of the proposed end-to-end trained DeepKSPD network in jointly
learning the local descriptors and the kernel-matrix-based SPD representation. As will be shown, our DeepKSPD network achieves the overall highest
3

classification accuracy on these benchmark datasets, when compared with
the related deep learning based methods.

2

Related Work

Let Xd×n = [x1 , x2 , · · · , xn ] denote a data matrix, in which each column
contains a local descriptor xi (xi ∈ Rd ), extracted from an image. In the days
when local invariant features such as SIFT are popularly used, methods like
BoF, VLAD, and Fisher vector have been developed to encode and pool these
descriptors to obtain a global image representation. VLAD and Fisher vector
methods have recently been applied to the deep local descriptors collected
from the activation feature maps of deep CNNs, demonstrating promising
image recognition performance [7, 5]. These methods usually need a sufficient
number of images to train a set of common bases (e.g., cluster centers or
Gaussian mixture models) for the encoding step.
The SPD representation takes a different approach. It traditionally computes a d × d covariance matrix over X as Σ = X̄ X̄ T (or simply XX T ),
where X̄ denotes the centered X. Originally, this covariance matrix is proposed as a region descriptor, for example, characterizing the covariance of
the color intensities of pixels in a local image patch. In the past several years,
it has been employed as a promising global image representation in a number
of visual recognition tasks. Recent research in this line aims to model the
nonlinear information in a set of descriptors. The approach proposed in [9]
implicitly maps each descriptor xi (i = 1, 2, · · · , n) onto a kernel-induced
feature space and computes a covariance matrix therein. Nevertheless, this
results in a high (or even infinite) dimensional covariance matrix that is difficult to manipulate explicitly or computationally. The other approach [23]
proposes to directly compute a kernel matrix K over X as follows. Let fj
denote the jth row of X, consisting of the n realizations of the jth component of x. The (i, j)th entry of K is calculated as k(fi , fj ), with a predefined
kernel function k such as a Gaussian kernel. In this way, the nonlinear relationship among the d components can be effectively and flexibly extracted.
The resulting kernel matrix K maintains the size of d × d and is more robust against the singularity issue caused by small sample. It is easy to see
that covariance matrix is a special case in which k reduces to a linear kernel
function. As reported in [23], this kernel-matrix-based SPD representation
achieves considerably better performance than its covariance counterpart and
4

that proposed in [9] on multiple visual recognition tasks.
Both covariance and kernel matrices are SPD and have a Riemannian
geometry. In order to work with the common classifiers that assume a Euclidean geometry, a variety of operations have been developed in the literature. Among them, the matrix logarithm operation, log(·), may be the most
commonly used one due to its simplicity and efficacy [1]. Conceptually, it
can be viewed as mapping an SPD matrix from the underlying manifold to
its tangent space in which Euclidean geometry can be applied. In practice,
after the matrix K is obtained, the matrix log(K) will be computed and
then reshaped into a long vector to be fed into a classifier.
Research on integrating the SPD representation with deep local descriptors or even into deep networks is still in its very early stage but has demonstrated both theoretical and practical values. In the recent work of Bilinear
CNN [14, 15], an outer product layer is applied to combine the activation
features maps from two CNNs, and this produces clear improvement in finegrained visual recognition. This outer product essentially leads to a covariance matrix (in the form of XX T ) when the two CNNs are set as the same.
Another work in [10] trains a deep network for image semantic segmentation,
in which the covariance-matrix-based SPD representation is used to represent
a set of local descriptors.
Nevertheless, to the best of our survey, all the existing few works on SPD
representation in deep learning focus on the covariance-matrix-based SPD
representation. None of them has considered the kernel-matrix-based one,
which can produce significantly better recognition performance. To address
this issue, we develop a deep network focusing on the kernel-matrix-based
SPD representation and jointly learn this representation with deep local descriptors. Also, the work in [10] derives the derivations of the matrix logarithm function from the scratch. Although instructive, it does not connect
this derivation with the operator theory on positive definite matrix [2]. In
this work, by establishing this interesting link, we not only readily obtain the
derivatives for matrix logarithm (and other general SPD matrix functions)
in a much concise way, but can also gain more insights by accessing the vast
knowledge in that field for future research.
At the end, it is worth noting that in this work the kernel matrix is
integrated into deep neural networks as a representation of a set of local
descriptors collected from the activation feature maps. This is fundamentally
different from the recent works that develop new CNNs with reproducing
kernels, supervised convolutional kernel networks, and deep kernel learning
5

Figure 1: The structure of the proposed DeepKSPD network
models [17, 16, 26].

3

The proposed network DeepKSPD

The proposed network DeepKSPD consists of three blocks, as shown in Fig. 1.
The leftmost block maps an input image into a set of deep local descriptors.
Since we deal with visual recognition, any convolutional neural network can
be used. It generates a set of activation feature maps for an image, from
which a set of deep local descriptors are collected. In this work we employ the
commonly used VGG-19 network pre-trained on the ImageNet dataset. The
rightmost block includes the commonly used fully connected and softmax
layers to produce the posteriori probability for each class. In between is
the KSPD block that contains the layers related to the kernel-matrix-based
representation and the matrix logarithm. In specific, the input of the KSPD
block is the output of the last convolutional layer (conv5 4) of the VGG-19
network. In this way, the input consists of d activation feature maps of the
size of w × h. These feature maps are reshaped along the depth dimension
d, and this gives rise to the matrix Xd×n with n = w × h. Afterwards,
the kernel matrix Kd×d is computed with X. It pools the n deep local
descriptors by capturing the pairwise nonlinear relationship among the d
feature maps. Following that is the matrix logarithm layer to handle the
Riemannian geometry of SPD matrix and this produces the matrix H =
log(K). Since H is symmetric, a layer that extracts the upper triangular
and diagonal entries of H is deployed next to avoid redundancy. We observe
that normalized KSPD representations usually perform better. Therefore,

6

an L2 normalization and a batch normalization layer are added at the two
ends of the KSPD block, respectively.

4

End-to-end training of DeepKSPD

4.1

Derivatives between X and the kernel matrix K

Recall that Xd×n denotes a set of local descriptors. Considering that Gaussian kernel is commonly used in the literature and that it is used in [23]
to demonstrate the advantage of the kernel-matrix-based representation, we
exemplify the proposed DeepKSPD with a Gaussian kernel and focus on this
case to derive the derivatives. Other kernels such as polynomial kernel can
be dealt with in a similar way.
Let Id×d and 1d×d denote an identity matrix and a matrix of 1s. Let
◦ denote the entrywise product (Hadamard product) of two matrices, and
exp[·] denote an exponential function applied to a matrix in an entrywise
manner. In this way, the Gaussian kernel matrix K computed on X can be
expressed as


K = exp −θ · (I ◦ XX T )1 + 1T (I ◦ XX T )T − 2XX T ,

(1)

where θ is the width of the Gaussian kernel. This expression is illustrated in
Fig. 2.

Xd×n

Ad×d

Ed×d

Kd×d

XX T

(I ◦ A)1 + 1T (I ◦ A)T − 2A

exp[−θ · E]

···

J

Figure 2: Illustration of the mapping from X to K (a Gaussian kernel
function is used).
Let J denote the objective function to be optimized by the DeepKSPD
∂J
network. By temporarily assuming that the derivative ∂K
has been known
∂J
(will be resolved in the next section), we now work out the derivative ∂X
and ∂J
. Note that J is a composition of functions applied to X and it can
∂θ

7

be equally expressed as a function of each of the intermediate variables as
follows.
J(X) = J1 (A) = J2 (E) = J3 (K),

(2)

where A, E, and K are defined as
A = XX T ,

E = (I ◦ A)1 + 1T (I ◦ A)T − 2A ,

(3)

K = exp[−θ · E].

Following the rules for differentiation, the following relationship can be obtained
δA = (δX) X T + X (δX)T ,
δE = (I ◦ δA)1 + 1T (I ◦ δA)T − 2δA,

(4)

δK = (−θK) ◦ δE.

Furthermore, it is known from the differentiation of a scalar-valued matrix
function that
!






∂J3
∂J3 T
δJ = vec
, vec(δK) = trace
δK ,
∂K
∂K

(5)

where vec(·) denotes the vectorization of a matrix and h·, ·i denotes the inner
product. Combining this result with δK = (−θK) ◦ δE in Eq.(4) and using
the identity that trace(AT (B ◦ C)) = trace((B ◦ A)T C), we can obtain

δJ = trace

∂J3
∂K

!

T
δK

!

∂J3 T
= trace
−θK ◦
δE
∂K
!


∂J2 T
= trace
δE .
∂E


(6)

The last equality
because
from Eq.(2) we know that δJ can also be
 holds


∂J2 T
written as trace ∂E δE . Noting that Eq.(6) is true for any δE, we can
therefore derive that
∂J2
∂J3
= (−θK) ◦
.
∂E
∂K

(7)

Repeating the above process by using the relationship of δE and δA and that

8

of δA and δX in Eq.(4), we can further have (proof is provided in Appendix)

 ! !
∂J2
∂J2 T
∂J2
+
1T − 2
;
∂E
∂E
∂E
 !

∂J1
∂J1 T
+
X.
∂A
∂A

∂J1
=I◦
∂A
∂J
=
∂X

In addition, the derivative

∂J
∂θ

can be obtained as

∂J
= trace
∂θ



∂J3
∂K

!

T

(−K ◦ E) .

3
Therefore, when ∂J
is available, we can work out
∂K
the above results.

4.2

(8)

∂J
∂X

(9)

and

∂J
∂θ

according to

Derivatives of the matrix logarithm on the kernel
matrix K

3
we deal with the matrix logarithm operation between K
Now, to obtain ∂J
∂K
and J, which can be written as

J(X) = J4 (H) = J4 (log(K)).

(10)

∂J4
Note that ∂H
is ready to obtain because it only involves the classification
layers like fully connected layer, softmax regression and cross-entropy computation. The key issue is to obtain ∂H
. In the following we introduce the
∂K
Daleckiǐ-Kreǐn formula [6] to give a concise and unified result on differentiating SPD matrix functions, of which the matrix logarithm is a special
case.
Theorem 1 (pp.60, [2]) Let Md be the set of d × d real symmetric matrices. Let I be an open interval and Md (I) is the set of all real symmetric
matrices whose eigenvalues belong to I. Let C 1 (I) be the space of continuously differentiable real functions on I. Every function f in C 1 (I) induces a
differentiable map from A in Md (I) to f (A) in Md . Let DfA (·) denote the
derivative of f (A) at A. It is a linear map from Md to itself. When applied
to B ∈ Md , DfA (·) is given by the Daleckiǐ-Kreǐn formula as

DfA (B) = U G ◦ U T BU

9



UT ,

(11)

where A = U DU T is the eigendecomposition of A with D = diag(λ1 , · · · , λd ),
and ◦ is the entrywise product. The entry of the matrix G is defined as
(
gij =

f (λi )−f (λj )
if λi 6=
λi −λj
0
f (λi ), otherwise.

λj

(12)

This theorem indicates that for a matrix function f (·) applied to A, perturbing A by a small amount B will vary f (A) by the quantity DfA (B) in
Eq.(11), where the variation is in the sense of the first-order approximation.
∂J4
3
Now we show how to derive the functional relationship between ∂H
and ∂J
∂K
based on Theorem 1. According to Eq.(2) and following the argument in
Eq.(5), we have

δJ = trace

∂J4
∂H

!

T
δH


= trace

∂J3
∂K

!

T
δK

.

(13)

Applying the Daleckiǐ-Kreǐn formula, we can explicitly represent δH to
be a function of δK as
δH = DfK (δK) = U G ◦ U T δKU



UT .

(14)

Replacing δH in Eq.(13) with the above result and again applying the prop∂J4
3
erties of trace(AT B), the relationship between ∂H
and ∂J
can be derived in
∂K
a similar way as in Eqs.(6) and (7)
∂J3
=U
∂K




T ∂J4
G◦ U
U
UT .
∂H

(15)

where U and G are obtained from the eigendecomposition of K = U DU T .
The matrix logarithm f (K) , log(K) is now just a special case in which gij
log λ −log λ
in Eq.(12) is computed as λii −λj j when i 6= j and λ−1
otherwise.
i
The work in [10] derives the derivative of the matrix logarithm from the
scratch with the basic facts of matrix differentiation, which is instructive.
However, as previously mentioned, that work does not connect this derivative with the well-established Daleckiǐ-Kreǐn formula. To consolidate this
connection and link with the work in [10], we prove the following proposition.
Proposition 1 The functional relationship obtained in [10] shown in Eq.(16)
(with the notation in this work for consistency) is equivalent to that in Eq.(15)

10

obtained by this work.
(
!!


∂J4
∂J3
T
=U
G̃ ◦ 2U
U log(D)
∂K
∂H sym
)



T ∂J4
−1
U
+ D
U
UT ,
∂H
diag

(16)

where K = U DU T ; g̃ij = (λi − λj )−1 when i 6= j and zero otherwise; Adiag
means the off-diagonal entries of A are all set to zeros; and Asym is defined
to represent (A + AT )/2.
∂J4
Proof.
Note that ∂H
is symmetric because H is symmetric. Therefore,

∂J4
∂J4
just equals ∂H . In this way, Eq.(16) can be written as
∂H sym





∂J3
>
> ∂J4
=U
G̃ ◦ 2U
U log(D)
∂K
∂H
)



∂J
4
+ D −1 U >
U
U>
∂H
diag


∂J4
Define Z =
for the sake of clarity
∂H
n


=U
G̃> ◦ 2U > ZU log(D)




U>
+ D −1 U > ZU
diag

[Define Pd×d = ((log λ1 )1, · · · , (log λd )1)]
n
 


o
=U
2G̃> ◦ P ◦ U > ZU + D −1 ◦ U > ZU
U>
 
o
n
U>
=U
2G̃> ◦ P + D −1 ◦ U > ZU
h
i
Define Q = 2G̃> ◦ P + D −1
n

o
= U Q ◦ U > ZU
U >.

Noting that
∂J3
1
=
∂K
2

∂J3
∂K

is symmetric because K is symmetric, it can be shown that

∂J3
+
∂K



∂J3
∂K

> !

(
=U

11

)
 

Q + Q>
>
◦ U ZU
U >.
2

(17)

Now let us examine the matrix of

1
2


Q + Q> .


1
Q + Q> = (G̃> ◦ P ) + (G̃> ◦ P )> + D −1
2h

i
Note that G̃> = −G̃ according to its definition

= (−G̃ ◦ P ) + (G̃ ◦ P > ) + D −1
= G̃ ◦ (P > − P ) + D −1 .

Noting that
(
log λi − log λj , i 6= j
(P > − P )ij =
,
0,
i=j

(18)

it can be obtained that


G̃ ◦ (P

>

− P) + D

−1

( log λ −log λ
i


ij

=

λi −λj
−1
λi ,

j

,

i 6= j

i=j

= (G)ij ,

(19)

where G is the matrix defined in Eq.(12). Therefore it can be obtained that

1
Q + Q> = G.
2

(20)

Combining this result with the last line of Eq.(17) in this proof gives rise to
∂J3
=U
∂K





> ∂J4
G◦ U
U
U >.
∂H

(21)

This completes the proof.

Connecting with the results in operator theory not only facilitates the
access to the derivatives of SPD matrix functions, but also provides us more
insight on these functions. For example, gij defined in Eq.(12) has a specific name of “first divided difference” of the function f (·), and G is called
“Löewner matrix” [2]. The positive semi-definiteness (PSD) of G guarantees
the operator monotonicity of f (·), that is f (A) − f (B) maintains to be PSD
if A − B is PSD. This applies to the matrix logarithm function log(·) because it can be proved that G in Eq.(12) is PSD. Properties like this could
be useful for the future research on SPD representations, for example, when
designing a deep Siamese network that involves the difference of two SPD
representations.

12

Figure 3: Example images from the datasets. The top three rows correspond to the fine-grained image recognition benchmarks of Birds, Cars and
Aircraft, respectively. The bottom row corresponds to the scene recognition
benchmark MIT Indoor.

5

Experimental Result

There are two tasks in this experiment: i) test the performance of KSPD built
upon deep local descriptors and ii) more importantly, test the performance of
the proposed end-to-end learning network DeepKSPD, on the tasks of finegrained image recognition and scene recognition, by following the literature.
In the Birds dataset, bounding boxes are not used. Example images of these
datasets are in Fig. 3.
Datasets Four benchmark data sets are employed in this experiment. For
scene recognition, the MIT Indoor data set is used, which has 67 classes
with predefined 5600 training and 1340 test images. For fine-grained image
recognition, three data sets of Cars [13], Birds [25], and Aircrafts [18] are
tested. The Cars dataset has 16185 images from 196 classes; the Aircrafts
dataset contains 10200 images of 100 classes (variants). The birds dataset
has 11788 samples of 200 bird species. All the datasets are the benchmarks
widely used by the recently developed deep learning based image recognition
methods. In the Birds dataset, bounding boxes are not used.
Setting of Proposed Methods For the first task, we put forward a
method called KSPD-VGG, which constructs kernel-matrix-based SPD rep13

resentation upon the deep local descriptors extracted from VGG-19 pretrained on ImageNet. Specifically, the 512 feature maps (of size 27 × 27)
of the last convolutional layer of VGG-19 are reshaped to form 512 vectors
with the dimensions of 729 (27 × 27). These vectors are further used to
compute the 512 × 512 Gaussian kernel matrix K. Then, after applying the
matrix logarithm to the kernel matrix, only the upper triangular and diagonal
parts of the resulting matrix are taken and vectorized to represent an image. The resulting KSPD representations of all images are further processed
by PCA dimensionality reduction (to 4096 dimensions), standardization (to
zero mean and unit standard deviation), and `2 normalization. Finally, a
nonlinear SVM classifier is employed to perform classification for this first
task.
For the second task, the proposed DeepKSPD network is trained and
tested. Note that the architecture of DeepKSPD consists of three blocks
(Fig. 1). In the local descriptor block, the network hyperparameters (e.g.,
the number of kernels and their sizes) are set by following VGG-19. In
the proposed KSPD representation block, no hyperparameter needs to be
preset (initial θ is set to 0.1 for all of the experiments). In the classification
block, the size of FC layer is set as the number of classes for each data
set. DeepKSPD is trained by Adaptive Moment Estimation (Adam) in minibatch mode (with the batch-size of 20). A two-step training procedure [3] is
applied as good performance is observed [3, 14]. Specifically, we first train
the last layer using softmax regression for 15 epochs, and then fine-tune the
whole system. The total training epochs are 30 ∼ 50, varied with the data
sets.
Methods in Comparison We compare the proposed KSPD-VGG and
DeepKSPD with a set of methods that are either comparable or competitive
in the literature. They are listed in the first column in Table 1, and can be
roughly grouped into the following three categories.
The first category can be deemed as feature extraction methods, to which
KSPD-VGG belongs. This category also includes FV-SIFT [19], FC-VGG [10],
FV-VGG [5], and COV-VGG (standing for covariance-matrix-based SPD representation). Except in FV-SIFT, the images are represented by features extracted from the pretrained deep CNN model (VGG-19) without fine-tuning,
which allows us to better focus on the sheer effectiveness of the methods in
comparison. In FC-VGG, features are extracted from the last FC layer of

14

VGG-19 for classification. FV-SIFT and FV-VGG construct Fisher vectors
based on local descriptors for classification. FV-SIFT uses the conventional
SIFT descriptors, while FV-VGG uses the deep local descriptors from the last
convolutional layer of VGG-19, following the literature. COV-VGG’s setting
is same as that of KSPD-VGG, except that a covariance matrix is constructed
instead of a kernel matrix. Note that, we directly quote the results of FVSIFT and FC-VGG from the literature, and provide our own implementation
of FV-VGG, COV-VGG, and KSPD-VGG to ensure the same setting for fair
comparison.
The second category includes three end-to-end learning methods, i.e.,
DeepCOV, DeepKSPD (proposed) and Bilinear CNN (denoted as B-CNN) [14].
DeepCOV follows the same network architecture as the proposed DeepKSPD,
but replaces the kernel matrix in the KSPD layer with a covariance matrix.
DeepCOV is conceptually the same as [10], but [10] is designed for segmentation. B-CNN is tested by using the code provided by [14]. The fine-tuned
B-CNN is employed for a fair comparison with DeepCOV and DeepKSPD
that involve an end-to-end training. Note that, in [14], it shows that some
engineering efforts can significantly improve the performance of B-CNN, such
as augmenting the data sets by flipping images and using a separate SVM
classifier instead of the softmax layer in the original deep model for classification, etc. To minimize the impacts of these engineering tricks, we switch off
the image flipping component in the downloaded code, and directly perform
the classification by the softmax layer as usual, same as what we do with
DeepCOV and DeepKSPD.
In the third category, additional methods previously reported on the involved data sets are quoted to further extend the comparison and provide a
whole picture.
Results and Discussion The result is summarized in Table 1 with the
following observations.
First, the proposed KSPD-VGG and DeepKSPD demonstrate their effectiveness for visual recognition. On every dataset, the end-to-end learning
method DeepKSPD achieves the best performance among all the methods.
Overall, DeepKSPD shows superior performance over KSPD-VGG (up to 7
percentage points on Cars) and other competitive methods, demonstrating
the essentials of the end-to-end learning of kernel-matrix-based representation.

15

Table 1: Comparison of Methods

(† indicates the results quoted from the literature.)

ACC (%)

MIT
Cars Aircraft Birds
indoor

Average

Symbiotic Model [4]

–

78.0† 72.5†

–

–

FV-revisit [8]

–

82.7† 80.7†

–

–

FV-SIFT [19]

–

59.2†

61.0†

18.8†

–

FC-VGG [14]

67.6†

36.5† 45.0†

61.0†

52.5
73.1

FV-VGG [5]

73.7

75.2

72.7

71.3†

COV-VGG

74.2

80.3

81.4

76

77.98

KSPD-VGG (proposed)

77.2

83.5

83.8

78.5

80.1

B-CNN [15]

77.6

87.5

82.5

83.5

82.5

DeepCOV

75.2

86.7

81.3

82.2

81.35

DeepKSPD (proposed)

79.6

90.1 86.3

84.5

85.13

Second, it can be seen that KSPD-based methods consistently win COVbased ones on all data sets, either based on feature extraction (KSPD-VGG
vs COV-VGG) or using end-to-end learning (DeepKSPD vs DeepCOV). To
ensure fair comparison, the KSPD-based and COV-based methods only differ
in the SPD representation.
Third, as analyzed above, conceptually B-CNN is very close to DeepCOV when the two paths used in B-CNN are set as the same. However,
DeepCOV performs slightly worse than BCNN in the experiment (around
1%). Looking into this result, we find that after attaining the outer product
matrix, B-CNN applies sign square-root on all entries of the matrix, rather
than performing the matrix logarithm as in DeepCOV and DeepKSPD. Sign
square-root can be efficiently computed by GPU, so that a much longer training procedure (up to 100 epochs) is tolerable. However, matrix logarithm
is currently implemented with CPU, whose calculation is slower than sign
square-root. Therefore, we only train DeepCOV and DeepKSPD for 30 ∼ 50
epochs, and even with this setting the proposed DeepKSPD has achieved
superior performance. Note that the incorporation of matrix logarithm is
necessary, as it is a principled way to handle the Riemannian geometry of
SPD matrix. We have observed that using more epochs and smaller learning
rate, the performance of DeepKSPD and DeepCOV can be further improved,

16

and the superiority of DeepKSPD over B-CNN will become more salient. In
future, we will explore GPU-based implementation of matrix logarithm.
Fourth, as shown, the SPD representation (being it based on an outer
product, covariance, or kernel matrix) outperforms Fisher vector representation in the given visual recognition tasks. The proposed DeepKSPD also
outperforms FV-VGG obtained from fine-tuned VGG-19. The latter attained
78.7% on Aircraft, on Birds 74.7% and 85.7% on Cars [14], which is worse
than 86.3%, 84.5% and 90.1% achieved by DeepKSPD.
Moreover, it is worth emphasizing that this experiment focuses on comparing the core of these methods. Therefore, we minimize the engineering
tricks that are detachable from the model. Certainly, e steps such as augmenting the data, fine-tuning the model for feature extraction, and applying
multi-scaling, as used in the literature, can effectively improve the performance of KSPD-VGG and DeepKSPD.

6

Conclusion

Motivated by the recent progress on SPD representation, we develop a deep
neural network that jointly learns local descriptors and kernel-matrix-based
SPD representation for fine-grained image recognition. The matrix derivatives required by the backpropagation process are derived and linked to the
established literature on the theory of positive definite matrix. Experimental result on benchmark datasets demonstrates the improved performance of
kernel-matrix-based SPD representation when built upon deep local descriptors and the superiority of the proposed DeepKSPD network. Future work
will further explore the effectiveness of this network on other recognition
tasks and develop the SPD representations in other forms.

7

Appendix: Proof for Eq.(8) in main text

According to Eq.(2) and following the argument in Eq.(5), it can be shown
that
!



T


δJ =

vec

∂J2
∂E

, vec(δE)

= trace

∂J2
∂E

δE

,

where vec(·) denotes the vectorization of a matrix and h·, ·i denotes the inner
product. Combining this result with δE = (I ◦ δA)1 + 1T (I ◦ δA)T − 2δA
17

in Eq.(4), it can be obtained that

trace

∂J2
∂E

!

T



δE

= trace

∂J2
∂E

!

T

T

T

(I ◦ δA)1 + 1 (I ◦ δA) − 2δA



.

Keeping applying the identity that trace(AT (B ◦ C)) = trace((B ◦ A)T C),
we can have

trace

∂J2
∂E



!

T

∂J2
+
∂E

= trace  I ◦

δE



∂J2
∂E

T !

Because we know δJ can also be expressed as trace
result is valid for any δA, it can be obtained that
∂J1
=I◦
∂A

∂J2
+
∂E



∂J2
∂E

T !



1

1T


∂J1 T
∂A

!
T

!

−2

∂J2
−2
∂E

!T


δA .


δA and the last

∂J2
.
∂E

This gives rise to the first half ofEq.(8). 
T
1
δA with δA = (δX) X T +X (δX)T
Again, combining δJ = trace ∂J
∂A
in Eq.(4), it can be obtained that

trace

∂J1
∂A

!

T


= trace

δA

∂J1
∂A

T 

T

T

(δX) X + X (δX)



!
.

Applying the identities that trace(ABC) = trace(CAB) and trace(ABC) =
trace((ABC)T ), we can obtain

trace

∂J1
∂A



!

T
δA

∂J1
+
∂A

= trace 



∂J1
∂A





!T

T !
X


∂J T
∂X

δX  .



δX and the last
Because we know δJ can also be expressed as trace
result is valid for any δX, it can therefore be obtained that
∂J
=
∂X

∂J1
+
∂A



∂J1
∂A

This gives rise to the second half of Eq.(8).
18

T !
X.

can be derived in a similar manner. As previous, δJ can
In addition, ∂J
∂θ
be equally written as

δJ = trace

∂J3
∂K

!

T
δK

,

and δJ =

∂J
· δθ,
∂θ

where θ is the width of the Gaussian kernel, a scalar. It is not difficult to see
that by regarding E as constant, δK = (−K ◦ E) · δθ. Therefore, it can be
obtained that

trace

∂J3
∂K

!

T
δK

!

∂J3 T
= trace
((−K ◦ E) · δθ)
∂K
!


∂J3 T
= trace
(−K ◦ E) · δθ.
∂K


Combining with the last equation, we have
∂J
= trace
∂θ

8



∂J3
∂K

T

!
(−K ◦ E) .

Appendix: Visualization of feature maps
learned by DeepKSPD network

(a) Input image

(b) Before learning
19

(c) After learning

(d) Difference

To gain more insight into the proposed DeepKSPD network, we visualize
the activation feature maps (accumulated along the depth dimension) obtained with and without DeepKSPD learning. In the following figure, the
four columns correspond to 1) the original input image; 2) the accumulated
activation feature maps before learning (obtained from pretrained VGG-19
network); 3) the accumulated activation feature maps after learning (obtained from the trained DeepKSPD network); and 4) the difference between
the two previous maps, where red color indicates increase and green color
indicates decrease.
As seen, the activations in the feature maps learned by DeepKSPD are
generally enhanced on the body of the cars while reduced on the surroundings
that are less relevant for car recognition. This shows that in the presence of
the kernel-matrix-based SPD representation block, the DeepKSPD network is
able to learn features that are meaningful from the perspective of recognition.
This provides additional support to the excellent performance observed for
DeepKSPD.

References
[1] V. Arsigny, P. Fillard, X. Pennec, and N. Ayache. Log-euclidean metrics
for fast and simple calculus on diffusion tensors. Magnetic Resonance
in Medicine, 56(2):411–421, 2006.
[2] R. Bhatia. Positive Definite Matrices. Princeton University Press, 2015.
[3] S. Branson, G. V. Horn, S. Belongie, and P. Perona. Bird species categorization using pose normalized deep convolutional nets. In British
Machine Vision Conference (BMVC), Nottingham, 2014.
[4] Y. Chai, V. Lempitsky, and A. Zisserman. Symbiotic segmentation and
part localization for fine-grained categorization. In IEEE International
Conference on Computer Vision, 2013.
[5] M. Cimpoi, S. Maji, and A. Vedaldi. Deep filter banks for texture
recognition and segmentation. In IEEE Conference on Computer Vision
and Pattern Recognition, CVPR 2015, pages 3828–3836, 2015.
[6] Y. L. Daleckiĭ and S. G. Kreĭn. Integration and differentiation of functions of hermitian operators and applications to the theory of pertur20

bations. (Russian) Vorone. Gos. Univ. Trudy Sem. Funkcional. Anal.
1, (1):81–105, 1956. English translation is in book Thirteen Papers on
Functional Analysis and Partial Differential Equations, American Mathematical Society Translations: Series 2, vol.47, 1965.
[7] Y. Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless
pooling of deep convolutional activation features. In Computer Vision ECCV 2014, pages 392–407, 2014.
[8] P.-H. Gosselin, N. Murray, H. Jégou, and F. Perronnin. Revisiting the
Fisher vector for fine-grained classification. Pattern Recognition Letters,
49:92–98, Nov. 2014.
[9] M. T. Harandi, M. Salzmann, and F. M. Porikli. Bregman divergences
for infinite dimensional covariance matrices. In 2014 IEEE Conference
on Computer Vision and Pattern Recognition, CVPR 2014, pages 1003–
1010, 2014.
[10] C. Ionescu, O. Vantzos, and C. Sminchisescu. Matrix backpropagation
for deep networks with structured layers. In 2015 IEEE International
Conference on Computer Vision, ICCV 2015, pages 2965–2973, 2015.
[11] S. Jayasumana, R. I. Hartley, M. Salzmann, H. Li, and M. T. Harandi.
Kernel methods on the riemannian manifold of symmetric positive definite matrices. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 73–80, 2013.
[12] H. Jegou, M. Douze, C. Schmid, and P. Pérez. Aggregating local descriptors into a compact image representation. In The Twenty-Third
IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2010, pages 3304–3311, 2010.
[13] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations
for fine-grained categorization. In 4th International IEEE Workshop on
3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[14] T. Lin, A. Roy Chowdhury, and S. Maji. Bilinear CNN models for finegrained visual recognition. In 2015 IEEE International Conference on
Computer Vision, ICCV 2015, pages 1449–1457, 2015.

21

[15] T.-Y. Lin, A. RoyChowdhury, and S. Maji. Bilinear cnns for fine-grained
visual recognition. In Transactions of Pattern Analysis and Machine
Intelligence (PAMI).
[16] J. Mairal. End-to-end kernel learning with supervised convolutional
kernel networks. In NIPS, pages 1399–1407, 2016.
[17] J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid. Convolutional
kernel networks. In NIPS, pages 2627–2635, 2014.
[18] S. Maji, E. Rahtu, J. Kannala, M. B. Blaschko, and A. Vedaldi. Finegrained visual classification of aircraft. CoRR, abs/1306.5151, 2013.
[19] F. Perronnin, J. Sánchez, and T. Mensink. Improving the fisher kernel
for large-scale image classification. In Proceedings of the 11th European
Conference on Computer Vision: Part IV, ECCV’10, pages 143–156.
Springer-Verlag, 2010.
[20] J. Sánchez, F. Perronnin, T. Mensink, and J. J. Verbeek. Image classification with the fisher vector: Theory and practice. International
Journal of Computer Vision, 105(3):222–245, 2013.
[21] J. Sivic and A. Zisserman. Video google: A text retrieval approach to
object matching in videos. In 9th IEEE International Conference on
Computer Vision (ICCV 2003), pages 1470–1477, 2003.
[22] J. Wang, J. Yang, K. Yu, F. Lv, T. S. Huang, and Y. Gong. Localityconstrained linear coding for image classification. In The Twenty-Third
IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2010, pages 3360–3367, 2010.
[23] L. Wang, J. Zhang, L. Zhou, C. Tang, and W. Li. Beyond covariance:
Feature representation with nonlinear kernel matrices. In 2015 IEEE
International Conference on Computer Vision, ICCV 2015, pages 4570–
4578, 2015.
[24] R. Wang, H. Guo, L. S. Davis, and Q. Dai. Covariance discriminative
learning: A natural and efficient approach to image set classification. In
2012 IEEE Conference on Computer Vision and Pattern Recognition,
pages 2496–2503, 2012.

22

[25] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and
P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010001, California Institute of Technology, 2010.
[26] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Stochastic
variational deep kernel learning. In NIPS, pages 2586–2594, 2016.

23

