1

Optimal Bayesian Transfer Learning

arXiv:1801.00857v1 [stat.ML] 2 Jan 2018

Alireza Karbalayghareh, Student Member, IEEE, Xiaoning Qian, Senior Member, IEEE,
and Edward R. Dougherty, Fellow, IEEE

Abstract—Transfer learning has recently attracted significant
research attention, as it simultaneously learns from different
source domains, which have plenty of labeled data, and transfers
the relevant knowledge to the target domain with limited labeled
data to improve the prediction performance. We propose a
Bayesian transfer learning framework where the source and
target domains are related through the joint prior density of
the model parameters. The modeling of joint prior densities
enables better understanding of the “transferability” between
domains. We define a joint Wishart density for the precision
matrices of the Gaussian feature-label distributions in the source
and target domains to act like a bridge that transfers the
useful information of the source domain to help classification
in the target domain by improving the target posteriors. Using
several theorems in multivariate statistics, the posteriors and
posterior predictive densities are derived in closed forms with
hypergeometric functions of matrix argument, leading to our
novel closed-form and fast Optimal Bayesian Transfer Learning
(OBTL) classifier. Experimental results on both synthetic and
real-world benchmark data confirm the superb performance
of the OBTL compared to the other state-of-the-art transfer
learning and domain adaptation methods.
Index Terms—Transfer learning, domain adaptation, optimal
Bayesian transfer learning, optimal Bayesian classifier

I. I NTRODUCTION
A basic assumption of traditional machine learning is that
data in the training and test sets are independently sampled
in one domain with the identical underlying distribution.
However, with the growing amount of heterogeneity in modern
data, the assumption of having only one domain may not
be reasonable. Transfer learning (TL) is a learning strategy
that enables us to learn from a source domain with plenty
of labeled data as well as a target domain with no or very
few labeled data in order to design a better classifier in the
target domain than the ones trained by target-only data for
its generalization performance. This can reduce the effort of
collecting labeled data for the target domain, which might be
very costly, if not impossible. Due to its importance, there has
been ongoing research on the topic of transfer learning and
many surveys in the recent years covering transfer learning
and domain adaptation methods from different perspectives
[1], [2], [3], [4], [5].
If we train a model in one domain and directly apply it in
another, the trained model may not generalize well, but if the
domains are related, appropriate transfer learning and domain
adaptation methods can borrow information from all the data
across the domains to develop better generalizable models
in the target domain. Transfer learning in medical genomics
is desirable, since the number of labeled data samples is
The authors are with the Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA.

often very limited due to the difficulty of having disease
samples and the prohibitive costs of human clinical trials.
However, it is relatively easier to obtain gene-expression data
for cell lines or other model species like mice or dogs. If
these different life systems share the same underlying disease
cellular mechanisms, we may utilize data in cell lines or model
species as our source domain to develop transfer learning
methods for more accurate human disease prognosis in the
target domain [6], [7].
A. Related Works
Domain adaptation (DA) is a specific case of transfer learning where the source and target domains have the same classes
or categories [2], [3], [5]. DA methods either adapt the model
learned in the source domain to be applied in the target domain
or adapt the source data so that the distribution can be close
to the one of the target data. Depending on the availability
of labeled target data, the DA methods are categorized as
unsupervised and semi-supervised algorithms. Unsupervised
DA problems applies to the cases where there are no labeled
target data and the algorithm uses only unlabeled data in
the target domain along with source labeled data [8]. Semisupervised DA methods use both the unlabeled and a few
labeled target data to learn a classifier in the target domain
with the help of source labeled data [9], [10], [11], [12].
Depending on whether the source and target domains have
the same feature space with the same feature dimension,
there are homogeneous and heterogeneous DA methods. The
first direction in homogeneous DA is instance re-weighting,
for which the most popular measure to re-weight the data
is Maximum Mean Discrepancy (MMD) [13] between the
two domains. Transfer Adaptive Boosting (TrAdaBoost) [14]
is another method that adaptively sets the weights for the
source and target samples during each iteration based on the
relevance of source and target data to help train the target
classifier. Another direction is model or parameter adaptation.
There are several efforts to adapt the SVM classifier designed
in the source domain for the target domain, for example,
based on residual error [15], [16]. Feature augmentation methods, such as Geodesic Flow Sampling (GFS) and Geodesic
Flow Kernel (GFK) [8], derive intermediate subspaces using
Geodesic flows, which interpolate between the source and
target domains. Finding an invariant latent domain in which
the distance between the empirical distributions of the source
and target data is minimized is another direction to tackle the
problem of domain adaptation, such as Invariant Latent Space
(ILS) in [17]. Authors in [17] proposed to learn an invariant
latent Hilbert space to address both the unsupervised and semisupervised DA problems, where a notion of domain variance is

2

simultaneously minimized while maximizing a measure of discriminatory power using Riemannian optimization techniques.
Max-Margin Domain Transform (MMDT) [10] is a semisupervised feature transformation DA method which uses a
cost function based on the misclassification loss and jointly
optimizes both the transformation and classifier parameters.
Another domain-invariant representation method [18] matches
the distributions in the source and target domains via a regularized optimal transportation model. Heterogeneous Feature
Augmentation (HFA) [9] is a heterogeneous DA method which
typically embeds the source and target data into a common
latent space prior to data augmentation.
Domain adaption has been recently studied in deep learning
frameworks like deep adaptation network (DAN) [19], residual
transfer networks (RTN) [20], and models based on generative
adversarial networks (GAN) such as domain adversarial neural
network (DaNN) [21] and coupled GAN (CoGAN) [22].
Although deep DA methods have shown promising results,
they require a fairly large amount of labeled data.
B. Main Contributions
This paper treats homogeneous transfer learning and domain
adaptation from Bayesian perspectives, a key aim being better
theoretical understanding when data in the source domain
are “transferrable” to help learning in the target domain.
When learning complex systems with limited data, Bayesian
learning can integrate prior knowledge to compensate the
generalization performance loss due to the lack of data for the
required sample complexity based on the predictive models
of choice. Rooted in Optimal Bayesian Classifiers (OBC)
[23], [24], which minimize the Bayesian error estimates of
classifiers over uncertainty classes of feature-label distributions, we propose a Bayesian transfer learning framework
and the corresponding Optimal Bayesian Transfer Learning
(OBTL) classifier to formulate the OBC in the target domain
by taking advantages of both the available data and the joint
prior knowledge in source and target domains. In this Bayesian
learning framework, transfer learning from the source to target
domain is through a joint prior probability density function
for the model parameters of the feature-label distributions
of the two domains. By explicitly modeling the dependency
of the model parameters of the feature-label distribution, the
posterior of the target model parameters can be updated via
the joint prior probability distribution function in conjunction
with the source and target data. Based on that, we derive the
effective class-conditional densities of the target domain, by
which the OBTL classifier is constructed.
Our problem definition is the same as the aforementioned
domain adaptation methods, where there are plenty of labeled
source data and few labeled target data. The source and target
data follow different multivariate Gaussian distributions with
arbitrary mean vectors and precision (inverse of covariance)
matrices. For the OBTL, we define a joint Normal-Wishart
prior distribution, where the two precision matrices in the two
domains are jointly connected. This joint prior distribution
for the two precision matrices of the two domains acts like
a bridge through which the useful knowledge of the source

domain can be transferred to the target domain, making the
posterior of the target parameters tighter with less uncertainty.
With such a Bayesian transfer learning framework and
several theorems from multivariate statistics, we define an
appropriate joint prior for the precision matrices using hypergeometric functions of matrix argument, whose marginal
distributions are Wishart as well. The corresponding closedform posterior distributions for the target model parameters
are derived by integrating out all the source model parameters.
Having closed-form posteriors facilitates closed-form effective
class-conditional densities. Hence, the OBTL classifier can
be derived based on the corresponding hypergeometric functions and does not need iterative and costly techniques like
MCMC sampling. Although the OBTL classifier has a closed
form, computing these hypergeometric functions involves the
computation of series of zonal polynomials, which is timeconsuming and not scalable to high dimension. To resolve this
issue, we use the Laplace approximations of these functions,
which preserves the good prediction performance of the OBTL
while making it efficient and scalable. The performance of
the OBTL is tested on both synthetic data and real-world
benchmark image datasets to show its superior performance
over state-of-the-art domain adaption methods.
The paper is organized as follows. Section II introduces the
Bayesian transfer learning framework. Section III derives the
closed-form posteriors of target parameters, via which Section
IV obtains the effective class-conditional densities in the target
domain. Section V derives the OBTL classifier, and Section
VI presents the OBC in the target domain and shows that the
OBTL classifier converts to the target-only OBC when there
is no interaction between the domains. Section VII introduces
the Laplace approximation of the hypergeometric functions of
matrix argument. Section VIII presents experimental results
using both synthetic and real-world benchmark data. Finally,
Section IX concludes the paper.
II. BAYESIAN T RANSFER L EARNING F RAMEWORK
We consider a supervised transfer learning problem in which
there are L common classes (labels) in each domain. Let Ds
and Dt denote the labeled datasets of the source and target
domains with the sizes
where
n of Ns and Nt , respectively,
o
Nt ≪ Ns . Let Dsl = xls,1 , xls,2 , · · · , xls,nl , l ∈ {1, · · · , L},
s
denote the nls data
o for the label l. Simin of the source domain
larly, let Dtl = xlt,1 , xlt,2 , · · · , xlt,nl , l ∈ {1, · · · , L}, denote
t

the nlt data ofSthe target domain
we
PL
SLfor the label l. Obviously,
L
l
n
,
and
have Ds = l=1 Dsl , Dt = l=1 Dtl , Ns =
l=1 s
PL
Nt = l=1 nlt . Since we consider the homogeneous transfer
learning scenario, where the feature spaces are the same in
both the source and target domains, xls and xlt are d×1 vectors
for d features of the source and target domains, respectively.
We use the Gaussian model for the feature-label distribution
in each domain:

−1 
xlz ∼ N µlz , Λlz
, l ∈ {1, · · · , L},
(1)
where z ∈ {s, t} denotes the source, s, or target, t, domains;
µls and µlt are the d × 1 mean vectors in the source and target

3

domains for label l, respectively, and Λls and Λlt are the d × d
precision matrices in the source and target domains for label
l, respectively. In the Bayesian framework, a Normal-Wishart
distribution is employed as a prior for mean and precision
matrices of the Gaussian models. Here we require a joint prior
distribution of the parameters of the source and target domains
to account for the dependency or “relatedness” between the
domains. We define the following joint prior distribution for
µls , µlt , Λls , and Λls :




p µls , µlt , Λls , Λlt = p µls |Λls p µlt |Λlt p Λls , Λlt , (2)
µls

µlt

for l ∈ {1, · · · , L}, where we assume
and
are conditionally independent given Λls and Λlt for any class l. For
making the priors conjugate, both p µls |Λls and p µlt |Λlt
are considered as Gaussian:

−1 
, l ∈ {1, · · · , L}, (3)
µlz |Λlz ∼ N mlz , κlz Λlz

for z ∈ {s, t}, where mlz is the d × 1 mean vector of µlz ,
and κlz is a positive scalar hyperparameter. We need to define
a joint distribution for Λls and Λlt . In the case of a prior for
either Λls or Λlt , we use a Wishart distribution as the conjugate
prior. Here we desire a joint distribution for Λls and Λlt , whose
marginal distributions for both Λls and Λlt are Wishart.
′
h ′
′i
Letting xl = xlt , xls
be the 2d × 1 augmented feature
′

vector and A denoting the transpose of matrix A, a joint
sampling model would take the form

−1 
, l ∈ {1, · · · , L},
(4)
xl ∼ N µl , Λl

where

µl =

"

µlt

µls

#

,

Λl =

"

Λlt

Λlts

Λlts

′

Λls

#

,

(5)

where µl is the 2d × 1 mean vector, and Λl is the 2d × 2d
precision matrix. Λlt and Λls account for the interactions of
features within the source and target domains, respectively,
and Λlts accounts for the interactions of the features across
the source and target domains, for any class l ∈ {1, · · · , L}.
The key point here is that in transfer learning joint sampling of
the source and target domains is not possible, and we always
assume that there are two datasets separately sampled from
the source and target domains. That is why we use (1) instead
of (4) for the sampling model. If we would use (4) as the
sampling model, then we would use a Wishart distribution as
a prior for the precision matrix Λl in (5). This is where we
get the idea of defining a joint prior distribution for Λls and
Λlt . If Λl in (5) has a Wishart distribution, by marginalizing
out the term Λlts one could derive a joint distribution for Λls
and Λlt .
We present some theorems and definitions that will be used
later in deriving the OBTL classifier.
Definition 1. A random d × d symmetric positive-definite
matrix Λ has a nonsingular Wishart distribution with ν degrees
of freedom, Wd (M, ν), if ν ≥ d and M is a d × d positive-

definite matrix (M > 0) and the density is


i
h νd  ν 
ν−d−1
ν −1
1
|M| 2
|Λ| 2 etr − M−1 Λ ,
p(Λ) = 2 2 Γd
2
2
(6)
where |A| is the determinant of A, etr(A) = exp (tr(A)) and
Γd (α) is the multivariate gamma function given by


d
d(d−1) Y
i−1
Γd (α) = π 4
Γ α−
.
(7)
2
i=1
Theorem 1. [25]: If Λ ∼ Wd (M, ν), and A is an r×d matrix
′
′
of rank r, where r ≤ d, then AΛA ∼ Wr (AMA , ν).
Λ Λ 
11
12
Corollary 1. If Λ ∼ Wd (M, ν) and Λ = Λ′ Λ ,
22
12
where Λ11 and Λ22 are d
1 × d1 and
 d2 × d2 submatrices,
M11 M12
is the corresponding
respectively, and if M = M′ M
22
12
partition of M with M11 and M22 being two d1 × d1 and
d2 × d2 submatrices, respectively, then Λ11 ∼ Wd1 (M11 , ν)
and Λ22 ∼ Wd2 (M22 , ν).
Using Corollary 1, we can ensure that using the Wishart
distribution for the precision matrix Λl (5) of the joint model
in (4) will lead to the Wishart marginal distributions for Λls
and Λlt in the source and target domains separately, which is
a desired property. Now we introduce a theorem, proposed in
[26], which gives the form of the joint distribution of the two
submatrices of a partitioned Wishart matrix.
Λ Λ 
11
12
be a (d1 +d2 )×(d1 +
Theorem 2. [26]: Let Λ = Λ′ Λ
22
12
d2 ) partitioned Wishart random matrix, where the diagonal
partitions are of sizes d1 × d1 and d2 × d2 , respectively.
The Wishart distribution of Λ has ν ≥ d1 + d2 degrees of

M11 M12
freedom and positive-definite scale matrix M = M′ M
22
12
partitioned in the same way as Λ. The joint distribution of the
two diagonal partitions Λ11 and Λ22 have the density function
given by
p(Λ11 , Λ22 ) =





′
1 −1
1  −1
K etr − M11 + F C2 F Λ11 etr − C2 Λ22
2
2


ν−d2 −1
ν−d1 −1
ν 1
× |Λ11 | 2
; G ,
|Λ22 | 2
0 F1
2 4
(8)
′
′
where C2 = M22 −M12 M−1
M12 , F = C−1
M12 M−1
,G=
11
2
11
1
1


(d1 +d2 )ν
′
ν
2
2
Λ22
FΛ11 F Λ22
, K −1 = 2 2 Γd1 ν2 Γd2 ν2 |M| 2 ,
and 0 F1 is the generalized matrix-variate hypergeometric
function.
Definition 2. [27]: The generalized hypergeometric function
of one matrix argument is defined by
p Fq (a1 , · · ·

, ap ; b1 , · · · , bq ; X)
∞ X
X
(a1 )κ · · · (ap )κ Cκ (X)
=
,
(b1 )κ · · · (bq )κ k!

(9)

k=0 κ⊢k

where ai , i = 1, · · · , p, and bj , j = 1, · · · , q, are arbitrary
complex (real in our case) numbers, Cκ (X) is the zonal
polynomial of d × d symmetric matrix X corresponding to
the ordered partition κ = (k1 , · · · , kd ), k1 ≥ · · · ≥ kd ≥ 0,

4

P
k1 + · · · kd = k and
κ⊢k denotes summation over all
partitions κ of k. The generalized hypergeometric coefficient
(a)κ is defined by

d 
Y
i−1
a−
(a)κ =
,
(10)
2
ki
i=1
where (a)r = a(a + 1) · · · (a + r − 1), r = 1, 2, · · · , with
(a)0 = 1.

d × d symmetric matrix, we have
Z
d+1
etr(−ZR)|R|α− 2
R>0

=

Z

×

p Fq (a1 , · · ·

etr(−ZR)|R|α−

d+1
2

, ap ; b1 , · · · , bq ; RX)dR

R>0

=

Conditions for convergence of the series in (9) are available
in the literature [28]. From (9) it follows

×

p Fq (a1 , · · · , ap ; b1 , · · · , bq ; R
−α
Γd (α)|Z|
p+1 Fq (a1 , · · · , ap , α; b1 , · · ·

1/2

XR1/2 )dR

, bq ; XZ−1 ).

Now, using Theorem 2, we define the joint prior distribution,
p(Λls , Λlt ) in (2), of the precision matrices of the source and
∞ X
∞
X
Cκ (X) X (tr(X))k
target domains for class l ∈ {1, · · · , L} as follows:
=
= etr(X),
0 F0 (X) =

 
k!
k!
′

1
k=0 κ⊢k
k=0
l
l
l
l −1
l
l l
p(Λt , Λs ) = K etr −
Mt
+ F C F Λlt
∞ X
X
2
(a)κ Cκ (X)


= |Im − X|−a , ||X|| < 1,
1 F0 (a; X) =
1 l −1 l
k!
Λs
× etr − C
(16)
k=0 κ⊢k
2
∞ X
X


Cκ (X)
ν l −d−1
ν l −d−1
νl 1 l
,
0 F1 (b; X) =
2
2
× Λlt
; G ,
Λls
(b)κ k!
0 F1
k=0 κ⊢k
2 4
∞
X X (a)κ Cκ (X)
 l

Mt Mlts
,
l
1 F1 (a; b; X) =
′
where
M
=
is a 2d × 2d positive defi(b)κ k!
l
l
2 F1 (a, b; c; X)

k=0 κ⊢k
∞ X
X

=

k=0 κ⊢k

Mts

(a)κ (b)κ Cκ (X)
,
(c)κ
k!

||X|| < 1,

(11)
where ||X|| < 1 means that the maximum of the absolute
values of the eigenvalues of X is less than 1. 1 F1 (a; b; X)
and 2 F1 (a, b; c; X) are respectively called Confluent and Gauss
hypergeometric functions of matrix argument.
Theorem 3. [25]: Let Z be a complex symmetric matrix
whose real part is positive-definite, and let X be an arbitrary
complex symmetric matrix. Then
Z
d+1
etr(−ZR)|R|α− 2 Cκ (RX)dR
(12)
R>0
= Γd (α)(α)κ |Z|−α Cκ (XZ−1 ),
the integration being over the space of positive-definite d × d
matrices, and valid for all complex numbers α satisfying
Re(α) > d−1
2 . Γd (α) is the multivariate gamma function
defined in (7).

Theorem 4. [29]: The zonal polynomials are invariant under
orthogonal transformation. That is, for a d × d symmetric
matrix X,
′
Cκ (X) = Cκ (HXH ),
(13)

Using Corollary 1, Λlt and Λls have the following Wishart
marginal distributions:
Λlz ∼ Wd (Mlz , ν l ),

p Fq (a1 , · · ·

=

, ap ; b1 , · · · , bq ; RX)
p Fq (a1 , · · ·

, ap ; b1 , · · · , bq ; R1/2 XR1/2 ).

Theorem 5. [30]: If Z > 0 and Re(α) >

d−1
2 ,

(15)

and X is a

z ∈ {s, t}.

(17)

1
L
= p(Dt1 , · · · , DtL |µ1t , · · · , µL
t , Λt , · · · , Λt )

1
L
× p(Ds1 , · · · , DsL |µ1s , · · · , µL
s , Λs , · · · , Λs ) (18)
L
L
Y
Y
=
p(Dtl |µlt , Λlt )
p(Dsl |µls , Λls ).

(14)

As a result, if R is a symmetric positive-definite matrix, the
hypergeometric function has the following property:

l ∈ {1, · · · , L},

III. P OSTERIORS OF TARGET PARAMETERS
Having defined the prior distributions in the previous section, we aim to derive the posterior distribution of the parameters of the target domain upon observing the training source Ds
and target Dt datasets. The likelihood of the datasets Dt and
Ds is conditionally independent given the parameters of the
target and source domains. The dependence between the two
domains is due to the dependence of the prior distributions of
the precision matrices, as shown in Fig 1. Within each domain,
source or target, the likelihoods of the different classes are also
conditionally independent given the parameters of the classes.
As such, the joint likelihood of the datasets Dt and Ds can
be written as
p(Dt ,Ds |µt , µs , Λt , Λs ) = p(Dt |µt , Λt )p(Ds |µs , Λs )

where H is an orthogonal matrix of order d. If R is a
symmetric positive-definite matrix of order d, then
Cκ (RX) = Cκ (R1/2 XR1/2 ).

Ms

nite scale matrix,′ ν l ≥ 2d denotes degrees of ′ freedom,

−1 l
−1 l
l −1
,
M
Mts , Fl = Cl
Cl = Mls − Mlts Mlt
ts  Mt

′
1
1
l
l
l 2 l l l
l ν2
l 2
l −1
dν l 2 ν l
G = Λs F Λt F Λs , and (K ) = 2 Γd 2 |M | .

l=1

l=1

The posterior of the parameters given Dt and Ds satisfies
p(µt , µs , Λt , Λs |Dt , Ds )
∝ p(Dt , Ds |µt , µs , Λt , Λs )p(µt , µs , Λt , Λs )
∝

L
Y

l=1

p(Dtl |µlt , Λlt )

L
Y
l=1

p(Dsl |µls , Λls )

L
Y
l=1

p(µlt , µls , Λlt , Λls ),
(19)

5

Dtl

µlt

Λlt

Λls

µls

Target Domain
Figure 1:

Source Domain

Dependency of the source and target domains through their precision matrices for any class l ∈ {1, · · · , L}.

where we assume that the priors of the parameters
in different classes are independent, p(µt , µs , Λt , Λs ) =
Q
L
l
l
l
l
l=1 p(µt , µs , Λt , Λs ). From (2) and (19),
p(µt , µs , Λt , Λs |Dt , Ds ) ∝

L
Y

p(Dtl |µlt , Λlt )p(Dsl |µls , Λls )

l=1



×p µls |Λls p µlt |Λlt p Λls , Λlt .
(20)
We can see that the posterior of the parameters is equal to the
product of the posteriors of the parameters of each class:
L
Y
p(µt , µs , Λt , Λs |Dt , Ds ) =
p(µlt , µls , Λlt , Λls |Dtl , Dsl ), (21)
l=1

where

p(µlt , µls , Λlt , Λls |Dtl , Dsl ) ∝ p(Dtl |µlt , Λlt )p(Dsl |µls , Λls )



×p µls |Λls p µlt |Λlt p Λls , Λlt . (22)

Since we are interested in the posterior of the parameters of the
target domain, we integrate out the parameters of the source
domain in (21):
Z
p(µt , µs , Λt , Λs |Dt , Ds )dµs dΛs
p(µt , Λt |Dt , Ds ) =
µs ,Λs

=
=

L Z
Y
l=1
L
Y

l=1

where

µls ,Λls

p(µlt , µls , Λlt , Λls |Dtl , Dsl )dµls dΛls

×

Z

µls ,Λls

µlt |Λlt

p(Dsl |µls , Λls )p



µ|Λ, D ∼ N (mn , (κn Λ)−1 ),
Λ|D ∼ Wd (Mn , νn ),

(27)

νn = ν + n,

n

x̄ =
(23)

1X
xi ,
n i=1

S=

n
X
i=1

mn =

′

(xi − x̄)(xi − x̄) .

(28)

(29)

Using Lemma 1 we can simplify (26) as
µls |Λls



p

From (1), for each domain z ∈ {s, t},
nlz
2

Λls , Λlt



dµls dΛls .



1
exp − Qlz , (24)
2
′
l


P
nz
l
l
l
l
l
where Qlz =
i=1 xz,i − µz Λz xz,i − µz . Moreover,
from (3), for each domain z ∈ {s, t},
1

d
d
p µlz |Λlz = (2π)− 2 κlz 2 Λlz 2

 l

′ l l
κz l
l
l
(25)
µz − m z Λ z µz − m z .
× exp −
2
p(Dzl |µlz , Λlz ) = (2π)−

(26)

Lemma 1. If D = {x1 , · · · , xn } where xi is a d × 1 vector
and xi ∼ N (µ, (Λ)−1 ), for i = 1, · · · , n, and (µ, Λ) has a
Normal-Wishart prior, such that, µ|Λ ∼ N (m, (κΛ)−1 ) and
Λ ∼ Wd (M, ν), then the posterior of (µ, Λ) upon observing
D is also a Normal-Wishart distribution:

κm + nx̄
,
κ+n
′
κn
−1
M−1
+S+
(m − x̄)(m − x̄) ,
n = M
κ+n
depending on the sample mean and covariance matrix

p(µlt , Λlt |Dtl , Dsl )
Z
=
p(µlt , µls , Λlt , Λls |Dtl , Dsl )dµls dΛls
∝



nlt
1
1
p(µlt , Λlt |Dtl , Dsl ) ∝ Λlt 2 exp − Qlt Λlt 2
2

 l
′


κ
× exp − t µlt − mlt Λlt µlt − mlt
2

 
ν l −d−1
′
−1
1
2
× Λlt
etr −
+ Fl Cl Fl Λlt
Mlt
2
(


Z
nls
1
1 l
l 2
Λs
exp − Qs Λls 2
×
2
µls ,Λls
 l


′
κ
× exp − s µls − mls Λls µls − mls
2


ν l −d−1
−1 l
1
2
etr − Cl
× Λls
Λs
2

 l
ν 1 l 21 l l l ′ l 21
dµls dΛls .
× 0 F1
; Λs F Λt F Λs
2 4

where
κn = κ + n,

p(µlt , Λlt |Dtl , Dsl ),

µls ,Λls
p(Dtl |µlt , Λlt )p

Dsl

dnlz
2

Λlz

From (16), (23), (24), and (25),

p(µlt , Λlt |Dtl , Dsl )

!
l
′


κ
t,n
µlt − mlt,n Λlt µlt − mlt,n
∝ Λlt exp −
2


ν l +nlt −d−1
1 l −1 l
l
2
Λt
× Λt
etr − Tt
2
(
!
Z
1

′ l l
κls,n l
l 2
l
l
Λs exp −
µs − ms,n Λs µs − ms,n
2
µls ,Λls


ν l +nls −d−1
1 l −1 l
l
2
Λs
etr − Ts
× Λs
2
 l

ν 1 l 21 l l l ′ l 21
dµls dΛls ,
; Λ F Λt F Λs
× 0 F1
2 4 s
(30)
1
2

6

where
κlt,n

= + nlt ,
κls,n = κls + nls ,
l
l
l l
κ m + nt x̄t
κls mls + nls x̄ls
l
,
mlt,n = t lt
,
m
=
s,n
κls + nls
κt + nlt
′
−1
−1
+ Fl Cl Fl + Slt
= Mlt
Tlt
′
κl nl
+ l t t l (mlt − x̄lt )(mlt − x̄lt ) ,
κt + nt


κl nl
−1
−1
Tls
= Cl
+ Sls + l s s l (mls − x̄ls )(mls
κs + ns

t,n

′

− x̄ls ) ,

(31)

with sample means and covariances for z ∈ {s, t} as
l

l

x̄lz

nz
1 X
= l
xl ,
nz i=1 z,i

Slz

=

nz
X
i=1

xlz,i − x̄lz



xlz,i − x̄lz

′

.

Using the equation


Z
′
d
1
1
exp − (x − µ) Λ(x − µ) dx = (2π) 2 |Λ|− 2 , (32)
2
x

and integrating out µls in (30) yields
p(µlt , Λlt |Dtl , Dsl )

!

′ l l
κlt,n l
l
l
exp −
µt − mt,n Λt µt − mt,n
∝
2


ν l +nlt −d−1
−1 l
1
2
× Λlt
Λt
etr − Tlt
(33)
2


Z (
ν l +nls −d−1
−1 l
1
2
Λls
×
Λs
etr − Tls
2
Λls

 l
ν 1 l 21 l l l ′ l 12
dΛls .
× 0 F1
; Λs F Λt F Λs
2 4
Λlt

1
2

The integral, I, in (33) can be done using Theorem 5 as
 l

ν + nls
I = Γd
2
(34)
 l

ν l +nls
ν + nls ν l 1 l l l ′ l
,
F
;
;
F
Λ
F
T
× 2Tls 2
1 1
t
s
2
2 2
where 1 F1 (a; b; X) is the Confluent hypergeometric function
with the matrix argument X. As a result, (33) becomes
p(µlt , Λlt |Dtl , Dsl ) =
Al Λlt

1
2

exp −

ν l +nlt −d−1
2

κlt,n
2

′


µlt − mlt,n Λlt µlt − mlt,n



1 l −1 l
×
Λt
etr − Tt
2
 l

ν + nls ν l 1 l l l ′ l
× 1 F1
; ; F Λt F Ts .
2
2 2
Λlt

!

′ l l
κlt,n l
l
l
µt − mt,n Λt µt − mt,n dµlt
exp −
×
2
µlt
 l

ν + nls ν l 1 l l l ′ l
×1 F1
; ; F Λt F Ts dΛlt .
2
2 2
(36)
1
d
Using (32), the inner integral equals to (2π) 2 |κlt,n Λlt |− 2 =

 d2
1
2π
|Λlt |− 2 . Hence,
κl
Z

κlt

!


(35)

where the constant of proportionality, Al , makes the integration of the posterior p(µlt , Λlt |Dtl , Dsl ) with respect to µlt and
Λlt equal to one. Hence,


Z
ν l +nlt −d−1
1
−1 l
−1
1
2
Λlt
Λt Λlt 2
Al
=
etr − Tlt
2
Λlt

! d2 Z

ν l +nlt −d−1
2



−1 l
1
etr − Tlt
Λt
2
Λlt
 l

l
l
′
ν + ns ν 1 l l l l
× 1 F1
; ; F Λt F Ts dΛlt .
2
2 2
(37)
′
Letting the variable change Ω = Fl Λlt Fl , we have
−1  l ′ −1
dΩ = |Fl |d+1 dΛlt and Λlt = Fl
Ω F
. Since
tr(ABCD) = tr(BCDA) = tr(CDAB) = tr(DABC)
and |ABC| = |A||B||C|, Al can be derived as
! d2
Z 
ν l +nlt −d−1

2π
l −(ν l +nlt )
l −1
2
|Ω|
|F
|
A
=
l
κt,n
Ω


1  l ′ −1 l −1 l −1
F Ω
Tt
× etr − F
2
 l

ν + nls ν l 1
l
× 1 F1
(38)
; ; ΩTs
dΩ
2
2 2
! d2
 l

ν l +nlt
d(ν l +nlt )
ν + nlt
2π
2
l
2
Γ
2
T
=
d
t
2
κlt,n
 l

ν + nls ν l + nlt ν l l l l l ′
× 2 F1
,
,
; ; Ts F Tt F
2
2
2

l −1

A

=

2π
κlt,n

Λlt

where the second equality follows from Theorem 5, and
2 F1 (a, b; c; X) is the Gauss hypergeometric function with the
matrix argument X. As such, we have derived the closed-form
posterior distribution of the target parameters (µlt , Λlt ) in (35),
where Al is given by (38).
IV. E FFECTIVE C LASS -C ONDITIONAL D ENSITIES
For optimal Bayesian classifier [23], [24], using the posterior predictive densities of the classes, called “effective
class-conditional densities”, leads to the optimal choices for
classifiers in order to minimize the Bayesian error estimates
of the classifiers. Similarly, we can derive the effective classconditional densities for defining the OBTL classifier in the
target domain, albeit with the posterior of the target parameters
derived from both the target and source datasets.
Suppose that x denotes a d × 1 new observed data point
in the target domain, where we aim to optimally classify
it to one of the classes l ∈ {1, · · · , L}. In the context of
the optimal Bayesian classifier, we need the effective classconditional densities for the L classes, defined as
Z
p(x|µlt , Λlt )π ⋆ (µlt , Λlt )dµlt dΛlt ,
(39)
p(x|l) =
µlt ,Λlt

7

for l ∈ {1, · · · , L}, where π ⋆ (µlt , Λlt ) = p(µlt , Λlt |Dtl , Dsl )
is the posterior of (µlt , Λlt ) upon observation of Dtl and Dsl .
The likelihood p(x|µlt , Λlt ) and posterior p(µlt , Λlt |Dtl , Dsl ) are
given in (1) and (35), respectively. Hence,
Z
n
1
d
|Λlt | 2
p(x|l) = (2π)− 2 Al
µlt ,Λlt


′


1
× exp − x − µlt Λlt x − µlt
2
!
1

′ l l
κlt,n l
l 2
l
l
× Λt exp −
µt − mt,n Λt µt − mt,n
(40)
2


ν l +nlt −d−1
1 l −1 l
l
2
etr − Tt
× Λt
Λt
2
 l

ν + nls ν l 1 l l l ′ l
dµlt dΛlt .
× 1 F1
; ; F Λt F Ts
2
2 2
Similarly, we can simplify (40) as
Z
n
d
1
p(x|l) = (2π)− 2 Al
|Λlt | 2

Tlx

−1

= Tlt

−1

+

κlt,n
κlt,n + 1

(45)

where ξt = (ξt1 , · · · , ξtL ) are the concentration parameters, and
ξtl > 0 for l ∈ {1, · · · , L}. As the Dirichlet distribution is a
conjugate prior for the categorical distribution, upon observing
n = (n1t , · · · , nL
t ) data for class l in the target domain, the
posterior has a Dirichlet distribution:
π ⋆ = (c1t , · · · , cL
t |n) ∼ Dir(L, ξt + n)

= Dir(L, ξt1 + n1t , · · · , ξtL + nL
t ),
(46)
with the posterior mean of clt as
ξtl + nlt
,
(47)
Nt + ξt0
PL
PL l
l
0
where Nt =
l=1 nt and ξt =
l=1 ξt . As such, the
optimal Bayesian transfer learning (OBTL) classifier for any
new unlabeled sample x in the target domain is defined as

(41)

where
κlx = κlt,n + 1 = κlt + nlt + 1,

(c1t , · · · , cL
t ) ∼ Dir(L, ξt ),

Eπ⋆ (clt ) =

µl ,Λl

t
t

 l

′ l l
κx l
l
µt − mx Λt µt − mlx
× exp −
2


ν l +nlt +1−d−1
−1 l
1
2
× Λlt
etr − Tlx
Λt
2
 l

l
l
ν + ns ν 1 l l l ′ l
dµlt dΛlt ,
× 1 F1
; ; F Λt F Ts
2
2 2

V. O PTIMAL BAYESIAN T RANSFER L EARNING C LASSIFIER
Let clt be the prior probability that the target sample x
l
belongs
PL l to the class l ∈ {1, · · · , L}. Since 0 < ct < 1 and
l=1 ct = 1, a Dirichlet prior is assumed:

κlt,n mlt,n + x
,
κt,n + 1

′
mlt,n − x mlt,n − x .
mlx =

(42)
The integration in (41) is similar to the one in (36). As a result,
using (38),

 l
  d2 d ν l +nl +1
(
)
t
d
ν + nlt + 1
2π
2
p(x|l) = (2π)− 2 Al
2
Γ
d
κlx
2


l
l
ν +nt +1
l
l
l
l
l
′
ν + ns ν + nt + 1 ν
2
l l l l
Tlx
F
.
,
;
;
T
F
T
F
2 1
s
x
2
2
2
(43)
By replacing the value of Al , we have the effective classconditional density. We denote OOBTL (x|l) = p(x|l), since it
is the objective function for the OBTL classifier. As such,
!d
 l

l
κlt,n 2
ν
+
n
+
1
t
−d
Γd
OOBTL (x|l) = π 2
κlx
2

 l
ν l +nlt
ν l +nlt +1
l
ν + nt
− 2
2
Tlt
Tlx
× Γ−1
d
2
(44)

 l
ν + nls ν l + nlt + 1 ν l l l l l ′
,
; ; Ts F Tx F
× 2 F1
2
2
2

 l
l
l
l
l
′
ν
+
n
ν
ν
+
n
t
−1
s
l l l l
,
; ; Ts F Tt F
× 2 F1
.
2
2
2

ΨOBTL (x) = arg

max

l∈{1,··· ,L}

Eπ⋆ (clt )OOBTL (x|l),

(48)

which minimizes the expected error of the classifier. If we
do not have any prior knowledge for the selection of classes,
we use the same concentration parameter for all the classes:
ξt = (ξ, · · · , ξ). Hence, if the number of samples in each class
is the same, n1t = · · · = nL
t , the first term Eπ ⋆ is the same
for all the classes and (48) is reduced to:
ΨOBTL (x) = arg

max

l∈{1,··· ,L}

OOBTL (x|l).

(49)

VI. OBC IN TARGET D OMAIN
To see how the source data can help improve the performance, we compare the OBTL classifier with the OBC
when there is only the target domain. Using exactly the same
modeling and parameters as the previous sections, the priors
for µlt and Λlt , from (3) and(17), are givenby
−1
,
µlt |Λlt ∼ N mlt , κlt Λlt
(50)
Λlt ∼ Wd (Mlt , ν l ).
Using Lemma 1, upon observing the dataset Dtl , the posteriors
of µlt and Λlt will be

−1 
,
µlt |Λlt , Dtl ∼ N mlt,n , κlt,n Λlt
(51)
l
l
l
l
Λt |Dt ∼ Wd (Mt,n , νt,n ),
where
κlt,n = κlt + nlt ,

l
νt,n
= ν l + nlt ,

mlt,n =

κlt mlt + nlt x̄lt
,
κlt + nlt

′
κlt nlt
(mlt − x̄lt )(mlt − x̄lt ) ,
κlt + nlt
(52)
with the corresponding sample mean and covariance:

Mlt,n

−1

= Mlt

−1

+ Slt +

l

l

x̄lt

nt
1 X
xl ,
= l
nt i=1 t,i

Slt

=

nt
X
i=1

xlt,i − x̄lt



′
xlt,i − x̄lt . (53)

8

Using (39) and similar integral steps, the effective classconditional densities p(x|l) = OOBC (x|l) for the OBC are
derived as [23]
! d2

 l
κlt,n
ν + nlt + 1
−d
OOBC (x|l) = π 2
Γd
2
κlt,n + 1
(54)
 l

ν l +nlt
ν l +nlt +1
l
ν
+
n
−
t
2
2
× Γ−1
Mlt,n
,
Mlx
d
2
where
Mlx

−1

= Mlt,n

−1

+

κlt,n
l
κt,n +

′

1

(mlt,n −x)(mlt,n −x) . (55)

The multi-class OBC [31], under a zero-one loss function, can
be defined as
ΨOBC (x) = arg

max

l∈{1,··· ,L}

Eπ⋆ (clt )OOBC (x|l).

max

l∈{1,··· ,L}

OOBC (x|l).

(57)

For binary classification, the definition of the OBC in (56) is
equivalent to the definition in [23], where it is defined to be
the binary classifier possessing the minimum Bayesian mean
square error estimate [32] relative to the posterior distribution.
Theorem 6. If

Mlts

= 0 for all l ∈ {1, · · · , L}, then

ΨOBTL (x) = ΨOBC (x),

Bd (α, β) =

where D ⊆ Rd is an open set and λ is a real parameter. If
g(λ) has a unique minimum over D at point ŷ ∈ D, then the
Laplace approximation to I is given by
′′
d
d
1
I˜ = (2π) 2 λ− 2 |g (ŷ)|− 2 h(ŷ) exp(−λg(ŷ)),

∂ g(y)
is the Hessian of g(y). The hywhere g (y) = ∂y∂y
T
pergeometric function 2 F1 (a, b; c; X) depends only on the
eigenvalues of the symmetric matrix X. Hence, without loss
of generality, it is assumed that X = diag{x1 , · · · , xd }. The
following g and h functions are used for (59):
′′

g(Y) = −a log |Y| − (c − a) log |Id − Y| + log |Id − XY|,
h(Y) = Bd−1 (a, c − a)|Y|−

d

2 F̃1 (a, b; c; X) =

2 F1 (a, b; c; X)

×

Z

0d <Y<Id

×

= Bd−1 (a, c − a)

|Y|a−

d+1
2

|Id − Y|c−a−

d+1
2

(59)

d+1
2

.

d
Y

i=1

d(d+1)

22 π 4
−1
J2,12
Bd (a, c − a)

{ŷia (1

c−a

− ŷi )

(64)
−b

(1 − xi ŷi )

},

2a
ŷi = p
,
2
τ − 4axi (c − b) − τ

(65)

with τ = xi (b − a) − c, and
d Y
d
Y

i=1 j=i

|Id − XY|−b dY,

|Id − Y|−

where ŷi is defined as

J2,1 =

We have derived the effective class-conditional densities
in closed forms (44). However, deriving the OBTL classifier
(48) requires computing the Gauss hypergeometric function
of matrix argument. Computing the exact values of hypergeometirc functions of matrix argument using the series of zonal
polynomial, as in (11), is time-consuming and is not scalable
to high dimension. To facilitate computation, we propose to
use the Laplace approximation of this function, as in [33],
which is computationally efficient and scalable.
The Gauss hypergeomeric function has the following integral representation:

d+1
2

(63)
Using (62) and (63), the Laplace approximation to
2 F1 (a, b; c; X) is given by [33]

l

VII. L APLACE A PPROXIMATION OF G AUSS
H YPERGEOMETRIC F UNCTION

(62)

2

(58)

Proof. If
= 0 for all l ∈ {1, · · · , L}, then F = 0. Since
2 F1 (a, b; c; 0) = 1 for any values of a, b, and c, the Gauss
hypergeometric functions will disappear in (44). From (31) and
(52), Tlt = Mlt,n . From (42) and (55), Tlx = Mlx . As a result,
OOBTL (x|l) = OOBC (x|l), and consequently, ΨOBTL (x) =
ΨOBC (x).

(60)

y∈D

meaning that if there is no interaction between the source
and target domains in all the classes a priori, then the OBTL
classifier turns to the OBC classifier in the target domain.
Mlts

Γd (α)Γd (β)
,
Γd (α + β)

where Γd (α) is the multivariate gamma function defined in
(7). The Laplace approximation is one common solution to
approximate the integral
Z
I=
h(y) exp(−λg(y))dy,
(61)

(56)

Similar to the OBTL, in the case of equal prior probabilities
for the classes,
ΨOBC (x) = arg

which is valid under the following conditions: X ∈ Cd×d
is symmetric and satisfies Re(X) < Id , Re(a) > d−1
2 , and
Re(c − a) > d−1
.
B
(α,
β)
is
the
multivariate
beta
function
d
2

{a(1− ŷi)(1− ŷj )+(c−a)ŷi ŷj −bLi Lj }, (66)

with
Li =

xi ŷi (1 − ŷi )
.
1 − xi ŷi

(67)

2 F1 (a, b; c; X) at X = 0 is 1, that is,
= 1. As a result, the Laplace approximation
in (64) is calibrated at X = 0 to give the calibrated Laplace
approximation [33]:

The value of

2 F1 (a, b; c; 0)

2 F̂1 (a, b; c; X)

=

2 F̃1 (a, b; c; X)

= ccd−

d(d+1)
4

−1

R2,12
)

2 F̃1 (a, b; c; 0)
(  
c−a
d
a
Y
ŷi
1 − ŷi
×
(1 − xi ŷi )−b
a
c
−
a
i=1

(68)
,

9

ns = 200, ν = 25

R2,1 =

i=1 j=i

ŷi ŷj
(1 − ŷi )(1 − ŷj )
+
a
c−a

bxi xj ŷi ŷj (1 − ŷi )(1 − ŷj )
−
(1 − xi ŷi )(1 − xj ŷj )a(c − a)

(69)


.

VIII. E XPERIMENTS
A. Synthetic datasets
We have considered a simulation setup and evaluated the
OBTL classifiers by the average classification error with
different joint prior densities modeling the relatedness of the
source and target domains. The setup is as follows. Unless
mentioned, the feature dimension is d = 10, the number of
classes in each domain is L = 2, the number of source training
data per class is ns = nls = 200, the number of target training
data per class is nt = nlt = 10, ν = ν l = 25, κt = κlt = 100,
κs = κls = 100, for both the classes l = 1, 2, m1t = 0d ,
m2t = 0.05 × 1d , m1s = m1t + 1d , and m2s = m2t + 1d , where
0d and 1d are d × 1 all-zero and all-one vectors, respectively.
For the scale matrices, we choose Mlt = kt Id , Mls = ks Id ,
and Mlts = kts Id for two classes l = 1, 2, where Id is the d×d
identity matrix. Note that choosing an identity matrix for Mlts
makes sense when the order of the features in the two domains
is the 
same. We 
have the constraint that the scale matrix
Mlt Mlts
l
′
should be positive definite for any class l.
M =
l
l
Mts

Ms

It is easy to check the following corresponding
√ constraints on
kt ks . We define
kt , ks , and
k
:
k
>
0,
k
>
0,
and
|k
|
<
ts
t
s
ts
√
kts = α kt ks , where |α| < 1. In this particular example,
the value of |α| shows the amount of relatedness between the
source and target domains. If |α| = 0, the two domains are not
related and if |α| is close to one, we have greater relatedness.
We set kt = ks = 1 and plot the average classification error
curves for different values of |α|. All the simulations assume
equal prior probabilities for the classes, so we use (49) and
(57) for the OBTL classifier and OBC, respectively.
We evaluate the prediction performance according to the
common evaluation procedure of Bayesian learning by average
classification errors. To sample from the prior (2) we first
sample from a Wishart
W2d (Ml , ν l ) to get a
 l distribution

l
Λt Λts
′
sample for Λl =
, for each class l = 1, 2, and
l
l
Λts

Λs

then pick (Λlt , Λls ), which is a joint sample from p(Λlt , Λls )
in (16). Then given Λlt and Λls , we sample from (3) to get
samples of µlt and µls for l = 1, 2. Once we have µlt , µls , Λlt ,
and Λls , we generate 100 different training and test sets from
(1). Training sets contain samples from both the target and
source domains, but the test set contains only samples from
the target domain. As the numbers of source and target training
data per class is ns and nt , there are Lns and Lnt source and
target training data in total, respectively. We assume the size of
the test set per class is 500 in the simulations, so 1000 in total.
For each training and test set, we use the OBTL and its targetonly version, OBC, and calculate the error. Then we average
all the errors for 100 different training and test sets. We further
repeat this whole process 100 times for different realizations
of Λlt and Λls , µlt , and µls for l = 1, 2, and finally average all

Average Classification Error

0.35

d Y
d 
Y

nt = 10, ν = 25

OBC, target-only
OBTL, α = 0.5
OBTL, α = 0.7
OBTL, α = 0.9

0.3

OBC, target only
OBTL, α = 0.5
OBTL, α = 0.7
OBTL, α = 0.9

0.26

Average Classification Error

where

0.25

0.2

0.24

0.22

0.2

0.18

0.15
0.16
10

20

30

40

50

50

100

150

200

nt

ns

(a)

(b)

250

300

350

400

Figure 2: (a) Average classification error versus the number of target training
data per class, nt , (b) Average classification error versus the number of source
training data per class, ns .

the errors and return the average classification error. Note that
in all figures, the hyperparameters used in the OBTL classifier
are the same as the ones used for simulating data, except for
the figures showing the sensitivity of the performance with
respect to different hyperparameters, in which case we assume
that true values of the hyperparameters used for simulating
data are unknown.
To examine how the source data improves the classifier
in target domain, we compare the performance of the OBTL
classifier with the OBC designed in the target domain alone.
The average classification error versus nt is depicted in Fig. 2a
for the OBC and OBTL with different values of α. When α is
close to one, the performance of the OBTL classifier is much
better than that of the OBC, this due to the greater relatedness
between the two domains and appropriate use of the source
data. This performance improvement is especially noticeable
when nt is small, which reflects the real-world scenario. In
Fig. 2a, we also observe that the errors of the OBTL classifier
and OBC are converging to a similar value when nt gets very
large, meaning that the source data are redundant when there is
a large amount of target data. When α is larger, the error curves
converge faster to the optimal error, which is the average Bayes
error of the target classifier. Recall that when α = 0, the OBTL
classifier reduces to the OBC. In this particular example, the
sign of α does not matter in the performance of the OBTL,
which can be verified by (44). Hence, we can use |α| in all
the cases.
Figure 2b depicts average classification error versus ns for
the OBC and OBTL with different values of α. The error of
the OBC is constant for all ns as it does not employ the source
data. The error of the OBTL classifier equals that of the OBC
when ns = 0 and starts to decrease as ns increases. In Fig.
2b when α is larger, the rate of improvement is greater since
the two domains are more related.
We investigate the sensitivity of the OBTL with respect to
the hyperparameters. Fig. 3 represents the average classification error of the OBTL with respect to |α|, where we assume
that we do not know the true value αtrue of the amount of
relatedness between source and target domains. In Figs. 3a3d we plot the error curves when αtrue = 0.3, 0.5, 0.7, 0.9,
respectively. We observe several important trends in these
figures. First of all, the performance gain of the OBTL

10

0.26
0.25
0.24
0.23
0.22

OBTL
OBC

0.27
0.26
0.25
0.24
0.23
0.22

0.6

0.8

1

0.2

0.4

(a)

0.8

1

0.22
0.21
0.2
0.19

60

80

0.6

|α|

(c)

Figure 3:

0.8

1

100

120

20

0.2

0.18

0.4

60

80

0.6

100

120

0.8

(b)
α = 0.9, νtrue = 50

0.32

0.28

0.26

0.24

0.22

1

OBTL
OBC

0.3

0.28

0.26

0.24

0.22
20

40

60

80

|α|

100

120

140

20

40

60

80

ν

(d)

(c)

120

140

(d)
Average classification error vs ν

α = 0.5, κtrue = 50

0.3

100

ν

Figure 4:

Average classification error vs |α|

towards the OBC depends heavily on the relatedness (value
of αtrue ) of source and target and the value of α used in
the classifier. Generally speaking, there exists an αmax in
(0, 1) such that for |α| < αmax , the OBTL has a performance
gain towards the OBC, where maximum gain is achieved at
|α| = αtrue . Second, the performance gain is higher when
the two domains are highly related (Fig. 3d). Third, when
the two domains are very related, for example, αtrue = 0.9
in Fig. 3d, αmax = 1, meaning that for any |α|, the OBTL
has performance gain towards the target-only OBC. However,
when the source and target domains are not related much, like
Figs. 3a-3b, αmax < 1, and choosing |α| greater than αmax
leads to performance loss compared to the OBC. This means
that exaggeration in the amount of relatedness between the two
domains can hurt the transfer learning classifier when the two
domains are not actually related, which refers to the concept
of negative transfer.
Figure 4 shows the errors versus ν, assuming unknown true
value νtrue , for different values of α (0.5 and 0.9) and νtrue
(25 and 50). The salient point here is that the performance of
the OBTL classifier is not so sensitive to ν if it is chosen in
its allowable range, that is, ν ≥ 2d. In Fig. 4, the error of the
OBTL does not change much for ν ≥ 2d = 20. As a result,
we can choose any arbitrary ν ≥ 2d in real datasets without
worrying about critical performance deterioration.
Figure 5 depicts average classification error versus κt for
two different values of α (0.5 and 0.9), where the true value of
κt is κtrue = 50. Similar to ν, if κt is greater than a value (20
in Fig. 5), the performance does not change much. According
to (31), it is better to choose κlt and κls to be proportional
to nt and ns , respectively, since the values of updated means
mlt,n and mls,n are weighted averages of our prior knowledge
about means, mlt and mls , and the sample means x̄lt and x̄ls .
Assuming that κt = βt nt and κs = βs ns , for some βt , βs > 0,

140

ν

0.2
0.2

40

OBTL
OBC

0.22

0

0.27

140

α = 0.9, νtrue = 25

0.3

α = 0.9, κtrue = 50

0.28
OBTL
OBC

Average Classification Error

0.4

0.28

(a)

0.16
0.2

0.29

ν

OBTL
OBC

0.24

0.23

0

40

(b)

OBTL
OBC

0.3

0.26
20

ν = 25, αtrue = 0.9

Average Classification Error

Average Classification Error

0.6

|α|

ν = 25, αtrue = 0.7

0.24

0.23

0.22
0

|α|

0.25

0.24

Average Classification Error

0.4

OBTL
OBC

OBTL
OBC

Average Classification Error

0.2

Average Classification Error

0

0.25

α = 0.5, νtrue = 50

0.31

OBTL
OBC

0.26

Average Classification Error

Average Classification Error

Average Classification Error

0.27

α = 0.5, νtrue = 25

ν = 25, αtrue = 0.5

0.28

OBTL
OBC

Average Classification Error

ν = 25, αtrue = 0.3

0.28

0.28

0.26

0.24

0.22

0.2

0.26
0.24
0.22
0.2
0.18
0.16

20

40

60

80

100

κt

40

60

80

100

κt

(a)

Figure 5:

20

(b)
Average classification error vs κt

if we have higher trust on our priors on means, we pick higher
βt and βs (like in Fig. 5); but for the untrustworthy priors, we
choose lower values for βt and βs .
Sensitivity results in Figs. 3, 4, and 5 reveal that in our
simulation setup the performance improvement of the OBTL
depends on the value of α and true relatedness (αtrue in
this example) between the two domains and is not affected
that much by the choices of other hyperparameters like ν,
κt , and κs . We could have a reasonable range of α to get
improved performance but the correct estimates of relatedness
or transferability are critical, which is an important future
research direction (see Conclusions in Section IX).
B. Real-world benchmark datasets
We test the OBTL classifier on Office [34] and Caltech256
[35] image datasets, which have been adopted to help benchmark different transfer learning algorithms in the literature. We
have used exactly the same evaluation setup and data splits of
MMDT (Max-Margin Domain Transform) [10].
• Office dataset: This dataset has images in three different
domains: amazon, webcam, and dslr. The dataset contains

11

Table I: Semi-supervised accuracy for different source and target domains in the Office+Clatech256 dataset using SURF features. Domain names are denoted
as a: amazon, w: webcam, d: dslr, c: Caltech256. The numbers in red show the best accuracy and the numbers in blue show the second best accuracy in each
column. The results of the first six methods have been adopted from [17]. Similar to [17], we have also used the evaluation setup of [10] for the OBTL.

1-NN-t
SVM-t
HFA [9]
MMDT [10]
CDLS [12]
ILS (1-NN) [17]
OBTL

Table II:

a → w a → d a → c w → a w → d w → c d → a d → w d → c c → a c → w c → d Mean
34.5
33.6
19.7
29.5
35.9
18.9
27.1
33.4
18.6
29.2
33.5
34.1
29.0
63.7
57.2
32.2
46.0
56.5
29.7
45.3
62.1
32.0
45.1
60.2
56.3
48.9
57.4
55.1
31.0
56.5
56.5
29.0
42.9
60.5
30.9
43.8
58.1
55.6
48.1
64.6
56. 7 36.4
47.7
67.0
32.2
46.9
74.1
34.1
49.4
63.8
56.5
52.5
68.7
60.4
35.3
51.8
60.7
33.5
50.7
68.5
34.9
50.9
66.3
59.8
53.5
59.7
49.8
43.6
54.3
70.8
38.6
55.0
80.1
41.0
55.1
62.9
56.2
55.6
72.3
60.8
43.1
55.0
75.8
38.1
54.3
85.2
40.6
54.7
70.6
61.4
59.3

The values of hyperparameters of the OBTL used in each experiment. nt and ns are based on the data splits provided by [10].

a→w a→d a→c w→a w→d w→c d→a d→w d→c c→a c→w c→d
nt
3
3
3
3
3
3
3
3
3
3
3
3
ns
20
20
20
8
8
8
8
8
8
8
8
8
βt
1
2
5
1
1
3
1
2
2
1
1
1
βs
1
2
5
1
1
3
1
2
2
1
1
1
kt 0.01
0.005 0.002 0.005
0.01
0.005 0.005
0.02
0.003 0.004 0.004 0.005
ks 0.01
0.004 0.002 0.005
0.01
0.005 0.005
0.02
0.003 0.004 0.004 0.005
α
0.7
0.6
0.99
0.9
0.99
0.99
0.99
0.99
0.99
0.8
0.8
0.7

31 classes including the office stuff like backpack, chair,
keyboard, etc. The three domains amazon, webcam, and dslr
contain images from Amazon’s website, a webcam, and a digital single-lens reflex (dslr) camera, respectively, with different
lighting and backgrounds. SURF [36] image features are used
in all the domains, which are of dimension 800.
• Office + Caltech256 dataset: This dataset has L = 10
common classes of both Office and Caltech256 datasets with
the same feature dimension d = 800. Accordingt to the data
splits of [10], the numbers of training data per class in the
source domain are ns = 20 for amazon and ns = 8 for the
other three domains, and in the target domain nt = 3 for all
the four domains. For this four-domain dataset, 20 random
train-test splits have been created by [10]. We run the OBTL
classifier on that 20 provided train-test splits and report the
average accuracy. Note that the test data are solely from the
target domains. Authors of MMDT [10] reduce the dimension
to d = 20 using PCA. We follow the same procedure for the
OBTL classifier.
Following the comparison framework of [17], which used
the same evaluation setup of [10], we compare the OBTL’s
performance in terms of accuracy (10-class) in Table 1
with two target-only classifiers and four state-of-the-art semisupervised transfer learning algorithms (including [17] itself).
The evaluation setup is exactly the same for the OBTL and all
the other six methods. As a result, we use the results of [17]
for the first six methods in Table 1 and compare them with
the OBTL classifier. The six methods are as follows.
• 1-NN-t and SVM-t: The Nearest Neighbor (1-NN) and
linear SVM classifiers designed using only the target data.
• HFA [9]: This Heterogeneous Feature Augmentation (HFA)
method learns a common latent space between source and
target domains using the max-margin approach and designs
a classifier in that common space.
• MMDT [10]: This Max-Margin Domain Transform
(MMDT) method learns a transformation between the source

and target domains and employs the weighted SVM for
classification.
• CDLS [12]: This Cross-Domain Landmark Selection
(CDLS) is a semi-supervised heterogeneous domain adaptation
method, which derives a domain-invariant feature space for
improved classification performance.
• ILS (1-NN) [17]: This is a recent method that learns
an Invariant Latent Space (ILS) to reduce the discrepancy
between the source and target domains and uses Riemannian
optimization techniques to match statistical properties between
samples projected into the latent space from different domains.
In Table 1, we have calculated the accuracy of the OBTL
classifier in 12 distinct experiments, where the source-target
pairs are different (source → target) in each experiment. We
have marked the best accuracy in each column with red and
the second best accuracy with blue. In 6 out of 12 cases, the
OBTL has the best accuracy. In the other 6 cases, the OBTL
has the second best accuracy, but its accuracy is very close to
the best accuracy. We have written the mean accuracy of each
method in the last column, which has been averaged over all
the 12 different experiments. The OBTL classifier has the best
mean accuracy and the ILS [17] has the second best accuracy
among all the methods.
• Hyperparameters of the OBTL: We assume the same
values of hyperparameters for all the 10 classes in each
domain, so we can drop the superscript l denoting the class
label. We set ν = 10d = 200 for all the experiments. We
choose α separately in each experiment since the relatedness
between distinct pairs of domains are different. For mt and
ms , we pool all the target and source data in all the 10 classes,
respectively, and use the sample means of the datasets. We
find reasonable values for kt , ks , α, βt (κt = βt nt ), and βs
(κs = βs ns ) using a cross-validation method. We show the
values of those hyperparameters used in each experiment in
Table 2. We have assumed equal prior probabilities for all the
classes and used (49) for the OBTL classifier.

12

IX. C ONCLUSIONS A ND F UTURE W ORKS
We have constructed a Bayesian transfer learning framework
to tackle the semi-supervised and supervised transfer learning
problems. The proposed Optimal Bayesian Transfer Learning
(OBTL) classifier can deal with the lack of labeled data in the
target domain and is optimal in this new Bayesian framework
since it minimizes the expected classification error. We have
obtained the closed-form posterior distribution of the target
parameters and accordingly the closed-form effective classconditional densities in the target domain to define the OBTL
classifier. As the OBTL’s objective function consists of hypergeometric functions of matrix argument, we use the Laplace
approximations of those functions to derive a computationally
efficient and scalable OBTL classifier, while preserving its
superior performance. We have compared the performance
of the OBTL with its target-only version, OBC, to see how
transferring from source to target domain can help. We have
tested the OBTL classifier with real-world benchmark image
datasets and demonstrated its excellent performance compared
to other state-of-the-art domain adaption methods.
This paper considers a Gaussian model, in which we can
derive closed-form solutions, as the case with the OBC.
Since many practical problems cannot be approximated by
a Gaussian model, an important aspect of OBC development
has been the utilization of MCMC methods [37], [38]. These
can be naturally extended to the OBTL setting, albeit, with
greater computational burden.
We have only considered two domains in this paper, assuming there is only one source domain. Having seen the good
performance of the OBTL classifier in two domains, in future
work, we are going to apply it to the multi-source transfer
learning problems, where we can benefit from the knowledge
of different related sources in order to further improve the
target classifier.
As in the case of the OBC, a basic engineering aspect
of the OBTL is prior construction. This has been studied
under different conditions in the context of the OBC: using
the data from unused features to infer a prior distribution
[39], deriving the prior distribution from models of the datagenerating technology [37], deriving the prior from the uncertainties in the mathematical model characterizing the relevant
physical system [40], and applying constraints based on prior
knowledge to map the prior knowledge into a prior distribution
via optimization [41], [42], [43]. The latter methods are very
general and have been placed into a formal mathematical
structure in [43], where the prior results from an optimization
involving the Kullback-Leibler (KL) divergence constrained
by conditional probability statements characterizing physical
knowledge, such as genetic pathways in genomic medicine. A
key focus of our future work will be to extend this general
framework to the OBTL, which will require a formulation
that incorporates knowledge relating the source and target
domains. It should be emphasized that with optimal Bayesian
classification, as well as with optimal Bayesian filtering [44],
[45], [46], the prior distribution is not on the operator to be
designed (classifier or filter) but on the underlying scientific
model (feature-label distribution, covariance matrix, or obser-

vation model) for which the operator is optimized. It is for this
reason that uncertainty in the scientific model can be mapped
into a prior distribution based on physical laws.
ACKNOWLEDGMENT
This work was funded in part by Award CCF-1553281 from
the National Science Foundation.
R EFERENCES
[1] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–
1359, 2010.
[2] H. Venkateswara, S. Chakraborty, and S. Panchanathan, “Deep-learning
systems for domain adaptation in computer vision: Learning transferable
feature representations,” IEEE Signal Processing Magazine, vol. 34,
no. 6, pp. 117–129, 2017.
[3] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa, “Visual domain
adaptation: A survey of recent advances,” IEEE signal processing
magazine, vol. 32, no. 3, pp. 53–69, 2015.
[4] K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer
learning,” Journal of Big Data, vol. 3, no. 1, p. 9, 2016.
[5] G. Csurka, “Domain adaptation for visual applications: A comprehensive
survey,” arXiv preprint arXiv:1702.05374, 2017.
[6] N. Zou, Y. Zhu, J. Zhu, M. Baydogan, W. Wang, and J. Li, “A transfer
learning approach for predictive modeling of degenerate biological
systems,” Technometrics, vol. 57, no. 3, pp. 362–373, 2015.
[7] P. Ganchev, D. Malehorn, W. L. Bigbee, and V. Gopalakrishnan,
“Transfer learning of classification rules for biomarker discovery and
verification from molecular profiling studies,” Journal of biomedical
informatics, vol. 44, pp. S17–S23, 2011.
[8] B. Gong, Y. Shi, F. Sha, and K. Grauman, “Geodesic flow kernel
for unsupervised domain adaptation,” in Computer Vision and Pattern
Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 2066–
2073.
[9] L. Duan, D. Xu, and I. Tsang, “Learning with augmented features for
heterogeneous domain adaptation,” ICML, 2012.
[10] J. Hoffman, E. Rodner, T. Darrell, J. Donahue, and K. Saenko, “Efficient
learning of domain-invariant image representations,” in International
Conference on Learning Representations (ICLR), 2013.
[11] J. Hoffman, E. Rodner, J. Donahue, B. Kulis, and K. Saenko, “Asymmetric and category invariant feature transformations for domain adaptation,” International journal of computer vision, vol. 109, no. 1-2, pp.
28–41, 2014.
[12] Y.-H. Hubert Tsai, Y.-R. Yeh, and Y.-C. Frank Wang, “Learning crossdomain landmarks for heterogeneous domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 5081–5090.
[13] K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf,
and A. J. Smola, “Integrating structured biological data by kernel
maximum mean discrepancy,” Bioinformatics, vol. 22, no. 14, pp. e49–
e57, 2006.
[14] W. Dai, Q. Yang, G.-R. Xue, and Y. Yu, “Boosting for transfer learning,” in Proceedings of the 24th international conference on Machine
learning. ACM, 2007, pp. 193–200.
[15] L. Duan, I. W. Tsang, D. Xu, and S. J. Maybank, “Domain transfer
svm for video concept detection,” in Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009,
pp. 1375–1381.
[16] L. Bruzzone and M. Marconcini, “Domain adaptation problems: A
dasvm classification technique and a circular validation strategy,” IEEE
transactions on pattern analysis and machine intelligence, vol. 32, no. 5,
pp. 770–787, 2010.
[17] S. Herath, M. Harandi, and F. Porikli, “Learning an invariant hilbert
space for domain adaptation,” in 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), July 2017, pp. 3956–3965.
[18] N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy, “Optimal
transport for domain adaptation,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 39, no. 9, pp. 1853–1865, Sept 2017.
[19] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable
features with deep adaptation networks,” in International Conference
on Machine Learning, 2015, pp. 97–105.
[20] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Unsupervised domain
adaptation with residual transfer networks,” in Advances in Neural
Information Processing Systems, 2016, pp. 136–144.

13

[21] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training
of neural networks,” Journal of Machine Learning Research, vol. 17,
no. 59, pp. 1–35, 2016.
[22] M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,” in
Advances in neural information processing systems, 2016, pp. 469–477.
[23] L. A. Dalton and E. R. Dougherty, “Optimal classifiers with minimum
expected error within a Bayesian framework—Part I: Discrete and
gaussian models,” Pattern Recognition, vol. 46, no. 5, pp. 1301 – 1314,
2013.
[24] ——, “Optimal classifiers with minimum expected error within a
Bayesian framework — Part II: Properties and performance analysis,”
Pattern Recognition, vol. 46, no. 5, pp. 1288 – 1300, 2013.
[25] R. J. Muirhead, Aspects of multivariate statistical theory. John Wiley
& Sons, 2009.
[26] K. Halvorsen, V. Ayala, and E. Fierro, “On the marginal distribution of
the diagonal blocks in a blocked Wishart random matrix,” International
Journal of Analysis, vol. 2016, pp. 1–5, 2016.
[27] D. K. Nagar and J. C. Mosquera-Benıtez, “Properties of matrix variate
hypergeometric function distribution,” Applied Mathematical Sciences,
vol. 11, no. 14, pp. 677–692, 2017.
[28] A. G. Constantine, “Some non-central distribution problems in multivariate analysis,” Ann. Math. Statist., vol. 34, no. 4, pp. 1270–1285, 12
1963.
[29] D. K. Nagar and S. Nadarajah, “Appell’s hypergeometric functions of
matrix arguments,” Integral Transforms and Special Functions, vol. 28,
no. 2, pp. 91–112, 2017.
[30] A. K. Gupta, D. K. Nagar, and L. E. Sánchez, “Properties of matrix
variate confluent hypergeometric function distribution,” Journal of Probability and Statistics, vol. 2016, 2016.
[31] L. A. Dalton and M. R. Yousefi, “On optimal Bayesian classification
and risk estimation under multiple classes,” EURASIP Journal on
Bioinformatics and Systems Biology, vol. 2015, no. 1, p. 8, 2015.
[32] L. A. Dalton and E. R. Dougherty, “Bayesian minimum mean-square error estimation for classification error-Part I: Definition and the Bayesian
MMSE error estimator for discrete classification,” IEEE Transactions on
Signal Processing, vol. 59, no. 1, pp. 115–129, Jan 2011.
[33] R. W. Butler and A. T. A. Wood, “Laplace approximations for hypergeometric functions with matrix argument,” The Annals of Statistics,
vol. 30, no. 4, pp. 1155–1177, 2002.
[34] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual category
models to new domains,” in Proceedings of the 11th European Conference on Computer Vision: Part IV, ser. ECCV’10. Berlin, Heidelberg:
Springer-Verlag, 2010, pp. 213–226.
[35] G. Griffin, A. Holub, and P. Perona, “Caltech-256 object category
dataset,” Technical Report 7694, California Institute of Technology,
2007.
[36] H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded up robust
features,” Computer vision–ECCV 2006, pp. 404–417, 2006.
[37] J. M. Knight, I. Ivanov, and E. R. Dougherty, “MCMC implementation of
the optimal Bayesian classifier for non-Gaussian models: Model-based
RNA-seq classification,” BMC bioinformatics, vol. 15, no. 1, p. 401,
2014.
[38] J. M. Knight, I. Ivanov, K. Triff, R. S. Chapkin, and E. R. Dougherty,
“Detecting multivariate gene interactions in RNA-seq data using optimal Bayesian classification,” IEEE/ACM transactions on computational
biology and bioinformatics, 2015.
[39] L. A. Dalton and E. R. Dougherty, “Application of the Bayesian MMSE
estimator for classification error to gene expression microarray data,”
Bioinformatics, vol. 27, no. 13, pp. 1822–1831, 2011.
[40] A. Zollanvari and E. R. Dougherty, “Incorporating prior knowledge
induced from stochastic differential equations in the classification of
stochastic observations,” EURASIP Journal on Bioinformatics and Systems Biology, vol. 2016, no. 1, p. 2, 2016.
[41] M. S. Esfahani and E. R. Dougherty, “Incorporation of biological
pathway knowledge in the construction of priors for optimal Bayesian
classification,” IEEE/ACM Transactions on Computational Biology and
Bioinformatics, vol. 11, no. 1, pp. 202–218, Jan 2014.
[42] ——, “An optimization-based framework for the transformation of
incomplete biological knowledge into a probabilistic structure and its application to the utilization of gene/protein signaling pathways in discrete
phenotype classification,” IEEE/ACM Transactions on Computational
Biology and Bioinformatics, vol. 12, no. 6, pp. 1304–1321, Nov 2015.
[43] S. Boluki, M. S. Esfahani, X. Qian, and E. R. Dougherty, “Incorporating biological prior knowledge for bayesian learning via maximal
knowledge-driven information priors,” BMC Bioinformatics, vol. 18,
no. 14, p. 552, Dec 2017.

[44] L. A. Dalton and E. R. Dougherty, “Intrinsically optimal Bayesian robust
filtering,” IEEE Transactions on Signal Processing, vol. 62, no. 3, pp.
657–670, Feb 2014.
[45] X. Qian and E. R. Dougherty, “Bayesian regression with network prior:
Optimal Bayesian filtering perspective,” IEEE Transactions on Signal
Processing, vol. 64, no. 23, pp. 6243–6253, Dec 2016.
[46] R. Dehghannasiri, M. S. Esfahani, and E. R. Dougherty, “Intrinsically
Bayesian robust Kalman filter: An innovation process approach,” IEEE
Transactions on Signal Processing, vol. 65, no. 10, pp. 2531–2546, May
2017.

