arXiv:1611.06714v1 [] 21 Nov 2016

On the Monotonicity of the Copula Entropy
Yaniv Tenzer, and Gal Elidan
Department of Statistics, The Hebrew University of Jerusalem

Abstract
Understanding the way in which random entities interact is of key interest in numerous scientific fields. This can range from a full characterization of the joint distribution to single scalar summary statistics. In this
work we identify a novel relationship between the ubiquitous Shannon’s
mutual information measure and the central tool for capturing real-valued
non-Gaussian distributions, namely the framework of copulas. Specifically, we establish a monotonic relationship between the mutual information and the copula dependence parameter, for a wide range of copula
families. In addition to the theoretical novelty, our result gives rise to
highly efficient proxy to the expected likelihood, which in turn allows
for scalable model selection (e.g. when learning probabilistic graphical
models).

1

Introduction

Understanding the joint behavior of random entities is of great importance in
essentially all scientific fields ranging from computational biology and health
care to economics and astronomy. Accordingly, the study of joint distributions
is fundamental to all the data sciences and goes back at least to the seminal
studies of Sir Francis Galton [Galton, 1888].
In multivariate modeling, our goal may range from the task of characterizing
the full joint distribution which can be difficult, to specific summary statistics
such as correlation measures, e.g. Pearson’s correlation or mutual information.
In this work, we identify a novel and useful relationship between two central
frameworks for these tasks, namely the frameworks of copulas and information
theory.
In real-valued domains, the most prominent general purpose framework for
going beyond the multivariate normal distribution is that of copulas [Joe, 1997,
Nelsen, 2007] pioneered by Sklar [Sklar, 1959]. In a nutshell, copulas allow us to
separate the modeling of the (possibly nonparametric) univariate marginals and
that of the dependence function. Formally, given a set of univariate marginal
cumulative distribution functions {FXi }, a copula function CU (u1 , . . . , un ) is a
joint distribution over variables that are marginally uniform in the [0, 1] range,
so that
FX (x1 , . . . , xn ) = C{FXi } (FX1 (x1 ), . . . , FXn (xn )) ,
1

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

2

is a valid joint distribution.
This separation between the marginal representation and the copula function
that links them allows us, for example, to easily capture multi-modal or heavytailed distributions. Indeed, the popularity of copulas as a flexible tool for
capturing dependence has grown substantially in recent years [Elidan, 2013].
Most if not all popular copula families are governed by a dependence parameter that spans the range (or part of it) between independence and full
dependence. In fact, in the bivariate case, this is captured by a well known and
fundamental relationship between the dependence parameter of the copula and
rank-based correlation measures, such as Spearman’s rho or Kendall’s tau (see,
for example, chapter 3 in [Nelsen, 2007]).
In the field of information theory, Shanon’s mutual information measures
the reduction in entropy that the knowledge of one variable induces on another.
Mutual information is a fundamental tool used to quantify the strength of dependence between random variables [Shannon, 2001], and is used throughout
the exact sciences. Formally, Shannon’s mutual information is defined as:
Z
fX,Y (x, y)
dxdy.
(1.1)
M I(X; Y ) = fX,Y (x, y) log
fX (x)fY (y)
Whether in machine learning or physics, mutual information is often the de
facto tool for measuring correlation and/or identifying independence [Cover and
Thomas, 2012]. A natural question is thus how does this statistic relate to the
framework of copulas.
In this work we establish a relationship between the mutual information
of two variables and their corresponding copula dependence parameter. Concretely, we prove that the mutual information (equivalently the copula entropy)
is monotonic in the bivariate copula dependence parameter for a wide range of
copula families, covering the vast majority of copulas used in practice. We also
extend our results for the popular class of Archimedean copulas to higher dimensions where other measures of dependence such as Spearman’s rho or Kendall
tau are not well defined.
The monotonicity result is a theoretical one and an obvious question is
whether it has practical merit in the statistical sense. In our simulations we
show that it holds in practice using a modest number of samples. An immediate
implication is that the mutual information between different pairs of variables
can be ranked by evaluating simple statistics that are substantially simpler than
the copula entropy, namely Spearman’s rho or Kendall’s tau.
Thus, for example, given gene expression data for a large number of genes, we
can identify the two genes that have the highest mutual information without ever
evaluating the mutual information or the copula density for all pairs of genes,
a computationally formidable task. More broadly, our results facilitate highly
efficient model selection in scenarios involving many interactions, e.g. when
learning probabilistic graphical models. This practical implication is studied
in depth in our earlier paper [Tenzer and Elidan, 2013], where the theoretical
results were substantially more limited.

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

3

The rest of the paper is organized as follows. In Section 2 we briefly review the necessary background on copulas, TP2, super-modular functions and
stochastic orders. In Section 3 we present our main theoretical result: broadcoverage sufficient conditions that guarantee the monotonicity of the mutual information (copula entropy) in the copula dependence parameter. We extend the
results to the class of bivariate two-parameter families in Section 3.3. We then
generalize the results for Archimedean copulas of any dimension in Section 4.
In Section 6 we show the applicability of our theoretical results in the empirical
finite-sample scenario. We finish with concluding remarks in Section 7.

2

Preliminaries

In this section we briefly review basic definitions and results related to copulas,
super modular functions and stochastic orders that will be needed in the sequel.

2.1

Copulas

A copula is a multivariate joint distribution whose univariate marginals are
uniformly distributed. Formally:
Definition 2.1. : Let U1 , . . . , Un be random variables marginally uniformly
distributed on [0, 1]. A copula function C : [0, 1]n → [0, 1] is a joint distribution
Cθ (u1 , . . . , un ) = P (U1 ≤ u1 , . . . , Un ≤ un ),
where θ are the parameters of the copula function.
Now consider an arbitrary set X = {X1 , . . . Xn } of real-valued random variables (typically not marginally uniformly distributed). Sklar’s seminal theorem
[Sklar, 1959] states that for any joint distribution FX (x), there exists a copula
function C such that
FX (x) = C(F1 (x1 ), . . . , Fn (xn )).
When the univariate marginals are continuous, C is uniquely defined.
The constructive converse, which is of central interest from a modeling perspective, is also true. Since Ui ≡ Fi is itself a random variable that is always
uniformly distributed in [0, 1], any copula function taking any marginal distributions {Ui } as its arguments, defines a valid joint distribution with marginals
{Ui }. Thus, copulas are “distribution generating” functions that allow us to
separate the choice of the univariate marginals and that of the dependence
structure.
n
F(x1 ,...,xn )
from the copula construction,
Deriving the joint density f (x) = ∂ ∂x
1 ...∂xn
assuming F has n-order partial derivatives (true almost everywhere when F is

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

4

continuous) is straightforward. Using the chain rule we can write
f (x)

=
≡

∂ n C(F1 (x1 ), . . . , Fn (xn )) Y
fi (xi )
∂F1 (x1 ) . . . ∂Fn (xn )
i
Y
c(F1 (x1 ), . . . , Fn (xn ))
fi (xi ),

(2.1)

i

where c(F1 (x1 ), . . . , Fn (xn )), is called the copula density function.
Copulas are intimately related to the fundamental concept of Mutual Information (MI) of random variables [Cover and Thomas, 2012]. In the bivariate
case, the MI of two random variables X and Y is defined as in Equation (1.1).
Denote U ≡ FX and V ≡ FY so that cU,V is the copula of the joint distribution
of X and Y Applying Equation (2.1) we get
Z
M I(X; Y ) = − cU,V (u, v) log cU,V (u, v)dudv
(2.2)
=

E[− log cU,V ] = H(CU,V ).

In words, the MI between two random variables equals to the entropy of the
corresponding copula. It is easy to see that this also applies to higher dimensions.
Copulas are also closely tied to other measures of association. The following
relationship between the copula function and Spearman’s ρX,Y is well known in
the bivariate case:
Z
ρX,Y = 12 C(FX (x), FY (y))dxdy − 3,
(2.3)
and a similar relationship is known for Kendall’s tau [Nelsen, 2007]

2.2

TP2 and Super-Modular Functions

Total positive of order two functions (TP2) [Olkin and Marshall, 2016] play a
central role in statistics and many common copula families have a TP2 density
function. Below we define the TP2 concept and provide a simply connection
to super-modular functions that will be useful for our developments. Formally,
denote u ∨ v = min(u, v), u ∧ v = max(u, v).
Definition 2.2.: A function Ψ : R2 ⇒ R is called TP2 if
∀ u, v ∈ R2

Ψ(u ∨ v) · Ψ(u ∧ v) ≥ Ψ(u) · Ψ(v).

When the inequality is reversed, the function is reverse rule of order 2 (RR2).
Definition 2.3.: A function Ψ : R2 ⇒ R is said to be super-modular if
∀ u, v ∈ R2 , Ψ(u ∨ v) + Ψ(u ∧ v) ≥ Ψ(u) + Ψ(v)

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

5

Note that if the inequality is reversed the function is called sub-modular [Olkin
and Marshall, 2016]. The following two simple results will be useful in the
sequel.
Lemma 2.4. : Let Ψ(u, v) be a positive TP2 (RR2) function. Then Φ(u, v) =
log(Ψ(u, v)) is super-modular (sub-modular).
Proof: From the definition of a TP2 function Ψ(u ∨ v) · Ψ(u ∧ v) ≥ Ψ(u) · Ψ(v),
we have
Φ(u ∨ v) + Φ(u ∧ v) ≡ log(Ψ(u ∨ v)) + log(Ψ(u ∧ v))
=

log(Ψ(u ∨ v)Ψ(u ∧ v))

≥ log(Ψ(u)Ψ(v)))
=

log(Ψ(u)) + log(Ψ(v))

≡ Φ(u) + Φ(v)

Lemma 2.5. : Let f1 (x, y), f2 (x, y) be two real non-negative TP2 (RR2) functions. Then Ψ(x, y) = f1 f2 is TP2 (RR2).
The proof is immediate and we omit the details.

2.3

PQD and Super-Modular Oredrings

Stochastic orderings introduce the notion of partial orders between random variables. Perhaps the most well known is the standard stochastic order, or using
the name more commonly used in the copula community, the positive quadrant
dependent (PQD) order:
Definition 2.6.: Let X, X0 be bivariate random vectors and let FX (u, v),
FX0 (u, v) be the corresponding distribution functions. X0 is said to be more
PQD then X if:
∀(u, v) ∈ R2 , FX (u, v) ≤ FX0 (u, v)
Another stochastic ordering that will be useful in our development is the
Super-modular (SM) ordering:
Definition 2.7.: Let X, X0 be bivariate random vectors, X0 is said to be greater
than X in the super-modular order, denoted by X 5SM X0 , if∀ Ψ such that Ψ
is super modular:
EX (Ψ(x)) 5 EX0 (Ψ(x0 ))
Note that since most bivariate families are PQD ordered by construction [Joe,
1997], an immediate result of Equation (2.3) is that Spearman’s rho is monotonic
in the copula dependence parameter, within a specific copula family.

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

3

6

The Monotonicity of Mutual Information in
the Dependence Parameter for Bivariate Copulas

We now present our central result and establish a novel and elegant connection
between the copula function of the distribution of two random variables and the
mutual information between these variables (equivalently the copula entropy).
In Section 4 we generalize some of these results to higher dimensions.
Theorem 3.1.: Let Cθ (u, v) be an absolutely continuous bivariate copula, and
let X, X’ be two bivariate random vectors distributed according to the same
copula family with two different parameterizations so that X ∼ Cθ1 (u, v), X0 ∼
Cθ2 (u, v), where θ2 ≥ θ1 ≥ 0. (θ2 ≤ θ1 ≤ 0). Then −H(X) 5 −H(X0 ) if one of
the following condition holds:
(a) Cθ (u, v) is increasing (decreasing) in <SM and the copula density cθ (u, v)
is TP2 (RR2) (for all θ).
(b) Cθ (u, v) is an Archimedean copula whose generator φθ is completely monotone and satisfies the boundary condition φθ (0) = 1. In addition the copula
is increasing in <SM .
(c) Cθ,δ (u, v) is a two-parameters Archimedean copula whose generator ηθ,δ (s)
is completely monotone and satisfies the boundary condition ηθ,δ (0) = 1. In
addition the copula is increasing in <SM , with respect to θ, for a fixed δ.
(d) Cθ (u, v) is an elliptical copula.
That is, the negative copula entropy defined in Equation (2.3) is monotonic
increasing (decreasing) in θ if any of the above conditions hold for the copula
family.
We note that in the bivariate case, SM ordering is equivalent to PQD ordering [Shaked and Shanthikumar, 2007] so that the above conditions (a)-(c) can
equivalently be stated using a PQD order condition. We also note that condition (d) was proved in [Elidan, 2012] using an explicit formula of the copula
entropy and is stated here for completeness.
An immediate consequence of Theorem 3.1 and the known monotonicity of
ρs in the dependence parameter θ for PQD ordered families [Nelsen, 2007], is
Corollary 3.2.: If any of the above conditions (a)-(d) hold for a copula family,
then the magnitude of Spearman’s ρs is monotonic in the copula entropy.
Importantly, one of the above conditions holds in most if not all commonly
used bivariate copula families so that our result is widely applicable. In the
sections below, we prove the result for each of the sufficient conditions (a)-(c).
In Section 5, we survey some example families.

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

3.1

7

Proof for (a): TP2 Density and SM/PQD Order

To prove the result, we are going to show that the following holds:
Z
Z
cθ1 (u, v)log(cθ1 (u, v))∂u..∂un 5
cθ2 (u, v)log(cθ1 (u, v))∂u∂v
Z
5
cθ2 (u, v)log(cθ2 (u, v))∂u∂v.
Let Ψ(u, v) = log(cθ1 (u, v)). Since cθ1 (u, v) is TP2, Ψ(u, v) is super modular.
X 5SM X0 . Thus, E(Ψ(x)) 5 E(Ψ(x0 )) for all super modular function f (u, v)
and in particular for f = Ψ(u, v), according to Lemma 2.4. Thus,
Z
Z
cθ1 (u, v)Ψ(u, v)∂u∂v 5 cθ2 (u, v)Ψ(u, v)∂u∂v.
Now, substituting the explicit form of Ψ we get the first inequality:
Z
Z
cθ1 (u, v)log(cθ1 (u, v))∂u∂v 5 cθ2 (u, v)log(cθ1 (u, v))∂u∂v.
To prove the second inequality we observe that
Z
Z
cθ2 (u, v)log(cθ1 (u, v))∂u∂v 5
cθ2 (u, v)log(cθ2 (u, v))∂u∂v


Z
cθ2 (u, v)
⇔
cθ2 (u, v)log
∂u∂v = 0
cθ1 (u, v)
⇔ KL(cθ2 ; cθ1 ) = 0
where KL(; ) is the Kullback-Leiber divergence [Cover and Thomas, 2012],
which is always non-negative. Putting these inequalities together we can conclude:
Z
−H(X) ≡
cθ1 (u, v)log(cθ1 (u, v))dudv
Z
5
cθ2 (u, v)log(cθ2 (u, v))dudv ≡ −H(X0 )

3.2

Proof for (b): Bivariate Archimedean Copulas

Archimedean copulas are probably the most common non-elliptical copulas that
allow for asymmetric or heavy-tail distributions.
Condition a in Theorem 3.1 is a fairly general one in the sense that it does not
put any restrictions on the copula family type. In particular, it does not require
the family to be elliptical or Archimedean. As we shall see below, Condition b
actually implies condition (a) in the special case of Archimedean copulas with
completely monotone generator. Among many examples of copula families for
which this holds are the Ali-Mikhail (AMH), Clayton, Frank and the Gumbel
families (see Section 5 for details).

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

8

Definition 3.3. : Let ψθ (x) : [0, ∞) → [0, ∞] be a strictly convex univariate
function that is parametrized by θ ∈ R. In addition assume that ψθ (∞) = 0.
The Archimedean copula that is generated by ψθ (x) is defined as:
Cθ (u, v) = ψθ (ψθ−1 (u1 ) + ψθ−1 (u2 )).

(3.1)

We say that ψθ (x) is the copula generator of Cθ (u, v) [Nelsen, 2007]. We consider
the subclass of copula generators:
L∞ = {ψθ : (−1)i ψθ (x)(i) ≥ 0 ∀i = 0, 1, 2.., ψθ (0) = 1}.
This is the class of generators whose derivatives alternate signs (this property
is widely known as completely monotonicity) and in addition these generators
satisfy the boundary condition ψθ (0) = 1.
The following lemma shows that an Archimedean copula whose generator
ψθ (x) is in L∞ , can be written as a mixture of two univariate CDFs. We will
then use this to show that b implies a.
Lemma 3.4.: Let Cθ (u, v) be an Archimedean copula and let ψθ be its generator
such that ψθ ∈ L∞ . Then there exists a CDF M (α), of a positive random
variable α, and unique CDFs G1 (u), G2 (v) such that:
Z ∞
Cθ (u, v) =
G1 (u)α · G2 (v)α dM (α).
0

For the sack of completeness, we give a simple proof here. Note that a
different proof can be found in [Joe, 1997].
Proof: From Bernstein’s theorem we have that each ψθ ∈ L∞ is a Laplace
transform of some distribution function of a positive random variable α. That
is:
Z ∞
ψθ (s) =
e−sα dM (α), s ≥ 0.
0

Thus
Cθ (u, v) =

ψθ (ψθ−1 (u)

+

ψθ−1 (v))

Z
=

∞

−1

e−α(ψθ

(u)+ψθ−1 (v))

dM (α).

(3.2)

0

In addition, for any arbitrary distribution function F , and any positive random
variable M (α), there exists a unique distribution function G such that [Joe,
1997, p.84]:
Z ∞
Z ∞
F (x) =
G(x)α dM (α) =
e−α(− ln G(x)) dM (α) = ψθ (− ln G(x)),
0

0
−ψθ−1 (F (x))

Thus, G(x) = e
. In particular, if U, V are uniform, then F (u) =
−1
u, F (v) = v and therefore there exist G1 (u), G2 (v) such that G1 (u) = e−ψθ (u)
−1
and G2 (v) = e−ψθ (v) . Substituting this into Equation (3.2), we get
Z ∞
Cθ (u, v) =
G1 (u)α · G2 (v)α dM (α).
0

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

9

Finally, a mixture representation implies a TP2 density [Joe, 1997] . Therefore
if in addition PQD/SM ordering holds for these copula families, then condition
a is implied.

3.3

Proof for (d): Bivariate Two-parameters Archimedean
Copulas

Two-parameter families are used to capture more than one type or aspect of
dependence. For example, one parameter may control the strength of uppertail dependence while the other indicates concordance or the strength of lower
tail dependence. Bivariate Archimedean copulas generalize the one-parameter
families. In particular, these families have the following form:
Cθ,δ (u, v) = ψθ (−log Kδ (e−ψθ (u) , e−ψθ (v) )),

(3.3)

where Kδ is an Archimedean copula, parametrised by δ, as defined in Equation (3.1), and ψθ is a Laplace transform. Let φδ be the generator of Kδ . The
resulting copula is then also an Archimedean copula with generator ηθ,δ (s) =
ψθ (−log φδ (s)). For further details see [Joe, 1997].
For certain families, if we fix the δ parameter, the copula is then increasing
in SM order, with respect to θ. Therefore if the copula generator ηθ,δ (s), is
completely monotone and in addition the boundary condition ηθ,δ (0) = 1 holds,
then we are back to the settings of condition c in Theorem 3.1. As a result we
have the following corollary:
Corollary 3.5.: Let Cθ,δ (u, v) be a two parameters bivariate copula of the above
form specified in 3.3. Let ηθ,δ (s) = ψθ (−log φδ (s)) be its associated generator.
Then, if ηθ,δ (s) is completely monotone such that ηθ,δ (0) = 1 and Cθ,δ (u, v) is
increasing in SM order with respect to θ for a fixed δ, then the copula entropy
is increasing in θ.
The proof is immediate given our previous results and we omit the details.
Examples of bivariate two-parameters families for which the conditions given in
Corollary 3.5 hold are the BB1, BB2 and BB6 families [Joe, 1997]. See Section 5
for details.

4

Multivariate Copulas

In this section we generalize some of our results for higher dimensions. Before
presenting the main results of this section, let us introduce the following class
of positive real univariate functions:
L∗∞ = {φ : R+ → R+ |φ(0) = 0, φ(∞) = ∞, (−1)j−1 φ(j) ≥ 0, j ≥ 1}.
As we shell see, this class plays a central role in our result regarding multivariate
Archimedean copulas. We are now ready to state our main result:

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

10

Theorem 4.1. : Let Cθ (u1 , . . . , un ) be an absolutely continuous bivariate
copula, and let X, X’ be two random n-dimensional vectors such that X ∼
Cθ1 (u1 , . . . , un ), X0 ∼ Cθ2 (u1 , . . . , un ), where θ2 ≥ θ1 ≥ 0. (θ2 ≤ θ1 ≤ 0). Then
−H(X) 5 −H(X0 ) if one of the following condition holds:
(a) Cθ (u1 , . . . , un ) is increasing (decreasing) in <SM and the copula density
cθ (u1 , . . . , un ) is TP2 (RR2) (for all θ).
(b) Cθ (u1 , . . . , un ) is an Archimedean copula whose generator, φθ , is completely
monotone and satisfies the boundary condition φθ (0) = 1. In addition for
∗
all θ1 ≤ θ2 , φ−1
θ1 φθ2 ∈ L∞ .
It can be easily shown that condition a in Theorem 4.1 is sufficient in any
dimension. We omit the proof since it is essentially identical to the bivariate
case. Generalizing condition b, however, requires more work which we present
in this section.

4.1

Proof for (b): Multivariate Archimedean Copulas

Recall that our proof for b in the bivariate case relies on the fact that a completely monotone generator that satisfies the boundary condition implies a mixture representation, which in turn implies a TP2 bivariate density. We now
show that both properties also hold in the general case of Acrhimedean copulas
with completely monotone generators for which the boundary condition holds.
To complete the proof, we will then also need to show that the SM ordering
∗
holds under the composition condition φ−1
θ1 φθ2 ∈ L∞ for θ1 ≤ θ2 .
The following lemma, proved in [Joe, 1997, p.85-89] substantiates the first
property:
Lemma 4.2.: Let Cθ (u1 , ·, un ) be Archimedean copula and let ψθ be its generator
such that ψθ is completely monotone and ψθ (0) = 1. Then there exists a CDF
M (α) of a positive
R ∞ r.v. and a unique CDF-s G1 (u1 ), . . . , Gn (un ) such that:
Cθ (u1 , ..., un ) = 0 G1 (u)α . . . Gn (un )α dM (α)
The following lemma shows that the second property also holds:
Lemma 4.3. : Let Cθ (u1 , . . . , un ) be a copula that can be represented as in
Lemma 4.2. Then Cθ (u1 , . . . , un ) has a TP2 density.
R∞
Proof: We assume that Cθ (u1 , . . . , un ) = 0 G1 (u1 )α . . . Gn (un )α dM (α). Taking the derivative with respect to each argument we then have that the copula
density is:
Z ∞
Y
n
cθ (u1 , . . . , un ) = α
gi (ui )
G1 (u1 )α−1 · .. · Gn (un )α−1 dM (α).
0

i

Qn−1

Gi (ui )α−1 , we can write
Z ∞
Y
n
cθ (u1 , . . . , un ) = α
gi (ui )
φ(u1 , . . . , un−1 , α) · Gn (un )α−1 dM (α).

Using φ(u1 , . . . , un−1 , α) =

i

i=1

0

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

11

Now, for each i, Gi (ui )α−1 is TP2 in (ui , α). Using Observation 2.5 we then have
that φ(u1 , . . . R, un−1 , α) is also TP2. From this and [Karlin and Rinott, 1980]
∞
we have that 0 Q
φ(u1 , . . . , un−1 , α) · Gn (un )α−1 dM (α) is TP2 (with respect to
u1 , . . . , un ). As i gi (ui ) is trivially TP2 [Joe, 1997], using Observation 2.5
again we have that the resulting density is TP2.
We have shown that, as in the bivariate case, a completely monotone generator that satisfies the boundary condition implies a TP2 copula density. We
now characterize the conditions that also ensure SM ordering. Concretely L∗∞
provides us with the needed condition via the following theorem [Wei and Hu,
2002]:
Theorem 4.4. : Let C1 , C2 be two n-dimensional Archimedean copulas and
let φ1 , φ2 be their associated generators, respectively. If φ1 , φ2 are two Laplace
transforms (equivalently φ1 , φ2 are completely monotone and satisfy the boundary condition φi (0) = 1, i = 1, 2, [Joe, 1997]), such that φ1 φ−1
∈ L∗∞ , then
2
C1 ≤SM C2 .
Putting this and Lemma 4.3 together we get the desired result.
The conditions of our theorem for multivariate copulas may seem somewhat
obscure and therefore not of practical interest. However, we note that they
actually apply to some of the most popular multivariate Archimedean copulas,
namely the Clayton, Gumbel, Frank and Joe copulas. See Section 5 for details.

5

Examples

Below we demonstrate the wide applicability of our theory. We begin with
condition a of Theorem 3.1 that provides the broadest coverage and then also
provide examples for the others cases.

5.1

Examples Satisfying Condition a of Theorem 3.1

In all the following examples, the copulas under consideration are known to be
increasing in the PQD order and have a TP2/RR2 density function [Joe, 1997].
Therefore the entropy for these families is monotonic in θ by Theorem 3.1:
• Bivariate Normal:
0 ≤ θ ≤ 1,

Cθ (u, v) = Φθ (Φ−1 (u), Φ−1 (v)),

where Φ is the N (0, 1) cdf, and Φθ is the BVSN cdf with correlation θ.
• Bivariate Farlie-Gumbel-Morgenstern (FGM):
−1 ≤ θ ≤ 1,

Cθ (u, v) = uv + θuv(1 − u)(1 − v).

Note that for this family the density is TP2 when θ ≥ 0 and RR2 when
θ ≤ 0 [Joe, 1997]. Thus its negative entropy is monotonic increasing
in θ, when θ ∈ (0, 1] and monotonic decreasing when θ ∈ [−1, 0) [Joe,

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

12

1997]. Therefore overall the FGM copula negative entropy is monotonic
increasing in |θ|.
• Bivariate Frank:
0 ≤ θ ≤ ∞,

Cθ (u, v) = −θ

−1







−θu
−θv
log τ − 1 − e
(1 − e
/τ ,

where τ = 1 − e−θ .
• Bivariate Gumbel:
1 ≤ θ ≤ ∞,

θ

Cθ (u, v) = e−[(û)

+(v̂)θ )]1/θ

,

where û = −log(u), v̂ = −log(v).
• Bivariate Clayton:
0 ≤ θ ≤ ∞,

5.2

Cθ (u, v) = (u−θ + v −θ − 1)(−1/θ) .

Archimedean Copula Examples Statisfying condition
b in Theorem 3.1

In the following we demonstrate condition b for several Archimedean families
that are not covered by the previous section such as the Clayton/Frank/Gumbel
copulas. These examples are slightly lesser known but still useful Archimedean
copula families. In all the following examples, the copulas under consideration
are known to be increasing in the PQD order [Joe, 1997]. In addition it can be
easily verified that the boundary condition ψθ (0) = 1 holds.
In order to establish the completely monotonicity (and hence also the total
positivity of the corresponding density function) of the copula generator, we use
the following sufficient conditions [Nelsen, 2007, Widder, 1942]:
Lemma 5.1. : Let f (x), g(x) be two real univariate functions and let h1 (x) =
f (x) ◦ g(x), h2 (x) = f (x)g(x). Then:
(i) If g is completely monotonic and f is absolutely monotonic, i.e.,
for k = 0,1,2,... then h1 (x) is completely monotone.

∂f (x)
∂xk

≥0

(ii) If f is completely monotonic and g is a positive function with a completely
monotone derivative, then h1 (x) is completely monotone.
(iii) If f and g are completely monotone, then so is h2 (x).
• Bivariate Ali-Mikhail (AMH):
θ ∈ [−1, 1],

Cθ (u, v) =

uv
,
1 − θ(1 − u)(1 − v)

The generator of this copula is given by:
ψθ (t) =

1−θ
.
exp(t) − θ

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

13

This generator is completely monotone for θ ∈ (0, 1] [Jaworski et al.,
2010]. When θ ∈ [−1, 0), it can easily been shown that the corresponding
densities are RR2, using Lemma 2.5. Therefore overall the AMH copula
negative entropy is monotonic increasing in |θ|.
• Bivariate Joe:

1/θ
Cθ (u, v) = 1 − (1 − u)θ + (1 − v)θ − (1 − u)θ (1 − v)θ
,

θ ∈ [1, ∞),

The generator of this copula is given by:
ψθ (t) = 1 − (1 − exp(−t))1/θ .
Taking f (t) = 1 − t1/θ and g(t) = 1 − exp(−t), we see that f is completely
monotone and g is a positive function whose first derivative completely
monotone. Thus their composite is also completely monotone.
• Family 4.14 [Nelsen, 2007]:

θ ∈ [1, ∞),

Cθ (u, v) =



−1/θ

1 + (u

θ

− 1) + (v

−1/θ

− 1)

θ

θ −θ

,

The generator of this copula is given by:
ψθ (t) = (t1/θ + 1)−θ .
By taking f = (1 + t)−θ , g = t1/θ and repeating the same arguments as
in previous examples, we get that this generator is completely monotone.
• Family 4.19 [Nelsen, 2007]:
θ ∈ (0, ∞),

Cθ (u, v) =

ln(eθ/u

θ
,
+ eθ/v − eθ )

The generator of this copula is given by:
ψθ(t) =

θ
.
ln (t + exp(θ))

By taking f = θ/t, g = ln(t+exp(θ)) and repeating the same arguments as
in previous examples, we get that this generator is completely monotone.

5.3

Examples of Bivariate Two-parameter Families

We now provide some examples of two-parameter copula families that satisfy
condition c of Theorem 3.1. That is, they are all positively PQD/SM ordered
with respect to θ and have a completely monotone generator that satisfies the
boundary condition [Joe, 1997]. Therefore by Corollary 3.5 for fixed δ, their
entropy is monotonic in θ.

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

14

Recall that bivariate two-paraneter families have the following form:
Cθ,δ (u, v) = ψ(−log Kδ (e−ψθ (u) , e−ψθ (v) )),

(5.1)

where Kδ is an Archimedean copula, parametrised by δ, as defined in Equation (3.1), and ψθ is a Laplace transform. Let φδ be the generator of Kδ . The
resulting copula is then also an Archimedean copula with generator ηθ,δ (s) =
ψθ (−log φδ (s)). For further details see [Joe, 1997].
• Family BB1 [Joe, 1997]: Taking K to be Gumbel copula and ψθ (s) =
(1 + s)−1/θ , θ ≥ 0. The resulting copula is:

1/δ −1/θ
,
Cθ,δ (u, v) = 1 + (u−θ − 1)δ + (v −θ − 1)δ
where ηθ,δ (s) = (1 + s1/δ )−1/θ .
• [Joe, 1997], bivariate BB2: Taking K to be Clayton copula and ψθ (s) =
(1 + s)−1/θ , θ ≥ 0 the resulting copula is:



−1/θ
−θ
−θ
Cθ,δ (u, v) = 1 + δ −1 log eδ(u −1) + eδ(v −1) − 1
,
where ηθ,δ (s) = [1 + δ −1 log(1 + s)]−1/θ .
• [Joe, 1997], bivariate BB6: Taking K to be Gumbel copula and ψθ (s) =
1 − (1 − e−s )1/θ , θ ≥ 1 the resulting copula is:


 1  θ1
,
Cθ,δ (u, v) = 1 − 1 − exp − (− log(1 − ûθ ))δ + (− log(1 − v̂ θ ))δ δ
where û = 1 − u, v̂ = 1 − v,

5.4


ηθ,δ (s) = 1 − [1 − exp −s1/δ ]1/θ .

Examples of Multivariate Copulas

We finish with some multivariate examples that satisfy the conditions of Theorem 4.1. That is, in each example the copula generator, φθ , is completely
∗
monotone and in addition φ−1
θ1 φθ2 ∈ L∞ for θ1 ≤ θ2 [Joe, 1997].
• Multivariate Clayton:
θ1 ≤ θ2 ∈ (0, ∞),

φ−1
θ 1 φθ 2 =

1
1
(θ2 + 1)θ1 /θ2 − .
θ1
θ1

• Multivariate Gumbel:
θ1 ≤ θ2 ∈ (1, ∞),

θ1 /θ2
φ−1
.
θ1 φθ2 = t

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

15

• Multivariate Frank:
θ1 ≤ θ2 ∈ (0, ∞),

φ−1
θ1 φθ2


= − ln

(e−t (e−θ2 − 1) + 1)θ1 /θ2 − 1
e−θ1 − 1


.

• Multivariate Joe :
θ1 ≤ θ2 ∈ (0, 1),

6



θ1 /θ2
φ−1
.
φ
=
−
ln
1
−
(1
−
exp(−t))
θ1 θ2

Simulations

The theory presented in this paper suggests that, in the case of an infinite
number of samples generated from a given copula family, the entropy curve is
monotonic in the copula dependence parameter. In practice, however, we almost
always have access only to a finite number of samples. An obvious empirical
question is thus whether monotonicity approximately holds given a reasonable
number of samples, and what is the impact of the sample size on the entropy
vs. dependence parameter curve.
To answer these questions we explore the finite-sample behaviour of the
copula entropy monotonicity via a simulation study. For each of the different
theoretical scenarios discussed in the previous section, we choose representative
popular copula families. We then generate M = 1000 samples from the copula
for different values of the dependence parameter. For each of these samples, we
compute the entropy using the standard empirical estimator
P
log cθ (u)
b
H(cθ ) = − u∈D
,
M
where D denotes the set of M samples generated from a copula Cθ .
For each value of the dependence parameter, we repeat the above 50 times,
and report the mean empirical entropy along with a 95% confidence interval vs.
the value of the dependence parameter. Figure 1, first two rows, show the results
for single-parameter bivariate copula families. It is clear that with as little as
1000 samples, near-monotonicity consistently holds for all families evaluated
including both elliptical ones (Gaussian, student-T) and Archmedian copula
families (Clayton, Frank, Gumbel, Joe, AMH).
Next we turn to bivariate two parameters families. In this case we also need
to test the monotonicity along the δ axis, since as formalized in Corollary 3.5,
monotonicity holds in θ for each fixed value of δ. Results are shown in Figure 1,
third row, for the BB1 and BB6 copula families, for two different values of δ.
Results for other values of δ as well as for the BB2 family were qualitatively
similar. As before, near monotonicity is evident.
To evaluate the finite-sample monotonicity in the multivariate case, we repeat the same evaluation for several copula families of dimension 5. Results
are shown in the last row of Figure 1, and are qualitatively similar for higher
dimensions. As before, the empirical monotonicity is quiet impressive.

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

16

Clayton

Frank

Gumbel

Joe

Gaussian

T, ν = 3

T, ν = 7

AMH

BB1, δ = 1.5

BB1, δ = 3

BB6, δ = 1.5

BB6, δ = 3

Clayton

Frank

Gumbel

Joe

d=2

d=2

d=2

d=5

Figure 1: Simulation study of the monotonicity of the empirical entropy in the copula dependence
parameter. Shown is the mean (red line) and 95% range (black lines) of the empirical entropy over
50 random computations using from M = 1000 samples for each value of the dependence parameter
(x-axis). We consider several popular bivariate (d=2) single parameter copula families (first and second
rows), bivariate two parameters families (third row), as well as multivariate (d=5) copula families
(fourth row).

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

17

Figure 2: The impact of sample size
on the monotonicity of the empirical
entropy as a function of the dependence parameter of the generating
copula. Shown is the mean monotonic trend measure (y-axis) as a
function of the sample size (x-axis).
The monotonicity is measured via
the proportion of consecutive values of the dependence parameter, as
defined in Equation (6.1). Result
shown is the average over 50 repetitions for the Clayton family. Results for other copula families are
qualitatively similar.

Finally, we evaluate the impact of the sample size on the extent to which
the empirical entropy is monotonic in the copula dependence parameter. We
repeat the same evaluation procedure for a wide range of sample sizes (from
500 to 10000). To measure empirical monotonicity, for a given sample size, we
measure the fraction of consecutive parameter values for which monotonicity
holds. That is, we measure the empirical rate at which the following inequality
holds:
P
P
log cθ2 (u)
u∈D log cθ1 (u)
θ1 ≤ θ2 ⇒
≤ u∈D
,
(6.1)
M
M
for two consecutive values of θ1 and θ2 . We repeat this process 50 times and
report the average. Results for the Clayton family are shown in Figure 2.
As expected, with a greater sample size, monotonicity holds more frequently
nearing 100% at just 5000 training instances. Appealingly, even at much smaller
sample sizes, monotonicity is quite appealing.

7

Conclusion

In this work we establish a novel theoretical relationship between the main general purpose framework for capturing non-Gaussian real-valued distributions,
namely copulas, and the ubiquitous Shannon’s mutual information measure that
is used throughout the exact science to quantify the dependence between random variables.
Our main result is that the mutual information between two variables (equivalently the copula entropy) is monotonic in the copula dependence parameter
for a (very) wide range of copula families. Concretely, we provide fairly general
sufficient conditions for this monotonicity that cover the vast majority of com-

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

18

monly used bivariate copulas, as well as a wide class of multivariate Archimedean
copulas.
As we demonstrated in our earlier work where the theory was substantially
less developed [Tenzer and Elidan, 2013], the monotonicity result also has practical merits. Specifically, it allows us to rank the entropy (equivalently the
expected log-likelihood) of copula-based probabilistic models by simple computation of association measures such as Spearman’s rho. This gives rise to highly
efficient model selection in scenarios involving many interactions, e.g. when
learning copula graphical models. The practical merit depends, of course, on
the finite sample behavior of the entropy. Fortunately, as our simulations clearly
show, near perfect monotonicity holds even with modest sample sizes.
Essentially all copula families are constructed so as to span some or all of the
range between the independence and full dependence copula via the dependence
parameter. We have not been able to identify a single family where the above
monotonicity does not hold empirically. However, monotonicity does appear
to hold for the Plackett copula familiy [Nelsen, 2007] and identifying further
sufficient conditions remains a future challenge. Another direction if interesting
is developing finite sample theory that will explicitly quantify the amount by
which the empirical entropy can deviate from the expected monotonic behavior.

References
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley &
Sons, 2012.
Gal Elidan. Lightning-speed structure learning of nonlinear continuous networks. pages
355–363, 2012.
Gal Elidan. Copulas in machine learning. In Copulae in mathematical and quantitative
finance, pages 39–60. Springer, 2013.
Francis Galton. Co-relations and their measurement, chiefly from anthropometric
data. Proceedings of the Royal Society of London, 45(273-279):135–145, 1888.
Piotr Jaworski, Fabrizio Durante, Wolfgang Karl Härdle, and Tomasz Rychlik. Copula
theory and its applications: proceedings of the workshop held in Warsaw, 25-26
September 2009, volume 198. Springer Science & Business Media, 2010.
H. Joe. Multivariate models and dependence concepts. Monographs on Statistics and
Applied Probability, 73, 1997.
Samuel Karlin and Yosef Rinott. Classes of orderings of measures and related correlation inequalities. i. multivariate totally positive distributions. Journal of Multivariate Analysis, 10(4):467–498, 1980.
R. Nelsen. An Introduction to Copulas. Springer, 2007.
Ingram Olkin and Albert W Marshall. Inequalities: theory of majorization and its
applications, volume 143. Academic press, 2016.

Y. Tenzer and G. Elidan, On the monotonicity of the copula entropy

19

M. Shaked and J. Shanthikumar. Stochastic Orders. Springer, 2007.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1):3–55, 2001.
A. Sklar. Fonctions de repartition a n dimensions et leurs marges. Publications de
l’Institut de Statistique de L’Universite de Paris, 8:229–231, 1959.
Y. Tenzer and G. Elidan. Speedy model selection (SMS) for copula models. Conference
on Uncertainty in Artificial Intelligence, 2013.
Gang Wei and Taizhong Hu. Supermodular dependence ordering on a class of multivariate copulas. Statistics & probability letters, 57(4):375–385, 2002.
DV Widder. Completely convex functions and lidstone series. Transactions of the
American Mathematical Society, 51(2):387–398, 1942.

