arXiv:1508.01146v2 [stat.ME] 1 Jun 2016

Minimizing the CDF Path Length: A Novel
Perspective on Uniformity and Uncertainty
of Bounded Distributions
Michael E. Beyer, PE, CAP
June 2, 2016

Abstract
An index of uniformity is developed as an alternative to the maximum-entropy
principle for selecting continuous, differentiable probability distributions P subject
to constraints C. The uniformity index developed in this paper is motivated by the
observation that among all differentiable probability distributions defined on a finite
interval [a, b] ∈ R, it is the uniform probability distribution that minimizes the path
length of the associated cumulative distribution function FP on [a, b]. This intuition
is extended to situations where there are constraints on the allowable probability
distributions. In particular, constraints on the first and second raw moments of a
distribution are discussed in detail, including the analytical form of the solutions and
numerical studies of particular examples. The resulting ”shortest path” distributions
are found to be decidedly more heavy-tailed than the associated maximum-entropy
distributions, suggesting that entropy and ”CDF path length” measure two different
aspects of uncertainty for bounded distributions.

Keywords: Entropy, prior, uninformative, ignorance, Bayesian

1

1

Introduction

There are numerous situations were we seek a probability model to represent uncertainty,
yet we only feel comfortable making a few assumptions about the nature of that uncertainty.
In such situations, the choice of probability distributions is underdetermined, requiring additional criteria to pick out a single distribution from among those consistent with the
assumed facts. A common criterion is the maximum entropy principle (MEP), which recommends selecting the distribution that maximizes the Shannon Entropy, which is defined
for discrete probability distributions (Pd ) as:
X
H(X) := EPd [− ln(Pd (X))] = −
Pd (xi ) ln(Pd (xi ))
xi ∈X

For continuous distributions, one uses the Shannon Differential Entropy, which is an
extension of the Shannon Entropy to probability density functions f (x):
Z
f (x) log(f (x))dx
h(X) := −
X

The Shannon Entropy has several intuitively appealing properties when viewed as a
measure of information content: continuity, additivity, and maximality under a uniform
distribution. It also has close connections to data processing and statistical mechanics,
and it can be shown to satisfy several types of consistency criteria (Uffink (1996)). In his
seminal paper introducing information theory Shannon (1948) cited three properties that
guided the development of the Shannon Entropy (H) for a discrete probability distribution
pn (xi ), defined on a finite set {xi }, i ∈ 1...n:
1. H should be continuous in the pn (xi )
2. If all pn (xi ) are equal, then H should be a monotonic increasing function of n: if X
has n equally likely outcomes and Y has m > n equally likely outcomes, then Y has
more uncertainty is than X.
3. The value of H should not depend on how we structure the process generating the
observable distribution. For example, if we have a random variable X ∈ {1, 2, 3, 4},
we can model a measurement of X as resulting from a single chance event with
four possible outcomes, or as a sequence of two dependent events, each with two
possibilities (conditional on the first outcome). If we are not allowed to look at the
intermediate process, then the entropy of X should be the same for each model.
Due to the intuitively appealing properties of entropy noted by Shannon and others,
both Shannon Entropy and Differential Shannon Entropy have been recommended as tools
for developing uncertainty distributions (Jaynes (1957), Jaynes (1982)). Maximum-entropy
(ME) distributions are defined by their constraints, with several common examples given
below:
• X ∼ Uniform if X ∈ [a, b], −∞ < a < b < ∞
2

• X ∼ Beta if X ∈ [a, b], −∞ < a < b < ∞, E[X] = µ, V [X] = σ 2
• X ∼ Exponential if X ∈ [0, ∞), E[X] = µ
• X ∼ Normal if X ∈ (−∞, ∞), E[X] = µ, V [X] = σ 2
The fact that the normal distribution emerges as a ME distribution is quite a satisfying
result from a statistical and mathematical standpoint, as it concords with the ubiquity of
this distribution and provides support for its use as a ”minimally informative” choice in
modeling.
However, while entropy as a measure of information content is well-established, its applicability to modeling uncertainty is less clear cut. In particular, Uffink (1996) notes that
the Shannon Entropy is just one of many ”entropy-like” measures out there. Other examples are distributions maximizing the class of Rényi-entropies, of which Shannon Entropy
is a special case, or distributions that are invariant under a pre-specified transformation.
What is interesting is that these forms of ”entropy” also satisfy very plausible assumptions
about how a measure of information should behave, although the details of these assumptions differ among the models (Uffink (1995)). Finally, even within the ME paradigm, the
form of the resulting entropy maximizer depends critically on the form of the constraints
(Uffink (1995)), implying that the MEP does not remove subjectivity as much as push it
down one level.
In addition, from a decision-theoretic viewpoint, it is not clear how a ME distribution
is affecting the decisions. Is it making us more conservative, less conservative? Are we
emphasizing particular outcomes by maximizing the entropy? If so, why are those outcomes
justified in being given more weight, and in particular the specific weights prescribed by
the MEP? The above questions are not intended to be a pointed critique, and certainly
not any form of rebuttal, of the MEP or the resulting distributions. Instead, the intent is
merely to highlight the ambiguities inherent in attempting to model ignorance as opposed
to information. The ambiguity is exacerbated by the existence of alternative principles
of similar intuitiveness or level of justification (e.g., Jeffrey’s invariance principle, Rényientropies).
The novel approach presented in this paper no doubt also possesses its own set of ambiguities and unanswered questions. However, like other methods for modeling uncertainty,
it has several features that may appeal to various practitioners and/or be particularly
applicable in certain contexts.

2

Uniformity

The approach taken in this paper seeks to maximize the uniformity of a probability distribution, where ”uniform” is meant in the sense of the ”equal probability” of a random
variable for various outcomes. In particular, this paper focuses on maximizing the uniformity of bounded, continuous-valued random variables with smooth distribution functions
(i.e., the cumulative distribution function (CDF) is differentiable over the entire domain).
Like other measures of uncertainty, uniformity is developed from a base of intuitively
3

plausible presuppositions and then extended to less familiar domains. In particular, any
uniformity measure for a continuous, bounded random variable should possess two very
basic properties:
1. A degenerate distribution (i.e., step function CDF) is the least uniform.
2. The uniform distribution (not surprisingly) is the most uniform distribution.
Given a measure of uniformity, we formulate the central ”principle” for choosing a
distribution using this measure:
Principle of Maximum Uniformity: Of all continuous, finite-domain distribution
functions that are consistent with the available information, choose the one with the highest measure of uniformity.
The rationale for this principle is that we want to choose a distribution ”as uniform as
possible” from among the available distributions, so that we are not, a priori, overemphasizing any regions of the sample space beyond what is necessary to be consistent with our
prior assumptions or available information.
To actually implement this principle, we need to quantify how close a given distribution
is to a uniform distribution. The following observation will prove instrumental to the
definition of such a quantity: the set of points {(x, FU (x)) : x ∈ [a, b]} generated by the
CDF of the uniform density function, FU , on [a, b] takes the shortest possible path from
(x0 , y0 ) = (a, 0) to (x1 , y1 ) = (b, 1). This is a defining feature of the uniform CDF (and of
straight lines in general). Hence, we used the path length of the CDF as the basis for our
quantity:
Definition: Uniformity Index Given a continuous random variable X ∈ [a, b], with
probability density function fX , the uniformity index of fX is defined as:
p
1 + (b − a)2
U(fX ) := R b p
1 + [fX (t)]2 dt
a
The uniformity index is simply the ratio of the path length of the uniform CDF to
the path length of the CDF of X (FX ) on [a, b]. It satisfies the two basic properties of a
uniformity measure listed earlier: the longest path is the step function CDF associated with
the degenerate density function δa (x) and√the shortest path is produced by the uniform
1+(b−a)2

CDF on [a, b]. In general, √12 ≈ 0.707 ≤ 1+b−a ≤ U(fX ) ≤ 1. Note that for a given
domain [a, b], the uniformity index varies only as a function of the path length of fX ; hence,
maximizing the uniformity is equivalent to finding the Shortest Path Distribution (SPD)
that satisfies a given set of constraints C.

4

3

SPD under Raw Moment Constraints

Given a domain [a, b] and set of constraints C, we are seeking to minimize the path length
L[F ] of the cumulative distribution function F defined on [a, b], subject to a set of constraints C (i.e., F is part of a family of distributions PC satisfying C):
s
 2
Z b
dF
1+
dx
argmin L[F ] ≡ argmin
dx
F ∈PC
F ∈PC
a
This type of problem is typical in the Calculus of Variations, Rwhere we are seeking to find
b
a univariate function f (x) that minimizes a functional I[f ] ≡ a G(x, f, f 0 )dx . Candidates
for optimal functions are identified as solutions to the Euler-Lagrange equation:


∂G
d ∂G
−
=0
∂f
dx ∂f 0
In general, the Euler-Lagrange equation is merely a necessary condition for a minimum.
Second-order conditions are required to verify that f is indeed a minimum and not a
maximum or merely a stationary point (think ”plateaus” of some cubic polynomials). A
necessary condition for f being a minimum of I[f ] is the Legendre Condition:
∂ 2G
≥ 0 for x ∈ [a, b]
∂(f 0 )2
This paper will focus on solving the above problem for the path length functional L[F ]
under constraints on the raw moments of f = F 0 and a basic non-negativity constraint on
the probability density function f (i.e., f (x) ≥ 0 ∀x ∈ [a, b]). The functions associated with
the SP D constraints on all raw moments up to m is given by the vector-valued function
C m:

m

Z

m

b
i

x f (x)dx − µi

C (f ; µ
~ ) :=
a

, where µ
~ = (µ0 , µ1 , ..., µm ) and µ0 ≡ 1

(1)

i=1

The constraints in this problem are referred to as isoperimetric constraints (alluding
to the motivating problem of finding a shape that maximizes the enclosed area given a
fixed perimeter length). As with classical constrained optimization, constraints can be
incorporated using Lagrange Multipliers, λi , which turn a constrained problem into an unconstrained problem. In particular, the integrand for isoperimetric constraint i is multiplied
by λi and added to the integrand of L[F ] to form the Lagrangian, φ, which will be used to
find the optimal function:
φ[x; f, λ] ≡

p

1+

f (x)2

+

m
X

λi xi f (x), where f (x) = F 0 (x)

(2)

i=0

Applying the Euler-Lagrange equation to φ, we can derive the general form of a solution:

5

d
∂φ
−
∂f
dx



∂φ
∂f 0



f

m
X

=p
+
λi xi = 0 =⇒
2
1 + (f )
i=0
Pm
i
d
i=0 λi x
F (x; λ) = q
f (x; λ) =
P
dx
i 2
1−( m
i=0 λi x )

(3)

Where in the last equation, we took the positive root of the numerator, as the sign
is inconsequential with the undetermined multipliers. We will now check the Legendre
condition for φ:
!
m
X
p
∂ 2φ
∂2
=
1 + f (x)2 +
λi xi f (x) = 0
∂(f 0 )2
∂(f 0 )2
i=0
This result indicates that our solution is not ruled out as a minimum. We can informally
check the actual solutions by comparing them to a trial solution. If the solution has a longer
path length than our trial solution, we know its not a minimum. However, if it has a shorter
length, then we are likely looking at a minimum. This is the approach we will use on the
specific solutions derived in this paper.
We will now need to solve the system of integral constraints for the vector of Lagrange
multipliers λ := (λ0 , λ1 , ..., λn ), such that the (3) is non-negative and satisfies the isoperimetric constraints (1) :

Find λ :

4




Pm



i

Pm



i




i=0 λi x
i=0 λi x
q
≥ 0, ∀x ∈ [a, b]; C m  q
;µ
~ = 0
P
Pm


2
2
m
i
i
1 − ( i=0 λi x )
1 − ( i=0 λi x )

Numerical Solution

The form of the density function precludes a simple analytical solution for λ. Instead,
we numerically estimated λ using a piecewise uniform approximation f ∗ (x; λ) of the true
density function f (x; λ). First, the domain of f was broken into n equal-length intervals
∆i = [xi−1 , xi ) = [a + (i − 1)δn , a + iδn ], i = 1...n, where δn = b−a
. Each interval has a
n
a+(i−1)δn +a+iδn
2a+(2i−1)δn
midpoint, pi =
=
, which is used for calculating the value of the
2
2
density over that interval. Using this approach, we were able to replace the isoperimetric
constraints (which are expressed as integrals) with Riemann sums over the partition {∆i }:

Ckm

Z

b
k

Z

x f (x)dx − µk ≈

=
a

b
k ∗

x f (x)dx − µk =
a

n
X
i=1

n Z
X
i=1


f (pi )x dx − µk =
k

∆i

xk+1
− xk+1
i
i−1
f (pi ; λ)
− µk = hf (p; λ), δnk i − µk := Mnk (λ)
k+1

6

(4)

where

k
δn,i

Z

xi

:=

z k dz

xi−1

Each summand on the left hand side of (4) represents the contribution of the interval
∆i to the k th raw moment of the piecewise uniform approximation to f . Note that this
reduces to the simple Riemann sum over f with partition length δn in the case of the
”zeroth” moment (k = 0). For a given partition size n, we can estimate λ using a least
squares solution to the constraints of the approximate problem:
argmin
λ

m
X

[Mnj (λ)]2

(5)

j=1

Theoretically, this is all we need. However, from a numerical perspective, the form of
(3) as a function of λ has two problems :
1. The numerator can become negative
2. The denominator can become complex
Therefore, we needed to define a feasible region for λ to ensure we got physically sensible
answers. This was be accomplished using the following constraint set on λ:
0≤

m
X

λi pij < 1, j = 1...n

(6)

i=0

Conveniently, (6) defines two linear half-spaces, which defines a convex polytope in Rm .
We solved the above formulation using the R statistical programming language to run
the auglag nonlinear optimization routine in the alabama package, with Nelder-Mead as
the selected optimization method. The algorithm converged in some specific instances;
however, numerical tests indicated that directly using the parameterized form of the density f (pi ; λ) in the optimization model induced instability and poor convergence for some
values of the constraints. Therefore, we developed a second approach, which took a more
direct, high-dimensional optimization route.
In the second approach, we used the same set of partitions {∆i } and midpoints pi as
in the first approach, but we treated the density values at each midpoint (i.e., f (pi )) as a
decision variable. This change resulted in an n variable minimization problem in f (pi ) := fi ,
where we are approximating the true density function f (x; λ) by a piecewise constant
function f ∗ (x) that takes value fi if x ∈ ∆i . If a smooth, continuous solution exists to the
true optimization P
problem, then the sequence of approximate solutions will also converge
(pointwise): lim ni=1 1(x)x∈∆i fi (x) = f (x) ∀x ∈ [a, b], so f ∗ (x) → f (x) pointwise. The
n→∞
new optimization model becomes:
argmin δn
fi

n q
X

(1 + fi2 ) s.t. Mnk = 0, 0 ≤ k ≤ m; fi ≥ 0, i = 1...n

i=1

7

(7)

Where Mnk := hf , δnk i − µk as in (4), but now the vector of density values are the actual
decision variables, not values parameterized over λ. This more direct approach (using R
with auglag and the nlminb optimization routine) had better convergence properties for
the problems we investigated in this paper, with first and second order KKT conditions
being consistently satisfied. However, we felt that the analytical solution to the EulerLagrange equation still has theoretical and conceptual value, despite not leading to the
best numerical approach.
The next section will show the typical shape of the resulting SPDs under low-order
moment constraints. Each of these will be graphically compared to their equivalent ME
distribution and the CDF lengths will be calculated for each. The paper will end with a
discussion of the key insights from this investigation.

5
5.1

Typical solutions for m ∈ {0, 1, 2}
Uniform (m = 0)

This is the simplest case, where we only require that the probability equal 1. The resulting
SPD is a uniform distribution, which is identical to the ME distribution. The uniform
distribution is trivially the shortest path distribution under these conditions.

5.2

Exponential (m = 1)

We start to see some more structure when we also constrain the mean. In this case, we
are searching for a SPD on [0, 0.1] with mean 0.04. The equivalent ME distribution is the
truncated exponential distribution T EXP (λ, b), where λ = 12.3, b = 0.1. (Fig. 1) The
path length for the truncated exponential solution is 1.017, whereas the path length of the
SPD distribution is 1.006, verifying that indeed the SPD distribution has a shorter path
length.
While absolute path-length differences appear slight, this is primarily due to an issue
of scale. To correct for this, we calculated the ratio of each distribution’s respective difference from the CDF path length of a uniform distribution on [0, .01] (i.e., this would be
p
1 + (0.1)2 = 1.005), to get a ”difference ratio” (r) of 11. The general formula is shown
below:
p
L[fM E ] − 1 + (b − a)2
p
Difference Ratio(fM E , fSP ) :=
(8)
L[fSP ] − 1 + (b − a)2
In this case, a difference ratio of 11 means that the SPD CDF path length is 11 times
closer to the straight-line distance (shortest possible path) than the CDF path length of
the associated truncated exponential distribution. Essentially, we are using the path length
of the uniform CDF over [0, .01] as the point of reference, as 0 is not a feasible path length.

8

Figure 1: SPD (solid line) and ME distribution (dotted line) on [0, 0.1] for m1 = 0.04
As an additional exploration, we also examined how the solutions change as the upper
bound b of the random variable is increased.

Figure 2: Sequence of SPDs for m1 = 0.04 with increasing upper bounds (solid lines) and
ME distribution for unbounded case (dotted line).
From Figure 2 it appears that qualitatively the SPD becomes increasingly more peaked,
suggesting that lim SP D(m1 = 0.04) =
˙ δ(0). However, E[δ(0)] = 0 6= 0.04, which would
b→∞

9

imply that there is no unbounded SPD (consistent with the fact that the sequence path
lengths of the associated CDFs is unbounded).

5.3

Bell and Bowl (m = 2)

This was the most complex case examined. We calculated two SPDs on [−0.1, 0.1] with
mean 0 and second-moments (here:variance) equal to 0.001 and 0.005, respectively. The
equivalent ME distributions are Beta distributions on [−0.1, 0.1] with parameters (α1 =
β1 = 4.5) and (α2 = β2 = 0.5) , respectively. We called the first case the ”Bell” and the
second case the ”Bowl” Beta distributions to indicate their qualitative shapes. We found
that the SPD with the same first and second moments as the Bell Beta distribution was
markedly more peaked (Figure 3), while the Bowl Beta distribution and moment-matched
SPD were almost identical (Figure 4).

Figure 3: SPD for m1 = 0, m2 = 0.001 on [−0.1, 0.1] (solid line) and Bell Beta (dotted line)

10

Figure 4: SPD for m1 = 0, m2 = 0.005 on [−0.1, 0.1] (solid line) and Bowl Beta (dotted
line)
The path length was 1.06 for the Bell Beta CDF and 1.04 for the moment-matched
SPD CDF, yielding a difference ratio of 2. The Bowl Beta and moment-matched SPD
CDFs had the same path length of 1.02. Similar to the case with m = 1, we examined the
sequence of Bell-shaped SPDs with increasing support (Figure 5) and compared them to
their moment-matched normal distribution.

11

Figure 5: Sequence of SPDs (m1 = 0, m2 = 0.001) with symmetrically increasing support
(solid lines). Moment-matched normal distribution with unbounded support (dotted line).
As shown in Figure 5, it appears that lim SP D(m1 = 0, m2 = 0.001) =
˙ δ(0), similar
b→∞

to the fixed-mean SPD in the previous section. Again, this suggests that the SPDs must
be bounded.

6

Discussion

We see that the SPDs do not, as a general rule, match the results given by the Maximum
Entropy Principle. Therefore, how can we interpret the difference between an SPD and
its associated ME distribution? In broad terms, it appears that SPDs emphasize ”fragile
stability” while the ME distributions exhibit ”stable instability”. A process operating according to a ME distribution will rapidly ”fill” its outcome space, reaching its equilibrium
distribution quickly due to its high entropy. In contrast, a process operating according to
an SPD will appear to be relatively stable and well contained, only to suddenly produce
an extreme outlier (relative to the central 95% of the distribution).
The punctuated/catastrophic behavior of SPDs is reminiscent of the types of behavior
discussed in Nassim Taleb’s The Black Swan, where Taleb argues that the variance between
outcomes is a poor measure of risk if the underlying distribution has high kurtosis (thus
high tail-risk). It is interesting that taking a very geometric/literal approach to uniformity
produces such different densities when compared to distributions based on maximizing an
abstract measure. The generality of this behavior is an open question: does it arise only
with path length, or does it arise whenever one is optimizing a metric as opposed to a
measure as a function of an input distribution. Regardless, the results of this analysis
12

point out that it is far from clear how ”uncertainty” is best quantified, with different,
plausible approaches yielding very different results. By emphasizing extreme behavior,
SPDs provide a novel perspective on the relationship between risk, ignorance, probability,
and uncertainty

References
Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review 106 (4),
620–630.
Jaynes, E. T. (1982). On the rationale of maximum-entropy methods. Proceedings of the
IEEE 70 (9), 939–952.
Shannon, C. E. (1948, July). A mathematical theory of communication. The Bell System
Technical Journal 27, 379–423.
Uffink, J. (1995). The constraint rule of the maximum entropy principle. Studies in History
and Philosophy of Modern Physics 26B, 223–261.
Uffink, J. (1996). The constraint rule of the maximum entropy principle. Studies in History
and Philosophy of Modern Physics 27, 47–79.

13

