Sharpening Jensen's Inequality
J. G. Liao Arthur Berg
math.ST stat.TH 60E15




#1 1




0
  
 
  Sharpening Jensen's Inequality
    J. G. Liao and Arthur Berg 
    

    Division of Biostatistics and Bioinformatics
 Penn State University College of Medicine
    December 30, 2023
=================================================================================================================================

 

1
  

    Title


   


 This paper proposes a new sharpened version of the Jensen's inequality. The proposed new bound is simple and insightful, is broadly applicable by imposing minimum assumptions, and provides fairly accurate result in spite of its simple form. Applications to the moment generating function, power mean inequalities, and Rao-Blackwell estimation are presented. This presentation can be incorporated in any calculus-based statistical course.



Keywords:  Jensen gap, Power mean inequality, Rao-Blackwell Estimator, Taylor series


1.45 





§ INTRODUCTION

Jensen's inequality is a fundamental inequality in mathematics and it underlies many important statistical proofs and concepts. Some standard applications include derivation of the arithmetic-geometric mean inequality, non-negativity of Kullback and Leibler divergence, and the convergence property of the expectation-maximization algorithm <cit.>. Jensen's inequality is covered in all major statistical textbooks such as <cit.> and <cit.> as a basic mathematical tool for statistics.

Let X be a random variable with finite expectation and let φ(x) be a convex function, then Jensen's inequality <cit.> establishes 

    𝔼[φ(X)] - φ(𝔼[X])≥ 0.

This inequality, however, is not sharp unless var(X)=0 or φ(x) is a linear function of x.  Therefore, there is substantial room for advancement. This paper proposes a new sharper bound for the Jensen gap [φ(X)]-φ([X]). Some other improvements of Jensen's inequality have been developed recently; see for example <cit.>, <cit.>; <cit.> and references cited therein. Our proposed bound, however, has the following advantages. First, it has a simple, easy to use, and insightful form in terms of the second derivative φ”(x) and var(X).  At the same time, it gives fairly accurate results in the several examples below. Many previously published improvements, however, are much more complicated in form, much more involved to use, and can even be more difficult to compute than E[φ(X)] itself as discussed in <cit.>. Second, our method requires only the existence of φ”(x) and is therefore broadly applicable. In contrast, some other methods require φ(x) to admit a power series representation with positive coefficients <cit.> or require φ(x) to be super-quadratic <cit.>. Third, we provide both a lower bound and an upper bound in a single formula. 

We have incorporated the materials in this paper in our classroom teaching. With only slightly increased technical level and lecture time, we are able to present a much sharper version of the Jensen's inequality that significantly enhances students' understanding of the underlying concepts. 



§ MAIN RESULT


Let X be a one-dimensional random variable with mean μ, and P(X∈(a,b))=1, where -∞≤ a<b≤∞.  Let φ(x) is a twice differentiable function on (a,b), and define function

    h(x;ν)≜φ(x)-φ(ν)/(x-ν)^2-φ '(ν)/x-ν.

Then

    x∈(a,b)inf{h(x;μ)}var(X) ≤ E[φ(X)]-φ(E[X])≤sup_x∈(a,b){h(x;μ)}var(X).




 Let F(x) be the cumulative distribution function of X. Applying Taylor's theorem to φ(x) about μ with a mean-value form of the remainder gives

    φ(x)=φ(μ)+φ'(μ)(x-μ)+φ”(g(x))/2(x-μ)^2,

where g(x) is between x and μ.  Explicitly solving for φ”(g(x))/2 gives φ”(g(x))/2=h(x;μ) as defined above.  Therefore

    𝔼[φ(X)] - φ(𝔼[X])
       =∫_a^b {φ(x)-φ(μ)} dF(x)
       =∫_a^b {φ'(μ)(x-μ)+h(x;μ)(x-μ)^2} dF(x)
       =∫_a^b h(x;μ)(x-μ)^2 dF(x),

and the result follows because inf_x∈(a,b) h(x;μ)≤ h(x;μ)≤sup_x∈(a,b) h(x;μ).  



Theorem 1 also holds when inf h(x;μ) is replaced by infφ”(x)/2 and sup h(x;μ) replaced by supφ”(x)/2 since

    infφ”(x)/2≤inf h(x;μ)   and  supφ”(x)/2≥sup h(x;μ).
 
These less tight bounds are implied in the economics working paper <cit.>.  Our lower and upper bounds have the general form J·var(X), where J depends on φ.  Similar forms of bounds are presented in <cit.>, but our J in Theorem 1 is much simpler and applies to a wider class of φ. 


Inequality (<ref>) implies Jensen's inequality when φ”(x)≥ 0. Note also that Jensen's inequality is sharp when φ(x) is linear, whereas inequality (<ref>) is sharp when φ(x) is a quadratic function of x.  

In some applications the moments of X present in (<ref>) are unknown, although a random sample x_1,…,x_n from the underlying distribution F is available.  A version of Theorem 1 suitable for this situation is given in the following corollary.  


  Let x_1,…,x_n be any n datapoints in (-∞,∞), and let 


    x̅=1/n∑_i=1^n x_i,  φ_x=1/n∑_i=1^nφ(x_i),   S^2=1/n∑_i=1^n(x_i-x̅)^2.

Then

    inf_x∈[a,b]h(x;x̅)S^2≤φ_x-φ(x̅)≤sup_x∈[a,b]h(x;x̅)S^2,

where a=min{x_1,…,x_n} and b=max{x_1,…,x_n}.  


Consider the discrete random variable X with probability distribution P(X=x_i)=1/n, i=1,…,n.  We have E[X]=x̅, E[φ(X)]=φ_x, and var(X)=S^2.  Then the corollary follows from application of Theorem 1.  






If φ '(x) is convex, then h(x;μ) is monotonically increasing in x, and if φ '(x) is concave, then h(x;μ) is monotonically decreasing in x.  


 We prove that h'(x;μ)≥0 when φ'(x) is convex. The analogous result for concave φ'(x) follows similarly.  Note that 

    𝑑ℎ(x;μ)/𝑑𝑥=φ '(x)+φ '(μ)/2-φ(x)-φ(μ)/x-μ/1/2(x-μ)^2,

so it suffices to prove 

    φ '(x)+φ '(μ)/2≥φ(x)-φ(μ)/x-μ.

Without loss of generality we assume x>μ.  Convexity of φ'(x) gives 

    φ '(y)≤φ'(μ)+φ'(x )-φ'(μ)/x
    -μ(y-μ)

for all  y∈(μ,x).  Therefore we have 

    φ(x)-φ(μ)   =∫ _μ^xφ '(y)𝑑𝑦
       ≤∫ _μ^x{φ'(μ)+φ'(x )-φ'(μ)/x
    -μ(y-μ)
    }𝑑𝑦
       =φ '(x)+φ '(μ)/2(x-μ).

and the result follows.  


Lemma 1 makes Theorem 1 easy to use as the follow results hold:

    inf h(x;μ)= lim_x→ ah(x;μ)
       sup h(x;μ)= lim_x→ bh(x;μ)
    ,    when φ'(x)  is convex
    
          inf h(x;μ)= lim_x→ bh(x;μ)
       sup h(x;μ)= lim_x→ ah(x;μ)
    ,    when φ'(x)  is concave.

Note the limits of h(x;μ) can be either finite or infinite. The proof of Lemma 1 borrows ideas from <cit.>.  Examples of functions φ(x) for which φ' is convex include φ(x)=exp(x) and φ(x)=x^p for p≥2 or p∈(0,1].  
  Examples of functions φ(x) for which φ' is concave include φ(x)=-log x and φ(x)=x^p for p<0 or p∈[1,2].   




§ EXAMPLES

[Moment Generating Function] For any random variable X supported on (a,b) with a finite variance, we can bound the moment generating function E[e^tX] using Theorem 1 to get 

    inf_x∈(a,b){h(x;μ)}var(X)≤ E[e^tX]-e^tE[X]≤sup_x∈(a,b){h(x;μ)}var(X),

where 

    h(x;μ)=e^tx-e^tμ/(x-μ)^2-te^tμ/x-μ.

For t>0 and (a,b)=(-∞,∞), we have 

    inf h(x;μ)=lim_x→-∞h(x;μ)=0  and  sup h(x;μ)=lim_x→∞h(x;μ)=∞.

So Theorem 1 provides no improvement over Jensen's inequality.  However, on a finite domain such as a non-negative random variable with (a,b)=(0,∞), a significant improvement in the lower bound is possible because

    inf h(x;μ)=h(0;μ)=1-e^tμ+tμ e^tμ/μ^2>0.

Similar results hold for t<0.  We apply this to an example from <cit.>, where X is an exponential random variable with mean 1 and φ(x)=e^tx with t=1/2.   Here the actual Jensen's gap is [e^tX]-e^t[X]= 2-√(e)≈.351. 
Since var(X)=1, we have  

    .176≈ h(0;μ)≤[e^tX]-e^t[X]≤lim_x→∞h(x;μ)=∞.

The less sharp lower bound using infφ”(x)/2 is 0.125. Utilizing elaborate approximations and numerical optimizations <cit.> yielded a more accurate lower bound of 0.271.  



[Arithmetic vs Geometric Mean] Let X be a positive random variable on interval (a,b) with mean μ. Note that -log(x) is convex whose derivative is concave.  Applying Theorem 1 and Lemma 1 leads to 

    lim_x→ bh(x;μ) var(X) ≤ -E{log(X)} + logμ≤lim_x→ ah(x;μ)var(X),

where

    h(x;μ)=-log x+logμ/(x-μ)^2+1/μ(x-μ).


Now consider a sample of n positive data points x_1,…,x_n.  Let x̅ be the arithmetic mean and x̅_g=(x_1x_2⋯ x_n)^1/n be the geometric mean.  Applying Corollary <ref> gives

    exp{S^2 h(b;x̅)}≤x̅/x̅_g≤exp{S^2 h(a;x̅)},

where a, b, S^2 are as defined in Corollary <ref>.  To give some numerical results, we generated 100 random numbers from uniform distribution on [10,100].  For these 100 numbers, the arithmetic mean x̅ is 54.830 and the geometric mean x̅_g is 47.509.  The above inequality becomes

    1.075≤x̅/(x_1x_2⋯ x_n)^1/n = 1.154≤ 1.331,

which are fairly tight bounds. Replacing h(x_n;x̅) by φ”(x_n)/2 and h(x_1;x̅) by φ”(x_1)/2 leads to a less accurate lower bound 1.0339 and upper bound 21.698.  



[Power Mean] Let X be a positive random variable on a positive interval (a, b) with mean μ. For any real number s≠0, define the power mean as 

    M_s(X)=(E X^s)^1/s

Jensen's inequality establishes that M_s(X) is an increasing function of s. We now give a sharper inequality by applying Theorem 1. Let r≠0, Y=x^r, μ_y=EY, p=s/r and φ(y)=y^p.    Note that E X^s = E{φ(Y)}.  Applying Theorem 1 leads to 

    inf h(y;μ_y)var(Y)≤ E[X^s] - (E X^r)^p ≤sup h(y;μ_y) var(Y),

where

    h(y;μ_y)=y^p-μ_y^p/(y-μ_y)^2-p μ_y^p-1/y-μ_y.

To apply Lemma 1, note that φ'(y) is convex for p≥ 2 or p∈(0,1] and is concave for p<0 or p∈[1,2] as noted in Section 2.  

Applying the above result to the case of r=1 and s=-1, we have Y=X, p=-1. Therefore

    ((EX)^-1 +lim_y→ ah(y;μ_y)var(X))^-1≤(E X^-1)^-1≤((EX)^-1 + lim_y→ bh(y;μ_y)var(X))^-1.


For the same sequence x_1,…, x_n generated in Example 2, we have x̅_harmonic=39.113.  Applying Corollary <ref> leads to 

    25.337≤x̅_harmonic=39.113≤ 48.905.

Note that the upper bound 48.905 is much smaller than the arithmetic mean x̅=54.830 by the Jensen's inequality.  Replacing h(b;x̅) by φ”(b)/2 and h(a;x̅) by φ”(a)/2 leads to a less accurate lower bound 0.8298 and 51.0839.  





In a recent article published in the American Statistician, <cit.> revisited Kolmogorov's formulation of generalized mean as 

    E_φ(X)=φ^-1(E[φ(X)]),

where φ is a continuous monotone function with inverse φ^-1. The Example 2 corresponds to φ(x)=-log(x) and Example 3 corresponds to φ(x)=x^s.  We can also apply Theorem 1 to bound φ^-1(E φ(X)) for a more general function φ(x).  

[Rao-Blackwell Estimator]
Rao-Blackwell theorem (Theorem 7.3.17 in Casella and Berger, 2002; Theorem 10.42 in Wasserman, 2013) is a basic result in statistical estimation. Let  θ̂ be an estimator of θ,  L(θ, θ̂) be a loss function convex in θ̂, and T a sufficient statistic. Then the Rao-Blackwell estimator, θ̂^*=E[θ̂| T], satisifies the following inequality in risk function 

    E[L(θ, θ̂)]≥ E[L(θ, θ̂^*)].

We can improve this inequality by applying Theorem 1 to φ(θ̂)=L(θ, θ̂) with respect to the conditional distribution of θ̂ given T:

    E[L(θ, θ̂)| T] - L(θ, θ̂^*)≥inf_x∈(a,b) h(x;θ̂^*) var(θ̂| T),
  
where function h is defined as in Theorem 1 for φ(θ̂) and P(θ̂∈(a,b)| T)=1. Further taking expectations over T gives

    E[L(θ, θ̂)] - E[L(θ, θ̂^*)]≥ E[inf_x∈(a,b) h(x;θ̂^*)var(θ̂| T)].

In particular for square-error loss, L(θ, θ̂)=(θ̂-θ)^2, we have

    E[(θ-θ̂)^2] - E[(θ-θ̂^*)^2] = E[var(θ̂| T)].

Using the original Jensen's inequality only establishes the cruder inequality in Equation (<ref>).






§ IMPROVED BOUNDS BY PARTITIONING



As discussed in Example 1 above, Theorem 1 does not improve on Jensen's inequality if inf h(x;μ) = 0.  In such cases, we can often sharpen the bounds by partitioning the domain (a,b) following an approach used in <cit.>. Let 

    a=x_0<x_1<⋯<x_m=b,

I_j=[x_j-1,x_j), η_j=P(X∈ I_j), and μ_j=E(X| X∈ I_j). It follows from the law of total expectation that 


    E[φ(X)]   =∑_j=1^m η_j E[φ(X) | X∈ I_j]
       =∑_j=1^m η_j φ(μ_j)+ ∑_j=1^m η_j (E[φ(X) | X∈ I_j]-φ(μ_j)).

Let Y be a discrete random variable with distribution P(Y=μ_j)=η_j, j=1,2,…,m.
It is easy to see that EY = EX.  It follows by Theorem 1 that

    ∑_j=1^mη_jφ(μ_j) = E[φ(Y)]≥φ(EY)+inf_y∈[μ_1,μ_m] h(y;μ_y)var(Y).

We can also apply Theorem 1 to each E[φ(X | X∈ I_j)]-φ(μ_j) term: 

    E[φ(X | X∈ I_j)]-φ(μ_j)≥inf_x∈ I_j h(x;μ_j)var(X| X∈ I_j).

Combining the above two equations, we have

    E[φ(X)]- φ(EX) ≥inf_y∈[μ_1,μ_m] h(y;μ_y)var(Y) + ∑_j=1^m η_j inf_x∈ I_j h(x;μ_j)var(X| X∈ I_j).

Replacing inf by sup in the righthand side gives the upper bound.  


The Jensen gap on the left side of (<ref>) is positive if any of the m+1 terms on the right is positive. In particular, the Jensen gap is positive if there exists an interval I⊂(a,b) that satisfies inf_x∈ Iφ”(x)>0, P(X∈ I)>0 and var(X| X∈ I)>0.  Note that a finer partition does not necessarily lead to a sharper lower bound in (<ref>). The focus of the partition should therefore be on isolating the part of interval (a,b) in which φ”(x) is close to 0. 

Consider example X∼N(μ ,σ ^2) with μ=0 and σ=1 and φ(x)=e^x. We divide (-∞,∞) into three intervals with equal probabilities. This gives

The actual Jensen gap is e^μ+σ/2-e^μ=0.649.  The lower bound from (<ref>)
 is 0.409, which is a huge improvement over Jensen's bound of 0.  The upper bound ∞, however, provides no improvement over Theorem 1.  

To summarize, this paper proposes a new sharpened version of the Jensen's inequality. The proposed bound is simple and insightful, is broadly applicable by imposing minimum assumptions on φ(x), and provides fairly accurate result in spite of its simple form. It can be incorporated in any calculus-based statistical course.  

chicago


