Large-sample theory for inferential models: a possibilistic Bernstein--von Mises theorem
Ryan Martin Jonathan P. Williams




A possibilistic Bernstein–von Mises theorem



R. Martin and J. P. Williams

North Carolina State University 
 Department of Statistics 
 Raleigh, North Carolina 27695, U.S.A. 
 
{rgmarti3, jwilli27}@ncsu.edu



Large-sample theory for inferential models: a possibilistic Bernstein–von Mises theorem
    Ryan Martin
Jonathan P. Williams
    April 29, 2024
=======================================================================================




The inferential model (IM) framework offers alternatives to the familiar probabilistic (e.g., Bayesian and fiducial) uncertainty quantification in statistical inference.  Allowing this uncertainty quantification to be imprecise makes it possible to achieve exact validity and reliability.  But is imprecision and exact validity compatible with attainment of the classical notions of statistical efficiency?  The present paper offers an affirmative answer to this question via a new possibilistic Bernstein–von Mises theorem that parallels a fundamental result in Bayesian inference.  Among other things, our result demonstrates that the IM solution is asymptotically efficient in the sense that its asymptotic credal set is the smallest that contains the Gaussian distribution whose variance agrees with the Cramér–Rao lower bound.  











§ INTRODUCTION



<cit.> writes that “the most important unresolved problem in statistical inference is the use of Bayes theorem in the absence of prior information.”  Numerous attempts have been made, including Bayesian inference with default priors, from Laplace's flat priors to the refinements by <cit.> and <cit.>, fiducial inference in Fisher's original sense <cit.> and its generalizations <cit.>, and the various imprecise-probabilistic proposals <cit.>.  One thing that many of these solutions offer is a large-sample result which states, roughly, that the associated credible sets achieve the nominal frequentist coverage probability when the sample size, n, is large.  Statisticians insist on methods that are demonstrably reliable in the sense of tending to report “correct inferences” across repeated uses, so results like this are high-value.  The résultat extraordinaire along these lines is the Bernstein–von Mises theorem which states the following: under certain conditions, the Bayesian or fiducial posterior distribution is approximately Gaussian, centered at the maximum likelihood estimator, and is efficient in the sense that its variance equals the Cramér–Rao lower bound; here “approximately” is in the sense that the total variation distance between the two distributions is vanishing in probability as n →∞.  For details, see <cit.>, <cit.>, or <cit.>. 

Still, none of the Bayesian- or fiducial-like solutions mentioned above have proved to be fully satisfactory.  One possible explanation is that large-sample confidence sets don't tell the whole story.  Indeed, the false confidence theorem <cit.> implies that, at every n, there exists false hypotheses about the unknown to which the (Bayesian or fiducial) posterior will tend to assign relatively high probability—that is, the posterior probability assignments have an inherent unreliability.  To avoid this pitfall, it's necessary that the Bayesian or fiducial precise probability be replaced by a data-dependent imprecise probability.  The inferential model (IM) framework, including the original developments in <cit.> and the more recent generalizations in <cit.>, does just this.  Moreover, the IM is provably valid in a sense that implies it's safe from false confidence and, e.g., it provides exact finite-sample confidence sets.  While the IM solution's efficiency compared to existing solutions—Bayesian, frequentist, or otherwise—has been determined empirically in specific applications, a general theoretical statement concerning its efficiency is lacking.  In this paper, for the class of IMs in <cit.>, we establish a possibilistic version of the Bernstein–von Mises theorem that parallels those mentioned above.  Among other things, this result confirms our conjecture that, despite the genuine imprecision in the IM's output needed to ensure exact validity, there's no loss of efficiency asymptotically.

















§ BACKGROUND




 §.§ Possibility theory



Possibility theory is among the simplest imprecise probability theories, corresponding to consonant belief structures <cit.>.  Other key references include <cit.>, <cit.>, and <cit.>. 


The simplicity of possibilistic uncertainty quantification comes from its parallels to precise probability theory.  A necessity–possibility measure pair (, ) that's intended to quantify uncertainty about an uncertain Z in  is determined by a possibility contour π: → [0,1], with sup_z ∈π(z) = 1, via the rules

    (B) = sup_z ∈ Bπ(z)   and  (B) = 1 - sup_z ∉Bπ(z),    B ⊆.

So, where ordinary probability is determined by integrating a density function, possibility is determined by optimizing a contour.  The above values are often interpreted subjectively as (coherent) upper and lower probabilities associated with the proposition “Z ∈ B” or as Shaferian degrees of belief.


One way to elicit a possibility measure, specifically relevant to us here, is via the probability-to-possibility transform <cit.>.  If p is a probability density function, which determines a random variable Z ∼, then the probability-to-possibility transform defines π as 

    π(z) = { p(Z) ≤ p(z) },    z ∈.

This defines the “best approximation” 

of  by a possibility measure in the sense that its corresponding credal set is the smallest one that contains .  If p is the (multivariate) standard normal density on , the probability-to-possibility transform has contour defined as γ as shown in Figure <ref>(a); the corresponding Gaussian possibility measure defined via optimization is denoted by .  



































































 §.§ Inferential models



The original IM constructions <cit.> were formulated in terms of (nested) random sets and, hence, the connection to possibility theory was indirect.  A more recent IM construction introduced in <cit.>—see, also, <cit.>—directly defines the IM's possibility contour using a version of the probability-to-possibility transform.  An advantage of this alternative perspective and construction is that it avoids the ambiguity in specifying both a data-generating equation and a so-called “predictive random set.”  





The statistical model assumes that X^n=(X_1,…,X_n) consists of iid samples from a distribution _Θ depending on an unknown/uncertain Θ∈ to be inferred.  The model and observed data X^n=x^n together determine a likelihood function θ↦ L_x^n(θ) and a corresponding relative likelihood function

    R(x^n,θ) = L_x^n(θ)/sup_ϑ L_x^n(ϑ).

The relative likelihood itself defines a data-dependent possibility contour that has been widely studied <cit.>.  Most appealing about the likelihood-based possibility contour is its shape: peak at the maximum likelihood estimator θ̂_x^n and consistent with Fisher's suggested likelihood-based preference order on the parameter space.  But it lacks a standard scale, i.e., what constitutes a “small” relative likelihood depends on aspects of the individual application.  A (literally) uniform scale of interpretation across applications can easily be obtained via what <cit.> calls “validification,” which is a sort of possibilistic transform.  In particular, for observed data X^n=x^n, the possibilistic IM's contour is

    π_x^n(θ) = _θ{ R(X^n,θ) ≤ R(x^n, θ) },   θ∈,

and the corresponding possibility measure is 

    _x^n(H) = sup_θ∈ Hπ_x^n(θ),    H ⊆.

Critical to the IM developments is the so-called validity property which can be succinctly described by the following expression:

    sup_Θ∈_Θ{π_X^n(Θ) ≤α}≤α,   for all α∈ [0,1].

This is precisely the universal scaling that the relative likelihood itself is missing, i.e., “π_x^n(θ) ≤α” has the same inferential meaning/force in every application.  It also implies that the IM is safe from false confidence.  Finally, it has some important and familiar statistical consequences:


  * a test that rejects the hypothesis “Θ∈ H” when _x^n(H) ≤α will control the frequentist Type I error rate at level α, and 

  * the set C_α(x^n) = {θ∈: π_x^n(θ) > α} is a 100(1-α)% frequentist confidence set in the sense that its coverage probability is at least 1-α.

Given that the IM is inherently imprecise and offers finite-sample guarantees, the reader might think that this comes with an associated loss of efficiency compared to those methods whose justification relies on asymptotic considerations.  The result in the next section aims to debunk this myth.   






























§ A POSSIBILISTIC BERNSTEIN–VON MISES THEOREM




 §.§ Preview


To build some intuition, consider a simple exactly Gaussian model where X^n = (X_1,…,X_n) is an iid sample from (Θ,σ^2), where the mean Θ is unknown  but σ^2 is known.  The maximum likelihood estimator is θ̂_x^n=x̅, the sample mean, and it is 



not difficult to show that the IM's possibility contour is 

    π_x^n(θ) = γ{ n^1/2(θ-θ̂_x^n) / σ},   θ∈,

where γ is the Gaussian possibility contour in Section <ref>.  Alternatively, if we switch to a local parametrization of θ, i.e., θ = θ̂_x^n + σ n^-1/2 z, then 

    π̌_x^n(z) := π_x^n(θ̂_x^n + σ n^-1/2 z) = γ(z),    z ∈.

For a generic H ⊆, the corresponding possibility measure is 

    _x^n(H)     = 
    
    
    { n^1/2(H - θ̂_x^n)/σ},

where  is the Gaussian possibility measure from Section <ref> and, for constants a and b, the set aH+b is defined as aH + b = {aθ + b: θ∈ H}. 

Once we step away from an exact Gaussian model, there is no longer an exact correspondence between the possibilistic IM's solution and the Gaussian possibility measure.  It's a similar story in the Bayesian and (generalized) fiducial case.  But the probabilistic Bernstein–von Mises theorem implies that, under certain conditions, as n →∞, the suitably centered and scaled Bayesian posterior distribution will be approximately Gaussian.  Our main result shows that the same holds true for the possibilistic IM solution: under certain regularity conditions, a suitably centered and scaled version of the IM's possibility measure will converge in a strong sense to the Gaussian possibility measure.  








 §.§ Main result



The regularity conditions stated below are exactly those assumed in textbook treatments of the asymptotic normality of maximum likelihood estimators—in particular, <cit.>. 


The probability measures _θ, indexed by θ∈⊆^d, have density functions p_θ with respect to a fixed dominating measure.  Recall that Θ denotes the special “true” parameter value.  


  * The support of _θ doesn't depend on θ.

  * θ↦log p_θ(x) is twice continuously differentiable for almost all x.

  * Differentiation can be passed under the integral sign. 

  * The second derivative of θ↦log p_θ(X_1) is Lipschitz continuous in a neighborhood of Θ with Lipschitz constant that has finite _Θ-expectation.





Under the above regularity conditions, we establish a possibilistic Bernstein–von Mises theorem for the IM solution presented in Section <ref>.  That is, a centered and scaled version of the IM's possibility contour, namely, 

    π̌_X^n(z) = π_X^n( θ̂_X^n + J_X^n^-1/2 z ),    z ∈^d,

where J_X^n denotes the observed Fisher information matrix, can be well approximated by the Gaussian possibility contour.  



Let X^n = (X_1,…,X_n) consist of iid observations from _Θ, where {_θ: θ∈} is the posited model and Θ is in the interior of .  If the above regularity conditions hold at Θ, and the maximum likelihood estimator θ̂_X^n is consistent, then the possibilistic IM's solution satisfies 

    sup_z ∈ K| π̌_X^n(z) - γ(z) | → 0,   in _Θ-probability as n →∞,

where K ⊂^d ∖{0} is an arbitrary compact set, π̌_X^n is as in (<ref>), and γ is the Gaussian contour.  This, in turn, implies that 

    | _X^n(H) - { J_X^n^1/2(H - θ̂_X^n) }| → 0,   in _Θ-probability as n →∞,

for all compact H with H ∌Θ. 


The proof of Theorem <ref> is lengthy and will be presented elsewhere.  While it follows immediately from <cit.> that π_X^n(Θ) →γ(Z) in distribution under _Θ, for Z ∼_d(0,I), getting the required in-probability convergence uniformly over parameter values requires significant care.  

Why does K in (<ref>) exclude the origin?  It's because there's a sort of singularity, i.e., π̌_X^n(0) = π_X^n(θ̂_X^n) is identically 1, no sampling variability!  Since we know that π̌_X^n(0) = γ(0) exactly, it suffices to focus on z values away from the origin (and from infinity). Similarly, open subsets H that contain Θ will also eventually contain θ̂_X^n and, consequently, _X^n(H)=1 with high probability when n is large.  Therefore, it suffices to focus on compact H that exclude Θ.

The take-away message is that there's now a mathematically rigorous sense in which this possibilistic IM—despite its imprecision and exact validity—is also statistically efficient.  This also shows that the connection between the possibilistic IMs and Bayes/fiducial solutions highlighted in <cit.> for a special class of models holds much more broadly, asymptotically.  Finally, other asymptotically equivalent scaling schemes can be employed in (<ref>).  For example, replacing J^-1/2 with n^-1/2 leads to a limiting Gaussian possibility with underlying covariance matrix matching the Cramér–Rao lower bound.






















§ NUMERICAL ILLUSTRATIONS



 
iid Bernoulli.  Figure <ref>(a) shows the exact centered and scaled possibilistic IM contour for Θ based on iid Bernoulli data at three different sample sizes, along with the limiting standard Gaussian possibility contour. 



Logistic regression.  Consider the data in Table 8.4 of <cit.> on the relationship between exposure to chloracetic acid and the death of mice. 
A total of n = 120 mice are exposed, ten at each of the twelve dose levels, and a binary death indicator is measured.  A simple logistic regression model is fit and Figure <ref>(b) shows the exact IM possibility contour (red)—which requires expensive but still noisy Monte Carlo evaluations—along with the simple, analytically tractable Gaussian approximation (black).

























§ CONCLUSION



This paper establishes a possibilistic Bernstein–von Mises theorem for the possibilistic IM solution which, in addition to providing new insights and connections to the more familiar Bayesian and fiducial solutions, leads to theoretically and practically important conclusions concerning the IM's efficiency.  Follow-up work will consider extensions of our present result to cases that involve nuisance parameters and/or partial prior information about the unknowns. 










  §.§.§ 
 RM's research is supported by the U.S. National Science Foundation, SES–2051225.



  §.§.§ 

The authors have no competing interests to declare that are
relevant to the content of this article.






apalike


